Classification with Deep Neural Networks and Logistic Loss
Deep neural networks (DNNs) trained with the logistic loss (i.e., the cross
entropy loss) have made impressive advancements in various binary
classification tasks. However, generalization analysis for binary
classification with DNNs and logistic loss remains scarce. The unboundedness of
the target function for the logistic loss is the main obstacle to deriving
satisfactory generalization bounds. In this paper, we aim to fill this gap by
establishing a novel and elegant oracle-type inequality, which enables us to
deal with the boundedness restriction of the target function, and using it to
derive sharp convergence rates for fully connected ReLU DNN classifiers trained
with logistic loss. In particular, we obtain optimal convergence rates (up to
log factors) only requiring the H\"older smoothness of the conditional class
probability $\eta$ of data. Moreover, we consider a compositional assumption
that requires $\eta$ to be the composition of several vector-valued functions
of which each component function is either a maximum value function or a
H\"older smooth function only depending on a small number of its input
variables. Under this assumption, we derive optimal convergence rates (up to
log factors) which are independent of the input dimension of data. This result
explains why DNN classifiers can perform well in practical high-dimensional
classification problems. Besides the novel oracle-type inequality, the sharp
convergence rates given in our paper also owe to a tight error bound for
approximating the natural logarithm function near zero (where it is unbounded)
by ReLU DNNs. In addition, we justify our claims for the optimality of rates by
proving corresponding minimax lower bounds. All these results are new in the
literature and will deepen our theoretical understanding of classification with
DNNs.
Zihan Zhang, Lei Shi, Ding-Xuan Zhou
# Classification with Deep Neural Networks and Logistic Loss +

###### Abstract

Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfying generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle-type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle-type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers \(\tilde{f}_{n}^{\mathbf{FNN}}\) trained by empirical logistic risk minimization with respect to i.i.d. samples of size \(n\), which lead to sharp rates of convergence as \(n\rightarrow\infty\). In particular, we obtain optimal convergence rates for \(\tilde{f}_{n}^{\mathbf{FNN}}\) (up to some logarithmic factor) only requiring the Holder smoothness of the conditional class probability \(\eta\) of data. Moreover, we consider a compositional assumption that requires \(\eta\) to be the composition of several vector-valued multivariate functions of which each component function is either a maximum value function or a Holder smooth function only depending on a small number of its input variables. Under this assumption, we can even derive optimal convergence rates for \(\tilde{f}_{n}^{\mathbf{FNN}}\) (up to some logarithmic factor) which are independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high-dimensional classification problems. Furthermore, we establish dimension-free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle-type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

**Keywords and phrases:** deep learning; deep neural networks; binary classification; logistic loss; generalization analysis
Introduction

In this paper, we study the binary classification problem using deep neural networks (DNNs) with the rectified linear unit (ReLU) activation function. Deep learning based on DNNs has recently achieved remarkable success in a wide range of classification tasks including text categorization [19], image classification [27], and speech recognition [17], which has become a cutting-edge learning method. ReLU is one of the most popular activation functions, as scalable computing and stochastic optimization techniques can facilitate the training of ReLU DNNs [15, 23]. Given a positive integer \(d\). Consider the binary classification problem where we regard \([0,1]^{d}\) as the input space and \(\{-1,1\}\) as the output space representing the two labels of input data. Let \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\), regarded as the data distribution (i.e., the joint distribution of the input and output data). The goal of classification is to learn a real-valued function from a _hypothesis space_\(\mathcal{F}\) (i.e., a set of candidate functions) based on the sample of the distribution \(P\). The predictive performance of any (deterministic) real-valued function \(f\) which has a Borel measurable restriction to \([0,1]^{d}\) (i.e., the domain of \(f\) contains \([0,1]^{d}\), and \([0,1]^{d}\ni x\mapsto f(x)\in\mathbb{R}\) is Borel measurable) is measured by the _misclassification error_ of \(f\) with respect to \(P\), given by

\[\mathcal{R}_{P}(f):=P\left(\left\{\left.(x,y)\in[0,1]^{d}\times\{-1,1\}\right| y\neq\mathrm{sgn}(f(x))\right\}\right), \tag{1.1}\]

or equivalently, the _excess misclassification error_

\[\mathcal{E}_{P}(f):=\mathcal{R}_{P}(f)-\inf\left\{\mathcal{R}_{P}(g)\left|g:[ 0,1]^{d}\to\mathbb{R}\text{ is Borel measurable}\right.\right\}. \tag{1.2}\]

Here \(\circ\) means function composition, and \(\mathrm{sgn}(\cdot)\) denotes the sign function which is defined as \(\mathrm{sgn}(t)=1\) if \(t\geq 0\) and \(\mathrm{sgn}(t)=-1\) otherwise. The misclassification error \(\mathcal{R}_{P}(f)\) characterizes the probability that the binary classifier \(\mathrm{sgn}\circ f\) makes a wrong prediction, where by a _binary classifier_ (or _classifier_ for short) we mean a \(\{-1,1\}\)-valued function whose domain contains the input space \([0,1]^{d}\). Since any real-valued function \(f\) with its domain containing \([0,1]^{d}\) determines a classifier \(\mathrm{sgn}\circ f\), we in this paper may call such a function \(f\) a _classifier_ as well.

Note that the function we learn in a classification problem is based on the sample, meaning that it is not deterministic but a random function. Thus we take the expectation to measure its efficiency using the (excess) misclassification error. More specifically, let \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) be an independent and identically distributed (i.i.d.) sample of the distribution \(P\) and the hypothesis space \(\mathcal{F}\) be a set of real-valued functions which have a Borel measurable restriction to \([0,1]^{d}\). We desire to construct an \(\mathcal{F}\)-valued statistic \(\hat{f}_{n}\) from the sample \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) and the classification performance of \(\hat{f}_{n}\) can be characterized by upper bounds for the expectation of the excess misclassification error \(\mathbb{E}\left[\mathcal{E}_{P}(\hat{f}_{n})\right]\). One possible way to produce \(\hat{f}_{n}\) is the _empirical risk minimization_ with some _loss function_\(\phi:\mathbb{R}\to[0,\infty)\), which is given by

\[\hat{f}_{n}\in\operatorname*{arg\,min}_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^ {n}\phi\left(Y_{i}f(X_{i})\right). \tag{1.3}\]

If \(\hat{f}_{n}\) satisfies (1.3), then we will call \(\hat{f}_{n}\) an _empirical \(\phi\)-risk minimizer_ (ERM with respect to \(\phi\), or \(\phi\)-ERM) over \(\mathcal{F}\). For any real-valued function \(f\) which has a Borel measurable restriction to \([0,1]^{d}\), the \(\phi\)-_risk_ and _excess \(\phi\)-risk_ of \(f\) with respect to \(P\), denoted by \(\mathcal{R}_{P}^{\phi}(f)\) and \(\mathcal{E}_{P}^{\phi}(f)\) respectively, are defined as

\[\mathcal{R}_{P}^{\phi}(f):=\int_{[0,1]^{d}\times\{-1,1\}}\phi(yf(x))\mathrm{d} P(x,y) \tag{1.4}\]

and

\[\mathcal{E}_{P}^{\phi}(f):=\mathcal{R}_{P}^{\phi}(f)-\inf\left\{\mathcal{R}_{ P}^{\phi}(g)\Big{|}g:[0,1]^{d}\to\mathbb{R}\text{ is Borel measurable}\right\}. \tag{1.5}\]

To derive upper bounds for \(\mathbb{E}\left[\mathcal{E}_{P}(\hat{f}_{n})\right]\), we can first establish upper bounds for \(\mathbb{E}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\), which are typically controlled by two parts, namely the sample error and the approximation error (e.g., cf. Chapter 2 of [6]). Then we are able to bound \(\mathbb{E}\left[\mathcal{E}_{P}(\hat{f}_{n})\right]\) by \(\mathbb{E}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\) through the so-called
calibration inequality_ (also known as _Comparison Theorem_, see, e.g., Theorem 10.5 of [6] and Theorem 3.22 of [40]). In this paper, we will call any upper bound for \(\mathbb{E}\left[\mathcal{E}_{P}(\hat{f}_{n})\right]\) or \(\mathbb{E}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\) a _generalization bound_.

Note that \(\lim\limits_{n\to\infty}\frac{1}{n}\sum_{i=1}^{n}\phi\left(Y_{i}f(X_{i})\right)= \mathcal{R}_{P}^{\phi}(f)\) almost surely for all measurable \(f\). Therefore, the empirical \(\phi\)-risk minimizer \(\hat{f}_{n}\) defined in (1.3) can be regarded as an estimation of the so-called _target function_ which minimizes the \(\phi\)-risk \(\mathcal{R}_{P}^{\phi}\) over all Borel measurable functions \(f\). The target function can be defined pointwise. Rigorously, we say a measurable function \(f^{*}:[0,1]^{d}\to[-\infty,\infty]\) is a target function of the \(\phi\)-risk under the distribution \(P\) if for \(P_{X}\)-almost all \(x\in[0,1]^{d}\) the value of \(f^{*}\) at \(x\) minimizes \(\int_{\{-1,1\}}\phi(yz)\mathrm{d}P(y|x)\) over all \(z\in[-\infty,\infty]\), i.e.,

\[f^{*}(x)\in\operatorname*{arg\,min}_{z\in[-\infty,\infty]}\int_{\{-1,1\}}\phi (yz)\mathrm{d}P(y|x)\text{ for }P_{X}\text{-almost all }x\in[0,1]^{d}, \tag{1.6}\]

where \(\phi(yz):=\varlimsup_{t\to yz}\phi(t)\) if \(z\in\{-\infty,\infty\}\), \(P_{X}\) is the marginal distribution of \(P\) on \([0,1]^{d}\), and \(P(\cdot|x)\) is the _regular conditional distribution_ of \(P\) on \(\{-1,1\}\) given \(x\) (cf. Lemma A.3.16 in [40]). In this paper, we will use \(f^{*}_{\phi,P}\) to denote the target function of the \(\phi\)-risk under \(P\). Note that \(f^{*}_{\phi,P}\) may take values in \(\{-\infty,\infty\}\), and \(f^{*}_{\phi,P}\) minimizes \(\mathcal{R}_{P}^{\phi}\) in the sense that

\[\mathcal{R}_{P}^{\phi}(f^{*}_{\phi,P}):=\int_{[0,1]^{d}\times\{ -1,1\}}\phi(yf^{*}_{\phi,P}(x))\mathrm{d}P(x,y) \tag{1.7}\] \[=\inf\left\{\left.\mathcal{R}_{P}^{\phi}(g)\right|g:[0,1]^{d}\to \mathbb{R}\text{ is Borel measurable}\right\},\]

where \(\phi(yf^{*}_{\phi,P}(x)):=\varlimsup_{t\to yf^{*}_{\phi,P}(x)}\phi(t)\) if \(yf^{*}_{\phi,P}(x)\in\{-\infty,\infty\}\) (cf. Lemma A.1).

In practice, the choice of the loss function \(\phi\) varies, depending on the classification method used. For neural network classification, the _logistic loss_\(\phi(t)=\log(1+\mathrm{e}^{-t})\), also known as the _cross entropy loss_, is commonly used. We now explain why the logistic loss is related to cross entropy. Let \(\mathcal{X}\) be an arbitrary nonempty countable set equipped with the sigma algebra consisting of all its subsets. For any two probability measures \(Q_{0}\) and \(Q\) on \(\mathcal{X}\), the _cross entropy_ of \(Q\) relative to \(Q_{0}\) is defined as \(\mathrm{H}(Q_{0},Q):=-\sum_{z\in\mathcal{X}}Q_{0}(\{z\})\cdot\log Q(\{z\})\), where \(\log 0:=-\infty\) and \(0\cdot(-\infty):=0\) (cf. (2.112) of [32]). One can show that \(\mathrm{H}(Q_{0},Q)\geq\mathrm{H}(Q_{0},Q_{0})\geq 0\) and

\[\{Q_{0}\}=\operatorname*{arg\,min}_{Q}\mathrm{H}(Q_{0},Q)\text{ if }\mathrm{H}(Q_{ 0},Q_{0})<\infty.\]

Therefore, roughly speaking, the cross entropy \(\mathrm{H}(Q_{0},Q)\) characterizes how close is \(Q\) to \(Q_{0}\). For any \(a\in[0,1]\), let \(\mathscr{M}_{a}\) denote the probability measure on \(\{-1,1\}\) with \(\mathscr{M}_{a}(\{1\})=a\) and \(\mathscr{M}_{a}(\{-1\})=1-a\). Recall that any real-valued Borel measurable function \(f\) defined on the input space \([0,1]^{d}\) can induce a classifier \(\mathrm{sgn}\circ f\). We can interpret the construction of the classifier \(\mathrm{sgn}\circ f\) from \(f\) as follows. Consider the _logistic function_

\[\bar{l}:\mathbb{R}\to(0,1),\;z\mapsto\frac{1}{1+\mathrm{e}^{-z}}, \tag{1.8}\]

which is strictly increasing. For each \(x\in[0,1]^{d}\), \(f\) induces a probability measure \(\mathscr{M}_{\bar{l}(f(x))}\) on \(\{-1,1\}\) via \(\bar{l}\), which we regard as a prediction made by \(f\) of the distribution of the output data (i.e., the two labels \(+1\) and \(-1\)) given the input data \(x\). Observe that the larger \(f(x)\) is, the closer the number \(\bar{l}(f(x))\) gets to \(1\), and the more likely the event \(\{1\}\) occurs under the distribution \(\mathscr{M}_{\bar{l}(f(x))}\). If \(\mathscr{M}_{\bar{l}(f(x))}(\{+1\})\geq\mathscr{M}_{\bar{l}(f(x))}(\{-1\})\), then \(+1\) is more likely to appear given the input data \(x\) and we thereby think of \(f\) as classifying the input \(x\) as class \(+1\). Otherwise, when \(\mathscr{M}_{\bar{l}(f(x))}(\{+1\})<\mathscr{M}_{\bar{l}(f(x))}(\{-1\})\), \(x\) is classified as \(-1\). In this way, \(f\) induces a classifier given by

\[x\mapsto\begin{cases}+1,&\text{ if }\mathscr{M}_{\bar{l}(f(x))}(\{1\})\geq \mathscr{M}_{\bar{l}(f(x))}(\{-1\}),\\ -1,&\text{ if }\mathscr{M}_{\bar{l}(f(x))}(\{1\})<\mathscr{M}_{\bar{l}(f(x))}(\{-1\}). \end{cases} \tag{1.9}\]
Indeed, the classifier in (1.9) is exactly \(\operatorname{sgn}\circ f\). Thus we can also measure the predictive performance of \(f\) in terms of \(\mathscr{M}_{\bar{l}(f(\cdot))}\) (instead of \(\operatorname{sgn}\circ f\)). To this end, one natural way is to compute the average "extent" of how close is \(\mathscr{M}_{\bar{l}(f(x))}\) to the true conditional distribution of the output given the input \(x\). If we use the cross entropy to characterize this "extent", then its average, which measures the classification performance of \(f\), will be \(\int_{[0,1]^{d}}\operatorname{H}(\mathscr{Y}_{x},\mathscr{M}_{\bar{l}(f(x))}) \mathrm{d}\mathscr{X}(x)\), where \(\mathscr{X}\) is the distribution of the input data, and \(\mathscr{Y}_{x}\) is the conditional distribution of the output data given the input \(x\). However, one can show that this quantity is just the logistic risk of \(f\). Indeed,

\[\int_{[0,1]^{d}}\operatorname{H}(\mathscr{Y}_{x},\mathscr{M}_{ \bar{l}(f(x))})\mathrm{d}\mathscr{X}(x)\] \[=\int_{[0,1]^{d}}\left(-\mathscr{Y}_{x}(\{1\})\cdot\log(\mathscr{ M}_{\bar{l}(f(x))}(\{1\}))-\mathscr{Y}_{x}(\{-1\})\log(\mathscr{M}_{\bar{l}(f(x))}( \{-1\}))\right)\mathrm{d}\mathscr{X}(x)\] \[=\int_{[0,1]^{d}}\left(-\mathscr{Y}_{x}(\{1\})\cdot\log(\bar{l}(f (x)))-\mathscr{Y}_{x}(\{-1\})\log(1-\bar{l}(f(x)))\right)\mathrm{d}\mathscr{X }(x)\] \[=\int_{[0,1]^{d}}\left(\mathscr{Y}_{x}(\{1\})\cdot\log(1+\mathrm{ e}^{-f(x)})+\mathscr{Y}_{x}(\{-1\})\log(1+\mathrm{e}^{f(x)})\right)\mathrm{d} \mathscr{X}(x)\] \[=\int_{[0,1]^{d}}\left(\mathscr{Y}_{x}(\{1\})\cdot\phi(f(x))+ \mathscr{Y}_{x}(\{-1\})\phi(-f(x))\right)\mathrm{d}\mathscr{X}(x)\] \[=\int_{[0,1]^{d}}\int_{\{-1,1\}}\phi(yf(x))\mathrm{d}\mathscr{Y}_ {x}(y)\mathrm{d}\mathscr{X}(x)=\int_{[0,1]^{d}\times\{-1,1\}}\phi(yf(x)) \mathrm{d}P(x,y)=\mathcal{R}_{P}^{\phi}(f),\]

where \(\phi\) is the logistic loss and \(P\) is the joint distribution of the input and output data, i.e., \(\mathrm{d}P(x,y)=\mathrm{d}\mathscr{Y}_{x}(y)\mathrm{d}\mathscr{X}(x)\). Therefore, the average cross entropy of the distribution \(\mathscr{M}_{\bar{l}(f(x))}\) induced by \(f\) to the true conditional distribution of the output data given the input data \(x\) is equal to the logistic risk of \(f\) with respect to the joint distribution of the input and output data, which explains why the logistic loss is also called the cross entropy loss. Compared with the misclassification error \(\mathcal{R}_{P}(f)\) which measures the performance of the classifier \(f(x)\) in correctly generating the class label \(\operatorname{sgn}(f(x))\) that equals the most probable class label of the input data \(x\) (i.e., the label \(y_{x}\in\{-1,+1\}\) such that \(\mathscr{Y}_{x}(\{y_{x}\})\geq\mathscr{Y}_{x}(\{-y_{x}\})\)), the logistic risk \(\mathcal{R}_{P}^{\phi}(f)\) measures how close is the induced distribution \(\mathscr{M}_{\bar{l}(f(x))}\) to the true conditional distribution \(\mathscr{Y}_{x}\). Consequently, in comparison with the (excess) misclassification error, the (excess) logistic risk is also a reasonable quantity for characterizing the performance of classifiers but from a different angle. When classifying with the logistic loss, we are essentially learning the conditional distribution \(\mathscr{Y}_{x}\) through the cross entropy and the logistic function \(\bar{l}\). Moreover, for any classifier \(\hat{f}_{n}:[0,1]^{d}\to\mathbb{R}\) trained with logistic loss, the composite function \(\bar{l}\circ\hat{f}_{n}(x)=\mathscr{M}_{\bar{l}_{0}f_{n}(x)}(\{1\})\) yields an estimation of the _conditional class probability function_\(\eta(x):=P(\{1\}\,|x)=\mathscr{Y}_{x}(\{1\})\). Therefore, classifiers trained with logistic loss essentially capture more information about the exact value of the conditional class probability function \(\eta(x)\) than we actually need to minimize the misclassification error \(\mathcal{R}_{P}(\cdot)\), since the knowledge of the sign of \(2\eta(x)-1\) is already sufficient for minimizing \(\mathcal{R}_{P}(\cdot)\) (see (2.49)). In addition, we point out that the excess logistic risk \(\mathcal{E}_{P}^{\phi}(f)\) is actually the average _Kullback-Leibler divergence_ (_KL divergence_) from \(\mathscr{M}_{\bar{l}(f(x))}\) to \(\mathscr{Y}_{x}\). Here for any two probability measures \(Q_{0}\) and \(Q\) on some countable set \(\mathcal{X}\), the KL divergence from \(Q\) to \(Q_{0}\) is defined as \(\mathrm{KL}(Q_{0}\|Q):=\sum_{z\in\mathcal{X}}Q_{0}(\{z\})\cdot\log\frac{Q_{0}( \{z\})}{Q(\{z\})}\), where \(Q_{0}(\{z\})\cdot\log\frac{Q_{0}(\{z\})}{Q(\{z\})}:=0\) if \(Q_{0}(\{z\})=0\) and \(Q_{0}(\{z\})\cdot\log\frac{Q_{0}(\{z\})}{Q(\{z\})}:=\infty\) if \(Q_{0}(\{z\})>0=Q(\{z\})\) (cf. (2.11) of [32] or Definition 2.5 of [44]).

In this work, we focus on the generalization analysis of binary classification with empirical risk minimization over ReLU DNNs. That is, the classifiers under consideration are produced by algorithm (1.3) in which the hypothesis space \(\mathcal{F}\) is generated by deep ReLU networks. Based on recent studies in complexity and approximation theory of DNNs (e.g., [5, 33, 47]), several researchers have derived generalization bounds for \(\phi\)-ERMs over DNNs in binary classification problems [9, 22, 38]. However, to the best of our knowledge, the existing literature fails to establish satisfactory generalization analysis if the target function \(f_{\phi,P}^{*}\) is unbounded. In particular, take \(\phi\) to be the logistic loss, i.e., \(\phi(t)=\log(1+\mathrm{e}^{-t})\). The target function is then explicitly given by \(f_{\phi,P}^{*}\xrightarrow[]{P_{X}\cdot\mathrm{a.s.}}\log\frac{\eta}{1-\eta}\) with \(\eta(x):=P(\{1\}\,|x)\) (\(x\in[0,1]^{d}\)) being the conditional class probability function of \(P\) (cf. Lemma A.2), where recall that
\(P(\cdot|x)\) denotes the conditional probability of \(P\) on \(\{-1,1\}\) given \(x\). Hence \(f_{\phi,P}^{*}\) is unbounded if \(\eta\) can be arbitrarily close to \(0\) or \(1\), which happens in many practical problems (see Section 3 for more details). For instance, we have \(\eta(x)=0\) or \(\eta(x)=1\) for a noise-free distribution \(P\), implying \(f_{\phi,P}^{*}(x)=\infty\) for \(P_{X}\)-almost all \(x\in[0,1]^{d}\), where \(P_{X}\) is the marginal distribution of \(P\) on \([0,1]^{d}\). DNNs trained with the logistic loss perform efficiently in various image recognition applications as the smoothness of the loss function can further simplify the optimization procedure [11, 27, 39]. However, due to the unboundedness of \(f_{\phi,P}^{*}\), the existing generalization analysis for classification with DNNs and the logistic loss either results in slow rates of convergence (e.g., the logarithmic rate in [38]) or can only be conducted under very restrictive conditions (e.g., [22, 9]) (cf. the discussions in Section 3). The unboundedness of the target function brings several technical difficulties to the generalization analysis. Indeed, if \(f_{\phi,P}^{*}\) is unbounded, it cannot be approximated uniformly by continuous functions on \([0,1]^{d}\), which poses extra challenges for bounding the approximation error. Besides, previous sample error estimates based on concentration techniques are no longer valid because these estimates usually require involved random variables to be bounded or to satisfy strong tail conditions (cf. Chapter 2 of [45]). Therefore, in contrast to empirical studies, the previous strategies for generalization analysis could not demonstrate the efficiency of classification with DNNs and the logistic loss.

To fill this gap, in this paper we develop a novel theoretical analysis to establish tight generalization bounds for training DNNs with ReLU activation function and logistic loss in binary classification. Our main contributions are summarized as follows.

* For \(\phi\) being the logistic loss, we establish an oracle-type inequality to bound the excess \(\phi\)-risk without using the explicit form of the target function \(f_{\phi,P}^{*}\). Through constructing a suitable bivariate function \(\psi:[0,1]^{d}\times\{-1,1\}\rightarrow\mathbb{R}\), generalization analysis based on this oracle-type inequality can remove the boundedness restriction of the target function. Similar results hold even for the more general case when \(\phi\) is merely Lipschitz continuous (see Theorem 2.1 and related discussions in Section 2.1).
* By using our oracle-type inequality, we establish tight generalization bounds for fully connected ReLU DNN classifiers \(\hat{f}_{n}^{\mathbf{FNN}}\) trained by empirical logistic risk minimization (see (2.14)) and obtain sharp convergence rates in various settings:
* We establish optimal convergence rates for the excess logistic risk of \(\hat{f}_{n}^{\mathbf{FNN}}\) only requiring the Holder smoothness of the conditional probability function \(\eta\) of the data distribution. Specifically, for Holder-\(\beta\) smooth \(\eta\), we show that the convergence rates of the excess logistic risk of \(\hat{f}_{n}^{\mathbf{FNN}}\) can achieve \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{\beta+d}})\), which is optimal up to the logarithmic term \((\log n)^{\frac{5\beta}{\beta+d}}\). From this we obtain the convergence rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}})\) of the excess misclassification error of \(\hat{f}_{n}^{\mathbf{FNN}}\), which is very close to the optimal rate, by using the calibration inequality (see Theorem 2.2). As a by-product, we also derive a new tight error bound for the approximation of the natural logarithm function (which is unbounded near zero) by ReLU DNNs (see Theorem 2.4). This bound plays a key role in establishing the aforementioned optimal rates of convergence.
* We consider a compositional assumption which requires the conditional probability function \(\eta\) to be the composition \(h_{q}\circ h_{q-1}\circ\cdots\circ h_{1}\circ h_{0}\) of several vector-valued multivariate functions \(h_{i}\), satisfying that each component function of \(h_{i}\) is either a Holder-\(\beta\) smooth function only depending on (a small number) \(d_{*}\) of its input variables or the maximum value function among some of its input variables. We show that under this compositional assumption the convergence rate of the excess logistic risk of \(\hat{f}_{n}^{\mathbf{FNN}}\) can achieve \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta) ^{d}}{d_{*}+\beta\cdot(1\wedge\beta)^{d}}})\), which is optimal up to the logarithmic term \((\log n)^{\frac{5\beta\cdot(1\wedge\beta)^{d}}{d_{*}+\beta\cdot(1\wedge\beta) ^{d}}}\). We then use the calibration inequality to obtain the convergence rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta) ^{d}}{2d_{*}+2\beta\cdot(1\wedge\beta)^{d}}})\) of the excess misclassification error of \(\hat{f}_{n}^{\mathbf{FNN}}\) (see Theorem 2.3). Note that the derived convergence rates
\(\mathcal{O}\big{(}\big{(}\frac{\log n\big{)}^{5}}{n}\big{)}^{\frac{\beta(\cdot( 1,\beta))^{6}}{2\alpha+\beta\cdot(1,\alpha)^{3}}}\) and \(\mathcal{O}\big{(}\big{(}\frac{\log n\big{)}^{5}}{n}\big{)}^{\frac{\beta(\cdot( 1,\beta))^{6}}{2\alpha+\beta\cdot(1,\alpha)^{3}}}\) are independent of the input dimension \(d\), thereby circumventing the well-known curse of dimensionality. It can be shown that the above compositional assumption is likely to be satisfied in practice (see comments before Theorem 2.3). Thus this result helps to explain the huge success of DNNs in practical classification problems, especially high-dimensional ones.
* We derive convergence rates of the excess misclassification error of \(\hat{f}_{n}^{\mathbf{FNN}}\) under the piecewise smooth decision boundary condition combining with the noise and margin conditions (see Theorem 2.5). As a special case of this result, we show that when the input data are bounded away from the decision boundary almost surely, the derived rates can also be dimension-free.
* We demonstrate the optimality of the convergence rates stated above by presenting corresponding minimax lower bounds (see Theorem 2.6 and Corollary 2.1).

The rest of this paper is organized as follows. In the remainder of this section, we first introduce some conventions and notations that will be used in this paper. Then we describe the mathematical modeling of fully connected ReLU neural networks which defines the hypothesis spaces in our setting. At the end of this section, we provide a symbol glossary for the convenience of readers. In Section 2, we present our main results in this paper, including the oracle-type inequality, several generalization bounds for classifiers obtained from empirical logistic risk minimization over fully connected ReLU DNNs, and two minimax lower bounds. Section 3 provides discussions and comparisons with related works and Section 4 concludes the paper. In the Appendix, we first present covering number bounds and some approximation bounds for the space of fully connected ReLU DNNs, and then give detailed proofs of results in the main body of this paper.

### Conventions and Notations

Throughout this paper, we follow the conventions that \(0^{0}:=1\), \(1^{\infty}:=1\), \(\frac{5}{6}:=\infty=:\infty^{c}\), \(\log(\infty):=\infty\), \(\log 0:=-\infty\), \(0\cdot w:=0=:w\cdot 0\) and \(\frac{a}{\infty}:=0=:b^{\infty}\) for any \(a\in\mathbb{R},b\in[0,1),c\in(0,\infty)\), \(z\in[0,\infty]\), \(w\in[-\infty,\infty]\) where we denote by \(\log\) the natural logarithm function (i.e. the base-e logarithm function). The terminology "measurable" means "Borel measurable" unless otherwise specified. Any Borel subset of some Euclidean space \(\mathbb{R}^{m}\) is equipped with the Borel sigma algebra by default. Let \(\mathcal{G}\) be an arbitrary measurable space and \(n\) be a positive integer. We call any sequence of \(\mathcal{G}\)-valued random variables \(\left\{Z_{i}\right\}_{i=1}^{n}\) a _sample_ in \(\mathcal{G}\) of size \(n\). Furthermore, for any measurable space \(\mathcal{F}\) and any sample \(\left\{Z_{i}\right\}_{i=1}^{n}\) in \(\mathcal{G}\), an \(\mathcal{F}\)-valued statistic on \(\mathcal{G}^{n}\) from the sample \(\left\{Z_{i}\right\}_{i=1}^{n}\) is a random variable \(\hat{\theta}\) together with a measurable map \(\mathcal{T}:\mathcal{G}^{n}\to\mathcal{F}\) such that \(\hat{\theta}=\mathcal{T}(Z_{1},\ldots,Z_{n})\), where \(\mathcal{T}\) is called _the map associated with_ the statistic \(\hat{\theta}\). Let \(\hat{\theta}\) be an arbitrary \(\mathcal{F}\)-valued statistic from some sample \(\left\{Z_{i}\right\}_{i=1}^{n}\) and \(\mathcal{T}\) is the map associated with \(\hat{\theta}\). Then for any measurable space \(\mathcal{D}\) and any measurable map \(\mathcal{T}_{0}:\mathcal{F}\to\mathcal{D}\), \(\mathcal{T}_{0}(\hat{\theta})=\mathcal{T}_{0}(\mathcal{T}(Z_{1},\ldots,Z_{n}))\) is a \(\mathcal{D}\)-valued statistic from the sample \(\left\{Z_{i}\right\}_{i=1}^{n}\), and \(\mathcal{T}_{0}\circ\mathcal{T}\) is the map associated with \(\mathcal{T}_{0}(\hat{\theta})\).

Next we will introduce some notations used in this paper. We denote by \(\mathbb{N}\) the set of all positive integers \(\left\{1,2,3,4,\ldots\right\}\). For \(d\in\mathbb{N}\), we use \(\mathcal{F}_{d}\) to denote the set of all Borel measurable functions from \([0,1]^{d}\) to \((-\infty,\infty)\), and use \(\mathcal{H}_{0}^{d}\) to denote the set of all Borel probability measures on \([0,1]^{d}\times\left\{-1,1\right\}\). For any set \(A\), the indicator function of \(A\) is given by

\[\mathbb{1}_{A}(x):=\begin{cases}0,&\text{if }x\notin A,\\ 1,&\text{if }x\in A,\end{cases} \tag{1.10}\]

and the number of elements of \(A\) is denoted by \(\#(A)\). For any finite dimensional vector \(v\) and any positive integer \(l\) less than or equal to the dimension of \(v\), we denote by \((v)_{l}\) the \(l\)-th component of \(v\). More generally, for any nonempty subset \(I=\left\{i_{1},i_{2},\ldots,i_{m}\right\}\) of \(\mathbb{N}\) with \(1\leq i_{1}<i_{2}<\cdots<i_{m}\leq\) the dimension of \(v\), we denote \((v)_{I}:=\big{(}(v)_{i_{1}},(v)_{i_{2}},\ldots,(v)_{i_{m}}\big{)}\), which is a \(\#(I)\)-dimensional vector. For any function \(f\), we use \(\mathbf{dom}(f)\) to denote the domain of \(f\), and use \(\mathbf{ran}(f)\) to denote the range of \(f\), that
is, \(\mathbf{ran}(f):=\left\{f(x)\big{|}x\in\mathbf{dom}(f)\right\}\). If \(f\) is a \([-\infty,\infty]^{m}\)-valued function for some \(m\in\mathbb{N}\) with \(\mathbf{dom}(f)\) containing a nonempty set \(\Omega\), then the uniform norm of \(f\) on \(\Omega\) is given by

\[\|f\|_{\Omega}:=\sup\left\{\left|\big{(}f(x)\big{)}_{i}\right|\Big{|}x\in \Omega,\,i\in\{1,2,\dots,m\}\right\}. \tag{1.11}\]

For integer \(m\geq 2\) and real numbers \(a_{1},\cdots,a_{m}\), define \(a_{1}\lor a_{2}\vee\cdots\lor a_{m}=\max\{a_{1},a_{2},\cdots a_{m}\}\) and \(a_{1}\wedge a_{2}\wedge\cdots\wedge a_{m}=\min\{a_{1},a_{2},\cdots a_{m}\}.\) Given a real matrix \(\boldsymbol{A}=(a_{i,j})_{i=1,\dots,m,j=1,\dots,l}\) and \(t\in[0,\infty]\), the \(\ell^{t}\)-norm of \(\boldsymbol{A}\) is defined by

\[\|\boldsymbol{A}\|_{t}:=\left\{\begin{aligned} &\sum_{i=1}^{m}\sum_{j=1}^{l} \mathbbm{1}_{(0,\infty)}(|a_{i,j}|),&\text{if }t=0,\\ &\left|\sum_{i=1}^{m}\sum_{j=1}^{l}|a_{i,j}|^{t}\right|^{1/t},& \text{if }0<t<\infty,\\ &\sup\left\{|a_{i,j}|\left|i\in\{1,\cdots,m\},j\in\{1,\cdots,l\} \right.\right\},&\text{if }t=\infty.\end{aligned}\right. \tag{1.12}\]

Note that a vector is exactly a matrix with only one column or one row. Consequently, (1.12) with \(l=1\) or \(m=1\) actually defines the \(\ell^{t}\)-norm of a real vector \(\boldsymbol{A}\). Let \(\mathcal{G}\) be a measurable space, \(\left\{Z_{i}\right\}_{i=1}^{n}\) be a sample in \(\mathcal{G}\) of size \(n\), \(\mathcal{P}_{n}\) be a probability measure on \(\mathcal{G}^{n}\), and \(\hat{\theta}\) be a \([-\infty,\infty]\)-valued statistic on \(\mathcal{G}^{n}\) from the sample \(\left\{Z_{i}\right\}_{i=1}^{n}\). Then we denote

\[\boldsymbol{E}_{\mathcal{P}_{n}}[\hat{\theta}]:=\int\mathcal{T}\mathrm{d} \mathcal{P}_{n} \tag{1.13}\]

provided that the integral \(\int\mathcal{T}\mathrm{d}\mathcal{P}_{n}\) exists, where \(\mathcal{T}\) is the map associated with \(\hat{\theta}\). Therefore,

\[\boldsymbol{E}_{\mathcal{P}_{n}}[\hat{\theta}]=\mathbb{E}\left[\mathcal{T}(Z _{1},\dots,Z_{n})\right]=\mathbb{E}[\hat{\theta}]\]

if the joint distribution of \((Z_{1},\dots,Z_{n})\) is exactly \(\mathcal{P}_{n}\). Let \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) and \(x\in[0,1]^{d}\). We use \(P(\cdot|x)\) to denote the regular conditional distribution of \(P\) on \(\{-1,1\}\) given \(x\), and \(P_{X}\) to denote the marginal distribution of \(P\) on \([0,1]^{d}\). For short, we will call the function \([0,1]^{d}\ni x\mapsto P(\{1\}\,|x)\in[0,1]\) the _conditional probability function_ (instead of the _conditional class probability function_) of \(P\). For any probability measure \(\mathscr{Q}\) defined on some measurable space \((\Omega,\mathcal{F})\) and any \(n\in\mathbb{N}\), we use \(\mathscr{Q}^{\otimes n}\) to denote the product measure \(\underbrace{\mathscr{Q}\times\mathscr{Q}\times\cdots\mathscr{Q}}_{n}\) defined on the product measurable space \((\underbrace{\Omega\times\Omega\times\cdots\Omega}_{n},\,\underbrace{ \mathcal{F}\otimes\mathcal{F}\otimes\cdots\mathcal{F}}_{n})\).

### Spaces of fully connected neural networks

In this paper, we restrict ourselves to neural networks with the ReLU activation function. Consequently, hereinafter, for simplicity, we sometimes omit the word "ReLU" and the terminology "neural networks" will always refer to "ReLU neural networks".

The ReLU function is given by \(\sigma:\mathbb{R}\rightarrow[0,\infty),\ t\mapsto\max\left\{t,0\right\}\). For any vector \(v\in\mathbb{R}^{m}\) with \(m\) being some positive integer, the \(v\)-shifted ReLU function is defined as \(\sigma_{v}:\mathbb{R}^{m}\rightarrow[0,\infty)^{m},\ x\mapsto\sigma(x-v)\), where the function \(\sigma\) is applied componentwise.

Neural networks considered in this paper can be expressed as a family of real-valued functions which take the form

\[f:\mathbb{R}^{d}\rightarrow\mathbb{R},\quad x\mapsto\boldsymbol{W}_{L}\sigma _{v_{L}}\boldsymbol{W}_{L-1}\sigma_{v_{L-1}}\cdots\boldsymbol{W}_{1}\sigma_{v _{1}}\boldsymbol{W}_{0}x, \tag{1.14}\]

where the depth \(L\) denotes the number of hidden layers, \(\mathrm{m}_{k}\) is the width of \(k\)-th layer, \(\boldsymbol{W}_{k}\) is an \(\mathrm{m}_{k+1}\times\mathrm{m}_{k}\) weight matrix with \(\mathrm{m}_{0}=d\) and \(\mathrm{m}_{L+1}=1\), and the shift vector \(v_{k}\in\mathbb{R}^{\mathrm{m}_{k}}\) is called a bias. The architecture of a neural network is parameterized by weight matrices \(\left\{\boldsymbol{W}_{k}\right\}_{k=0}^{L}\) and biases
\(\{v_{k}\}_{k=1}^{L}\), which will be estimated from data. Throughout the paper, whenever we talk about a neural network, we will explicitly associate it with a function \(f\) of the form (1.14) generated by \(\{\mathbf{W}_{k}\}_{k=0}^{L}\) and \(\{v_{k}\}_{k=1}^{L}\).

The space of fully connected neural networks is characterized by their depth and width, as well as the number of nonzero parameters in weight matrices and bias vectors. In addition, the complexity of this space is also determined by the \(\|\cdot\|_{\infty}\)-bounds of neural network parameters and \(\|\cdot\|_{[0,1]^{d}}\)-bounds of associated functions in form (1.14). Concretely, let \((G,N)\in[0,\infty)^{2}\) and \((S,B,F)\in[0,\infty]^{3}\), the space of fully connected neural networks is defined as

\[\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F):=\left\{f:\mathbb{R}^{d}\to \mathbb{R}\begin{vmatrix}f\text{ is defined in (\ref{eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eqeq:eq:eq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eqeq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eq:eqeq:eq:eq:eq:eqeqeq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eq:eqeq:eqeq:eqeq:eqeq:eq:eq:eqeq:eqeq:eqeq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eqeq:eq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eqeq:eq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eqeq:eq:eq:eq:eqeq:eqeq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eq:eqeq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eq:
\begin{tabular}{c l l} \(\left\|\cdot\right\|_{\Omega}\) & The uniform norm on a set \(\Omega\) & Eq. (1.11) \\ \(\left\|\cdot\right\|_{\Omega}\) & The \(\ell^{*}\)-norm. & Eq. (1.12) \\ \(\left\|\cdot\right\|_{\Omega^{k,\lambda}(\Omega)}\) & The Holder norm. & Eq. (2.12) \\ \(\left\|\cdot\right\|_{S\Omega}\) & The sign function. & Below Eq. (1.2) \\ \(\left\|\cdot\right\|_{\Omega}\) & The ReLU function, that is, \(\mathbb{R}\ni t\mapsto\max\left\{0,t\right\}\in[0,\infty)\). & Above Eq. (1.14) \\ \(\left\|\cdot\right\|_{\Omega^{*}}\) & The \(\upsilon\)-shifted ReLU function. & Above Eq. (1.14) \\ \(\left\|\cdot\right\|_{\Omega}\) & The probability measure on \(\{-1,1\}\) with \(\mathscr{M}_{a}(\{1\})=a\). & Above Eq. (1.8) \\ \(P_{X}\) & The marginal distribution of \(P\) on \([0,1]^{d}\). & Below Eq. (1.6) \\ \(P(\cdot|x)\) & The regular conditional distribution of \(P\) on \(\{-1,1\}\) given & Below Eq. (1.6) \\ \(P_{\eta,2}\) & The probability on \([0,1]^{d}\times\{-1,1\}\) of which the marginal distribution on \([0,1]^{d}\) is \(\mathscr{Q}\) and the conditional probability function is \(\eta\). & Eq. (2.57) \\ \(P_{\eta}\) & The probability on \([0,1]^{d}\times\{-1,1\}\) of which the marginal distribution on \([0,1]^{d}\) is the Lebesgue measure and the conditional probability function is \(\eta\). & Below Eq. (2.57) \\ \(\left\|\cdot\right\|_{\Omega^{*}}\) & The expectation of a statistic \(\theta\) when the joint distribution & Eq. (1.13) \\ \(\mathscr{Q}^{\otimes m}\) & The product measure \(\underline{\mathscr{Q}\times\mathscr{Q}\times\cdots\mathscr{Q}}\). & Below Eq. (1.13) \\ \(\left\|\cdot\right\|_{\Omega^{*}}\) & The misclassification error of \(f\) with respect to \(P\). & Eq. (1.1) \\ \(\mathscr{E}_{P}(f)\) & The excess misclassification error of \(f\) with respect to \(P\). & Eq. (1.2) \\ \(\left\|\cdot\right\|_{\Omega^{*}}\) & The \(\phi\)-risk of \(f\) with respect to \(P\). & Eq. (1.4) \\ \(\mathscr{E}_{P}^{\dagger}(f)\) & The excess \(\phi\)-risk of \(f\) with respect to \(P\). & Eq. (1.5) \\ \(f_{\phi,P}^{\dagger}\) & The target function of the \(\phi\)-risk under some distribution & Eq. (1.6) \\ \(P\). & \(P\). & Eq. (1.6) \\ \(\mathscr{N}(\mathcal{F},\gamma)\) & The covering number of a class of real-valued functions \(\mathcal{F}\) & Eq. (2.1) \\  & with radius \(\gamma\) in the uniform norm. & Eq. (1.2) \\ \(\mathscr{B}_{r}^{\circ}(\Omega)\) & The closed ball of radius \(r\) centered at the origin in the & Eq. (2.13) \\  & Holder space of order \(\beta\) on \(\Omega\). & Eq. (2.27) \\ \(\mathscr{Q}_{d}^{\mathsf{CH}}(d_{*})\) & The set of all functions from \([0,1]^{d}\) to \(\mathbb{R}\) which compute the maximum value of up to \(d_{*}\) components of their input & Eq. (2.28) \\ \(\mathscr{G}_{d}^{\mathsf{CH}}(d_{*},\beta,r)\) & The set of all functions in \(\mathcal{B}_{r}^{\circ}([0,1]^{d})\) whose output values & Eq. (2.28) \\  & depend on exactly \(d_{*}\) components of their input vectors. & Above Eq. (2.30) \\ \(\mathscr{G}_{\infty}^{\mathsf{CH}}(d_{*})\) & \(\mathscr{G}_{\infty}^{\mathsf{CH}}(d_{*}):=[\underline{\cup}_{\Omega^{*}}^{ \mathsf{CH}}(d_{*},\beta,r)\) & Above Eq. (2.31) \\ \(\mathscr{G}_{d}^{\mathsf{CH}}(\cdots)\) & \(\mathscr{G}_{d}^{\mathsf{CH}}(q,K,d_{*},\beta,r)\) consists of compositional functions \(h_{q}\circ\) & Eq. (2.30) \\  & \(\cdots\circ h_{0}\) satisfying that each component function of \(h_{i}\) belongs to \(\mathscr{G}_{\infty}^{\mathsf{CH}}(d_{*},\beta,r)\). & Eq. (2.32) \\ \(\mathscr{G}_{d}^{\mathsf{CHOM}}(\cdots)\) & \(\mathscr{G}_{d}^{\mathsf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\) consists of compositional functions & Eq. (2.32) \\  & \(h_{q}\circ\cdots\circ h_{0}\) satisfying that each component function of \(h_{i}\) belongs to \(\mathscr{G}_{\infty}^{\mathsf{CH}}(d_{*},\beta,r)\cup\mathscr{G}_{\infty}^{ \mathsf{CH}}(d_{*})\). & \\ \(\mathscr{C}^{d,\beta,r,\Gamma,P}\) & The set of binary classifiers \(\mathsf{C}:[0,1]^{d}\rightarrow\{-1,+1\}\) such that \(\left\{x\in[0,1]^{d}\,|\,\mathsf{C}(x)=+1\right\}\) is the union of some disjoint closed regions with piecewise Holder smooth boundary. & Eq. (2.48) \\ \(\Delta_{\mathsf{C}}(x)\) & The distance from some point \(x\in[0,1]^{d}\) to the decision & Eq. (2.48) \\  & boundary of some classifier \(\mathsf{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\). & Above Eq. (1.10) \\ \(\mathscr{F}_{d}\) & The set of all Borel measurable functions from \([0,1]^{d}\) to & Above Eq. (1.10) \\ \((-\infty,\infty)\). & & \\ \(\mathscr{F}_{d}^{\mathsf{NNN}}(\cdots)\) & The class of ReLU neural networks defined on \(\mathbb{R}^{d}\). & Eq. (1.15) \\ \hline \end{tabular}
## 2 Main results

### Main Upper Bounds

In this subsection, we state our main results about upper bounds for the (excess) logistic risk or (excess) misclassification error of empirical logistic risk minimizers. The first result, given in Theorem 2.1, is an oracle-type inequality which provides upper bounds for the logistic risk of empirical logistic risk minimizers. Oracle-type inequalities have been extensively studied in the literature of nonparametric statistics (see [21] and references therein). As one of the main contributions in this paper, this inequality deserves special attention in its own right, allowing us to establish a novel strategy for generalization analysis. Before we state Theorem 2.1, we introduce some notations. For any pseudometric space \(\left(\mathcal{F},\rho\right)\) (cf. Section 10.5 of [1]) and \(\gamma\in\left(0,\infty\right)\), the covering number of \(\left(\mathcal{F},\rho\right)\) with radius \(\gamma\) is defined as

\[\mathcal{N}\left(\left(\mathcal{F},\rho\right),\gamma\right):=\inf\left\{\# \left(\mathcal{A}\right)\begin{vmatrix}\mathcal{A}\subset\mathcal{F},\,\text{and for any }f\in\mathcal{F}\text{ there}\\ \text{exists }g\in\mathcal{A}\text{ such that }\rho(f,g)\leq\gamma\end{vmatrix}\right\},\]

where we recall that \(\#\left(\mathcal{A}\right)\) denotes the number of elements of the set \(\mathcal{A}\). When the pseudometric \(\rho\) on \(\mathcal{F}\) is clear and no confusion arises, we write \(\mathcal{N}(\mathcal{F},\gamma)\) instead of \(\mathcal{N}\left(\left(\mathcal{F},\rho\right),\gamma\right)\) for simplicity. In particular, if \(\mathcal{F}\) consists of real-valued functions which are bounded on \(\left[0,1\right]^{d}\), we will use \(\mathcal{N}(\mathcal{F},\gamma)\) to

\begin{table}
\begin{tabular}{p{142.3pt} p{142.3pt} p{142.3pt}} \(\mathcal{H}_{0}^{d}\) & The set of all Borel probability measures on \(\left[0,1\right]^{d}\times\{-1,1\}\) & Above Eq. (1.10) \\ \(\mathcal{H}_{1}^{d,\beta,\gamma}\) & The set of all probability measures \(P\in\mathcal{H}_{0}^{d}\) whose conditional probability function coincides with some function in \(\mathcal{B}_{\gamma}^{\beta}\left(\left[0,1\right]^{d}\right)\)\(P_{X}\)-a.s.. \\ \(\mathcal{H}_{2,s_{1},c_{1},t_{1}}^{d,\beta,\gamma}\) & The set of all probability measures \(P\) in \(\mathcal{H}_{1}^{d,\beta,\gamma}\) satisfying the noise condition (2.24). \\ \(\mathcal{H}_{3,A}^{d,\beta,\gamma}\) & The set of all probability measures \(P\in\mathcal{H}_{0}^{d}\) whose marginal distribution on \(\left[0,1\right]^{d}\) is the Lebesgue measure and whose conditional probability function is in \(\mathcal{B}_{\gamma}^{\beta}\left(\left[0,1\right]^{d}\right)\) and bounded away from \(\frac{1}{2}\) almost surely. \\ \(\mathcal{H}_{4,q,K,d,\epsilon}^{d,\beta,\gamma}\) & The set of all probability measures \(P\in\mathcal{H}_{0}^{d}\) whose conditional probability function coincides with some function in \(\mathcal{G}_{d}^{\text{CHOM}}(q,K,d_{\ast},d_{\ast},\beta,r)\)\(P_{X}\)-a.s.. \\ \(\mathcal{H}_{5,A,q,K,d_{\ast}}^{d,\beta,\gamma}\) & The set of all probability measures \(P\in\mathcal{H}_{0}^{d}\) whose marginal distribution on \(\left[0,1\right]^{d}\) is the Lebesgue measure and whose conditional probability function is in \(\mathcal{G}_{d}^{\text{CH}}(q,K,d_{\ast},\beta,r)\) and bounded away from \(\frac{1}{2}\) almost surely. \\ \(\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma,\gamma, \gamma,\gamma
denote

\[\mathcal{N}\left(\left(\mathcal{F},\rho:(f,g)\mapsto\sup_{x\in[0,1]^{d}}|f(x)-g(x) |\,\right),\gamma\right) \tag{2.1}\]

unless otherwise specified. Recall that the \(\phi\)-risk of a measurable function \(f:[0,1]^{d}\to\mathbb{R}\) with respect to a distribution \(P\) on \([0,1]^{d}\times\{-1,1\}\) is denoted by \(\mathcal{R}_{P}^{\phi}(f)\) and defined in (1.4).

**Theorem 2.1**.: _Let \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) be an i.i.d. sample of a probability distribution \(P\) on \([0,1]^{d}\times\{-1,1\}\), \(\mathcal{F}\) be a nonempty class of uniformly bounded real-valued functions defined on \([0,1]^{d}\), and \(\hat{f}_{n}\) be an ERM with respect to the logistic loss \(\phi(t)=\log(1+\mathrm{e}^{-t})\) over \(\mathcal{F}\), i.e.,_

\[\hat{f}_{n}\in\operatorname*{arg\,min}_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1 }^{n}\phi\left(Y_{i}f(X_{i})\right). \tag{2.2}\]

_If there exists a measurable function \(\psi:[0,1]^{d}\times\{-1,1\}\to\mathbb{R}\) and a constant triple \((M,\Gamma,\gamma)\in(0,\infty)^{3}\) such that_

\[\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\leq\inf_{f\in \mathcal{F}}\int_{[0,1]^{d}\times\{-1,1\}}\phi(yf(x))\mathrm{d}P(x,y), \tag{2.3}\]

\[\sup\left\{\phi(t)\left|\left|t\right|\leq\sup_{f\in\mathcal{F}}\|f\|_{[0,1]^{ d}}\right\}\vee\sup\left\{\left|\psi(x,y)\right|\left|(x,y)\in[0,1]^{d} \times\{-1,1\}\right.\right\}\leq M, \tag{2.4}\]

\[\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(yf(x))-\psi\left(x,y)\right)^{2} \mathrm{d}P(x,y)\right. \tag{2.5}\]

_and_

\[W:=\max\left\{3,\,\mathcal{N}\left(\mathcal{F},\gamma\right)\right\}<\infty.\]

_Then for any \(\varepsilon\in(0,1)\), there holds_

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right] \tag{2.6}\] \[\leq 80\cdot\frac{(1+\varepsilon)^{2}}{\varepsilon}\cdot\frac{ \Gamma\log W}{n}+(20+20\varepsilon)\cdot\frac{M\log W}{n}+(20+20\varepsilon) \cdot\sqrt{\gamma}\cdot\sqrt{\frac{\Gamma\log W}{n}}\] \[\quad+4\gamma+(1+\varepsilon)\cdot\inf_{f\in\mathcal{F}}\left( \mathcal{R}_{P}^{\phi}(f)-\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P( x,y)\right).\]

According to its proof in Appendix A.3.2, Theorem 2.1 remains true when the logistic loss is replaced by any nonnegative function \(\phi\) satisfying

\[\left|\phi(t)-\phi(t^{\prime})\right|\leq\left|t-t^{\prime}\right|,\;\forall \;t,t^{\prime}\in\left[-\sup_{f\in\mathcal{F}}\|f\|_{[0,1]^{d}},\sup_{f\in \mathcal{F}}\|f\|_{[0,1]^{d}}\right].\]

Then by rescaling, Theorem 2.1 can be further generalized to the case when \(\phi\) is any nonnegative locally Lipschitz continuous loss function such as the exponential loss or the LUM (large-margin unified machine) loss (cf. [29]). Generalization analysis for classification with these loss functions based on oracle-type inequalities similar to Theorem 2.1 has been studied in our coming work [48].

Let us give some comments on conditions (2.3) and (2.5) of Theorem 2.1. To our knowledge, these two conditions are introduced for the first time in this paper, and will play pivotal roles in our estimates. Let \(\phi\) be the logistic loss and \(P\) be a probability measure on \([0,1]^{d}\times\{-1,1\}\). Recall that \(f_{\phi,P}^{*}\) denotes the target function of the logistic risk. If

\[\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)=\inf\left\{\left. \mathcal{R}_{P}^{\phi}(f)\right|f:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\}, \tag{2.7}\]
then condition (2.3) is satisfied and the left hand side of (2.6) is exactly \(\mathbb{E}\left[\mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}\right)\right]\). Therefore, Theorem 2.1 can be used to establish excess \(\phi\)-risk bounds for the \(\phi\)-ERM \(\hat{f}_{n}\). In particular, one can take \(\psi(x,y)\) to be \(\phi(yf_{\phi,P}^{*}(x))\) to ensure the equality (2.7) (recalling (1.7)). It should be pointed out that if \(\psi(x,y)=\phi(yf_{\phi,P}^{*}(x))\), inequality (2.5) is of the same form as the following inequality with \(\tau=1\), which asserts that there exist \(\tau\in[0,1]\) and \(\Gamma>0\) such that

\[\int_{[0,1]^{d}\times\{-1,1\}}\Big{(}\phi(yf(x))-\phi\left(yf_{\phi}^{*}(x) \right)\Big{)}^{2}\mathrm{d}P(x,y)\leq\Gamma\cdot\Big{(}\mathcal{E}_{P}^{\phi }(f)\Big{)}^{\tau},\,\forall\,\,f\in\mathcal{F}. \tag{2.8}\]

This inequality appears naturally when bounding the sample error by using concentration inequalities, which is of great importance in previous generalization analysis for binary classification (cf. condition (A4) in [22] and Definition 10.15 in [6]). In [9], the authors actually prove that if the target function \(f_{\phi,P}^{*}\) is bounded and the functions in \(\mathcal{F}\) are uniformly bounded by some \(F>0\), the inequality (2.5) holds with \(\psi(x,y)=\phi(yf_{\phi,P}^{*}(x))\) and

\[\Gamma=\frac{2}{\inf\left\{\phi^{\prime\prime}(t)\left|t\in\mathbb{R},\,\,|t| \leq\max\left\{F,\left\|f_{\phi,P}^{*}\right\|_{[0,1]^{d}}\right\}\right\}}.\]

Here \(\phi^{\prime\prime}(t)\) denotes the second order derivative of \(\phi(t)=\log(1+\mathrm{e}^{-t})\) which is given by \(\phi^{\prime\prime}(t)=\frac{e^{t}}{(1+\mathrm{e}^{t})^{2}}\). The boundedness of \(f_{\phi,P}^{*}\) is a key ingredient leading to the main results in [9] (see Section 3 for more details). However, \(f_{\phi,P}^{*}\) is explicitly given by \(\log\frac{n}{1-\eta}\) with \(\eta(x)=P(\{1\}\,|x)\), which tends to infinity when \(\eta\) approaches to \(0\) or \(1\). In some cases, the uniformly boundedness assumption on \(f_{\phi,P}^{*}\) is too restrictive. When \(f_{\phi,P}^{*}\) is unbounded, i.e., \(\|f_{\phi,P}^{*}\|_{[0,1]^{d}}=\infty\), condition (2.5) will not be satisfied by simply taking \(\psi(x,y)=\phi(yf_{\phi,P}^{*}(x))\). Since in this case we have \(\inf_{t\in(-\infty,+\infty)}\phi^{\prime\prime}(t)=0\), one cannot find a finite constant \(\Gamma\) to guarantee the validity of (2.5), i.e., the inequality (2.8) cannot hold for \(\tau=1\), which means the previous strategy for generalization analysis in [9] fails to work. In Theorem 2.1, the requirement for \(\psi(x,y)\) is much more flexible, we don't require \(\psi(x,y)\) to be \(\phi(yf_{\phi,P}^{*}(x))\) or even to satisfy (2.7). In this paper, by recurring to Theorem 2.1, we carefully construct \(\psi\) to avoid using \(f_{\phi,P}^{*}\) directly in the following estimates. Based on this strategy, under some mild regularity conditions on \(\eta\), we can develop a more general analysis to demonstrate the performance of neural network classifiers trained with the logistic loss regardless of the unboundedness of \(f_{\phi,P}^{*}\). The derived generalization bounds and rates of convergence are stated in Theorem 2.2, Theorem 2.3, and Theorem 2.5, which are new in the literature and constitute the main contributions of this paper. It is worth noticing that in Theorem 2.2 and Theorem 2.3, we use Theorem 2.1 to obtain optimal rates of convergence (up to some logarithmic factor), which demonstrates the tightness and power of the inequality (2.6) in Theorem 2.1. To obtain these optimal rates from Theorem 2.1, a delicate construction of \(\psi\) which allows small constants \(M\) and \(\Gamma\) in (2.4) and (2.5) is necessary. One frequently used form of \(\psi\) in this paper is

\[\psi: [0,1]^{d}\times\{-1,1\}\to\mathbb{R}, \tag{2.9}\] \[(x,y)\mapsto\begin{cases}\phi\left(y\log\frac{\eta(x)}{1-\eta(x )}\right),&\eta(x)\in[\delta_{1},1-\delta_{1}],\\ 0,&\eta(x)\in\{0,1\},\\ \eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)},&\eta(x)\in(0, \delta_{1})\cup(1-\delta_{1},1),\end{cases}\]

which can be regarded as a truncated version of \(\phi(yf_{\phi,P}^{*}(x))=\phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right)\), where \(\delta_{1}\) is some suitable constant in \((0,1/2]\). However, in Theorem 2.5 we use a different form of \(\psi\), which will be specified later.

The proof of Theorem 2.1 is based on the following error decomposition

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)-\Psi\right]\leq \mathrm{T}_{\varepsilon,\psi,n}+(1+\varepsilon)\cdot\inf_{g\in\mathcal{F}} \left(\mathcal{R}_{P}^{\phi}(g)-\Psi\right),\,\forall\,\,\varepsilon\in[0,1), \tag{2.10}\]
where \(\mathrm{T}_{\varepsilon,\psi,n}:=\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_ {n}\right)-\Psi-(1+\varepsilon)\cdot\frac{1}{n}\sum_{i=1}^{n}\left(\phi\left(Y_ {i}\hat{f}_{n}(X_{i})\right)-\psi(X_{i},Y_{i})\right)\right]\) and \(\Psi=\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\) (see (A.17)). Although (2.10) is true for \(\varepsilon=0\), it's better to take \(\varepsilon>0\) in (2.10) to obtain sharp rates of convergence. This is because bounding the term \(\mathrm{T}_{\varepsilon,\psi,n}\) with \(\varepsilon\in(0,1)\) is easier than bounding \(\mathrm{T}_{0,\psi,n}\). To see this, note that for \(\varepsilon\in(0,1)\) we have

\[\mathrm{T}_{\varepsilon,\psi,n}=(1+\varepsilon)\cdot\mathrm{T}_{0,\psi,n}- \varepsilon\cdot\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n} \right)-\Psi\right]\leq(1+\varepsilon)\cdot\mathrm{T}_{0,\psi,n},\]

meaning that we can always establish tighter upper bounds for \(\mathrm{T}_{\varepsilon,\psi,n}\) than for \(\mathrm{T}_{0,\psi,n}\) (up to the constant factor \(1+\varepsilon<2\)). Indeed, \(\varepsilon>0\) is necessary in establishing Theorem 2.1, as indicated in its proof in Appendix A.3.2. We also point out that, setting \(\varepsilon=0\) and \(\psi\equiv 0\) (hence \(\Psi=0\)) in (2.10), and subtracting \(\inf\left\{\mathcal{R}_{P}^{\phi}(g)\middle|g:[0,1]^{d}\to\mathbb{R}\text{ measurable}\right\}\) from both sides, we will obtain a simpler error decomposition

\[\begin{split}&\mathbb{E}\left[\mathcal{E}_{P}^{\phi}\left(\hat{f}_ {n}\right)\right]\leq\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n} \right)-\frac{1}{n}\sum_{i=1}^{n}\left(\phi\left(Y_{i}\hat{f}_{n}(X_{i}) \right)\right)\right]+\inf_{g\in\mathcal{F}}\mathcal{E}_{P}^{\phi}(g)\\ &\leq\mathbb{E}\left[\sup_{g\in\mathcal{F}}\left|\mathcal{R}_{P}^ {\phi}\left(g\right)-\frac{1}{n}\sum_{i=1}^{n}\left(\phi\left(Y_{i}g(X_{i}) \right)\right)\right|\right]+\inf_{g\in\mathcal{F}}\mathcal{E}_{P}^{\phi}(g), \end{split} \tag{2.11}\]

which is frequently used in the literature (see e.g., Lemma 2 in [25] and the proof of Proposition 4.1 in [31]). Note that (2.11) does not require the explicit form of \(f_{\phi,P}^{*}\), which means that we can also use this error decomposition to establish rates of convergence for \(\mathbb{E}\left[\mathcal{E}_{P}^{\phi}\{\hat{f}_{n}\}\right]\) regardless of the unboundedness of \(f_{\phi,P}^{*}\). However, in comparison with Theorem 2.1, using (2.11) may result in slow rates of convergence because of the absence of the positive parameter \(\varepsilon\) and a carefully constructed function \(\psi\).

We now state Theorem 2.2 which establishes generalization bounds for empirical logistic risk minimizers over DNNs. In order to present this result, we need the definition of Holder spaces [7]. The Holder space \(\mathcal{C}^{k,\lambda}(\Omega)\), where \(\Omega\subset\mathbb{R}^{d}\) is a closed domain, \(k\in\mathbb{N}\cup\{0\}\) and \(\lambda\in(0,1]\), consists of all those functions from \(\Omega\) to \(\mathbb{R}\) which have continuous derivatives up to order \(k\) and whose \(k\)-th partial derivatives are Holder-\(\lambda\) continuous on \(\Omega\). Here we say a function \(g:\Omega\to\mathbb{R}\) is Holder-\(\lambda\) continuous on \(\Omega\), if

\[\left|g\right|_{\mathcal{C}^{0,\lambda}(\Omega)}:=\sup_{\Omega\ni x\neq z\in \Omega}\frac{\left|g(x)-g(z)\right|}{\left\|x-z\right\|_{2}^{\lambda}}<\infty.\]

Then the Holder spaces \(\mathcal{C}^{k,\lambda}(\Omega)\) can be assigned the norm

\[\left\|f\right\|_{\mathcal{C}^{k,\lambda}(\Omega)}:=\max_{\left\|\mathbf{m} \right\|_{1}\leq k}\left\|\mathrm{D}^{\mathbf{m}}f\right\|_{\Omega}+\max_{\left\| \mathbf{m}\right\|_{1}=k}\left|\mathrm{D}^{\mathbf{m}}f\right|_{\mathcal{C}^{0, \lambda}(\Omega)}, \tag{2.12}\]

where \(\mathbf{m}=(m_{1},\cdots,m_{d})\in(\mathbb{N}\cup\{0\})^{d}\) ranges over multi-indices (hence \(\left\|\mathbf{m}\right\|_{1}=\sum_{i=1}^{d}m_{i}\)) and \(\mathrm{D}^{\mathbf{m}}f(x_{1},\ldots,x_{d})=\frac{\partial^{m_{1}}}{\partial x_{1 }^{m_{1}}}\cdots\frac{\partial^{m_{d}}}{\partial x_{d}^{m_{d}}}f(x_{1},\ldots,x _{d})\). Given \(\beta\in(0,\infty)\), we say a function \(f:\Omega\to\mathbb{R}\) is Holder-\(\beta\) smooth if \(f\in\mathcal{C}^{k,\lambda}(\Omega)\) with \(k=\left\lceil\beta\right\rceil-1\) and \(\lambda=\beta-\left\lceil\beta\right\rceil+1\), where \(\left\lceil\beta\right\rceil\) denotes the smallest integer than or equal to \(\beta\). For any \(\beta\in(0,\infty)\) and any \(r\in(0,\infty)\), let

\[\mathcal{B}_{r}^{\beta}\left(\Omega\right):=\left\{f:\Omega\to\mathbb{R}\begin{ }\left|f\in\mathcal{C}^{k,\lambda}(\Omega)\text{ and }\left\|f\right\|_{\mathcal{C}^{k,\lambda}(\Omega)}\leq r \text{ for }k=\\ -1+\left\lceil\beta\right\rceil\text{ and }\lambda=\beta-\left\lceil\beta \right\rceil+1\end{array}\right\} \tag{2.13}\]

denote the closed ball of radius \(r\) centered at the origin in the Holder space of order \(\beta\) on \(\Omega\). Recall that the space \(\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\) generated by fully connected neural networks is given in (1.15), which is parameterized by the depth and width of neural networks (bounded by \(G\) and \(N\)), the number of nonzero entries in weight matrices and bias vectors (bounded by \(S\)), and the upper bounds of neural network parameters and associated functions of form (1.14) (denoted by \(B\) and \(F\)). In the following theorem, we show that to ensure the rate of convergence as the sample size \(n\) becomes large, all these parameters should be taken within certain ranges scaling with \(n\). For two positive sequences \(\{\lambda_{n}\}_{n\geq 1}\) and \(\{\nu_{n}\}_{n\geq 1}\), we say \(\lambda_{n}\lesssim_{n}\nu_{n}\) holds if there exist \(n_{0}\in\mathbb{N}\) and a positive constant \(c\) independent of
such that \(\lambda_{n}\leq c\nu_{n},\forall\ n\geq n_{0}\). In addition, we write \(\lambda_{n}\asymp\nu_{n}\) if and only if \(\lambda_{n}\lesssim\nu_{n}\) and \(\nu_{n}\lesssim\lambda_{n}\). Recall that the excess misclassification error of \(f:\mathbb{R}^{d}\to\mathbb{R}\) with respect to some distribution \(P\) on \([0,1]^{d}\times\{-1,1\}\) is defined as

\[\mathcal{E}_{P}(f)=\mathcal{R}_{P}(f)-\inf\left\{\mathcal{R}_{P}(g)\left|g:[0, 1]^{d}\to\mathbb{R}\text{ is Borel measurable}\right.\right\},\]

where \(\mathcal{R}_{P}(f)\) denotes the misclassification error of \(f\) given by

\[\mathcal{R}_{P}(f)=P\left(\left\{\left.(x,y)\in[0,1]^{d}\times\{-1,1\}\right|y \neq\operatorname{sgn}(f(x))\right\}\right).\]

**Theorem 2.2**.: _Let \(d\in\mathbb{N}\), \((\beta,r)\in(0,\infty)^{2}\), \(n\in\mathbb{N}\), \(\nu\in[0,\infty)\), \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) be an i.i.d. sample in \([0,1]^{d}\times\{-1,1\}\) and \(\hat{f}_{n}^{\mathbf{FNN}}\) be an ERM with respect to the logistic loss \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) over \(\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\), i.e.,_

\[\hat{f}_{n}^{\mathbf{FNN}}\in\operatorname*{arg\,min}_{f\in\mathcal{F}_{d}^{ \mathbf{FNN}}(G,N,S,B,F)}\frac{1}{n}\sum_{i=1}^{n}\phi\left(Y_{i}f(X_{i}) \right). \tag{2.14}\]

_Define_

\[\mathcal{H}_{1}^{d,\beta,r}:=\left\{P\in\mathcal{H}_{0}^{d}\begin{bmatrix}P_{ X}(\left\{\left.z\in[0,1]^{d}\right|P(\left\{1\right\}|z)=\hat{\eta}(z)\right\}) \right.=\ 1\ \text{for}\\ \text{some }\hat{\eta}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\end{bmatrix}. \tag{2.15}\]

_Then there exists a constant \(\mathrm{c}\in(0,\infty)\) only depending on \((d,\beta,r)\), such that the estimator \(\hat{f}_{n}^{\mathbf{FNN}}\) defined by (2.14) with_

\[\begin{split}\mathrm{c}\log n&\leq G\lesssim\log n,\ N\asymp\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d}{d+\beta}},\ S\asymp \left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d}{d+\beta}}\cdot\log n,\\ & 1\leq B\lesssim n^{\nu},\text{ and }\ \frac{\beta}{d+\beta} \cdot\log n\leq F\lesssim\log n\end{split} \tag{2.16}\]

_satisfies_

\[\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E} _{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim\left(\frac {(\log n)^{5}}{n}\right)^{\frac{\beta}{\beta+d}} \tag{2.17}\]

_and_

\[\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E} _{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim\left(\frac{(\log n )^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}}. \tag{2.18}\]

Theorem 2.2 will be proved in Appendix A.3.4. As far as we know, for classification with neural networks and the logistic loss \(\phi\), generalization bounds presented in (2.17) and (2.18) establish fastest rates of convergence among the existing literature under the Holder smoothness condition on the conditional probability function \(\eta\) of the data distribution \(P\). Note that to obtain such generalization bounds in (2.17) and (2.18) we do not require any assumption on the marginal distribution \(P_{X}\) of the distribution \(P\). For example, we dot not require that \(P_{X}\) is absolutely continuous with respect to the Lebesgue measure. The rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+d}})\) in (2.17) for the convergence of excess \(\phi\)-risk is indeed optimal (up to some logarithmic factor) in the minimax sense (see Corollary 2.1 and comments therein). However, the rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}})\) in (2.18) for the convergence of excess misclassification error is not optimal. According to Theorem 4.1, Theorem 4.2, Theorem 4.3 and their proofs in [3], there holds

\[\inf_{\hat{f}_{n}}\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P\otimes n} \left[\mathcal{E}_{P}(\hat{f}_{n})\right]\asymp n^{-\frac{\beta}{2\beta+d}}, \tag{2.19}\]

where the infimum is taken over all \(\mathcal{F}_{d}\)-valued statistics from the sample \(\left\{(X_{i},Y_{i})\right\}_{i=1}^{n}\). Therefore, the rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}})\) in (2.18) does not match the minimax optimal rate \(\mathcal{O}(\left(\frac{1}{n}\right)^{\frac{\beta}{2\beta+d}})\). Despite
suboptimality, the rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}})\) in (2.18) is fairly close to the optimal rate \(\mathcal{O}(\left(\frac{1}{n}\right)^{\frac{\beta}{2\beta+d}})\), especially when \(\beta>>d\) because the exponents satisfy

\[\lim_{\beta\rightarrow+\infty}\frac{\beta}{2\beta+2d}=\frac{1}{2}=\lim_{\beta \rightarrow+\infty}\frac{\beta}{2\beta+d}.\]

In our proof of Theorem 2.2, the rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}})\) in (2.18) is derived directly from the rate \(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{\beta+d}}\) in (2.17) via the so-called calibration inequality which takes the form

\[\mathcal{E}_{P}(f)\leq c\cdot\left(\mathcal{E}_{P}^{\phi}(f)\right)^{\frac{1}{ 2}}\text{ for any }f\in\mathcal{F}_{d}\text{ and any }P\in\mathcal{H}_{0}^{d} \tag{2.20}\]

with \(c\) being a constant independent of \(P\) and \(f\) (see (A.102)). Indeed, it follows from Theorem 8.29 of [40] that

\[\mathcal{E}_{P}\left(f\right)\leq 2\sqrt{2}\cdot\left(\mathcal{E}_{P}^{\phi} \left(f\right)\right)^{\frac{1}{2}}\text{ for any }f\in\mathcal{F}_{d}\text{ and any }P\in\mathcal{H}_{0}^{d}. \tag{2.21}\]

In other words, (2.20) holds when \(c=2\sqrt{2}\). Interestingly, we can use Theorem 2.2 to obtain that the inequality (2.20) is optimal in the sense that the exponent \(\frac{1}{2}\) cannot be replaced by a larger one. Specifically, by using (2.17) of our Theorem 2.2 together with (2.19), we can prove that \(\frac{1}{2}\) is the largest number \(s\) such that there holds

\[\mathcal{E}_{P}(f)\leq c\cdot\left(\mathcal{E}_{P}^{\phi}(f)\right)^{s}\text{ for any }f\in\mathcal{F}_{d}\text{ and any }P\in\mathcal{H}_{0}^{d} \tag{2.22}\]

for some constant \(c\) independent of \(P\) or \(f\). We now demonstrate this by contradiction. Fix \(d\in\mathbb{N}\). Suppose there exists an \(s\in(1/2,\infty)\) and a \(c\in(0,\infty)\) such that (2.22) holds. Since

\[\lim_{\beta\rightarrow+\infty}\frac{(\frac{2}{3}\wedge s)\cdot\beta}{d+\beta} =\frac{2}{3}\wedge s>1/2=\lim_{\beta\rightarrow+\infty}\frac{\beta}{2\beta+d},\]

we can choose \(\beta\) large enough such that \(\frac{(\frac{2}{3}\wedge s)\cdot\beta}{d+\beta}>\frac{\beta}{2\beta+d}\). Besides, it follows from \(\mathcal{E}_{P}(f)\leq 1\) and (2.22) that

\[\mathcal{E}_{P}(f)\leq|\mathcal{E}_{P}(f)|^{\frac{2}{3}\wedge s}\leq\left|c \cdot\left(\mathcal{E}_{P}^{\phi}(f)\right)^{s}\right|^{\frac{2}{3}\wedge s }\leq(1+c)\cdot\left(\mathcal{E}_{P}^{\phi}(f)\right)^{(\frac{2}{3}\wedge s)} \tag{2.23}\]

for any \(f\in\mathcal{F}_{d}\) and any \(P\in\mathcal{H}_{0}^{d}\). Let \(r=3\) and \(\hat{f}_{n}^{\mathbf{FNN}}\) be the estimator in Theorem 2.2. Then it follows from (2.17), (2.19), (2.23) and Holder's inequality that

\[n^{-\frac{\beta}{2\beta+d}}\leq\inf_{\tilde{f}_{n}}\sup_{P\in \mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}(\hat{ f}_{n})\right]\leq\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}} \left[\mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\] \[\leq(1+c)\cdot\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\left(\mathbf{E} _{P^{\otimes n}}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n}^{\mathbf{FNN}})\right] \right)^{(\frac{2}{3}\wedge s)}\] \[\leq(1+c)\cdot\left(\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E} _{P^{\otimes n}}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n}^{\mathbf{FNN}}) \right]\right)^{(\frac{2}{3}\wedge s)}\] \[\lesssim\left(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{ \beta+d}}\right)^{(\frac{2}{3}\wedge s)}=\left(\frac{(\log n)^{5}}{n}\right)^{ (\frac{2}{3}\wedge s)\cdot\beta+d}.\]
Hence \(n^{-\frac{\beta}{2\beta+d}}\lesssim\left(\frac{\left(\log n\right)^{5}}{n}\right)^ {\frac{\left(\frac{\beta}{2\beta+2d}\right)\beta}{\beta+d}}\), which contradicts the fact that \(\frac{\left(\frac{\beta}{2\beta+\beta}\right)\cdot\beta}{d+\beta}>\frac{\beta} {2\beta+d}\). This proves the desired result. Due to the optimality of (2.20) and the minimax lower bound \(\mathcal{O}(n^{-\frac{\beta}{d+\beta}})\) for rates of convergence of the excess \(\phi\)-risk stated in Corollary 2.1, we deduce that rates of convergence of the excess misclassification error obtained directly from those of the excess \(\phi\)-risk and the calibration inequality which takes the form of (2.22) can never be faster than \(\mathcal{O}(n^{-\frac{\beta}{2\beta+2d}})\). Therefore, the convergence rate \(\mathcal{O}(\left(\frac{\left(\log n\right)^{5}}{n}\right)^{\frac{\beta}{2 \beta+2d}})\) of the excess misclassification error in (2.18) is the fastest one (up to the logarithmic term \(\left(\log n\right)^{\frac{5\beta}{2\beta+2d}}\)) among all those that are derived directly from the convergence rates of the excess \(\phi\)-risk and the calibration inequality of the form (2.22), which justifies the tightness of (2.18).

It should be pointed out that the rate \(\mathcal{O}(\left(\frac{\left(\log n\right)^{5}}{n}\right)^{\frac{\beta}{2 \beta+2d}})\) in (2.18) can be further improved if we assume the following noise condition (cf. [43]) on \(P\): there exist \(c_{1}>0\), \(t_{1}>0\) and \(s_{1}\in[0,\infty]\) such that

\[P_{X}\left(\left\{\left.x\in[0,1]^{d}\right|\left|2\cdot P(\left\{1\right\} \left.x\right)-1\right|\leq t\right\}\right)\leq c_{1}t^{s_{1}},\quad\forall\;0< t\leq t_{1}. \tag{2.24}\]

This condition measures the size of high-noisy points and reflects the noise level through the exponent \(s_{1}\in[0,\infty]\). Obviously, every distribution satisfies condition (2.24) with \(s_{1}=0\) and \(c_{1}=1\), whereas \(s_{1}=\infty\) implies that we have a low amount of noise in labeling \(x\), i.e., the conditional probability function \(P(\left\{1\right\}|x)\) is bounded away from \(1/2\) for \(P_{X}\)-almost all \(x\in[0,1]^{d}\). Under the noise condition (2.24), the calibration inequality for logistic loss \(\phi\) can be refined as

\[\mathcal{E}_{P}\left(f\right)\leq\bar{c}\cdot\left(\mathcal{E}_{P}^{\phi} \left(f\right)\right)^{\frac{s_{1}+1}{s_{1}+2}}\text{ for all }f\in\mathcal{F}_{d}, \tag{2.25}\]

where \(\bar{c}\in(0,\infty)\) is a constant only depending on \((s_{1},c_{1},t_{1})\), and \(\frac{s_{1}+1}{s_{1}+2}:=1\) if \(s_{1}=\infty\) (cf. Theorem 8.29 in [40] and Theorem 1.1 in [46]). Combining this calibration inequality (2.25) and (2.17), we can obtain an improved generalization bound given by

\[\sup_{P\in\mathcal{H}_{2,s_{1},c_{1},t_{1}}^{d,\beta,r}}\boldsymbol{E}_{P^{ \otimes n}}\left[\mathcal{E}_{P}\left(f_{n}^{\mathbf{FNN}}\right)\right]\lesssim \left(\frac{\left(\log n\right)^{5}}{n}\right)^{\frac{\left(s_{1}+1\right) \beta}{\left(s_{1}+2\right)\left(s+d\right)}},\]

where

\[\mathcal{H}_{2,s_{1},c_{1},t_{1}}^{d,\beta,r}:=\left\{\left.P\in\mathcal{H}_{ 1}^{d,\beta,r}\right|P\text{ satisfies \eqref{eq:2.24}}\right\}. \tag{2.26}\]

One can refer to Section 3 for more discussions about comparisons between Theorem 2.2 and other related results.

In our Theorem 2.2, the rates \(\left(\frac{\left(\log n\right)^{5}}{n}\right)^{\frac{\beta}{3+d}}\) and \(\left(\frac{\left(\log n\right)^{5}}{n}\right)^{\frac{\beta}{2\beta+2d}}\) become slow when the dimension \(d\) is large. This phenomenon, known as the curse of dimensionality, arises in our Theorem 2.2 because our assumption on the data distribution \(P\) is very mild and general. Except for the Holder smoothness condition on the conditional probability function \(\eta\) of \(P\), we do not require any other assumptions in our Theorem 2.2. The curse of dimensionality cannot be circumvented under such general assumption on \(P\), as shown in Corollary 2.1 and (2.19). Therefore, to overcome the curse of dimensionality, we need other assumptions. In our next theorem, we assume that \(\eta\) is the composition of several multivariate vector-valued functions \(h_{q}\circ\cdots\circ h_{1}\circ h_{0}\) such that each component function of \(h_{i}\) is either a Holder smooth function whose output values only depend on a small number of its input variables, or the function computing the maximum value of some of its input variables (see (2.32) and (2.34)). Under this assumption, the curse of dimensionality is circumvented because each component function of \(h_{i}\) is either essentially defined on a low-dimensional space or a very simple maximum value function. Our hierarchical composition assumption on the conditional probability function is convincing and likely to be met in practice because many phenomena in natural sciences can be "described well by processes that take place at a sequence of increasing scales and are local at each scale, in the sense that they can be described well by neighbor-to-neighbor interactions" (Appendix 2 of [35]). Similar compositional
assumptions have been adopted in many works such as [37, 25, 24]. One may refer to [34, 35, 36, 24] for more discussions about the reasonableness of such compositional assumptions.

In our compositional assumption mentioned above, we allow the component function of \(h_{i}\) to be the maximum value function, which is not Holder-\(\beta\) smooth when \(\beta>1\). The maximum value function is incorporated because taking the maximum value is an important operation to pass key information from lower scale levels to higher ones, which appears naturally in the compositional structure of the conditional probability function \(\eta\) in practical classification problems. To see this, let us consider the following example. Suppose the classification problem is to determine whether an input image contains a cat. We assume the data is perfectly classified, in the sense that the conditional probability function \(\eta\) is equal to zero or one almost surely. It should be noted that the assumption "\(\eta=0\) or \(1\) almost surely" does not conflict with the continuity of \(\eta\) because the support of the distribution of the input data may be unconnected. This classification task can be done by human beings through considering each subpart of the input image and determining whether each subpart contains a cat. Mathematically, let \(\mathcal{V}\) be a family of subset of \(\{1,2,\ldots,d\}\) which consists of all the index sets of those (considered) subparts of the input image \(x\in[0,1]^{d}\). \(\mathcal{V}\) should satisfy

\[\bigcup_{J\in\mathcal{V}}J=\{1,2,\ldots,d\}\]

because the union of all the subparts should cover the input image itself. For each \(J\in\mathcal{V}\), let

\[\eta_{J}((x)_{J})=\begin{cases}1,&\text{ if the subpart $(x)_{J}$ of the input image $x$ contains a cat},\\ 0,&\text{ if the subpart $(x)_{J}$ of the input image $x$ doesn't contains a cat}.\end{cases}\]

Then we will have \(\eta(x)=\max_{J\in\mathcal{V}}\left\{\eta_{J}((x)_{J})\right\}\) a.s. because

\[\eta(x)=1\xleftarrow{\text{a.s.}}x\text{ contains a cat}\Leftrightarrow\text{ at least one of the subpart $(x)_{J}$ contains a cat}\] \[\Leftrightarrow\eta_{J}((x)_{J})=1\text{ for at least one }J\in\mathcal{V}\Leftrightarrow\max_{J\in\mathcal{V}}\left\{\eta_{J}((x)_{J}) \right\}=1.\]

Hence the maximum value function emerges naturally in the expression of \(\eta\).

We now give the specific mathematical definition of our compositional model. For any \((d,d_{\star},d_{\star},\beta,r)\in\mathbb{N}\times\mathbb{N}\times(0,\infty) \times(0,\infty)\), define

\[\mathcal{G}_{d}^{\mathbf{M}}(d_{\star}):=\left\{f:[0,1]^{d}\to\mathbb{R} \begin{vmatrix}\exists\;I\subset\{1,2,\ldots,d\}\text{ such that }1\leq\#(I)\leq d_{\star}\text{ and }\\ f(x)=\max\left\{(x)_{i}\left|i\in I\right.\right\},\,\forall\,x\in[0,1]^{d} \end{vmatrix}\right\}, \tag{2.27}\]

and

\[\mathcal{G}_{d}^{\mathbf{H}}(d_{\star},\beta,r) \tag{2.28}\] \[:=\left\{f:[0,1]^{d}\to\mathbb{R}\begin{vmatrix}\exists\;\;I\subset \{1,2,\ldots,d\}\text{ and }g\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d_{\star}}\right)\text{ such that }\\ \#(I)=d_{\star}\text{ and }f(x)=g\left((x)_{I}\right)\text{ for all }x\in[0,1]^{d}\end{vmatrix}\right\}.\]

Thus \(\mathcal{G}_{d}^{\mathbf{M}}(d_{\star})\) consists of all functions from \([0,1]^{d}\) to \(\mathbb{R}\) which compute the maximum value of at most \(d_{\star}\) components of their input vectors, and \(\mathcal{G}_{d}^{\mathbf{H}}(d_{\star},\beta,r)\) consists of all functions from \([0,1]^{d}\) to \(\mathbb{R}\) which only depend on \(d_{\star}\) components of the input vector and are Holder-\(\beta\) smooth with corresponding Holder-\(\beta\) norm less than or equal to \(r\). Obviously,

\[\mathcal{G}_{d}^{\mathbf{H}}(d_{\star},\beta,r)=\varnothing,\;\forall\;(d,d_{ \star},\beta,r)\in\mathbb{N}\times\mathbb{N}\times(0,\infty)\times(0,\infty) \text{ with }d<d_{\star}. \tag{2.29}\]

Next, for any \((d_{\star},d_{\star},\beta,r)\in\mathbb{N}\times\mathbb{N}\times(0,\infty) \times(0,\infty)\), define \(\mathcal{G}_{\infty}^{\mathbf{H}}(d_{\star},\beta,r):=\bigcup_{d=1}^{\infty} \mathcal{G}_{d}^{\mathbf{H}}(d_{\star},\beta,r)\) and \(\mathcal{G}_{\infty}^{\mathbf{M}}(d_{\star}):=\bigcup_{d=1}^{\infty}\mathcal{G }_{d}^{\mathbf{M}}(d_{\star})\). Finally, for any \(\beta\in\mathbb{N}\cup\{0\}\), any \((\beta,r)\in(0,\infty)^{2}\) and any \((d,d_{\star},d_{\star},K)\in\mathbb{N}^{4}\) with

\[d_{\star}\leq\min\left\{d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\right\}, \tag{2.30}\]
define

\[\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r) \tag{2.31}\] \[:=\left\{h_{q}\circ\cdots\circ h_{1}\circ h_{0}\right.\]

and

\[\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},\beta,r) \tag{2.32}\] \[:=\left\{h_{q}\circ\cdots\circ h_{1}\circ h_{0}\right.\]

Obviously, we always have that \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\subset\mathcal{G}_{d}^{ \mathbf{CHOM}}(q,K,d_{*},\beta,r)\). The condition (2.30), which is equivalent to

\[d_{*}\leq\begin{cases}d,&\text{if }q=0,\\ d\wedge K,&\text{if }q>0,\end{cases}\]

is required in the above definitions because it follows from (2.29) that

\[\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)=\varnothing\text{ if }d_{*}> \min\left\{d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\right\}.\]

Thus we impose the condition (2.30) simply to avoid the trivial empty set. The space \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\) consists of composite functions \(h_{q}\circ\cdots h_{1}\circ h_{0}\) satisfying that each component function of \(h_{i}\) only depends on \(d_{*}\) components of its input vector and is Holder-\(\beta\) smooth with corresponding Holder-\(\beta\) norm less than or equal to \(r\). For example, the function \([0,1]^{4}\ni x\mapsto\sum\limits_{1\leq i<j\leq 4}(x)_{i}\cdot(x)_{j}\in \mathbb{R}\) belongs to \(\mathcal{G}_{4}^{\mathbf{CH}}(2,4,2,2,8)\) (cf. Figure 2.1). The definition of \(\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},\beta,r)\) is similar to that of \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\). The only difference is that, in comparison to \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\), we in the definition of \(\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},\beta,r)\) additionally allow the component function of \(h_{i}\) to be the function which computes the maximum value of at most \(d_{*}\) components of its input vector. For example, the function \([0,1]^{4}\ni x\mapsto\max\limits_{1\leq i<j\leq 4}(x)_{i}\cdot(x)_{j}\in \mathbb{R}\) belongs to \(\mathcal{G}_{4}^{\mathbf{CHOM}}(2,6,3,2,2,2)\) (cf. Figure 2.2). From the above description of the spaces \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\) and \(\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\), we see that the condition (2.30) is very natural because it merely requires the essential input dimension \(d_{*}\) of the Holder-\(\beta\) smooth component function of \(h_{i}\) to be less than or equal to its actual input dimension,
which is \(d\) (if \(i=0\)) or \(K\) (if \(i>0\)). At last, we point out that the space \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\) reduces to the Holder ball \(\mathcal{B}_{r}^{\beta}([0,1]^{d})\) when \(q=0\) and \(d_{*}=d\). Indeed, we have that

\[\begin{split}&\mathcal{B}_{r}^{\beta}([0,1]^{d})=\mathcal{G}_{d}^ {\mathbf{H}}(d,\beta,r)=\mathcal{G}_{d}^{\mathbf{CH}}(0,K,d,\beta,r)\\ &\subset\mathcal{G}_{d}^{\mathbf{CHOM}}(0,K,d_{*},d,\beta,r),\; \forall\;K\in\mathbb{N},\;d\in\mathbb{N},\;d_{*}\in\mathbb{N},\;\beta\in(0, \infty),r\in(0,\infty).\end{split} \tag{2.33}\]

Now we are in a position to state our Theorem 2.3, where we establish sharp convergence rates, which are free from the input dimension \(d\), for fully connected DNN classifiers trained with the logistic loss under the assumption that the conditional probability function \(\eta\) of the data distribution belongs to \(\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\). In particular, it can be shown the convergence rate of the excess logistic risk stated in (2.36) in Theorem 2.3 is optimal (up to some logarithmic term). Since \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\subset\mathcal{G}_{d}^{ \mathbf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\), the same convergences rates as in Theorem 2.3 can also be achieved under the slightly narrower assumption that \(\eta\) belongs to \(\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r)\). The results of Theorem 2.3 break the curse of dimensionality and help explain why deep neural networks perform well, especially in high-dimensional problems.

**Theorem 2.3**.: _Let \(q\in\mathbb{N}\cup\{0\}\), \((d,d_{*},d_{*},K)\in\mathbb{N}^{4}\) with \(d_{*}\leq\min\left\{d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\right\}\), \((\beta,r)\in(0,\infty)^{2}\), \(n\in\mathbb{N}\), \(\nu\in[0,\infty)\), \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) be an i.i.d. sample in \([0,1]^{d}\times\{-1,1\}\) and \(f_{n}^{\mathbf{FNN}}\) be an ERM with respect to the logistic loss \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) over the space \(\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\), which is given by (2.14). Define_

\[\mathcal{H}_{4,q,K,d_{*},d_{*}}^{d,\beta,r}:=\left\{P\in\mathcal{H}_{0}^{d} \left|P_{X}(\left\{\,z\in[0,1]^{d}\right]P(\left\{1\right\}|z)=\hat{\eta}(z) \right\})\;=\;1\;\;\text{for}\;\right\}. \tag{2.34}\]

_Then there exists a constant \(\mathrm{c}\in(0,\infty)\) only depending on \((d_{*},d_{*},\beta,r,q)\), such that the estimator
\(\hat{f}_{n}^{\mathbf{FNN}}\) defined by (2.14) with_

\[\begin{split}&\mathrm{c}\log n\leq G\lesssim\log n,\ N\asymp\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{ \star}}{d_{\star}+\beta\cdot(1\wedge\beta)^{q}}},\ S\asymp\left(\frac{(\log n)^{5 }}{n}\right)^{\frac{-d_{\star}}{d_{\star}+\beta\cdot(1\wedge\beta)^{q}}}\cdot \log n,\\ & 1\leq B\lesssim n^{\nu},\text{ and }\ \frac{\beta\cdot(1\wedge\beta)^{q}}{d_{\star}+\beta\cdot(1\wedge\beta)^{q}} \cdot\log n\leq F\lesssim\log n\end{split} \tag{2.35}\]

_satisfies_

\[\sup_{P\in\mathcal{H}_{4,q}^{d,\beta,r},\mathcal{E}_{P}\otimes n}\left[ \mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim \left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{ \star}+\beta\cdot(1\wedge\beta)^{q}}} \tag{2.36}\]

_and_

\[\sup_{P\in\mathcal{H}_{4,q}^{d,\beta,r},\mathcal{E}_{P}\otimes n}\left[ \mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim\left( \frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)^{q}}{2d_{\star}+ \beta\cdot(1\wedge\beta)^{q}}}. \tag{2.37}\]

The proof of Theorem 2.3 is given in Appendix A.3.4. Note that Theorem 2.3 directly leads to Theorem 2.2 because it follows from (2.33) that

\[\mathcal{H}_{1}^{d,\beta,r}\subset\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d, \beta,r}\ \text{if }q=0,\,d_{\star}=d\text{ and }d_{\star}=K=1.\]

Consequently, Theorem 2.3 can be regarded as a generalization of Theorem 2.2. Note that both the rates \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge \beta)^{q}}{d_{\star}+\beta\cdot(1\wedge\beta)^{q}}})\) and \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta) ^{q}}{2d_{\star}+2\beta\cdot(1\wedge\beta)^{q}}})\) in (2.36) and (2.37) are independent of the input dimension \(d\), thereby overcoming the curse of dimensionality. Moreover, according to Theorem 2.6 and the comments therein, the rate \(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{ \star}+\beta\cdot(1\wedge\beta)^{q}}}\) in (2.36) for the convergence of the excess logistic risk is even optimal (up to some logarithmic factor). This justifies the sharpness of Theorem 2.3.

Next, we would like to demonstrate the main idea of the proof of Theorem 2.3. The strategy we adopted is to apply Theorem 2.1 with a suitable \(\psi\) satisfying (2.7). Let \(P\) be an arbitrary probability in \(\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d,\beta,r}\) and denote by \(\eta\) the conditional probability function \(P(\{1\}\,|\cdot)\) of \(P\). According to the previous discussions, we cannot simply take \(\psi(x,y)=\phi(yf_{\phi,P}^{*}(x))\) as the target function \(f_{\phi,P}^{*}=\log\frac{\eta}{1-\eta}\) is unbounded. Instead, we define \(\psi\) by (2.9) for some carefully selected \(\delta_{1}\in(0,1/2]\). For such \(\psi\), we prove

\[\int_{[0,1]^{d}\times\{-1,1\}}\psi\left(x,y\right)\mathrm{d}P(x,y)=\inf\left\{ \mathcal{R}_{P}^{\phi}(f)\Big{|}\,f:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\} \tag{2.38}\]

in Lemma A.3, and establish a tight inequality of form (2.5) with \(\Gamma=\mathcal{O}((\log\frac{1}{\delta_{1}})^{2})\) in Lemma A.10. We then calculate the covering numbers of \(\mathcal{F}:=\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\) by Corollary A.1 and use Lemma A.15 to estimate the approximation error

\[\inf_{f\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi}(f)-\int_{[0,1]^{d}\times\{- 1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right)\]

which is essentially \(\inf_{f\in\mathcal{F}}\mathcal{E}_{P}^{\phi}(f)\). Substituting the above estimations into the right hand side of (2.6) and taking supremum over \(P\in\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d,\beta,r}\), we obtain (2.36). We then derive (2.37) from (2.36) through the calibration inequality (2.21).

We would like to point out that the above scheme for obtaining generalization bounds, which is built on our novel oracle-type inequality in Theorem 2.1 with a carefully constructed \(\psi\), is very general. This scheme can be used to establish generalization bounds for classification in other settings, provided that the estimation for the corresponding approximation error is given. For example, one can expect to establish generalization bounds for convolutional neural network (CNN) classification with the logistic loss by using Theorem 2.1 together with recent results about CNN approximation. CNNs perform convolutions instead of matrix multiplications in at least one of their layers (cf. Chapter 9 of
[11]). Approximation properties of various CNN architectures have been intensively studied recently. For instance, 1D CNN approximation is studied in [51, 50, 30, 8], and 2D CNN approximation is investigated in [24, 16]. With the help of these CNN approximation results and classical concentration techniques, generalization bounds for CNN classification have been established in many works such as [25, 24, 38, 10]. In our coming work [48], we will derive generalization bounds for CNN classification with logistic loss on spheres under the Sobolev smooth conditional probability assumption through the novel framework developed in our paper.

In our proof of Theorem 2.2 and Theorem 2.3, a tight error bound for neural network approximation of the logarithm function \(\log(\cdot)\) arises as a by-product. Indeed, for a given data distribution \(P\) on \([0,1]^{d}\times\{-1,1\}\), to estimate the approximation error of \(\mathcal{F}_{d}^{\mathbf{FNN}}\), we need to construct neural networks to approximate the target function \(f_{\phi,P}^{*}=\log\frac{\eta}{1-\eta}\), where \(\eta\) denotes the conditional probability function of \(P\). Due to the unboundedness of \(f_{\phi,P}^{*}\), one cannot approximate \(f_{\phi,P}^{*}\) directly. To overcome this difficulty, we consider truncating \(f_{\phi,P}^{*}\) to obtain an efficient approximation. We design neural networks \(\tilde{\eta}\) and \(\tilde{l}\) to approximate \(\eta\) on \([0,1]^{d}\) and \(\log(\cdot)\) on \([\delta_{n},1-\delta_{n}]\) respectively, where \(\delta_{n}\in(0,1/4]\) is a carefully selected number which depends on the sample size \(n\) and tends to zero as \(n\to\infty\). Let \(\Pi_{\delta_{n}}\) denote the clipping function given by \(\Pi_{\delta_{n}}:\mathbb{R}\to[\delta_{n},1-\delta_{n}],t\mapsto\arg\min_{t^{ \prime}\in[\delta_{n},1-\delta_{n}]}|t^{\prime}-t|\). Then \(\tilde{L}:t\mapsto\tilde{l}(\Pi_{\delta_{n}}(t))-\tilde{l}(1-\Pi_{\delta_{n}} (t))\) is a neural network which approximates the function

\[\overline{L}_{\delta_{n}}:t\mapsto\begin{cases}\log\frac{t}{1-t},&\text{if }t \in[\delta_{n},1-\delta_{n}],\\ \log\frac{1-\delta_{n}}{\delta_{n}},&\text{if }t>1-\delta_{n},\\ \log\frac{\delta_{n}}{1-\delta_{n}},&\text{if }t<\delta_{n},\end{cases} \tag{2.39}\]

meaning that the function \(\tilde{L}(\tilde{\eta}(x))=\tilde{l}\left(\Pi_{\delta_{n}}\left(\tilde{\eta} (x)\right)\right)-\tilde{l}\left(\Pi_{\delta_{n}}\left(1-\tilde{\eta}(x) \right)\right)\) is a neural network which approximates the truncated \(f_{\phi,P}^{*}\) given by

\[\overline{L}_{\delta_{n}}\circ\eta:x\mapsto\overline{L}_{\delta_{n}}(\eta(x)) =\begin{cases}f_{\phi,P}^{*}(x),&\text{if }\left|f_{\phi,P}^{*}(x)\right|\leq\log\frac{1- \delta_{n}}{\delta_{n}},\\ \operatorname{sgn}(f_{\phi,P}^{*}(x))\log\frac{1-\delta_{n}}{\delta_{n}},& \text{otherwise}.\end{cases}\]

One can build \(\tilde{\eta}\) by applying some existing results on approximation theory of neural networks (see Appendix A.2). However, the construction of \(\tilde{l}\) requires more effort. Since the logarithm function \(\log(\cdot)\) is unbounded near \(0\), which leads to the blow-up of its Holder norm on \([\delta_{n},1-\delta_{n}]\) when \(\delta_{n}\) is becoming small, existing conclusions, e.g., the results in Appendix A.2, cannot yield satisfactory error bounds for neural network approximation of \(\log(\cdot)\) on \([\delta_{n},1-\delta_{n}]\). To see this, let us consider using Theorem A.2 to estimate the approximation error directly. Note that approximating \(\log(\cdot)\) on \([\delta_{n},1-\delta_{n}]\) is equivalent to approximating \(l_{\delta_{n}}(t):=\log((1-2\delta_{n})t+\delta_{n})\) on \([0,1]\). For \(\beta_{1}>0\) with \(k=\lceil\beta_{1}-1\rceil\) and \(\lambda=\beta_{1}-\lceil\beta_{1}-1\rceil\), denote by \(l_{\delta_{n}}^{(k)}\) the \(k\)-th derivative of \(l_{\delta_{n}}\). Then there holds

\[\|l_{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\geq\sup_{0 \leq t<t^{\prime}\leq 1}\frac{\left|l_{\delta_{n}}^{(k)}(t)-l_{\delta_{n}}^{(k)}(t^{\prime}) \right|}{\left|t-t^{\prime}\right|^{\lambda}}\] \[\geq\frac{\left|l_{\delta_{n}}^{(k)}(0)-l_{\delta_{n}}^{(k)} \left(\frac{\delta_{n}}{1-2\delta_{n}}\right)\right|}{\left|0-\frac{\delta_{n} }{1-2\delta_{n}}\right|^{\lambda}}\geq\inf_{t\in\left[0,\frac{\delta_{n}}{1-2 \delta_{n}}\right]}\frac{\left|l_{\delta_{n}}^{(k+1)}(t)\right|\cdot\left|0- \frac{\delta_{n}}{1-2\delta_{n}}\right|}{\left|0-\frac{\delta_{n}}{1-2\delta_{ n}}\right|^{\lambda}}\] \[=\inf_{t\in[0,\frac{\delta_{n}}{1-2\delta_{n}}]}\frac{\left|\frac {k!}{((1-2\delta_{n})t+\delta_{n})^{k+1}}\right|\cdot\left(1-2\delta_{n}\right) ^{k+1}\cdot\left|0-\frac{\delta_{n}}{1-2\delta_{n}}\right|}{\left|0-\frac{ \delta_{n}}{1-2\delta_{n}}\right|^{\lambda}}=\frac{k!}{2^{k+1}}\cdot(1-2 \delta_{n})^{k+\lambda}\cdot\frac{1}{\delta_{n}^{k+\lambda}}.\]

Hence it follows from \(\delta_{n}\in(0,1/4]\) that

\[\|l_{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\geq\frac{\lceil\beta_{1}-1 \rceil!}{2^{\lceil\beta_{1}\rceil}}\cdot(1-2\delta_{n})^{\beta_{1}}\cdot \frac{1}{\delta_{n}^{\beta_{1}}}\geq\frac{\lceil\beta_{1}-1\rceil!}{4^{\lceil \beta_{1}\rceil}}\cdot\frac{1}{\delta_{n}^{\beta_{1}}}\geq\frac{3}{128}\cdot \frac{1}{\delta_{n}^{\beta_{1}}}.\]
By Theorem A.2, for any positive integers \(m\) and \(M^{\prime}\) with

\[M^{\prime}\geq\max\left\{(\beta_{1}+1),\left(\|l_{\delta_{n}}\|_{\mathcal{C}^{k, \lambda}([0,1])}\,\lceil\beta_{1}\rceil+1\right)\cdot\mathrm{e}\right\}\geq\|l _{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\geq\frac{3}{128}\cdot\frac{1}{ \delta_{n}^{\beta_{1}}}, \tag{2.40}\]

there exists a neural network

\[\tilde{f}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(14m(2+\log_{2}\left(1\lor \beta_{1}\right)),6\left(1+\lceil\beta_{1}\rceil\right)M^{\prime},987(2+\beta _{1})^{4}M^{\prime}m,1,\infty\right) \tag{2.41}\]

such that

\[\sup_{x\in[0,1]}\left|l_{\delta_{n}}(x)-\tilde{f}(x)\right| \leq\|l_{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\cdot\lceil \beta_{1}\rceil\cdot 3^{\beta_{1}}M^{\prime-\beta_{1}}\] \[\qquad+\left(1+2\,\|l_{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\cdot\lceil\beta_{1}\rceil\right)\cdot 6\cdot(2+\beta_{1}^{2})\cdot M^{ \prime}\cdot 2^{-m}.\]

To make this error less than or equal to a given error threshold \(\varepsilon_{n}\) (depending on \(n\)), there must hold

\[\varepsilon_{n}\geq\|l_{\delta_{n}}\|_{\mathcal{C}^{k,\lambda}([0,1])}\cdot \lceil\beta_{1}\rceil\cdot 3^{\beta_{1}}M^{\prime-\beta_{1}}\geq\|l_{\delta_{n}}\|_{ \mathcal{C}^{k,\lambda}([0,1])}\cdot M^{\prime-\beta_{1}}\geq M^{\prime-\beta _{1}}\cdot\frac{3}{128}\cdot\frac{1}{\delta_{n}^{\beta_{1}}}.\]

This together with (2.40) gives

\[M^{\prime}\geq\max\left\{\frac{3}{128}\cdot\frac{1}{\delta_{n}^{\beta_{1}}}, \varepsilon_{n}^{-1/\beta_{1}}\cdot\left|\frac{3}{128}\right|^{1/\beta_{1}} \cdot\frac{1}{\delta_{n}}\right\}. \tag{2.42}\]

Consequently, the width and the number of nonzero parameters of \(\tilde{f}\) are greater than or equal to the right hand side of (2.42), which may be too large when \(\delta_{n}\) is small (recall that \(\delta_{n}\to 0\) as \(n\to\infty\)). In this paper, we establish a new sharp error bound for approximating the natural logarithm function \(\log(\cdot)\) on \([\delta_{n},1-\delta_{n}]\), which indicates that one can achieve the same approximation error by using a much smaller network. This refined error bound is given in Theorem 2.4 which is critical in our proof of Theorem 2.2 and also deserves special attention in its own right.

**Theorem 2.4**.: _Given \(a\in(0,1/2]\), \(b\in(a,1]\), \(\alpha\in(0,\infty)\) and \(\varepsilon\in(0,1/2]\), there exists_

\[\tilde{f}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(A_{1}\log\frac{1} {\varepsilon}+139\log\frac{1}{a}\,,\;A_{2}\left|\frac{1}{\varepsilon}\right|^{ \frac{1}{\alpha}}\cdot\log\frac{1}{a}\,,\right.\] \[\left.A_{3}\left|\frac{1}{\varepsilon}\right|^{\frac{1}{\alpha}} \cdot\left|\log\frac{1}{\varepsilon}\right|\cdot\log\frac{1}{a}+65440\left| \log\frac{1}{a}\right|^{2},1,\infty\right)\]

_such that_

\[\sup_{z\in[a,b]}\left|\log z-\tilde{f}(z)\right|\leq\varepsilon\text{ and }\log a\leq\tilde{f}(t)\leq\log b,\;\forall\;t\in\mathbb{R},\]

_where \((A_{1},A_{2},A_{3})\in(0,\infty)^{3}\) are constants depending only on \(\alpha\)._

In Theorem 2.4, we show that for each fixed \(\alpha\in(0,\infty)\) one can construct a neural network to approximate the natural logarithm function \(\log(\cdot)\) on \([a,b]\) with error \(\varepsilon\), where the depth, width and number of nonzero parameters of this neural network are in the same order of magnitude as \(\log\frac{1}{\varepsilon}+\log\frac{1}{a}\), \(\left(\frac{1}{\varepsilon}\right)^{\frac{1}{\alpha}}\left(\log\frac{1}{a}\right)\) and \(\left(\frac{1}{\varepsilon}\right)^{\frac{1}{\alpha}}\left(\log\frac{1}{ \varepsilon}\right)\left(\log\frac{1}{a}\right)+\left(\log\frac{1}{a}\right)^{2}\) respectively. Recall that in our generalization analysis we need to approximate \(\log\) on \([\delta_{n},1-\delta_{n}]\), which is equivalent to approximating \(l_{\delta_{n}}(t)=\log((1-2\delta_{n})t+\delta_{n})\) on \([0,1]\). Let \(\varepsilon_{n}\in(0,1/2]\) denote the desired accuracy of the approximation of \(l_{\delta_{n}}\) on \([0,1]\), which depends on the sample size \(n\) and converges to zero as \(n\to\infty\). Using Theorem 2.4 with \(\alpha=2\beta_{1}\), we deduce that for any \(\beta_{1}>0\) one can approximate \(l_{\delta_{n}}\) on \([0,1]\) with error \(\varepsilon_{n}\) by a network of which the width and the number of nonzero parameters are less than \(C_{\beta_{1}}\varepsilon_{n}^{-\frac{1}{2\beta_{1}}}\left|\log\varepsilon_{n} \right|\cdot\left|\log\delta_{n}\right|^{2}\) with some constant \(C_{\beta_{1}}>0\) (depending only on \(\beta_{1}\)). The complexity of this neural network is
much smaller than that of \(\tilde{f}\) defined in (2.41) with (2.42) as \(n\to\infty\) since \(\left|\log\delta_{n}\right|^{2}=\mathrm{o}\left(1/\delta_{n}\right)\) and \(\varepsilon_{n}^{-\frac{1}{2\beta_{1}}}\left|\log\varepsilon_{n}\right|= \mathrm{o}\left(\varepsilon_{n}^{-1/\beta_{1}}\right)\) as \(n\to\infty\). In particular, when

\[\frac{1}{n^{\theta_{2}}}\lesssim\varepsilon_{n}\wedge\delta_{n}\leq \varepsilon_{n}\vee\delta_{n}\lesssim\frac{1}{n^{\theta_{1}}}\text{ for some }\theta_{2}\geq\theta_{1}>0\text{ independent of }n\text{ or }\beta_{1}, \tag{2.43}\]

which occurs in our generalization analysis (e.g., in our proof of Theorem 2.3, we essentially take \(\varepsilon_{n}=\delta_{n}\asymp\left(\frac{(\log n)^{5}}{n}\right)^{\frac{ \beta+(1+\beta)\tilde{\eta}}{4+\beta+(1+\beta)\tilde{\eta}^{3}}}\), meaning that \(n^{\frac{-\beta-(1+\beta)\tilde{\eta}}{4+\beta+(1+\beta)\tilde{\eta}^{3}}} \lesssim\varepsilon_{n}=\delta_{n}\lesssim n^{\frac{-\beta-(1+\beta)\tilde{ \eta}}{24+\beta+(1+\beta)\tilde{\eta}^{3}}}\) (cf. (A.78), (A.82), (A.91) and (A.96)), we will have that the right hand side of (2.42) grows no slower than \(n^{\theta_{1}+\theta_{1}/\beta_{1}}\). Hence, in this case, no matter what \(\beta_{1}\) is, the width and the number of nonzero parameters of the network \(\tilde{f}\), which approximates \(l_{\delta_{n}}\) on \([0,1]\) with error \(\varepsilon_{n}\) and is obtained by using Theorem A.2 directly (cf. (2.41)), will grow faster than \(n^{\theta_{1}}\) as \(n\to\infty\). However, it follows from Theorem 2.4 that there exists a network \(\overline{f}\) of which the width and the number of nonzero parameters are less than \(C_{\beta_{1}}\varepsilon_{n}^{-\frac{1}{2\beta_{1}}}\left|\log\varepsilon_{n} \right|\cdot\left|\log\delta_{n}\right|^{2}\lesssim n^{\frac{\theta_{2}}{2 \beta_{1}}}\left|\log n\right|^{3}\) such that it achieves the same approximation error as that of \(\tilde{f}\). By taking \(\beta_{1}\) large enough we can make the growth (as \(n\to\infty\)) of the width and the number of nonzero parameters of \(\overline{f}\) slower than \(n^{\theta}\) for arbitrary \(\theta\in(0,\theta_{1}]\). Therefore, in the usual case when the complexity of \(\tilde{\eta}\) is not too small in the sense that the width and the number of nonzero parameters of \(\tilde{\eta}\) grow faster than \(n^{\theta_{3}}\) as \(n\to\infty\) for some \(\theta_{3}\in(0,\infty)\) independent of \(n\) or \(\beta_{1}\), we can use Theorem 2.4 with a large enough \(\alpha=2\beta_{1}\) to construct the desired network \(\tilde{l}\) of which the complexity is insignificant in comparison to that of \(\tilde{L}\circ\tilde{\eta}\). In other words, the neural network approximation of logarithmic function based on Theorem 2.4 brings little complexity in approximating the target function \(f_{\phi,p}^{\ast}\). The above discussion demonstrates the tightness of the inequality in Theorem 2.4 and the advantage of Theorem 2.4 over those general results on approximation theory of neural networks such as Theorem A.2.

It is worth mentioning that an alternative way to approximate the function \(\overline{L}_{\delta_{n}}\) defined in (2.39) is by simply using its piecewise linear interpolation. For example, in [25], the authors express the piecewise linear interpolation of \(\overline{L}_{\delta_{n}}\) at equidistant points by a neural network \(\tilde{L}\), and construct a CNN \(\tilde{\eta}\) to approximate \(\eta\), leading to an approximation of the truncated target function of the logistic risk \(\tilde{L}\circ\tilde{\eta}\). It follows from Proposition 3.2.4 of [2] that

\[h_{n}^{2}\lesssim\left\|\tilde{L}-\overline{L}_{\delta_{n}}\right\|_{|\delta_ {n},1-\delta_{n}|}\lesssim\frac{h_{n}^{2}}{\delta_{n}^{2}}, \tag{2.44}\]

where \(h_{n}\) denotes the step size of the interpolation. Therefore, to ensure the error bound \(\varepsilon_{n}\) for the approximation of \(\overline{L}_{\delta_{n}}\) by \(\tilde{L}\), we must have \(h_{n}\lesssim\sqrt{\varepsilon_{n}}\), implying that the number of nonzero parameters of \(\tilde{L}\) will grow no slower than \(\frac{1}{h_{n}}\gtrsim\frac{1}{\sqrt{\varepsilon_{n}}}\) as \(n\to\infty\). Consequently, in the case (2.43), we will have that the number of nonzero parameters of \(\tilde{L}\) will grow no slower than \(n^{\theta_{1}/2}\). Therefore, in contrast to using Theorem 2.4, we cannot make the number of nonzero parameters of the network \(\tilde{L}\) obtained from piecewise linear interpolation grow slower than \(n^{\theta}\) for arbitrarily small \(\theta>0\). As a result, using piecewise linear interpolation to approximate \(\overline{L}_{\delta_{n}}\) may bring extra complexity in establishing the approximation of the target function. However, the advantage of using piecewise linear interpolation is that one can make the depth or width of the network \(\tilde{L}\) which expresses the desired interpolation bounded as \(n\to\infty\) (cf. Lemma 7 in [25] and its proof therein).

The proof of Theorem 2.4 is in Appendix A.3.3. The key observation in our proof is the fact that for all \(k\in\mathbb{N}\), the following holds true:

\[\log x=\log(2^{k}\cdot x)-k\log 2,\ \ \ \ \forall\ x\in(0,\infty). \tag{2.45}\]

Then we can use the values of \(\log(\cdot)\) which are taken far away from zero (i.e., \(\log(2^{k}\cdot x)\) in the right hand side of (2.45)) to determine its values taken near zero, while approximating the former is more efficient as the Holder norm of the natural logarithm function on domains far away from zero can be well controlled.

In the next theorem, we show that if the data distribution has a piecewise smooth decision boundary, then DNN classifiers trained by empirical logistic risk minimization can also achieve dimension-free
rates of convergence under the noise condition (2.24) and a margin condition (see (2.51) below). Before stating this result, we need to introduce this margin condition and relevant concepts.

We first define the set of (binary) classifiers which have a piecewise Holder smooth decision boundary. We will adopt similar notations from [22] to describe this set. Specifically, let \(\beta,r\in(0,\infty)\) and \(I,\Theta\in\mathbb{N}\). For \(g\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d-1}\right)\) and \(j=1,2,\cdots,d\), we define horizon function \(\Psi_{g,j}:[0,1]^{d}\to\{0,1\}\) as \(\Psi_{g,j}(x):=\mathbb{1}_{\{(x)_{j}\geq g(x_{-j})\}}\), where \(x_{-j}:=((x)_{1},\cdots,(x)_{j-1},(x)_{j+1},\cdots,(x)_{d})\in[0,1]^{d-1}\). For each horizon function, the corresponding basis piece \(\Lambda_{g,j}\) is defined as \(\Lambda_{g,j}:=\left\{x\in[0,1]^{d}\left|\Psi_{g,j}(x)=1\right.\right\}\). Note that \(\Lambda_{g,j}=\left\{x\in[0,1]^{d}\right|(x)_{j}\geq\max\left\{0,g(x_{-j}) \right\}\right\}\). Thus \(\Lambda_{g,j}\) is enclosed by the hypersurface \(\mathcal{S}_{g,j}:=\left\{x\in[0,1]^{d}\right|(x)_{j}=\max\left\{0,g(x_{-j}) \right\}\right\}\) and (part of) the boundary of \([0,1]^{d}\). We then define the set of pieces which are the intersection of \(I\) basis pieces as

\[\mathcal{A}^{d,\beta,r,I}:=\left\{A\left|A=\bigcap_{k=1}^{I}\Lambda_{g_{k},j_ {k}}\text{ for some }j_{k}\in\{1,2,\cdots,d\}\text{ and }g_{k}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d-1}\right)\ \right\},\]

and define \(\mathcal{C}^{d,\beta,r,I,\Theta}\) to be a set of binary classifiers as

\[\mathcal{C}^{d,\beta,r,I,\Theta} \tag{2.46}\] \[:=\left\{\begin{array}{l}\mathtt{C}(x)=2\sum_{i=1}^{\Theta} \mathbb{1}_{A_{i}}(x)-1:[0,1]^{d}\to\{-1,1\}\Bigg{|}\begin{array}{l}A_{1},A _{2},A_{3},\cdots,A_{\Theta}\text{ are }\\ \text{disjoint sets in }\mathcal{A}^{d,\beta,r,I}\end{array}\right\}.\]

Thus \(\mathcal{C}^{d,\beta,r,I,\Theta}\) consists of all binary classifiers which are equal to \(+1\) on some disjoint sets \(A_{1},\ldots,A_{\Theta}\) in \(\mathcal{A}^{d,\beta,r,I}\) and \(-1\) otherwise. Let \(A_{t}=\cap_{k=1}^{I}\Lambda_{g_{t,k},j_{t,k}}\) (\(t=1,2,\ldots,\Theta\)) be arbitrary disjoint sets in \(\mathcal{A}^{d,\beta,r,I}\), where \(j_{t,k}\in\{1,2,\ldots,d\}\) and \(g_{t,k}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d-1}\right)\). Then \(\mathtt{C}:[0,1]^{d}\to\{-1,1\},x\mapsto 2\sum_{i=1}^{\Theta}\mathbb{1}_{A_{i}}(x)-1\) is a classifier in \(\mathcal{C}^{d,\beta,r,I,\Theta}\). Recall that \(\Lambda_{g_{t,k},j_{t,k}}\) is enclosed by \(\mathcal{S}_{g_{t,k},j_{t,k}}\) and (part of) the boundary of \([0,1]^{d}\) for each \(t,k\). Hence for each \(t\), the region \(A_{t}\) is enclosed by hypersurfaces \(\mathcal{S}_{g_{t,k},j_{t,k}}\) (\(k=1,\ldots,I\)) and (part of) the boundary of \([0,1]^{d}\). We say the piecewise Holder smooth hypersurface

\[D_{\mathtt{C}}^{*}:=\bigcup_{t=1}^{\Theta}\bigcup_{k=1}^{I}\left(\mathcal{S}_ {g_{t,k},j_{t,k}}\cap A_{t}\right) \tag{2.47}\]

is the _decision boundary_ of the classifier \(\mathtt{C}\) because intuitively, points on different sides of \(D_{\mathtt{C}}^{*}\) are classified into different categories (i.e. \(+1\) and \(-1\)) by \(\mathtt{C}\) (cf. Figure 2.3). Denote by \(\Delta_{\mathtt{C}}(x)\) the distance from \(x\in[0,1]^{d}\) to the decision boundary \(D_{\mathtt{C}}^{*}\), i.e.,

\[\Delta_{\mathtt{C}}(x):=\inf\left\{\left\|x-x^{\prime}\right\|_{2}\left|x^{ \prime}\in D_{\mathtt{C}}^{*}\right\}. \tag{2.48}\]

We then describe the margin condition mentioned above. Let \(P\) be a probability measure on \([0,1]^{d}\times\{-1,1\}\), which we regard as the joint distribution of the input and output data, and \(\eta(\cdot)=P(\{1\}\mid\cdot)\) is the conditional probability function of \(P\). The corresponding Bayes classifier is the sign of \(2\eta-1\) which minimizes the misclassification error over all measurable functions, i.e.,

\[\mathcal{R}_{P}(\operatorname{sgn}(2\eta-1))=\mathcal{R}_{P}(2\eta-1)=\inf \left\{\mathcal{R}_{P}(f)\left|f:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right.\right\}. \tag{2.49}\]

We say the distribution \(P\) has a piecewise smooth decision boundary if

\[\exists\mathtt{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\text{ s.t. }\text{ sgn}(2\eta-1)\xrightarrow{P_{\mathtt{X}\text{-a.s.}}}\mathtt{C},\]

that is,

\[P_{X}\left(\left\{x\in[0,1]^{d}\right|\operatorname{sgn}(2\cdot P(\{1\}\mid x )-1)=\mathtt{C}(x)\right\}\right)=1 \tag{2.50}\]

for some \(\mathtt{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\). Suppose \(\mathtt{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\) and (2.50) holds. We call \(D_{\mathtt{C}}^{*}\) the _decision boundary_ of \(P\), and for \(c_{2}\in(0,\infty)\), \(t_{2}\in(0,\infty)\), \(s_{2}\in[0,\infty]\), we use the following condition

\[P_{X}\left(\left\{x\in[0,1]^{d}\right|\Delta_{\mathtt{C}}(x)\leq t\right\} \right)\leq c_{2}t^{s_{2}},\quad\forall\;0<t\leq t_{2}, \tag{2.51}\]

which we call the _margin condition_, to measure the concentration of the input distribution \(P_{X}\) near the decision boundary \(D_{\mathtt{C}}^{*}\) of \(P\). In particular, when the input data are bounded away from the decision boundary \(D_{\mathtt{C}}^{*}\) of \(P\) (\(P_{X}\)-a.s.), (2.51) will hold for \(s_{2}=\infty\).

Now we are ready to give our next main theorem.
**Theorem 2.5**.: _Let \(d\in\mathbb{N}\cap[2,\infty)\), \((n,I,\Theta)\in\mathbb{N}^{3}\), \((\beta,r,t_{1},t_{2},c_{1},c_{2})\in(0,\infty)^{6}\), \((s_{1},s_{2})\in[0,\infty]^{2}\), \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) be a sample in \([0,1]^{d}\times\{-1,1\}\) and \(\hat{f}_{n}^{\mathbf{FNN}}\) be an ERM with respect to the logistic loss \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) over \(\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\) which is given by (2.14). Define_

\[\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_{2}}:= \left\{P\in\mathcal{H}_{0}^{d}\Big{|}\begin{subarray}{c}\eqref{eq:2.4}\text{, \eqref{eq:2.50} and \eqref{eq:2.51} hold}\\ \text{for some }\mathcal{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\end{subarray} \right\}. \tag{2.52}\]

_Then the following statements hold true:_

1. _For_ \(s_{1}\in[0,\infty]\) _and_ \(s_{2}=\infty\)_, the_ \(\phi\)_-ERM_ \(\hat{f}_{n}^{\mathbf{FNN}}\) _with_ \[G =G_{0}\log\frac{1}{t_{2}\wedge\frac{1}{2}},\ N=N_{0}\left(\frac{ 1}{t_{2}\wedge\frac{1}{2}}\right)^{\frac{d-1}{\beta}},\ S=S_{0}\left(\frac{1}{ t_{2}\wedge\frac{1}{2}}\right)^{\frac{d-1}{\beta}}\log\left(\frac{1}{t_{2} \wedge\frac{1}{2}}\right),\] \[B =B_{0}\left(\frac{1}{t_{2}\wedge\frac{1}{2}}\right),\text{ and }\ F\asymp \left(\frac{\log n}{n}\right)^{\frac{1}{s_{1}+2}}\] _satisfies_ \[\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,s,r,I,\Theta,s_{1},s_{2},s_{2}}}\boldsymbol{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^ {\mathbf{FNN}}\right)\right]\lesssim\left(\frac{\log n}{n}\right)^{\frac{s_{1} }{s_{1}+2}},\] (2.53) _where_ \(G_{0},N_{0},S_{0},B_{0}\) _are positive constants only depending on_ \(d,\beta,r,I,\Theta\)_;_
2. _For_ \(s_{1}=\infty\) _and_ \(s_{2}\in[0,\infty)\)_, the_ \(\phi\)_-ERM_ \(\hat{f}_{n}^{\mathbf{FNN}}\) _with_ \[G \asymp\log n,\ N\asymp\left(\frac{n}{(\log n)^{3}}\right)^{\frac{d-1}{s _{2}+d-1}},\ S\asymp\left(\frac{n}{(\log n)^{3}}\right)^{\frac{d-1}{s_{2} \beta+d-1}}\log n,\] \[B \asymp\left(\frac{n}{(\log n)^{3}}\right)^{\frac{1}{s_{2}+\frac{ d-1}{\beta}}},\text{ and }\ F=t_{1}\wedge\frac{1}{2}\] _satisfies_ \[\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,s,r,I,\Theta,s_{1},s_{2},s_{2}}}\boldsymbol{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^ {\mathbf{FNN}}\right)\right]\lesssim\left(\frac{(\log n)^{3}}{n}\right)^{\frac {1}{s_{1}+\frac{1}{s_{2}s_{2}}}};\] (2.54)
_._
3. _For_ \(s_{1}\in[0,\infty)\) _and_ \(s_{2}\in[0,\infty)\)_, the_ \(\phi\)_-ERM_ \(\hat{f}_{n}^{\mathbf{FNN}}\) _with_ \[G\asymp\log n,N\asymp\left(\frac{n}{(\log n)^{3}}\right)^{\frac{(d-1)(s_{ 1}+1)}{s_{2}\beta+(s_{1}+1)(s_{2}\beta+d-1)}},S\asymp\left(\frac{n}{(\log n)^{3 }}\right)^{\frac{(d-1)(s_{1}+1)}{s_{2}\beta+(s_{1}+1)(s_{2}\beta+d-1)}}\log n,\] \[B\asymp\left(\frac{n}{(\log n)^{3}}\right)^{\frac{s_{1}+1}{s_{2}+(s_{ 1}+1)\left(s_{2}+\frac{d-1}{\beta}\right)}},\text{ and }\ F\asymp\left(\frac{(\log n)^{3}}{n}\right)^{\frac{s_{2}}{s_{2}+(s_{ 1}+1)\left(s_{2}+\frac{d-1}{\beta}\right)}}\] _satisfies_ \[\sup_{P\in\mathcal{H}_{d,s_{1},s_{2}}^{d,\beta,r,I,\theta,s_{1},s_{2}}}\mathbf{E} _{P\otimes n}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right) \right]\lesssim\left(\frac{(\log n)^{3}}{n}\right)^{\frac{s_{1}}{1+(s_{1}+1) \left(1+\frac{d-1}{\beta s_{2}}\right)}}.\] (2.55)

It is worth noting that the rate \(\mathcal{O}(\left(\frac{\log n}{n}\right)^{\frac{s_{1}}{s_{1}+2}})\) established in (2.53) does not depend on the dimension \(d\), and dependency of the rates in (2.54) and (2.55) on the dimension \(d\) diminishes as \(s_{2}\) increases, which demonstrates that the condition (2.51) with \(s_{2}=\infty\) helps circumvent the curse of dimensionality. In particular, (2.53) will give a fast dimension-free rate of convergence \(\mathcal{O}(\frac{\log n}{n})\) if \(s_{1}=s_{2}=\infty\). One may refer to Section 3 for more discussions about the result of Theorem 2.5.

The proof of Theorem 2.5 is in Appendix A.3.5. Our proof relies on Theorem 2.1 and the fact that the ReLU networks are good at approximating indicator functions of bounded regions with piecewise smooth boundary [18, 33]. Let \(P\) be an arbitrary probability in \(\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\theta,s_{1},s_{2}}\) and denote by \(\eta\) the condition probability function \(P(\{1\}\,|\cdot)\) of \(P\). To apply Theorem 2.1 and make good use of the noise condition (2.24) and the margin condition (2.51), we define another \(\psi\) (which is different from that in (2.9)) as

\[\psi:[0,1]^{d}\times\{-1,1\}\to\mathbb{R},\quad(x,y)\mapsto\begin{cases} \phi\left(yF_{0}\text{sgn}(2\eta(x)-1)\right),&\text{if }\left|2\eta(x)-1 \right|>\eta_{0},\\ \phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right),&\text{if }\left|2\eta(x)-1 \right|\leq\eta_{0}\end{cases}\]

for some suitable \(\eta_{0}\in(0,1)\) and \(F_{0}\in\left(0,\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)\). For such \(\psi\), Lemma A.17 guarantees that inequality (2.3) holds as

\[\int_{[0,1]^{d}\times\{-1,1\}}\psi\left(x,y\right)\text{d}P(x,y)\leq\inf\left\{ \left.\mathcal{R}_{P}^{\phi}(f)\right|f:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\},\]

and (2.4), (2.5) of Theorem 2.1 are satisfied with \(M=\frac{2}{1-\eta_{0}}\) and \(\Gamma=\frac{8}{1-\eta_{0}^{2}}\). Moreover, we utilize the noise condition (2.24) and the margin condition (2.51) to bound the approximation error

\[\inf_{f\in\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)}\left(\mathcal{R}_{P}^{ \phi}(f)-\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\text{d}P(x,y)\right) \tag{2.56}\]

(see (A.115), (A.116), (A.117)). Then, as in the proof of Theorem 2.2, we combine Theorem 2.1 with estimates for the covering number of \(\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\) and the approximation error (2.56) to obtain an upper bound for \(\mathbf{E}_{P^{\otimes n}}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{ FNN}}\right)-\int\psi\text{d}P\right]\), which, together with the noise condition (2.24), yields an upper bound for \(\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}(\hat{f}_{n}^{\mathbf{FNN}})\right]\) (see (A.113)). Finally taking the supremum over all \(P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\theta,s_{1},s_{2}}\) gives the desired result. The proof of Theorem 2.5 along with that of Theorem 2.2 and Theorem 2.3 indicates that Theorem 2.1 is very flexible in the sense that it can be used in various settings with different choices of \(\psi\).

### Main Lower Bounds

In this subsection, we will give our main results on lower bounds for convergence rates of the logistic risk, which will justify the optimality of our upper bounds established in the last subsection. To state these results, we need some notations.
Recall that for any \(a\in[0,1]\), \(\mathscr{M}_{a}\) denotes the probability measure on \(\{-1,1\}\) with \(\mathscr{M}_{a}(\{1\})=a\) and \(\mathscr{M}_{a}(\{-1\})=1-a\). For any measurable \(\eta:[0,1]^{d}\to[0,1]\) and any Borel probability measure \(\mathscr{Q}\) on \([0,1]^{d}\), we denote

\[\begin{split} P_{\eta,\mathscr{Q}}:&\{\text{ Borel subsets of }[0,1]^{d}\times\{-1,1\}\}\to[0,1],\\ S&\mapsto\int_{[0,1]^{d}}\int_{\{-1,1\}}\mathbb{1} _{S}(x,y)\mathrm{d}\mathscr{M}_{\eta(x)}(y)\mathrm{d}\mathscr{Q}(x).\end{split} \tag{2.57}\]

Therefore, \(P_{\eta,\mathscr{Q}}\) is the (unique) probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the marginal distribution on \([0,1]^{d}\) is \(\mathscr{Q}\) and the conditional probability function is \(\eta\). If \(\mathscr{Q}\) is the Lebesgue measure on \([0,1]^{d}\), we will write \(P_{\eta}\) for \(P_{\eta,\mathscr{Q}}\).

For any \(\beta\in(0,\infty)\), \(r\in(0,\infty)\), \(A\in[0,1)\), \(q\in\mathbb{N}\cup\{0\}\), and \((d,d_{*},K)\in\mathbb{N}^{3}\) with \(d_{*}\leq\min\left\{d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\right\}\), define

\[\begin{split}&\mathcal{H}_{3,A}^{d,\beta,r}:=\left\{P_{\eta} \Big{|}\begin{subarray}{c}\eta\in\mathcal{B}_{r}^{\beta}([0,1]^{d}),\ \mathbf{ran}(\eta)\subset[0,1],\text{ and}\\ \int_{[0,1]^{d}}\mathbb{1}_{[0,A]}([2\eta)-1]\mathrm{d}x=0\end{subarray} \right\},\\ &\mathcal{H}_{5,A,q,K,d_{*}}^{d,\beta,r}:=\left\{P_{\eta}\left| \begin{subarray}{c}\eta\in\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r), \mathbf{ran}(\eta)\subset[0,1],\text{ and}\\ \int_{[0,1]^{d}}\mathbb{1}_{[0,A]}([2\eta(x)-1])\mathrm{d}x=0\end{subarray} \right.\right.\end{split} \tag{2.58}\]

Now we can state our Theorem 2.6. Recall that \(\mathcal{F}_{d}\) is the set of all measurable real-valued functions defined on \([0,1]^{d}\).

**Theorem 2.6**.: _Let \(\phi\) be the logistic loss, \(n\in\mathbb{N}\), \(\beta\in(0,\infty)\), \(r\in(0,\infty)\), \(A\in[0,1)\), \(q\in\mathbb{N}\cup\{0\}\), and \((d,d_{*},K)\in\mathbb{N}^{3}\) with \(d_{*}\leq\min\left\{d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\right\}\). Suppose \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) is a sample in \([0,1]^{d}\times\{-1,1\}\) of size \(n\). Then there exists a constant \(\mathrm{c}_{0}\in(0,\infty)\) only depending on \((d_{*},\beta,r,q)\), such that_

\[\inf_{f_{n}}\sup_{P\in\mathcal{H}_{5,A,q,K,d_{*}}^{d,\beta,r}}\mathbf{E}_{P^{ \otimes n}}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\geq\mathrm{c}_{0} n^{-\frac{\beta\cdot(1,\beta)q}{d_{*}+\beta\cdot(1,\beta)q}}\text{ provided that }n>\left|\frac{7}{1-A}\right|^{\frac{d_{*}+\beta\cdot(1,\beta)q}{\beta \cdot(1,\beta)q}},\]

_where the infimum is taken over all \(\mathcal{F}_{d}\)-valued statistics on \(([0,1]^{d}\times\{-1,1\})^{n}\) from the sample \(\{(X_{i},Y_{i})\}_{i=1}^{n}\)._

Taking \(q=0\), \(K=1\), and \(d_{*}=d\) in Theorem 2.6, we immediately obtain the following corollary:

**Corollary 2.1**.: _Let \(\phi\) be the logistic loss, \(d\in\mathbb{N}\), \(\beta\in(0,\infty)\), \(r\in(0,\infty)\), \(A\in[0,1)\), and \(n\in\mathbb{N}\). Suppose \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) is a sample in \([0,1]^{d}\times\{-1,1\}\) of size \(n\). Then there exists a constant \(\mathrm{c}_{0}\in(0,\infty)\) only depending on \((d,\beta,r)\), such that_

\[\inf_{\hat{f}_{n}}\sup_{P\in\mathcal{H}_{3,A}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n }}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\geq\mathrm{c}_{0}n^{-\frac {\beta}{d+\beta}}\text{ provided that }n>\left|\frac{7}{1-A}\right|^{\frac{d+\beta}{\beta}},\]

_where the infimum is taken over all \(\mathcal{F}_{d}\)-valued statistics on \(([0,1]^{d}\times\{-1,1\})^{n}\) from the sample \(\{(X_{i},Y_{i})\}_{i=1}^{n}\)._

Theorem 2.6, together with Corollary 2.1, is proved in Appendix A.3.6.

Obviously, \(\mathcal{H}_{5,A,q,K,d_{*}}^{d,\beta,r}\subset\mathcal{H}_{4,q,K,d_{*}}^{d, \beta,r}\). Therefore, it follows from Theorem 2.6 that

\[\inf_{\hat{f}_{n}}\sup_{P\in\mathcal{H}_{4,q,K,d_{*},d_{*}}^{d,\beta,r}}\mathbf{E}_{P^ {\otimes n}}\left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\geq\inf_{\hat{f}_{ n}}\sup_{P\in\mathcal{H}_{5,A,q,K,d_{*}}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[ \mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\gtrsim n^{-\frac{\beta\cdot(1\wedge \beta)q}{2\star+\beta\cdot(1\wedge\beta)q}}.\]

This justifies that the rate \(\mathcal{O}(\left(\frac{\log n}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)q}{2 \star+\beta\cdot(1\wedge\beta)q}})\) in (2.36) is optimal (up to the logarithmic factor \((\log n)^{\frac{5\beta\cdot(1\wedge\beta)q}{2\star+\beta\cdot(1\wedge\beta)q}}\)). Similarly, it follows from \(\mathcal{H}_{3,A}^{d,\beta,r}\subset\mathcal{H}_{1}^{d,\beta,r}\) and Corollary 2.1 that

\[\inf_{\hat{f}_{n}}\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}} \left[\mathcal{E}_{P}^{\phi}(\hat{f}_{n})\right]\geq\inf_{\hat{f}_{n}}\sup_{P \in\mathcal{H}_{3,A}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}^{ \phi}(\hat{f}_{n})\right]\gtrsim n^{-\frac{\beta}{2\star+\beta}},\]
which justifies that the rate \(\mathcal{O}(\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta}{\beta+d}})\) in (2.17) is optimal (up to the logarithmic factor \((\log n)^{\frac{5\beta}{\beta+d}}\)). Moreover, note that any probability \(P\) in \(\mathcal{H}^{d,\beta,r}_{3,A}\) must satisfy the noise condition (2.24) provided that \(s_{1}\in[0,\infty]\), \(t_{1}\in(0,A]\), and \(c_{1}\in(0,\infty)\). In other words, for any \(s_{1}\in[0,\infty]\), \(t_{1}\in(0,A]\), and \(c_{1}\in(0,\infty)\), there holds \(\mathcal{H}^{d,\beta,r}_{3,A}\subset\mathcal{H}^{d,\beta,r}_{2,s_{1},c_{1},t_ {1}}\), meaning that

\[n^{-\frac{\beta}{d+\beta}}\lesssim\inf_{f_{n}}\sup_{P\in\mathcal{H}^{d,\beta,r }_{3,A}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right] \leq\inf_{f_{n}}\sup_{P\in\mathcal{H}^{d,\beta,r}_{2,s_{1},c_{1},t_{1}}}\mathbf{E }_{P^{\otimes n}}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\]

\[\leq\inf_{f_{n}}\sup_{P\in\mathcal{H}^{d,\beta,r}_{1}}\mathbf{E}_{P^{\otimes n}} \left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\leq\sup_{P\in\mathcal{H}^{d, \beta,r}_{1}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}^{\phi}_{P}\left(\hat{f}_ {n}^{\textbf{FNN}}\right)\right]\lesssim\left(\frac{(\log n)^{5}}{n}\right)^{ \frac{\beta}{\beta+d}},\]

where \(\hat{f}_{n}^{\textbf{FNN}}\) is the estimator defined in Theorem 2.2. From above inequalities we see that the noise condition (2.24) does little to help improve the convergence rate of the excess \(\phi\)-risk in classification.

The proof of Theorem 2.6 and Corollary 2.1 is based on a general scheme for obtaining lower bounds, which is given in Section 2 of [44]. However, the scheme in [44] is stated for a class of probabilities \(\mathcal{H}\) that takes the form \(\mathcal{H}=\{Q_{\theta}|\theta\in\Theta\}\) with \(\Theta\) being some pseudometric space. In our setting, we do not have such pseudometric space. Instead, we introduce another quantity

\[\inf_{f\in\mathcal{F}_{d}}\left|\mathcal{E}^{\phi}_{P}(f)+\mathcal{E}^{\phi} _{Q}(f)\right| \tag{2.59}\]

to characterize the difference between any two probability measures \(P\) and \(Q\) (see (A.130)). Estimating lower bounds for the quantity defined in (2.59) plays a key role in our proof of Theorem 2.6 and Corollary 2.1.

## 3 Discussions on Related Work

In this section, we compare our results with some existing ones in the literature. We first compare Theorem 2.2 and Theorem 2.5 with related results about binary classification using fully connected DNNs and logistic loss in [22] and [9] respectively. Then we compare our work with [20], in which the authors carry out generalization analysis for estimators obtained from gradient descent algorithms.

Throughout this section, we will use \(\phi\) to denote the logistic loss (i.e., \(\phi(t)=\log(1+\mathrm{e}^{-t})\)) and \(\{(X_{i},Y_{i})\}_{i=1}^{n}\) to denote an i.i.d. sample in \([0,1]^{d}\times\{-1,1\}\). The symbols \(d\), \(\beta\), \(r\), \(I\), \(\Theta\), \(t_{1}\), \(c_{1}\), \(t_{2}\), \(c_{2}\) and \(c\) will denote arbitrary numbers in \(\mathbb{N}\), \((0,\infty)\), \((0,\infty)\), \(\mathbb{N}\), \((0,\infty)\), \((0,\infty)\), \((0,\infty)\), \((0,\infty)\), \((0,\infty)\) and \([0,\infty)\), respectively. The symbol \(P\) will always denote some probability measure on \([0,1]^{d}\times\{-1,1\}\), regarded as the data distribution, and \(\eta\) will denote the corresponding conditional probability function \(P(\{1\}\left|\cdot\right)\) of \(P\).

Recall that \(\mathcal{C}^{d,\beta,r,I,\Theta}\), defined in (2.46), is the space consisting of classifiers which are equal to \(+1\) on the union of some disjoint regions with piecewise Holder smooth boundary and \(-1\) otherwise. In Theorem 4.1 of [22], the authors conduct generalization analysis when the data distribution \(P\) satisfies the piecewise smooth decision boundary condition (2.50), the noise condition (2.24), and the margin condition (2.51) with \(s_{1}=s_{2}=\infty\) for some \(\mathsf{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\). They show that there exist constants \(G_{0},N_{0},S_{0},B_{0},F_{0}\) not depending on the sample size \(n\) such that the \(\phi\)-ERM

\[\hat{f}_{n}^{\textbf{FNN}}\in\operatorname*{arg\,min}_{f\in\mathcal{F}^{ \textbf{FNN}}_{d}(G_{0},N_{0},S_{0},B_{0},F_{0})}\frac{1}{n}\sum_{i=1}^{n}\phi \left(Y_{i}f(X_{i})\right)\]

satisfies

\[\sup_{P\in\mathcal{H}^{d,\beta,r,I,\Theta,\infty}_{d,1},c_{1},t_{2},c_{2}, \infty}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^{\textbf{FNN}} \right)\right]\lesssim\frac{(\log n)^{1+\epsilon}}{n} \tag{3.1}\]

for any \(\epsilon>0\). Indeed, the noise conditions (2.24) and the margin condition (2.51) with \(s_{1}=s_{2}=\infty\) are equivalent to the following two conditions: there exist \(\eta_{0}\in(0,1)\) and \(\overline{\Delta}>0\) such that

\[P_{X}\left(\left\{\left.x\in[0,1]^{d}\right|\left|2\eta(x)-1\right|\leq\eta_{0 }\right\}\right)=0\]
\[P_{X}\left(\left\{x\in[0,1]^{d}\mid\Delta_{\mathcal{C}}(x)\leq\overline{\Delta} \right\}\right)=0\]

(cf. conditions (N\({}^{\prime}\)) and (M\({}^{\prime}\)) in [22]). Under the two conditions above, combining with the assumption \(\mathrm{sgn}(2\eta-1)\xrightarrow{P_{X}\cdot\mathrm{a.s.}}\mathfrak{C}\in \mathcal{C}^{d,\beta,r,I,\Theta}\), Lemma A.7 of [22] asserts that there exists \(f_{0}^{*}\in\mathcal{F}_{d}^{\mathbf{FNN}}(G_{0},N_{0},S_{0},B_{0},F_{0})\) such that

\[f_{0}^{*}\in\operatorname*{arg\,min}_{f\in\mathcal{F}_{d}^{\mathbf{FNN}}(G_{0},N_{0},S_{0},B_{0},F_{0})}\mathcal{R}_{P}^{\phi}(f)\]

and

\[\mathcal{R}_{P}(f_{0}^{*})=\mathcal{R}_{P}(2\eta-1)=\inf\left\{\mathcal{R}_{P }(f)\left|f:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right.\right\}.\]

The excess misclassification error of \(f:[0,1]^{d}\to\mathbb{R}\) is then given by \(\mathcal{E}_{P}(f)=\mathcal{R}_{P}(f)-\mathcal{R}_{P}(f_{0}^{*})\). Since \(f_{0}^{*}\) is bounded by \(F_{0}\), the authors in [22] can apply classical concentration techniques developed for bounded random variables (cf. Appendix A.2 of [22]) to deal with \(f_{0}^{*}\) (instead of the target function \(f_{\phi,P}^{*}\)), leading to the generalization bound (3.1). In this paper, employing Theorem 2.1, we extend Theorem 4.1 of [22] to much less restrictive cases in which the noise exponent \(s_{1}\) and the margin exponent \(s_{2}\) are allowed to be taken from \([0,\infty]\). The derived generalization bounds are presented in Theorem 2.5. In particular, when \(s_{1}=s_{2}=\infty\) (i.e., let \(s_{1}=\infty\) in statement (1) of Theorem 2.5), we obtain a refined generalization bound under the same conditions as those of Theorem 4.1 in [22], which asserts that the \(\phi\)-ERM \(f_{n}^{\mathbf{FNN}}\) over \(\mathcal{F}_{d}^{\mathbf{FNN}}(G_{0},N_{0},S_{0},B_{0},F_{0})\) satisfies

\[\sup_{P\in\mathcal{H}_{0,t_{1},e_{1},e_{1},e_{2},e_{2}}^{d,\beta,r,I,\Theta, \infty}}\boldsymbol{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim \frac{\log n}{n}, \tag{3.2}\]

removing the \(\epsilon\) in their bound (3.1). The above discussion indicates that Theorem 2.1 can lead to sharper estimates in comparison with classical concentration techniques, and can be applied in very general settings.

The recent work [9] considers estimation and inference using fully connected DNNs and the logistic loss in which their setting can cover both regression and classification. For any probability measure \(P\) on \([0,1]^{d}\times\{-1,1\}\) and any measurable function \(f:[0,1]^{d}\to[-\infty,\infty]\), define \(\left\|f\right\|_{\mathcal{L}^{2}_{P_{X}}}:=\left(\int_{[0,1]^{d}}\left|f(x) \right|^{2}\mathrm{d}P_{X}(x)\right)^{\frac{1}{2}}\). Recall that \(\mathcal{B}_{r}^{\beta}\left(\Omega\right)\) is defined in (2.13). Let \(\mathcal{H}_{7}^{d,\beta}\) be the set of all probability measures \(P\) on \([0,1]^{d}\times\{-1,1\}\) such that the target function \(f_{\phi,P}^{*}\) belongs to \(\mathcal{B}_{1}^{\beta}\left([0,1]^{d}\right)\). In Corollary 1 of [9], the authors claimed that if \(P\in\mathcal{H}_{7}^{d,\beta}\) and \(\beta\in\mathbb{N}\), then with probability at least \(1-\mathrm{e}^{-v}\) there holds

\[\left\|\hat{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{L}^{2}_{P_ {X}}}^{2}\lesssim n^{-\frac{2\beta}{2\beta+d}}\log^{4}n+\frac{\log\log n+v}{n}, \tag{3.3}\]

where the estimator \(\hat{f}_{n}^{\mathbf{FNN}}\in\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,\infty,F)\) is defined by (2.14) with

\[G\asymp\log n,\ N\asymp n^{\frac{d}{d+2\beta}},\ S\asymp n^{\frac{d}{d+2 \beta}}\log n,\ \text{and}\ \ F=2. \tag{3.4}\]

Note that \(f_{\phi,P}^{*}\in\mathcal{B}_{1}^{\beta}\left([0,1]^{d}\right)\) implies \(\|f_{\phi,P}^{*}\|_{\infty}\leq 1\). From Lemma 8 of [9], bounding the quantity \(\left\|\hat{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{L}^{2}_{P_ {X}}}^{2}\) on the left hand side of (3.3) is equivalent to bounding \(\mathcal{E}_{P}^{\phi}(\hat{f}_{n}^{\mathbf{FNN}})\), since

\[\frac{1}{2(\mathrm{e}+\mathrm{e}^{-1}+2)}\left\|\hat{f}_{n}^{\mathbf{FNN}}-f_{ \phi,P}^{*}\right\|_{\mathcal{L}^{2}_{P_{X}}}^{2}\leq\mathcal{E}_{P}^{\phi}( \hat{f}_{n}^{\mathbf{FNN}})\leq\frac{1}{4}\left\|\hat{f}_{n}^{\mathbf{FNN}}-f_{ \phi,P}^{*}\right\|_{\mathcal{L}^{2}_{P_{X}}}^{2}. \tag{3.5}\]

Hence (3.3) actually establishes the same upper bound (up to a constant independent of \(n\) and \(P\)) for the excess \(\phi\)-risk of \(\hat{f}_{n}^{\mathbf{FNN}}\), leading to upper bounds for the excess misclassification error \(\mathcal{E}_{P}(\hat{f}_{n}^{\mathbf{FNN}})\) through the calibration inequality. The authors in [9] apply concentration techniques based on (empirical) _Rademacher complexity_ (cf. Section A.2 of [9] or [4, 26]) to derive the bound (3.3), which allows for removing the restriction of uniformly boundedness on the weights and biases in the neural network
models, i.e., the hypothesis space generated by neural networks in their analysis can be of the form \(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,\infty,F\right)\). In our paper, we employ the covering number to measure the complexity of hypothesis space. Due to the lack of compactness, the covering numbers of \(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,\infty,F\right)\) are in general equal to infinity. Consequently, in our convergence analysis, we require the neural networks to possess bounded weights and biases. The assumption of bounded parameters may lead to additional optimization constraints in the training process. However, it has been found that the weights and biases of a trained neural network are typically around their initial values (cf. [11]). Thus the boundedness assumption matches what is observed in practice and has been adopted by most of the literature (see, e.g., [22, 37]). In particular, the work [37] considers nonparametric regression using neural networks with all parameters bounded by one (i.e., \(B=1\)). This assumption can be realized by projecting the parameters of the neural network onto \([-1,1]\) after each updating. Though the framework developed in this paper would not deliver generalization bounds without restriction of uniformly bounded parameters, we weaken this constraint in Theorem 2.2 by allowing the upper bound \(B\) to grow polynomially with the sample size \(n\), which simply requires \(1\leq B\lesssim n^{\nu}\) for any \(\nu>0\). It is worth mentioning that in our coming work [48], we actually establish oracle-type inequalities analogous to Theorem 2.1, with the covering number \(\mathcal{N}\left(\mathcal{F},\gamma\right)\) replaced by the supremum of some empirical \(L_{1}\)-covering numbers. These enable us to derive generalization bounds for the empirical \(\phi\)-risk minimizer \(\hat{f}_{n}^{\mathbf{FNN}}\) over \(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,\infty,F\right)\) because empirical \(L_{1}\)-covering numbers of \(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,\infty,F\right)\) can be well-controlled, as indicated by Lemma 4 and Lemma 6 of [9] (see also Theorem 9.4 of [14] and Theorem 7 of [5]). In addition, note that (3.3) can lead to probability bounds (i.e., confidence bounds) for the excess \(\phi\)-risk and misclassification error of \(\hat{f}_{n}^{\mathbf{FNN}}\), while the generalization bounds presented in this paper are only in expectation. Nonetheless, in [48], we obtain both probability bounds and expectation bounds for the empirical \(\phi\)-risk minimizer.

As discussed in Section 1, the boundedness assumptions on the target function \(f_{\phi,P}^{*}\) and its derivatives, i.e., \(f_{\phi,P}^{*}\in\mathcal{B}_{1}^{\beta}\left([0,1]^{d}\right)\), are too restrictive. This assumption actually requires that there exists some \(\delta\in(0,1/2)\) such that the conditional class probability \(\eta(x)=P(\{1\}|x)\) satisfies \(\delta<\eta(x)<1-\delta\) for \(P_{X}\)-almost all \(x\in[0,1]^{d}\), which rules out the case when \(\eta\) takes values in \(0\) or \(1\) with positive probabilities. However, it is believed that the conditional class probability should be determined by the patterns that make the two classes mutually exclusive, implying that \(\eta(x)\) should be closed to either \(0\) or \(1\). This is also observed in many benchmark datasets for image recognition. For example, it is reported in [22], the conditional class probabilities of CIFAR10 data set estimated by neural networks with the logistic loss almost solely concentrate on \(0\) or \(1\) and very few are around \(0.5\) (see Fig.2 in [22]). Overall, the boundedness restriction on \(f_{\phi,P}^{*}\) is not expected to hold in binary classification as it would exclude the well classified data. We further point out that the techniques used in [9] cannot deal with the case when \(f_{\phi,P}^{*}\) is unbounded, or equivalently, when \(\eta\) can take values close to \(0\) or \(1\). Indeed, the authors apply approximation theory of neural networks developed in [47] to construct uniform approximations of \(f_{\phi,P}^{*}\), which requires \(f_{\phi,P}^{*}\in\mathcal{B}_{1}^{\beta}\left([0,1]^{d}\right)\) with \(\beta\in\mathbb{N}\). However, if \(f_{\phi,P}^{*}\) is unbounded, uniformly approximating \(f_{\phi,P}^{*}\) by neural networks on \([0,1]^{d}\) is impossible, which brings the essential difficulty in estimating the approximation error. Besides, the authors use Bernstein's inequality to bound the quantity \(\frac{1}{n}\sum_{i=1}^{n}\left(\phi(Y_{i}f_{1}^{*}(X_{i}))-\phi(Y_{i}f_{\phi,P }^{*}(X_{i}))\right)\) appearing in the error decomposition for \(\left\|\hat{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{L}_{P_{X}} ^{2}}^{2}\) (see (A.1) in [9]), where \(f_{1}^{*}\in\arg\min_{f\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,\infty,2 \right)}\|f-f_{\phi,P}^{*}\|_{[0,1]^{d}}\).

We can see that the unboundedness of \(f_{\phi,P}^{*}\) will lead to the unboundedness of the random variable \(\left(\phi(Yf_{1}^{*}(X))-\phi(Yf_{\phi,P}^{*}(X))\right)\), which makes Bernstein's inequality invalid to bound its empirical mean by the expectation. In addition, the boundedness assumption on \(f_{\phi,P}^{*}\) ensures the inequality (3.5) on which the entire framework of convergence estimates in [9] is built (cf. Appendix A.1 and A.2 of [9]). Without this assumption, most of the theoretical arguments in [9] are not feasible. In contrast, we require \(\eta\xrightarrow{P_{X}\text{-a.s.}}\hat{\eta}\) for some \(\hat{\eta}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\) and \(r\in(0,\infty)\) in Theorem 2.2. This Holder smoothness condition on \(\eta\) is well adopted in the study of binary classifiers (see [3] and references therein). Note that \(f_{\phi,P}^{*}\in\mathcal{B}_{1}^{\beta}\left([0,1]^{d}\right)\) indeed implies \(\eta\xrightarrow{P_{X}\text{-a.s.}}\hat{\eta}\) for some \(\hat{\eta}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\) and \(r\in(0,\infty)\) which only depends on \((d,\beta)\). Therefore, the settings considered in Theorem 2.2 is more general than that of [9]. Moreover, the condition \(\eta\xrightarrow{P_{X}\text{-a.s.}}\hat{\eta}\in\mathcal{B}_{r}^{\beta}\left([0, 1]^{d}\right)\) is more nature, allowing \(\eta\) to take values
close to \(0\) and \(1\) with positive probabilities. We finally point out that, under the same assumption (i.e., \(P\in\mathcal{H}_{7}^{d,\beta}\)), one can use Theorem 2.1 to establish a convergence rate which is slightly improved compared with (3.3). Actually, we can show that there exists a constant \(\mathrm{c}\in(0,\infty)\) only depending on \((d,\beta)\), such that for any \(\mu\in[1,\infty)\), and \(\nu\in[0,\infty)\), there holds

\[\sup_{P\in\mathcal{H}_{7}^{d,\beta}}\boldsymbol{E}_{P^{\otimes n}}\left[\left\| \hat{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{L}_{P_{X}}^{2}}^{2} \right]\lesssim\left(\frac{(\log n)^{3}}{n}\right)^{\frac{2\beta}{2\beta+d}}, \tag{3.6}\]

where the estimator \(\hat{f}_{n}^{\mathbf{FNN}}\in\mathcal{F}_{d}^{\mathbf{FNN}}(G,N,S,B,F)\) is defined by (2.14) with

\[\begin{split}&\mathrm{c}\log n\leq G\asymp\log n,\ N\asymp\left(\frac{n}{\log^{3}n}\right)^{\frac{d}{d+2\beta}},\ S\asymp\left(\frac{n}{\log^{3}n}\right)^{\frac{d}{d+2\beta}} \cdot\log n,\\ & 1\leq B\lesssim n^{\nu},\ \mathrm{and}\ 1\leq F\leq\mu.\end{split} \tag{3.7}\]

Though we restrict the weights and biases to be bounded by \(B\), both the convergence rate and the network complexities in the result above refine the previous estimates established in (3.3) and (3.4). In particular, since \(\frac{6\beta}{2\beta+d}<3<4\), the convergence rate in (3.6) is indeed faster than that in (3.3) due to a smaller power exponent of the term \(\log n\). The proof of this claim is in Appendix A.3.7. We also remark that the convergence rate in (3.6) achieves the minimax optimal rate established in [41] up to log factors (so does the rate in (3.3)), which confirms that generalization analysis developed in this paper is also rate-optimal for bounded \(f_{\phi,P}^{*}\).

In our work, we have established generalization bounds for ERMs over hypothesis spaces consisting of neural networks. However, such ERMs cannot be obtained in practice because the correspoding optimization problems (e.g., (2.2)) cannot be solved explicitly. Instead, practical neural network estimators are obtained from algorithms which numerically solve the empirical risk minimization problem. Therefore, it is better to conduct generalization analysis for estimators obtained from such algorithms. One typical work in this direction is [20].

In [20], for classification tasks, the authors establish excess \(\phi\)-risk bounds to show that classifiers obtained from solving empirical risk minimization with respect to the logistic loss over shallow neural networks using gradient descent with (or without) early stopping are consistent. Note that the setting of [20] is quite different from ours: We consider deep neural network models in our work, while [20] considers shallow ones. Besides, we utilize the smoothness of the conditional probability function \(\eta(\cdot)=P(\{1\}\,|\cdot)\) to characterize the regularity (or complexity) of the data distribution \(P\). Instead, in [20], for each \(\overline{U}_{\infty}:\mathbb{R}^{d}\to\mathbb{R}^{d}\), the authors construct a function

\[f(\ \cdot\ ;\overline{U}_{\infty}):\mathbb{R}^{d}\to\mathbb{R},x\mapsto\int_{ \mathbb{R}^{d}}x^{\top}\overline{U}_{\infty}(v)\cdot\mathbb{1}_{[0,\infty)}(v ^{\top}x)\cdot\frac{1}{(2\pi)^{n/2}}\cdot\exp(-\frac{\|v\|_{2}^{2}}{2})\mathrm{ d}v\]

called infinite-width random feature model. Then they use the norm of \(\overline{U}_{\infty}\) which makes \(\mathcal{E}_{P}^{\phi}(f(\ \cdot\ ;\overline{U}_{\infty}))\) small to characterize the regularity of data: the data distribution is regarded as simple if there is a \(\overline{U}_{\infty}\) with \(\mathcal{E}_{P}^{\phi}(f(\ \cdot\ ;\overline{U}_{\infty}))\approx 0\) and moreover has a low norm. More rigorously, the slower the quantity

\[\inf\left\{\left\|\overline{U}_{\infty}\right\|_{\mathbb{R}^{d}}\left| \mathcal{E}_{P}^{\phi}(f(\ \cdot\ ;\overline{U}_{\infty}))\leq\varepsilon\right.\right\} \tag{3.8}\]

grows as \(\varepsilon\to 0\), the more regular (simpler) the data distribution \(P\) is. In [20], the established excess \(\phi\)-risk bounds depend on the quantity \(\mathcal{E}_{P}^{\phi}(f(\ \cdot\ ;\overline{U}_{\infty}))\) and the norm \(\left\|\overline{U}_{\infty}\right\|_{\mathbb{R}^{d}}\). Hence by assuming certain growth rates of the quantity in (3.8) as \(\varepsilon\to 0\), we can obtain specific rates of convergence from the excess \(\phi\)-risk bounds in [20]. It is natural to ask is there any relation between these two characterizations of data regularity, that is, the smoothness of conditional probability function, and the rate of growth of the quantity in (3.8) as \(\varepsilon\to 0\). For example, will Holder smoothness of the conditional probability function imply certain growth rates of the quantity in (3.8) as \(\varepsilon\to 0\)? This question is worth considering because once we prove the equivalence of these two characterizations, then the generalization analysis in [20] will be able to be used in other settings requiring smoothness of
the conditional probability function and vice versa. In addition, it is also interesting to study how can we use our new techniques developed in this paper to establish generalization bounds for deep neural network estimators obtained from learning algorithms (e.g., gradient descent) within the settings in this paper.

## 4 Conclusion

In this paper, we develop a novel generalization analysis for binary classification with DNNs and logistic loss. The unboundedness of the target function in logistic classification poses challenges for the estimates of sample error and approximation error when deriving generalization bounds. To overcome these difficulties, we introduce a bivariate function \(\psi:[0,1]^{d}\times\{-1,1\}\rightarrow\mathbb{R}\) to establish an elegant oracle-type inequality, aiming to bound the excess risk with respect to the logistic loss. This inequality incorporates the estimation of sample error and enables us to propose a framework for generalization analysis, which avoids using the explicit form of the target function. By properly choosing \(\psi\) under this framework, we can eliminate the boundedness restriction of the target function and establish sharp rates of convergence. In particular, for fully connected DNN classifiers trained by minimizing the empirical logistic risk, we obtain an optimal (up to some logarithmic factor) rate of convergence of the excess logistic risk (which further yields a rate of convergence of the excess misclassification error via the calibration inequality) merely under the Holder smoothness assumption on the conditional probability function. If we instead assume that the conditional probability function is the composition of several vector-valued multivariate functions of which each component function is either a maximum value function of some of its input variables or a Holder smooth function only depending on a small number of its input variables, we can even establish dimension-free optimal (up to some logarithmic factor) convergence rates for the excess logistic risk of fully connected DNN classifiers, further leading to dimension-free rates of convergence of their excess misclassification error through the calibration inequality. This result serves to elucidate the remarkable achievements of DNNs in high-dimensional real-world classification tasks. In other circumstances such as when the data distribution has a piecewise smooth decision boundary and the input data are bounded away from it (i.e., \(s_{2}=\infty\) in (2.51)), dimension-free rates of convergence can also be derived. Besides the novel oracle-type inequality, the sharp estimates presented in our paper also owe to a tight error bound for approximating the natural logarithm function (which is unbounded near zero) by fully connected DNNs. All the claims for the optimality of rates in our paper are justified by corresponding minimax lower bounds. As far as we know, all these results are new to the literature, which further enrich the theoretical understanding of classification using deep neural networks. At last, we would like to emphasize that our framework of generalization analysis is very general and can be extended to many other settings (e.g., when the loss function, the hypothesis space, or the assumption on the data distribution is different from that in this current paper). In particular, in our forthcoming research [48], we have investigated generalization analysis for CNN classifiers trained with the logistic loss, exponential loss, or LUM loss on spheres under the Sobolev smooth conditional probability assumption. Motivated by recent work [12, 13, 28, 49], we will also study more efficient implementations of deep logistic classification for dealing with big data.

## Appendix

In this appendix, we first bound the covering numbers of spaces of fully connected neural networks. Then we will provide some results on the approximation theory of fully connected neural networks. At last, we give proofs of results in the main body of this paper.

### Covering Numbers of Spaces of Fully Connected Neural Networks

Recall that if \(\mathcal{F}\) consists of bounded real-valued functions defined on a domain containing \([0,1]^{d}\), the covering number of \(\mathcal{F}\) with respect to the radius \(\gamma\) and the metric given by \(\rho(f,g)=\sup_{x\in[0,1]^{d}}\left|f(x)-g(x)\right|,\forall f,g\in\mathcal{F}\) is denoted by \(\mathcal{N}(\mathcal{F},\gamma)\). For the space \(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right)\) defined by (1.15), the covering number
\(\mathcal{N}\left(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right),\gamma\right)\) can be bounded from above in terms of \(G,N,S,B\), and the radius of covering \(\gamma\). The related results are stated below.

**Theorem A.1**.: _For \(G\in[1,\infty)\), \((N,S,B)\in[0,\infty)^{3}\), and \(\gamma\in(0,1)\), there holds_

\[\log\left(\mathcal{N}\left(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,\infty\right),\gamma\right)\right)\] \[\leq(S+Gd+1)(2G+5)\cdot\log\frac{(\max\left\{N,d\right\}+1)(B \lor 1)(G+1)}{\gamma}.\]

Theorem A.1 can be proved in the same manner as in the proof of Lemma 5 in [37]. Therefore, we omit the proof here. Similar results are also presented in Proposition A.1 of [22] and Lemma 3 of [42]. Corollary A.1 follows immediately from Theorem A.1 and Lemma 10.6 of [1].

**Corollary A.1**.: _For \(G\in[1,\infty)\), \((N,S,B)\in[0,\infty)^{3}\), \(F\in[0,\infty]\) and \(\gamma\in(0,1)\), there holds_

\[\log\left(\mathcal{N}\left(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right),\gamma\right)\right)\] \[\leq(S+Gd+1)(2G+5)\cdot\log\frac{(\max\left\{N,d\right\}+1)(B\lor 1 )(2G+2)}{\gamma}.\]

### Approximation Theory of Fully Connected Neural Networks

Theorem A.2 below gives error bounds for approximating Holder continuous functions by fully connected neural networks. Since it can be derived straightforwardly from Theorem 5 of [37], we omit its proof.

**Theorem A.2**.: _Suppose that \(f\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\) with some \((\beta,r)\in(0,\infty)^{2}\). Then for any positive integers \(m\) and \(M^{\prime}\) with \(M^{\prime}\geq\max\left\{(\beta+1)^{d},\left(r\sqrt{d}\left\lceil\beta\right \rceil^{d}+1\right)\mathrm{e}^{d}\right\}\), there exists_

\[\tilde{f}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(14m(2+\log_{2}\left(d\lor \beta\right)),6\left(d+\left\lceil\beta\right\rceil\right)M^{\prime},987(2d+ \beta)^{4d}M^{\prime}m,1,\infty\right)\]

_such that_

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|\] \[\leq r\sqrt{d}\left\lceil\beta\right\rceil^{d}\cdot 3^{\beta}M^{ \prime-\beta/d}+\left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d}\right) \cdot 6^{d}\cdot(1+d^{2}+\beta^{2})\cdot M^{\prime}\cdot 2^{-m}.\]

Corollary A.2 follows directly from Theorem A.2.

**Corollary A.2**.: _Suppose that \(f\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\) with some \((\beta,r)\in(0,\infty)^{2}\). Then for any \(\varepsilon\in(0,1/2]\), there exists_

\[\tilde{f}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(D_{1}\log\frac{1}{\varepsilon },D_{2}\varepsilon^{-\frac{d}{\beta}},D_{3}\varepsilon^{-\frac{d}{\beta}}\log \frac{1}{\varepsilon},1,\infty\right)\]

_such that_

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|\leq\varepsilon,\]

_where \((D_{1},D_{2},D_{3})\in(0,\infty)^{3}\) are constants depending only on \(d\), \(\beta\) and \(r\)._

Proof.: Let

\[E_{1} =\max\left\{(\beta+1)^{d},\left(r\sqrt{d}\left\lceil\beta\right \rceil^{d}+1\right)\mathrm{e}^{d},\left(\frac{1}{2r}\cdot 3^{-\beta}\cdot\frac{1}{ \sqrt{d}\left\lceil\beta\right\rceil^{d}}\right)^{-d/\beta}\right\},\] \[E_{2} =3\max\left\{1+\frac{d}{\beta},\frac{\log\left(4E_{1}\cdot \left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d}\right)\left(1+d^{2}+\beta^{2 }\right)\cdot 6^{d}\right)}{\log 2}\right\},\]
\[D_{1} =14\cdot(2+\log_{2}{(d\lor\beta)})\cdot(E_{2}+2),\] \[D_{2} =6\cdot(d+\lceil\beta\rceil)\cdot(E_{1}+1),\] \[D_{3} =987\cdot(2d+\beta)^{4d}\cdot(E_{1}+1)\cdot(E_{2}+2).\]

Then \(D_{1},D_{2},D_{3}\) are constants only depending on \(d,\beta,r\).

For \(f\in\mathcal{B}_{r}^{\beta}\left(\left[0,1\right]^{d}\right)\) and \(\varepsilon\in(0,1/2]\), choose \(M^{\prime}=\left\lceil E_{1}\cdot\varepsilon^{-d/\beta}\right\rceil\) and \(m=\lceil E_{2}\log(1/\varepsilon)\rceil\). Then \(m\) and \(M^{\prime}\) are positive integers satisfying that

\[1 \leq\max\left\{(\beta+1)^{d},\left(r\sqrt{d}\left\lceil\beta \right\rceil^{d}+1\right)\mathrm{e}^{d}\right\}\leq E_{1}\leq E_{1}\cdot \varepsilon^{-d/\beta}\] (A.1) \[\leq M^{\prime}\leq 1+E_{1}\cdot\varepsilon^{-d/\beta}\leq(E_{1}+ 1)\cdot\varepsilon^{-d/\beta},\] \[M^{\prime-\beta/d}\leq\left(E_{1}\cdot\varepsilon^{-d/\beta} \right)^{-\beta/d}\leq\varepsilon\cdot\frac{1}{2r}\cdot 3^{-\beta}\cdot \frac{1}{\sqrt{d}\left\lceil\beta\right\rceil^{d}},\] (A.2)

and

\[m\leq E_{2}\log(1/\varepsilon)+2\log 2\leq E_{2}\log(1/\varepsilon)+2\log(1/ \varepsilon)=(2+E_{2})\cdot\log(1/\varepsilon).\] (A.3)

Moreover, we have that

\[2 \cdot\left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d}\right) \cdot 6^{d}\cdot(1+d^{2}+\beta^{2})\cdot M^{\prime}\cdot\frac{1}{\varepsilon}\] (A.4) \[\leq 2\cdot\left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d} \right)\cdot 6^{d}\cdot(1+d^{2}+\beta^{2})\cdot(E_{1}+1)\cdot\varepsilon^{-1-d/\beta}\] \[\leq 2\cdot\left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d} \right)\cdot 6^{d}\cdot(1+d^{2}+\beta^{2})\cdot 2E_{1}\cdot\varepsilon^{-1-d/\beta}\] \[\leq 2^{\frac{1}{3}E_{2}}\cdot\varepsilon^{-1-d/\beta}\leq 2^{ \frac{1}{3}E_{2}}\cdot\varepsilon^{-\frac{1}{3}E_{2}}\leq\varepsilon^{-\frac{1 }{3}E_{2}}\cdot\varepsilon^{-\frac{1}{3}E_{2}}\] \[\leq\varepsilon^{-E_{2}\cdot\log 2}=2^{E_{2}\log(1/ \varepsilon)}\leq 2^{m}.\]

Therefore, from (A.1), (A.2), (A.3), (A.4), and Theorem A.2, we conclude that there exists

\[\tilde{f} \in\mathcal{F}_{d}^{\mathbf{FNN}}(14m(2+\log_{2}{(d\lor\beta)}),6 \left(d+\lceil\beta\rceil\right)M^{\prime},987(2d+\beta)^{4d}M^{\prime}m,1,\infty)\] \[=\mathcal{F}_{d}^{\mathbf{FNN}}\left(\frac{D_{1}}{E_{2}+2}\cdot m,\frac{D_{2}}{E_{1}+1}\cdot M^{\prime},\frac{D_{3}}{(E_{1}+1)\cdot(E_{2}+2)} \cdot M^{\prime}m,1,\infty\right)\] \[\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left(D_{1}\log\frac{1}{ \varepsilon},D_{2}\varepsilon^{-\frac{d}{\beta}},D_{3}\varepsilon^{-\frac{d}{ \beta}}\log\frac{1}{\varepsilon},1,\infty\right)\]

such that

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|\] \[\leq r\sqrt{d}\left\lceil\beta\right\rceil^{d}\cdot 3^{\beta}M^{ \prime-\beta/d}+\left(1+2r\sqrt{d}\left\lceil\beta\right\rceil^{d}\right) \cdot 6^{d}\cdot(1+d^{2}+\beta^{2})\cdot M^{\prime}\cdot 2^{-m}\leq\frac{ \varepsilon}{2}+\frac{\varepsilon}{2}=\varepsilon.\]

Thus we complete the proof. 

### Proofs of Results in the Main Body

The proofs in this section will be organized in logical order in the sense that each result in this section is proved without relying on results that are presented after it.

Throughout this section, we use

\[C_{\mathtt{Parameter}_{1},\mathtt{Parameter}_{2},\cdots,\mathtt{Parameter}_{m}}\]

to denote a positive constant only depending on \(\mathtt{Parameter}_{1}\), \(\mathtt{Parameter}_{2}\), \(\cdots\), \(\mathtt{Parameter}_{m}\). For example, we may use \(C_{d,\beta}\) to denote a positive constant only depending on \((d,\beta)\). The values of such constants appearing in the proofs may be different from line to line or even in the same line. Besides, we may use the same symbol with different meanings in different proofs. For example, the symbol \(I\) may denote a number in one proof, and denote a set in another proof. To avoid confusion, we will explicitly redefine these symbols in each proof.
#### a.3.1 Proofs of some properties of the target function

The following lemma justifies our claim in (1.7).

**Lemma A.1**.: _Let \(d\in\mathbb{N}\), \(P\) be a probability measure on \([0,1]^{d}\times\{-1,1\}\), and \(\phi:\mathbb{R}\to[0,\infty)\) be a measurable function. Define_

\[\overline{\phi}:[-\infty,\infty]\to[0,\infty],\;z\mapsto\begin{cases} \varlimsup_{t\to+\infty}\phi(t),&\text{ if }z=\infty,\\ \phi(z),&\text{ if }z\in\mathbb{R},\\ \varlimsup_{t\to-\infty}\phi(t),&\text{ if }z=-\infty,\end{cases}\]

_which is an extension of \(\phi\) to \([-\infty,\infty]\). Suppose \(f^{*}:[0,1]^{d}\to[-\infty,\infty]\) is a measurable function satisfying that_

\[f^{*}(x)\in\operatorname*{arg\,min}_{z\in[-\infty,\infty]}\int_{\{-1,1\}} \overline{\phi}(yz)\mathrm{d}P(y|x)\text{ for }P_{X}\text{-almost all }x\in[0,1]^{d}.\] (A.5)

_Then there holds_

\[\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf^{*}(x))\mathrm{d}P(x,y)=\inf \left\{\left.\mathcal{R}^{\phi}_{P}(g)\right|g:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\}.\]

Proof.: Let \(\Omega_{0}:=\left\{\left.x\in[0,1]^{d}\right|f^{*}(x)\in\mathbb{R}\right\} \times\{-1,1\}\). Then for any \(m\in\mathbb{N}\) and any \((i,j)\in\{-1,1\}^{2}\), define

\[f_{m}:[0,1]^{d}\to\mathbb{R},\;x\mapsto\begin{cases}m,&\text{ if }f^{*}(x)= \infty,\\ f^{*}(x),&\text{ if }f^{*}(x)\in\mathbb{R},\\ -m,&\text{ if }f^{*}(x)=-\infty,\end{cases}\]

and \(\Omega_{i,j}=\left\{\left.x\in[0,1]^{d}\right|f^{*}(x)=i\cdot\infty\right\} \times\{j\}\). Obviously, \(yf^{*}(x)=ij\cdot\infty\) and \(yf_{m}(x)=ijm\) for any \((i,j)\in\{-1,1\}^{2}\), any \(m\in\mathbb{N}\), and any \((x,y)\in\Omega_{i,j}\). Therefore,

\[\begin{split}&\varlimsup_{m\to+\infty}\int_{\Omega_{i,j}}\phi( yf_{m}(x))\mathrm{d}P(x,y)\\ &=\varlimsup_{m\to+\infty}\int_{\Omega_{i,j}}\phi(ijm)\mathrm{d}P (x,y)=P(\Omega_{i,j})\cdot\varlimsup_{m\to+\infty}\phi(ijm)\\ &\leq P(\Omega_{i,j})\cdot\varlimsup_{t\to ij\cdot\infty}\phi(t)= P(\Omega_{i,j})\cdot\overline{\phi}(ij\cdot\infty)=\int_{\Omega_{i,j}} \overline{\phi}(ij\cdot\infty)\mathrm{d}P(x,y)\\ &=\int_{\Omega_{i,j}}\overline{\phi}(yf^{*}(x))\mathrm{d}P(x,y), \;\forall\;(i,j)\in\{-1,1\}^{2}\,.\end{split}\] (A.6)

Besides, it is easy to verify that \(yf_{m}(x)=yf^{*}(x)\in\mathbb{R}\) for any \((x,y)\in\Omega_{0}\) and any \(m\in\mathbb{N}\), which means that

\[\int_{\Omega_{0}}\phi(yf_{m}(x))\mathrm{d}P(x,y)=\int_{\Omega_{0}}\overline{ \phi}(yf^{*}(x))\mathrm{d}P(x,y),\;\forall\;m\in\mathbb{N}.\] (A.7)
Combining (A.6) and (A.7), we obtain

\[\begin{split}&\inf\left\{\mathcal{R}^{\phi}_{P}(g)\Big{|}\,g:[0,1]^ {d}\to\mathbb{R}\text{ is measurable}\right\}\\ &\leq\varlimsup_{m\to+\infty}\mathcal{R}^{\phi}_{P}(f_{m})= \varlimsup_{m\to+\infty}\int_{[0,1]^{d}\times\{-1,1\}}\phi(yf_{m}(x))\mathrm{d }P(x,y)\\ &=\varlimsup_{m\to+\infty}\left(\int_{\Omega_{0}}\phi(yf_{m}(x)) \mathrm{d}P(x,y)+\sum_{i\in\{-1,1\}}\sum_{j\in\{-1,1\}}\int_{\Omega_{i,j}}\phi( yf_{m}(x))\mathrm{d}P(x,y)\right)\\ &\leq\varliminf_{m\to+\infty}\int_{\Omega_{0}}\phi(yf_{m}(x)) \mathrm{d}P(x,y)+\sum_{i\in\{-1,1\}}\sum_{j\in\{-1,1\}}\varlimsup_{m\to+ \infty}\int_{\Omega_{i,j}}\phi(yf_{m}(x))\mathrm{d}P(x,y)\\ &\leq\int_{\Omega_{0}}\overline{\phi}(yf^{*}(x))\mathrm{d}P(x,y )+\sum_{i\in\{-1,1\}}\sum_{j\in\{-1,1\}}\int_{\Omega_{i,j}}\overline{\phi}(yf ^{*}(x))\mathrm{d}P(x,y)\\ &=\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf^{*}(x)) \mathrm{d}P(x,y).\end{split}\] (A.8)

On the other hand, for any measurable \(g:[0,1]^{d}\to\mathbb{R}\), it follows from (A.5) that

\[\begin{split}&\int_{\{-1,1\}}\overline{\phi}(yf^{*}(x))\mathrm{d }P(y|x)=\inf_{z\in[-\infty,\infty]}\int_{\{-1,1\}}\overline{\phi}(yz)\mathrm{d }P(y|x)\leq\int_{\{-1,1\}}\overline{\phi}(yg(x))\mathrm{d}P(y|x)\\ &=\int_{\{-1,1\}}\phi(yg(x))\mathrm{d}P(y|x)\text{ for $P_{X}$- almost all $x\in[0,1]^{d}$}.\end{split}\]

Integrating both sides, we obtain

\[\begin{split}&\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf ^{*}(x))\mathrm{d}P(x,y)=\int_{[0,1]^{d}}\int_{\{-1,1\}}\overline{\phi}(yf^{*} (x))\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\\ &\leq\int_{[0,1]^{d}}\int_{\{-1,1\}}\phi(yg(x))\mathrm{d}P(y|x)P _{X}(x)=\int_{[0,1]^{d}\times\{-1,1\}}\phi(yg(x))\mathrm{d}P(x,y)=\mathcal{R} ^{\phi}_{P}(g).\end{split}\]

Since \(g\) is arbitrary, we deduce that

\[\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf^{*}(x))\mathrm{d}P(x,y)\leq \inf\left\{\,\mathcal{R}^{\phi}_{P}(g)\Big{|}\,g:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\},\]

which, together with (A.8), proves the desired result. 

The next lemma gives the explicit form of the target function of the logistic risk.

**Lemma A.2**.: _Let \(\phi(t)=\log(1+\mathrm{e}^{-t})\) be the logistic loss, \(d\in\mathbb{N}\), \(P\) be a probability measure on \([0,1]^{d}\times\{-1,1\}\), and \(\eta\) be the conditional probability function \(P(\{1\}\,|\cdot)\) of \(P\). Define_

\[f^{*}:[0,1]^{d}\to[-\infty,\infty],\;x\mapsto\begin{cases}\infty,&\text{if $ \eta(x)=1$},\\ \log\frac{\eta(x)}{1-\eta(x)},&\text{if $\eta(x)\in(0,1)$},\\ -\infty,&\text{if $\eta(x)=0$},\end{cases}\] (A.9)

_which is a natural extension of the map_

\[\left\{\,z\in[0,1]^{d}\right|\eta(z)\in(0,1)\right\}\ni x\mapsto\log\frac{\eta (x)}{1-\eta(x)}\in\mathbb{R}\]

_to all of \([0,1]^{d}\). Then \(f^{*}\) is a target function of the \(\phi\)-risk under \(P\), i.e., (1.6) holds. In addition, the target function of the \(\phi\)-risk under \(P\) is unique up to a \(P_{X}\)-null set. In other words, for any target function \(f^{*}\) of the \(\phi\)-risk under \(P\), we must have_

\[P_{X}\left(\left\{\,x\in[0,1]^{d}\right|f^{*}(x)\neq f^{*}(x)\right\}\right)=0.\]
Proof.: Define

\[\overline{\phi}:[-\infty,\infty]\to[0,\infty],\ z\mapsto\begin{cases}0,&\text{if }z= \infty,\\ \phi(z),&\text{if }z\in\mathbb{R},\\ \infty,&\text{if }z=-\infty,\end{cases}\] (A.10)

which is a natural extension of the logistic loss \(\phi\) to \([-\infty,\infty]\), and define

\[V_{a}:[-\infty,\infty]\to[0,\infty],z\mapsto a\overline{\phi}(z)+(1-a) \overline{\phi}(-z)\]

for any \(a\in[0,1]\). Then we have that

\[\begin{split}&\int_{\{-1,1\}}\overline{\phi}(yz)\mathrm{d}P(y|x )=\eta(x)\overline{\phi}(z)+(1-\eta(x))\overline{\phi}(-z)\\ &=V_{\eta(x)}(z),\ \forall\ x\in[0,1]^{d},\ z\in[-\infty, \infty].\end{split}\] (A.11)

For any \(a\in[0,1]\), we have that \(V_{a}\) is smooth on \(\mathbb{R}\), and an elementary calculation gives

\[V_{a}^{\prime\prime}(t)=\frac{1}{2+\mathrm{e}^{t}+\mathrm{e}^{-t}}>0,\ \forall\ t\in\mathbb{R}.\]

Therefore, \(V_{a}\) is strictly convex on \(\mathbb{R}\) and

\[\begin{split}&\operatorname*{arg\,min}_{z\in\mathbb{R}}V_{a}(z)= \left\{z\in\mathbb{R}\big{|}V_{a}^{\prime}(z)=0\right\}=\left\{z\in\mathbb{R} \left|a\phi^{\prime}(z)-(1-a)\phi^{\prime}(-z)=0\right.\right\}\\ &=\left\{z\in\mathbb{R}\left|-a+\frac{\mathrm{e}^{z}}{1+\mathrm{ e}^{z}}=0\right.\right\}=\begin{cases}\left\{\log\frac{a}{1-a}\right\},&\text{if }a\in(0,1),\\ \varnothing,&\text{if }a\in\left\{0,1\right\}.\end{cases}\end{split}\] (A.12)

Besides, it is easy to verify that

\[V_{a}(z)=\infty,\ \forall\ a\in(0,1),\ \forall\ z\in\left\{\infty,-\infty \right\},\]

which, together with (A.12), yields

\[\operatorname*{arg\,min}_{z\in[-\infty,\infty]}V_{a}(z)=\operatorname*{arg\, min}_{z\in\mathbb{R}}V_{a}(z)=\left\{\log\frac{a}{1-a}\right\},\ \forall\ a\in(0,1).\] (A.13)

In addition, it follows from

\[\overline{\phi}(z)>0=\overline{\phi}(\infty),\ \forall\ z\in[-\infty,\infty)\]

that

\[\operatorname*{arg\,min}_{z\in[-\infty,\infty]}V_{1}(z)=\operatorname*{arg\, min}_{z\in[-\infty,\infty]}\overline{\phi}(z)=\left\{\infty\right\}\] (A.14)

and

\[\operatorname*{arg\,min}_{z\in[-\infty,\infty]}V_{0}(z)=\operatorname*{arg\, min}_{z\in[-\infty,\infty]}\overline{\phi}(-z)=\left\{-\infty\right\}.\] (A.15)

Combining (A.11), (A.14) and (A.15), we obtain

\[\begin{split}&\operatorname*{arg\,min}_{z\in[-\infty,\infty]} \int_{\{-1,1\}}\overline{\phi}(yz)\mathrm{d}P(y|x)=\operatorname*{arg\,min} _{z\in[-\infty,\infty]}V_{\eta(x)}(z)=\begin{cases}\left\{+\infty\right\},& \text{if }\eta(x)=1,\\ \left\{\log\frac{\eta(x)}{1-\eta(x)}\right\},&\text{if }\eta(x)\in(0,1),\\ \left\{-\infty\right\},&\text{if }\eta(x)=0\end{cases}\\ &=\left\{f^{*}(x)\right\},\ \forall\ x\in[0,1]^{d},\end{split}\]

which implies (1.6). Therefore, \(f^{*}\) is a target function of the \(\phi\)-risk under the distribution \(P\). Moreover, the uniqueness of the target function of the \(\phi\)-risk under \(P\) follows immediately from the fact that for all \(x\in[0,1]^{d}\) the set

\[\operatorname*{arg\,min}_{z\in[-\infty,\infty]}\int_{\{-1,1\}}\overline{\phi} (yz)\mathrm{d}P(y|x)=\left\{f^{*}(x)\right\}\]

contains exactly one point and the uniqueness (up to some \(P_{X}\)-null set) of the conditional distribution \(P(\cdot|\cdot)\) of \(P\). This completes the proof.
The Lemma A.3 below provides a formula for computing the infimum of the logistic risk over all real-valued measurable functions.

**Lemma A.3**.: _Let \(\phi(t)=\log(1+\mathrm{e}^{-t})\) be the logistic loss, \(\delta\in(0,1/2]\), \(d\in\mathbb{N}\), \(P\) be a probability measure on \([0,1]^{d}\times\{-1,1\}\), \(\eta\) be the conditional probability function \(P(\{1\}\,|\cdot)\) of \(P\), \(f^{*}\) be defined by (A.9), \(\overline{\phi}\) be defined by (A.10), \(H\) be defined by_

\[H:[0,1]\to[0,\infty),t\mapsto\begin{cases}t\log\left(\frac{1}{t}\right)+(1-t) \log\left(\frac{1}{1-t}\right),&\text{ if }t\in(0,1),\\ 0,&\text{ if }t\in\{0,1\},\end{cases}\]

_and \(\psi\) be defined by_

\[\psi:[0,1]^{d}\times\{-1,1\}\to[0,\infty),\] \[(x,y)\mapsto\begin{cases}\phi\left(y\log\frac{\eta(x)}{1-\eta(x) }\right),&\text{ if }\eta(x)\in[\delta,1-\delta],\\ 0,&\text{ if }\eta(x)\in\{0,1\},\\ \eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)},&\text{ if }\eta(x)\in(0,\delta)\cup(1-\delta,1).\end{cases}\]

_Then there holds_

\[\inf\Big{\{}\,\mathcal{R}_{P}^{\phi}(g)\Big{|}\,g:[0,1]^{d}\to \mathbb{R}\text{ is measurable}\Big{\}}=\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf^{*}(x ))\mathrm{d}P(x,y)\] \[=\int_{[0,1]^{d}}H(\eta(x))\mathrm{d}P_{X}(x)=\int_{[0,1]^{d} \times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y).\]

Proof.: According to Lemma A.2, \(f^{*}\) is a target function of the \(\phi\)-risk under the distribution \(P\), meaning that

\[f^{*}(x)\in\operatorname*{arg\,min}_{z\in[-\infty,\infty]}\int_{\{-1,1\}} \overline{\phi}(yz)\mathrm{d}P(y|x)\text{ for }P_{X}\text{-almost all }x\in[0,1]^{d}.\]

Then it follows from Lemma A.1 that

\[\inf\Big{\{}\,\mathcal{R}_{P}^{\phi}(g)\Big{|}\,g:[0,1]^{d}\to \mathbb{R}\text{ is measurable}\Big{\}}=\int_{[0,1]^{d}\times\{-1,1\}}\overline{\phi}(yf^{*}(x ))\mathrm{d}P(x,y)\] (A.16) \[=\int_{[0,1]^{d}}\int_{\{-1,1\}}\overline{\phi}(yf^{*}(x)) \mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\] \[=\int_{[0,1]^{d}}\Big{(}\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta (x))\overline{\phi}(-f^{*}(x))\Big{)}\mathrm{d}P_{X}(x).\]

For any \(x\in[0,1]^{d}\), if \(\eta(x)=1\), then we have

\[\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta(x))\overline{\phi}(-f^{ *}(x))=\overline{\phi}(f^{*}(x))=\overline{\phi}(+\infty)=0=H(\eta(x))=0\] \[=1\cdot 0+(1-1)\cdot 0=\eta(x)\psi(x,1)+(1-\eta(x))\psi(x,-1)= \int_{\{-1,1\}}\psi(x,y)\mathrm{d}P(y|x);\]

If \(\eta(x)=0\), then we have

\[\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta(x))\overline{\phi}(-f^{ *}(x))=\overline{\phi}(-f^{*}(x))=\overline{\phi}(+\infty)=0=H(\eta(x))=0\] \[=0\cdot 0+(1-0)\cdot 0=\eta(x)\psi(x,1)+(1-\eta(x))\psi(x,-1)= \int_{\{-1,1\}}\psi(x,y)\mathrm{d}P(y|x);\]

If \(\eta(x)\in(0,\delta)\cup(1-\delta,1)\), then we have

\[\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta(x))\overline{\phi}(-f^{*}(x))\]
\[=\eta(x)\phi\left(\log\frac{\eta(x)}{1-\eta(x)}\right)+(1-\eta(x)) \phi\left(-\log\frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\log\left(1+\frac{1-\eta(x)}{\eta(x)}\right)+(1-\eta(x)) \log\left(1+\frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)}\] \[=H(\eta(x))=\int_{\{-1,1\}}\left(\eta(x)\log\frac{1}{\eta(x)}+(1- \eta(x))\log\frac{1}{1-\eta(x)}\right)\mathrm{d}P(y|x)\] \[=\int_{\{-1,1\}}\psi(x,y)\mathrm{d}P(y|x);\]

If \(\eta(x)\in[\delta,1-\delta]\), then we have that

\[\eta(x)\overline{\phi}\left(f^{*}(x)\right)+(1-\eta(x))\overline {\phi}(-f^{*}(x))\] \[=\eta(x)\phi\left(\log\frac{\eta(x)}{1-\eta(x)}\right)+(1-\eta(x ))\phi\left(-\log\frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\log\left(1+\frac{1-\eta(x)}{\eta(x)}\right)+(1-\eta(x) )\log\left(1+\frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)}\] \[=H(\eta(x))=\eta(x)\phi\left(\log\frac{\eta(x)}{1-\eta(x)}\right) +(1-\eta(x))\phi\left(-\log\frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\psi(x,1)+(1-\eta(x))\psi(x,-1)=\int_{\{-1,1\}}\psi(x,y) \mathrm{d}P(y|x).\]

In conclusion, we always have that

\[\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta(x))\overline{\phi}(-f^{*}(x))=H(\eta (x))=\int_{\{-1,1\}}\psi(x,y)\mathrm{d}P(y|x).\]

Since \(x\) is arbitrary, we deduce that

\[\int_{[0,1]^{d}}\left(\eta(x)\overline{\phi}(f^{*}(x))+(1-\eta(x ))\overline{\phi}(-f^{*}(x))\right)\mathrm{d}P_{X}(x)=\int_{[0,1]^{d}}H(\eta (x))\mathrm{d}P_{X}(x)\] \[=\int_{[0,1]^{d}}\int_{\{-1,1\}}\psi(x,y)\mathrm{d}P(y|x)\mathrm{ d}P_{X}(x)=\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y),\]

which, together with (A.16), proves the desired result. 

#### a.3.2 Proof of Theorem 2.1

This subsection is devoted to the proof of Theorem 2.1.

Proof of Theorem 2.1.: Throughout this proof, we denote

\[\Psi:=\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y).\]

Then it follows from (2.3) and (2.4) that \(0\leq\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)-\Psi\leq 2M<\infty\). Let \(\{(X_{k}^{\prime},Y_{k}^{\prime})\}_{k=1}^{n}\) be an i.i.d. sample from distribution \(P\) which is independent of \(\left\{(X_{k},Y_{k})\right\}_{k=1}^{n}\). By independence, we have

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)-\Psi\right]= \frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\phi\left(Y_{i}^{\prime}\hat{f}_{n}(X_ {i}^{\prime})\right)-\psi\left(X_{i}^{\prime},Y_{i}^{\prime}\right)\right]\]
with its empirical counterpart given by

\[\hat{R}:=\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\phi\left(Y_{i}\hat{f}_{n}(X_{i}) \right)-\psi(X_{i},Y_{i})\right].\]

Then we have

\[\hat{R}-\left(\mathcal{R}_{P}^{\phi}(g)-\Psi\right) =\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\phi\left(Y_{i}\hat{f}_ {n}(X_{i})\right)-\phi(Y_{i}g(X_{i}))\right]\] \[=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}\phi\left(Y_{i}\hat{f}_ {n}(X_{i})\right)-\frac{1}{n}\sum_{i=1}^{n}\phi\left(Y_{i}g(X_{i})\right) \right]\leq 0,\ \forall\ g\in\mathcal{F},\]

where the last inequality follows from the fact that \(\hat{f}_{n}\) is an empirical \(\phi\)-risk minimizer which minimizes \(\frac{1}{n}\sum_{i=1}^{n}\phi\left(Y_{i}g(X_{i})\right)\) over all \(g\in\mathcal{F}\). Hence \(\hat{R}\leq\inf_{g\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi}(g)-\Psi\right)\), which means that

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi\right]=\left(\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n} \right)-\Psi\right]-(1+\varepsilon)\cdot\hat{R}\right)+(1+\varepsilon)\cdot \hat{R}\] (A.17) \[\leq\left(\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n }\right)-\Psi\right]-(1+\varepsilon)\cdot\hat{R}\right)+(1+\varepsilon)\cdot \inf_{g\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi}(g)-\Psi\right),\ \forall\ \varepsilon\in[0,1).\]

We then establish an upper bound for \(\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)-\Psi\right]-(1+ \varepsilon)\cdot\hat{R}\) by using a similar argument to that in the proof of Lemma 4 of [37]. The desired inequality (2.6) will follow from this bound and (A.17). Recall that \(W=\max\left\{3,\ \mathcal{N}\left(\mathcal{F},\gamma\right)\right\}\). From the definition of \(W\), there exist \(f_{1},\cdots,f_{W}\in\mathcal{F}\) such that for any \(f\in\mathcal{F}\), there exists some \(j\in\{1,\cdots,W\}\), such that \(\|f-f_{j}\|_{\infty}\leq\gamma\). Therefore, there holds \(\left\|\hat{f}_{n}-f_{j^{*}}\right\|_{[0,1]^{d}}\leq\gamma\) where \(j^{*}\) is a \(\{1,\cdots,W\}\)-valued statistic from the sample \(\{(X_{i},Y_{i})\}_{i=1}^{n}\). Denote

\[A:=M\cdot\sqrt{\frac{\log W}{\Gamma n}}.\] (A.18)

And for \(j=1,2\cdots,W\), let

\[h_{j,1} :=\mathcal{R}_{P}^{\phi}(f_{j})-\Psi,\] (A.19) \[h_{j,2} :=\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(yf_{j}(x))-\psi(x,y) \right)^{2}\mathrm{d}P(x,y),\] \[V_{j} :=\left|\sum_{i=1}^{n}\left(\phi\left(Y_{i}f_{j}(X_{i})\right)- \psi\left(X_{i},Y_{i}\right)-\phi\left(Y_{i}^{\prime}f_{j}(X_{i}^{\prime}) \right)+\psi\left(X_{i}^{\prime},Y_{i}^{\prime}\right)\right)\right|,\] \[r_{j} :=A\vee\sqrt{h_{j,1}}.\]

Define \(T:=\max\limits_{j=1,\cdots,W}\frac{V_{j}}{r_{j}}\). Denote by \(\mathbb{E}\left[\cdot|\left(X_{i},Y_{i}\right)_{i=1}^{n}\right]\) the conditional expectation with respect to \(\{(X_{i},Y_{i})\}_{i=1}^{n}\). Then

\[r_{j^{*}} \leq A+\sqrt{h_{j^{*},1}}=A+\sqrt{\mathbb{E}\left[\phi\left(Y^{ \prime}f_{j^{*}}(X^{\prime})\right)-\psi(X^{\prime},Y^{\prime})\right]\left(X_ {i},Y_{i}\right)_{i=1}^{n}}\] \[\leq A+\sqrt{\gamma+\mathbb{E}\left[\phi\left(Y^{\prime}\hat{f}_ {n}(X^{\prime})\right)-\psi(X^{\prime},Y^{\prime})\right]\left(X_{i},Y_{i} \right)_{i=1}^{n}}\] \[=A+\sqrt{\gamma+\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi}\leq A+\sqrt{\gamma}+\sqrt{\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n} \right)-\Psi},\]

where \((X^{\prime},Y^{\prime})\) is an i.i.d. copy of \((X_{i},Y_{i})\left(1\leq i\leq n\right)\) and the second inequality follows from

\[|\phi(t_{1})-\phi(t_{2})|\leq|t_{1}-t_{2}|\,,\ \forall\ t_{1},t_{2}\in\mathbb{R}\] (A.20)
and \(\left\|f_{j^{*}}-\hat{f}\right\|_{[0,1]^{d}}\leq\gamma\). Consequently,

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi\right]-\hat{R}\leq\left|\hat{R}-\mathbb{E}\left[\mathcal{R}_{P}^{\phi} \left(\hat{f}_{n}\right)-\Psi\right]\right|\] (A.21) \[=\frac{1}{n}\left|\mathbb{E}\left[\sum_{i=1}^{n}\left(\phi\left(Y _{i}\hat{f}_{n}(X_{i})\right)-\psi(X_{i},Y_{i})-\phi\left(Y_{i}^{\prime}\hat{f }_{n}(X_{i}^{\prime})\right)+\psi(X_{i}^{\prime},Y_{i}^{\prime})\right)\right]\right|\] \[\leq\frac{1}{n}\mathbb{E}\left[\left|\sum_{i=1}^{n}\left(\phi \left(Y_{i}f_{j^{*}}(X_{i})\right)-\psi(X_{i},Y_{i})-\phi\left(Y_{i}^{\prime} f_{j^{*}}(X_{i}^{\prime})\right)+\psi(X_{i}^{\prime},Y_{i}^{\prime})\right) \right|\right]+2\gamma\] \[=\frac{1}{n}\mathbb{E}\left[Y_{j^{*}}\right]+2\gamma\leq\frac{1 }{n}\mathbb{E}\left[T\cdot r_{j^{*}}\right]+2\gamma\] \[\leq\frac{1}{n}\mathbb{E}\left[T\cdot\sqrt{\mathcal{R}_{P}^{\phi }\left(\hat{f}_{n}\right)-\Psi}\right]+\frac{A+\sqrt{\gamma}}{n}\cdot\mathbb{E }\left[T\right]+2\gamma\] \[\leq\frac{\varepsilon\mathbb{E}\left[\mathcal{R}_{P}^{\phi} \left(\hat{f}_{n}\right)-\Psi\right]}{2+2\varepsilon}+\frac{(1+\varepsilon) \mathbb{E}\left[T^{2}\right]}{2\varepsilon\cdot n^{2}}+\frac{A+\sqrt{\gamma} }{n}\mathbb{E}\left[T\right]+2\gamma,\;\forall\;\varepsilon\in(0,1),\]

where the last inequality follows from \(2\sqrt{ab}\leq\frac{1}{1+\varepsilon}a+\frac{1+\varepsilon}{d}\), \(\forall a>0,b>0\). We then bound \(\mathbb{E}\left[T\right]\) and \(\mathbb{E}\left[T^{2}\right]\) by Bernstein's inequality (see e.g., Chapter 3.1 of [6] and Chapter 6.2 of [40]). Indeed, it follows from (2.5) and (A.19) that

\[h_{j,2}\leq\Gamma\cdot h_{j,1}\leq\Gamma\cdot\left(r_{j}\right)^{2},\;\forall \;j\in\{1,\cdots,W\}.\]

For any \(j\in\{1,\cdots,W\}\) and \(t\geq 0\), we apply Bernstein's inequality to the zero mean i.i.d. random variables

\[\left\{\phi\left(Y_{i}f_{j}(X_{i})\right)-\psi(X_{i},Y_{i})-\phi\left(Y_{i}^{ \prime}f_{j}(X_{i}^{\prime})\right)+\psi(X_{i}^{\prime},Y_{i}^{\prime})\right\} _{i=1}^{n}\]

and obtain

\[\mathbb{P}(V_{j}\geq t)\] \[=\mathbb{P}\left(\left|\sum_{i=1}^{n}\left(\phi\left(Y_{i}f_{j}(X _{i})\right)-\psi(X_{i},Y_{i})-\phi\left(Y_{i}^{\prime}f_{j}(X_{i}^{\prime}) \right)+\psi(X_{i}^{\prime},Y_{i}^{\prime})\right)\right|\geq t\right)\] \[\leq 2\exp\left(\frac{-t^{2}/2}{Mt+\sum_{i=1}^{n}\mathbb{E}\left[ \left(\phi\left(Y_{i}f_{j}(X_{i})\right)-\psi(X_{i},Y_{i})-\phi\left(Y_{i}^{ \prime}f_{j}(X_{i}^{\prime})\right)+\psi(X_{i}^{\prime},Y_{i}^{\prime})\right) ^{2}\right]}\right)\] \[\leq 2\exp\left(\frac{-t^{2}/2}{Mt+2\sum_{i=1}^{n}\mathbb{E} \left[\left(\phi\left(Y_{i}f_{j}(X_{i})\right)-\psi(X_{i},Y_{i})\right)^{2}+ \left(\phi\left(Y_{i}^{\prime}f_{j}(X_{i}^{\prime})\right)-\psi(X_{i}^{\prime},Y_{i}^{\prime})\right)^{2}\right]}\right)\] \[=2\exp\left(-\frac{t^{2}/2}{Mt+4\sum_{i=1}^{n}h_{j,2}}\right)=2 \exp\left(-\frac{t^{2}}{2Mt+8nh_{j,2}}\right)\] \[\leq 2\exp\left(-\frac{t^{2}}{2Mt+8n\Gamma\cdot\left(r_{j}\right)^ {2}}\right).\]

Hence

\[\mathbb{P}(T\geq t) \leq\sum_{j=1}^{W}\mathbb{P}(V_{j}/r_{j}\geq t)=\sum_{j=1}^{W} \mathbb{P}(V_{j}\geq tr_{j})\] \[\leq 2\sum_{j=1}^{W}\exp\left(-\frac{(tr_{j})^{2}}{2Mtr_{j}+8n \Gamma\cdot r_{j}^{2}}\right)=2\sum_{j=1}^{W}\exp\left(-\frac{t^{2}}{2Mt/r_{j}+ 8n\Gamma}\right)\]
\[\leq 2\sum_{j=1}^{W}\exp\left(-\frac{t^{2}}{2Mt/A+8n\Gamma}\right)=2W\exp \left(-\frac{t^{2}}{2Mt/A+8n\Gamma}\right),\;\forall\;t\in[0,\infty).\]

Therefore, for any \(\theta\in\{1,2\}\), by taking

\[B:=\left(\frac{M}{A}\cdot\log W+\sqrt{\left(\frac{M}{A}\cdot\log W\right)^{2} +8n\Gamma\log W}\right)^{\theta}=4^{\theta}\cdot\left(n\Gamma\log W\right)^{ \theta/2},\]

we derive

\[\mathbb{E}\left[T^{\theta}\right] =\int_{0}^{\infty}\mathbb{P}\left(T\geq t^{1/\theta}\right) \mathrm{d}t\leq B+\int_{B}^{\infty}\mathbb{P}\left(T\geq t^{1/\theta}\right) \mathrm{d}t\] \[\leq B+\int_{B}^{\infty}\left(2W\exp\left(-\frac{t^{2/\theta}}{2 Mt^{1/\theta}/A+8n\Gamma}\right)\right)\mathrm{d}t\] \[\leq B+\int_{B}^{\infty}\left(2W\exp\left(-\frac{B^{1/\theta} \cdot t^{1/\theta}}{2MB^{1/\theta}/A+8n\Gamma}\right)\right)\mathrm{d}t\] \[=B+2WB\theta\cdot\left(\log W\right)^{-\theta}\int_{\log W}^{ \infty}\mathrm{e}^{-u}u^{\theta-1}\mathrm{d}u\] \[\leq B+2WB\theta\cdot\left(\log W\right)^{-\theta}\cdot\theta \cdot\mathrm{e}^{-\log W}\left(\log W\right)^{\theta-1}\] \[\leq 5\theta B\leq 5\theta\cdot 4^{\theta}\cdot\left(n\Gamma \log W\right)^{\theta/2}.\]

Plugging the inequality above and (A.18) into (A.21), we obtain

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi\right]-\hat{R}\leq\left|\hat{R}-\mathbb{E}\left[\mathcal{R}_{P}^{\phi} \left(\hat{f}_{n}\right)-\Psi\right]\right|\] \[\leq\frac{\varepsilon\mathbb{E}\left[\mathcal{R}_{P}^{\phi} \left(\hat{f}_{n}\right)-\Psi\right]}{2+2\varepsilon}+\frac{(1+\varepsilon) \mathbb{E}\left[T^{2}\right]}{2\varepsilon\cdot n^{2}}+\frac{A+\sqrt{\gamma}} {n}\mathbb{E}\left[T\right]+2\gamma\] \[\leq\frac{\varepsilon}{1+\varepsilon}\mathbb{E}\left[\mathcal{R} _{P}^{\phi}\left(\hat{f}_{n}\right)-\Psi\right]+20\cdot\sqrt{\gamma}\cdot\sqrt {\frac{\Gamma\log W}{n}}\] \[\quad+20M\cdot\frac{\log W}{n}+80\cdot\frac{\Gamma\log W}{n} \cdot\frac{1+\varepsilon}{\varepsilon}+2\gamma,\;\forall\;\varepsilon\in(0,1).\]

Multiplying the above inequality by \((1+\varepsilon)\) and then rearranging, we obtain that

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi\right]-(1+\varepsilon)\cdot\hat{R}\leq 20\cdot(1+\varepsilon)\cdot \sqrt{\gamma}\cdot\sqrt{\frac{\Gamma\log W}{n}}\] (A.22) \[\quad+20\cdot(1+\varepsilon)\cdot M\cdot\frac{\log W}{n}+80\cdot \frac{\Gamma\log W}{n}\cdot\frac{(1+\varepsilon)^{2}}{\varepsilon}+(2+2 \varepsilon)\cdot\gamma,\;\forall\;\varepsilon\in(0,1).\]

Combining (A.22) and (A.17), we deduce that

\[\mathbb{E}\left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}\right)- \Psi\right]\leq(1+\varepsilon)\cdot\inf_{g\in\mathcal{F}}\left(\mathcal{R}_{P} ^{\phi}(g)-\Psi\right)+20\cdot(1+\varepsilon)\cdot\sqrt{\gamma}\cdot\sqrt{ \frac{\Gamma\log W}{n}}\] \[\quad+20\cdot(1+\varepsilon)\cdot M\cdot\frac{\log W}{n}+80\cdot \frac{\Gamma\log W}{n}\cdot\frac{(1+\varepsilon)^{2}}{\varepsilon}+(2+2 \varepsilon)\cdot\gamma,\;\forall\;\varepsilon\in(0,1).\]

This proves the desired inequality (2.6) and completes the proof of Theorem 2.1. 

#### a.3.3 Proof of Theorem 2.4

To prove Theorem 2.4, we need the following Lemma A.4 and Lemma A.5.

Lemma A.4, which describes neural networks that approximate the multiplication operator, can be derived directly from Lemma A.2 of [37]. Thus we omit its proof. One can also find a similar result to Lemma A.4 in the earlier paper [47] (cf. Proposition 3 therein).
**Lemma A.4**.: _For any \(\varepsilon\in(0,1/2]\), there exists a neural network_

\[\mathrm{M}\in\mathcal{F}_{2}^{\mathbf{FNN}}\left(15\log\frac{1}{\varepsilon},6,9 00\log\frac{1}{\varepsilon},1,1\right)\]

_such that for any \(t,t^{\prime}\in[0,1]\), there hold \(\mathrm{M}(t,t^{\prime})\in[0,1]\), \(\mathrm{M}(t,0)=\mathrm{M}(0,t^{\prime})=0\) and_

\[\left|\mathrm{M}(t,t^{\prime})-t\cdot t^{\prime}\right|\leq\varepsilon.\]

In Lemma A.5, we construct a neural network which performs the operation of multiplying the inputs by \(2^{k}\).

**Lemma A.5**.: _Let \(k\) be a positive integer and \(f\) be a univariate function given by \(f(x)=2^{k}\cdot\max\left\{x,0\right\}\). Then_

\[f\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(k,2,4k,1,\infty\right).\]

Proof.: For any \(1\leq i\leq k-1\), let \(v_{i}=(0,0)^{\top}\) and

\[\mathbf{W}_{i}=\begin{pmatrix}1&1\\ 1&1\end{pmatrix}.\]

In addition, take

\[\mathbf{W}_{0}=(1,1)^{\top},\mathbf{W}_{k}=(1,1),\text{ and }v_{k}=(0,0)^{\top}.\]

Then we have

\[f=x\mapsto\mathbf{W}_{k}\sigma_{v_{k}}\mathbf{W}_{k-1}\sigma_{v_{k-1}}\cdots\mathbf{W}_{1} \sigma_{v_{1}}\mathbf{W}_{0}x\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(k,2,4k,1, \infty\right),\]

which proves this lemma. 

Now we are in the position to prove Theorem 2.4.

Proof of Theorem 2.4.: Given \(a\in(0,1/2]\), let \(I:=\left\lceil-\log_{2}a\right\rceil\) and \(J_{k}:=\left\lfloor\frac{1}{3\cdot 2^{k}},\frac{1}{2k}\right\rfloor\) for \(k=0,1,2,\cdots\). Then \(1\leq I\leq 1-\log_{2}a\leq 4\log\frac{1}{a}\). The idea of proof is to construct neural networks \(\left\{\tilde{h}_{k}\right\}_{k}\) which satisfy that \(0\leq\tilde{h}_{k}\left(t\right)\leq 1\) and \((8\log a)\cdot\tilde{h}_{k}\) approximates the natural logarithm function on \(J_{k}\). Then the function

\[x\mapsto(8\log a)\cdot\sum_{k}\mathrm{M}\left(\tilde{h}_{k}(x),\tilde{f}_{k}( x)\right)\]

is the desired neural network in Theorem 2.4, where \(\mathrm{M}\) is the neural network that approximates multiplication operators given in Lemma A.4 and \(\{\tilde{f}_{k}\}_{k}\) are neural networks representing piecewise linear function supported on \(J_{k}\) which constitutes a partition of unity.

Specifically, given \(\alpha\in(0,\infty)\), there exists some \(r_{\alpha}>0\) only depending on \(\alpha\) such that

\[x\mapsto\log\left(\frac{2x}{3}+\frac{1}{3}\right)\in\mathcal{B}_{r_{\alpha}}^{ \alpha}\left([0,1]\right).\]

Hence it follows from Corollary A.2 that there exists

\[\tilde{g}_{1} \in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{2}{ \varepsilon},C_{\alpha}\left(\frac{2}{\varepsilon}\right)^{1/\alpha},C_{ \alpha}\left(\frac{2}{\varepsilon}\right)^{1/\alpha}\log\frac{2}{\varepsilon},1,\infty\right)\] \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{ \varepsilon},C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{1/\alpha},C_{ \alpha}\left(\frac{1}{\varepsilon}\right)^{1/\alpha}\log\frac{1}{\varepsilon},1,\infty\right)\]

such that

\[\sup_{x\in[0,1]}\left|\tilde{g}_{1}(x)-\log\left(\frac{2x}{3}+\frac{1}{3} \right)\right|\leq\varepsilon/2.\]
Recall that the ReLU function is given by \(\sigma(t)=\max\left\{t,0\right\}\). Let

\[\tilde{g}_{2}:\mathbb{R}\rightarrow\mathbb{R},\quad x\mapsto-\sigma\left(- \sigma\left(\tilde{g}_{1}(x)+\log 3\right)+\log 3\right).\]

Then

\[\tilde{g}_{2}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{ \varepsilon},C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{1/\alpha},C_{ \alpha}\left(\frac{1}{\varepsilon}\right)^{1/\alpha}\log\frac{1}{\varepsilon},1,\infty\right),\] (A.23)

and for \(x\in\mathbb{R}\), there holds

\[-\log 3\leq\tilde{g}_{2}(x)=\begin{cases}-\log 3,&\text{ if }\tilde{g}_{1}(x)<-\log 3,\\ \tilde{g}_{1}(x),&\text{ if }-\log 3\leq\tilde{g}_{1}(x)\leq 0,\\ 0,&\text{ if }\tilde{g}_{1}(x)>0.\end{cases}\]

Moreover, since \(-\log 3\leq\log\left(\frac{2\pi}{3}+\frac{1}{3}\right)\leq 0\) whenever \(x\in[0,1]\), we have

\[\sup_{x\in[0,1]}\left|\tilde{g}_{2}(x)-\log\left(\frac{2x}{3}+\frac{1}{3} \right)\right|\leq\sup_{x\in[0,1]}\left|\tilde{g}_{1}(x)-\log\left(\frac{2x}{3 }+\frac{1}{3}\right)\right|\leq\varepsilon/2.\]

Let \(x=\frac{3\cdot 2^{k}\cdot t-1}{2}\) in the above inequality, we obtain

\[\sup_{t\in J_{k}}\left|\tilde{g}_{2}\left(\frac{3\cdot 2^{k}\cdot t-1}{2} \right)-k\log 2-\log t\right|\leq\varepsilon/2,\quad\forall\ k=0,1,2,\cdots.\] (A.24)

For any \(0\leq k\leq I\), define

\[\tilde{h}_{k}:\mathbb{R}\rightarrow\mathbb{R},\quad t\mapsto\sigma\left( \frac{\sigma\left(-\tilde{g}_{2}\left(\sigma\left(\frac{3}{4\cdot 2^{I-k}}\cdot 2^{I+1} \cdot\sigma(t)-\frac{1}{2}\right)\right)\right)}{8\log\frac{1}{a}}+\frac{k\log 2 }{8\log\frac{1}{a}}\right).\]

Then we have

\[0 \leq\tilde{h}_{k}(t)\leq\left|\frac{\sigma\left(-\tilde{g}_{2} \left(\sigma\left(\frac{3}{4\cdot 2^{I-k}}\cdot 2^{I+1}\cdot\sigma(t)-\frac{1}{2} \right)\right)\right)}{8\log\frac{1}{a}}+\frac{k\log 2}{8\log\frac{1}{a}}\right|\] (A.25) \[\leq\frac{\sup_{x\in\mathbb{R}}\left|\tilde{g}_{2}(x)\right|}{8 \log\frac{1}{a}}+\frac{I}{8\log\frac{1}{a}}\leq\frac{\log 3+4\log\frac{1}{a}}{8 \log\frac{1}{a}}\leq 1,\ \forall\ t\in\mathbb{R}.\]

Therefore, it follows from (A.23), the definition of \(\tilde{h}_{k}\), and Lemma A.5 that (cf. Figure A.1)

\[\tilde{h}_{k} \in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{ \varepsilon}+I,C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{\alpha} },C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{\alpha}}\log\frac{1}{ \varepsilon}+4I,1,1\right)\] (A.26) \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{ \varepsilon}+4\log\frac{1}{a},C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{ \frac{1}{\alpha}},C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{ \alpha}}\log\frac{1}{\varepsilon}+16\log\frac{1}{a},1,1\right)\]

for all \(0\leq k\leq I\). Besides, according to (A.24), it is easy to verify that for \(0\leq k\leq I\), there holds

\[\left|(8\log a)\cdot\tilde{h}_{k}(t)-\log t\right|=\left|\tilde{g}_{2}\left( \frac{3}{2}\cdot 2^{k}\cdot t-1/2\right)-k\log 2-\log t\right|\leq\varepsilon/2,\quad \forall\ t\in J_{k}.\]

Define

\[\tilde{f}_{0}:\mathbb{R}\rightarrow[0,1],\quad x\mapsto\begin{cases}0,&\text{ if }x\in(-\infty,1/3),\\ 6\cdot\left(x-\frac{1}{3}\right),&\text{ if }x\in[1/3,1/2],\\ 1,&\text{ if }x\in(1/2,\infty),\end{cases}\]
and for \(k\in\mathbb{N}\),

\[\tilde{f}_{k}:\mathbb{R}\to[0,1],\quad x\mapsto\begin{cases}0,&\text{ if }x\in \mathbb{R}\setminus J_{k},\\ 6\cdot 2^{k}\cdot\left(x-\frac{1}{3\cdot 2^{k}}\right),&\text{ if }x\in \left[\frac{1}{3\cdot 2^{k}},\frac{1}{2^{k+1}}\right),\\ 1,&\text{ if }x\in\left[\frac{1}{2^{k+1}},\frac{1}{3\cdot 2^{k-1}}\right],\\ -3\cdot 2^{k}\cdot\left(x-\frac{1}{2^{k}}\right),&\text{ if }x\in \left(\frac{1}{3\cdot 2^{k-1}},\frac{1}{2^{k}}\right].\end{cases}\]

Figure A.1: Networks representing functions \(\tilde{h}_{k}\).

Figure A.2: Graphs of functions \(\tilde{f}_{k}\).
Then it is easy to show that for any \(x\in\mathbb{R}\) and \(k\in\mathbb{N}\), there hold

\[\tilde{f}_{k}(x) =\frac{6}{2^{I-k+3}}\cdot 2^{I+3}\cdot\sigma\left(x-\frac{1}{3\cdot 2 ^{k}}\right)-\frac{6}{2^{I-k+3}}\cdot 2^{I+3}\cdot\sigma\left(x-\frac{1}{2^{k+1}}\right)\] \[\quad+\frac{6}{2^{I-k+4}}\cdot 2^{I+3}\cdot\sigma\left(x-\frac{1}{ 2^{k}}\right)-\frac{6}{2^{I-k+3}}\cdot 2^{I+3}\cdot\sigma\left(x-\frac{1}{3 \cdot 2^{k-1}}\right),\]

and

\[\tilde{f}_{0}(x)=\frac{6}{2^{I+3}}\cdot 2^{I+3}\cdot\sigma(x-1/3)-\frac{6}{2^{ I+3}}\cdot 2^{I+3}\cdot\sigma(x-1/2).\]

Hence it follows from Lemma A.5 that (cf. Figure A.3)

\[\tilde{f}_{k} \in\mathcal{F}_{1}^{\mathbf{FNN}}\left(I+5,8,16I+60,1,\infty\right)\] (A.27) \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(12\log\frac{1}{a},8,1 52\log\frac{1}{a},1,\infty\right),\quad\ \forall\ 0\leq k\leq I.\]

Next, we show that

\[\sup_{t\in[a,1]}\left|\log(t)+8\log\left(\frac{1}{a}\right)\sum_{k=0}^{I} \tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|\leq\varepsilon/2.\] (A.28)

Indeed, we have the following inequalities:

\[\left|\log(t)+8\log\left(\frac{1}{a}\right)\sum_{k=0}^{I}\tilde{h }_{k}(t)\tilde{f}_{k}(t)\right| =\left|\log t+8\log\left(\frac{1}{a}\right)\tilde{h}_{0}(t)\tilde {f}_{0}(t)\right|\] (A.29) \[=\left|\log t+8\log\left(\frac{1}{a}\right)\tilde{h}_{0}(t) \right|\leq\varepsilon/2,\ \forall\ t\in[1/2,1];\]

Figure A.3: Networks representing functions \(\tilde{f}_{k}\).
\[\begin{split}\left|\log(t)+8\log\left(\frac{1}{a}\right)\sum_{k=0}^{I }\tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|&=\left|\log(t)+8\log \left(\frac{1}{a}\right)\tilde{h}_{m-1}(t)\right|\leq\varepsilon/2,\\ &\forall t\in\left[\frac{1}{2^{m}},\frac{1}{3\cdot 2^{m-2}} \right]\cap[a,1]\text{ with }2\leq m\leq I;\end{split}\] (A.30)

and

\[\begin{split}&\left|\log(t)+8\log\left(\frac{1}{a}\right)\sum_{k=0} ^{I}\tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|\\ &=\left|\log\left(t\right)(\tilde{f}_{m}(t)+\tilde{f}_{m-1}(t))-8 \log\left(a\right)\left(\tilde{h}_{m}(t)\tilde{f}_{m}(t)+\tilde{h}_{m-1}(t) \tilde{f}_{m-1}(t)\right)\right|\\ &\leq\tilde{f}_{m}(t)\left|\log(t)-8\log\left(a\right)\tilde{h}_{ m}(t)\right|+\tilde{f}_{m-1}(t)\left|\log(t)-8\log\left(a\right)\tilde{h}_{m-1}(t) \right|\\ &\leq\tilde{f}_{m}(t)\cdot\frac{\varepsilon}{2}+\tilde{f}_{m-1}( t)\cdot\frac{\varepsilon}{2}=\frac{\varepsilon}{2},\quad\forall\;t\in\left[ \frac{1}{3\cdot 2^{m-1}},\frac{1}{2^{m}}\right]\cap[a,1]\text{ with }1\leq m\leq I.\end{split}\] (A.31)

Note that

\[[a,1]\subset[1/2,1]\cup\left(\bigcup_{m=1}^{I}\left[\frac{1}{3\cdot 2^{m-1}},\frac{1}{2^{m}}\right]\right)\cup\left(\bigcup_{m=2}^{I}\left[\frac{1}{2^{m}},\frac{1}{3\cdot 2^{m-2}}\right]\right).\]

Consequently, (A.28) follows immediately from (A.29), (A.30) and (A.31).

From Lemma A.4 we know that there exists

\[\mathrm{M}\in\mathcal{F}_{2}^{\mathbf{FNN}}\left(15\log\frac{96\left(\log a \right)^{2}}{\varepsilon},6,900\log\frac{96\left(\log a\right)^{2}}{ \varepsilon},1,1\right)\] (A.32)

such that for any \(t,t^{\prime}\in[0,1]\), there hold \(\mathrm{M}(t,t^{\prime})\in[0,1]\), \(\mathrm{M}(t,0)=\mathrm{M}(0,t^{\prime})=0\) and

\[\left|\mathrm{M}(t,t^{\prime})-t\cdot t^{\prime}\right|\leq\frac{\varepsilon }{96\left(\log a\right)^{2}}.\] (A.33)

Define

\[\tilde{g}_{3}:\mathbb{R}\rightarrow\mathbb{R},\quad x\mapsto\sum_{k=0}^{I} \mathrm{M}\left(\tilde{h}_{k}(x),\tilde{f}_{k}(x)\right),\]

and

\[\begin{split}\tilde{f}:\mathbb{R}&\rightarrow\mathbb{ R},\\ x&\mapsto\sum_{k=1}^{8I}\left[\frac{\log(a)}{I}\cdot \sigma\left(\frac{\log b}{8\log a}+\sigma\left(\sigma\left(\tilde{g}_{3}(x) \right)-\frac{\log b}{8\log a}\right)-\sigma\left(\sigma\left(\tilde{g}_{3}(x) \right)-\frac{1}{8}\right)\right)\right].\end{split}\]
Then it follows from (A.25),(A.33), (A.28), the definitions of \(\tilde{f}_{k}\) and \(\tilde{g}_{3}\) that

\[\begin{split}&|\log t-8\log(a)\cdot\tilde{g}_{3}(t)|\\ &\leq 8\log\left(\frac{1}{a}\right)\cdot\left|\tilde{g}_{3}(t)-\sum_{k= 0}^{I}\tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|+\left|\log t+8\log\left(\frac{1} {a}\right)\sum_{k=0}^{I}\tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|\\ &\leq 8\log\left(\frac{1}{a}\right)\cdot\left|\tilde{g}_{3}(t)-\sum_{ k=0}^{I}\tilde{h}_{k}(t)\tilde{f}_{k}(t)\right|+\varepsilon/2\\ &\leq\varepsilon/2+|8\log a|\cdot\sum_{k=0}^{I}\left|\mathrm{M} \left(\tilde{h}_{k}(t),\tilde{f}_{k}(t)\right)-\tilde{h}_{k}(t)\tilde{f}_{k}( t)\right|\\ &\leq\varepsilon/2+|8\log a|\cdot(I+1)\cdot\frac{\varepsilon}{96 \left(\log a\right)^{2}}\leq\varepsilon,\ \forall\ t\in[a,1].\end{split}\] (A.34)

However, for any \(t\in\mathbb{R}\), by the definition of \(\tilde{f}\), we have

\[\begin{split}\tilde{f}(t)=\begin{cases}8\log(a)\cdot\tilde{g}_{3 }(t),&\text{ if }8\log(a)\cdot\tilde{g}_{3}(t)\in[\log a,\log b],\\ \log a,&\text{ if }8\log(a)\cdot\tilde{g}_{3}(t)<\log a,\\ \log b,&\text{ if }8\log(a)\cdot\tilde{g}_{3}(t)>\log b, \end{cases}\\ \text{satisfying }\ \log a\leq\tilde{f}(t)\leq\log b\leq 0.\end{split}\] (A.35)

Then by (A.34), (A.35) and the fact that \(\log t\in[\log a,\log b],\ \forall\ t\in[a,b]\), we obtain

\[\left|\log t-\tilde{f}(t)\right|\leq|\log t-8\log(a)\cdot\tilde{g}_{3}(t)|\leq \varepsilon,\ \forall\ t\in[a,b].\]

That is,

\[\sup_{t\in[a,b]}\left|\log t-\tilde{f}(t)\right|\leq\varepsilon.\] (A.36)

Figure A.4: The network representing the function \(\tilde{g}_{3}\).
On the other hand, it follows from (A.26), (A.27), (A.32), the definition of \(\tilde{g}_{3}\), and \(1\leq I\leq 4\log\frac{1}{a}\) that

\[\tilde{g}_{3}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{ \varepsilon}+I+15\log\left(96\left(\log a\right)^{2}\right),C_{\alpha}\left( \frac{1}{\varepsilon}\right)^{\frac{1}{\alpha}}I,\right.\]

\[\left.\left(I+1\right)\cdot\left(20I+C_{\alpha}\left(\frac{1}{\varepsilon} \right)^{\frac{1}{\alpha}}\cdot\log\frac{1}{\varepsilon}+900\log\left(96 \left(\log a\right)^{2}\right)\right),1,\infty\right)\]

\[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log\frac{1}{\varepsilon} +139\log\frac{1}{a},C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{a} }\log\frac{1}{a},\right.\]

\[\left.C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{\alpha}}\cdot \left(\log\frac{1}{\varepsilon}\right)\cdot\left(\log\frac{1}{a}\right)+65440 \left(\log a\right)^{2},1,\infty\right).\]

Then by the definition of \(\tilde{f}\) we obtain (cf. Figure A.5)

\[\tilde{f}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{\alpha}\log \frac{1}{\varepsilon}+139\log\frac{1}{a},C_{\alpha}\left(\frac{1}{\varepsilon }\right)^{\frac{1}{\alpha}}\log\frac{1}{a},\right.\] \[\left.C_{\alpha}\left(\frac{1}{\varepsilon}\right)^{\frac{1}{ \alpha}}\cdot\left(\log\frac{1}{\varepsilon}\right)\cdot\left(\log\frac{1}{a} \right)+65440\left(\log a\right)^{2},1,\infty\right).\]

This, together with (A.35) and (A.36), completes the proof of Theorem 2.4. 

#### a.3.4 Proof of Theorem 2.2 and Theorem 2.3

This subsection is devoted to the proof of Theorem 2.2 and Theorem 2.3. We will first establish several lemmas. We then use these lemmas to prove Theorem 2.3. Finally, we derive Theorem 2.2 by applying Theorem 2.3 with \(q=0\), \(d_{*}=d\) and \(d_{*}=K=1\).

Figure A.5: The network representing the function \(\tilde{f}\).
**Lemma A.6**.: _Let \(\phi(t)=\log(1+\mathrm{e}^{-t})\) be the logistic loss. Suppose real numbers \(a,f,A,B\) satisfy that \(0<a<1\) and \(A\leq\min\left\{f,\log\frac{a}{1-a}\right\}\leq\max\left\{f,\log\frac{a}{1-a} \right\}\leq B\). Then there holds_

\[\min\left\{\frac{1}{4+2\mathrm{e}^{A}+2\mathrm{e}^{-A}},\frac{1}{4 +2\mathrm{e}^{B}+2\mathrm{e}^{-B}}\right\}\cdot\left|f-\log\frac{a}{1-a} \right|^{2}\] \[\leq a\phi(f)+(1-a)\phi(-f)-a\log\frac{1}{a}-(1-a)\log\frac{1}{1-a}\] \[\leq\sup\left\{\frac{1}{4+2\mathrm{e}^{z}+2\mathrm{e}^{-z}} \middle|z\in[A,B]\right\}\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\leq\frac{1} {8}\cdot\left|f-\log\frac{a}{1-a}\right|^{2}.\]

Proof.: Consider the map \(G:\mathbb{R}\to[0,\infty),z\mapsto a\phi(z)+(1-a)\phi(-z)\). Obviously \(G\) is twice continuously differentiable on \(\mathbb{R}\) with \(G^{\prime}\left(\log\frac{a}{1-a}\right)=0\) and \(G^{\prime\prime}(z)=\frac{1}{2+\mathrm{e}^{z}+\mathrm{e}^{-z}}\) for any real number \(z\). Then it follows from Taylor's theorem that there exists a real number \(\xi\) between \(\log\frac{a}{1-a}\) and \(f\), such that

\[a\phi(f)+(1-a)\phi(-f)-a\log\frac{1}{a}-(1-a)\log\frac{1}{1-a}=G (f)-G\left(\log\frac{a}{1-a}\right)\] (A.37) \[=\left(f-\log\frac{a}{1-a}\right)\cdot G^{\prime}\left(\log \frac{a}{1-a}\right)+\frac{G^{\prime\prime}(\xi)}{2}\cdot\left|f-\log\frac{a} {1-a}\right|^{2}\] \[=\frac{G^{\prime\prime}(\xi)}{2}\cdot\left|f-\log\frac{a}{1-a} \right|^{2}=\frac{\left|f-\log\frac{a}{1-a}\right|^{2}}{4+2\mathrm{e}^{\xi}+2 \mathrm{e}^{-\xi}}.\]

Since \(A\leq\min\left\{f,\log\frac{a}{1-a}\right\}\leq\max\left\{f,\log\frac{a}{1-a} \right\}\leq B\), we must have \(\xi\in[A,B]\), which, together with (A.37), yields

\[\min\left\{\frac{1}{4+2\mathrm{e}^{A}+2\mathrm{e}^{-A}},\frac{1}{ 4+2\mathrm{e}^{B}+2\mathrm{e}^{-B}}\right\}\cdot\left|f-\log\frac{a}{1-a} \right|^{2}\] (A.38) \[=\left(\inf_{t\in[A,B]}\frac{1}{4+2\mathrm{e}^{t}+\mathrm{e}^{-t} }\right)\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\leq\frac{\left|f-\log\frac{a }{1-a}\right|^{2}}{4+2\mathrm{e}^{\xi}+2\mathrm{e}^{-\xi}}\] \[=a\phi(f)+(1-a)\phi(-f)-a\log\frac{1}{a}-(1-a)\log\frac{1}{1-a}= \frac{\left|f-\log\frac{a}{1-a}\right|^{2}}{4+2\mathrm{e}^{\xi}+2\mathrm{e}^{- \xi}}\] \[\leq\sup\left\{\frac{1}{4+2\mathrm{e}^{z}+2\mathrm{e}^{-z}} \middle|z\in[A,B]\right\}\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\leq\frac{1} {8}\cdot\left|f-\log\frac{a}{1-a}\right|^{2}.\]

This completes the proof. 

**Lemma A.7**.: _Let \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) be the logistic loss, \(f\) be a real number, \(d\in\mathbb{N}\), and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the conditional probability function \([0,1]^{d}\ni z\mapsto P(\{1\}\,|z)\in[0,1]\) is denoted by \(\eta\). Then for \(x\in[0,1]^{d}\) such that \(\eta(x)\notin\{0,1\}\), there holds_

\[\left|\inf_{t\in\left[f\wedge\log\frac{\eta(x)}{1-\eta(x)},f\lor \log\frac{\eta(x)}{1-\eta(x)}\right]}\frac{1}{2(2+\mathrm{e}^{t}+\mathrm{e}^{-t })}\right|\cdot\left|f-\log\frac{\eta(x)}{1-\eta(x)}\right|^{2}\] \[\leq\int_{\{-1,1\}}\left(\phi\left(yf\right)-\phi\left(y\log \frac{\eta(x)}{1-\eta(x)}\right)\right)\mathrm{d}P(y|x)\] \[\leq\left|\sup_{t\in\left[f\wedge\log\frac{\eta(x)}{1-\eta(x)},f \lor\log\frac{\eta(x)}{1-\eta(x)}\right]}\frac{1}{2(2+\mathrm{e}^{t}+\mathrm{e }^{-t})}\right|\cdot\left|f-\log\frac{\eta(x)}{1-\eta(x)}\right|^{2}\leq\frac{1 }{4}\left|f-\log\frac{\eta(x)}{1-\eta(x)}\right|^{2}.\]

Proof.: Given \(x\in[0,1]^{d}\) such that \(\eta(x)\notin\{0,1\}\), define

\[V_{x}:\mathbb{R}\to(0,\infty),\quad t\mapsto\eta(x)\phi(t)+(1-\eta(x))\phi(-t).\]
Then it is easy to verify that

\[\int_{\{-1,1\}}\phi\left(y{t}\right)\mathrm{d}P(y|x)=\phi(t)P(Y=1|X=x)+\phi(-t)P( Y=-1|X=x)=V_{x}(t)\]

for all \(t\in\mathbb{R}\). Consequently,

\[\int_{\{-1,1\}}\left(\phi\left(y{f}\right)-\phi\left(y\log\frac{ \eta(x)}{1-\eta(x)}\right)\right)\mathrm{d}P(y|x)=V_{x}(f)-V_{x}\left(\log \frac{\eta(x)}{1-\eta(x)}\right)\] \[=\eta(x)\phi(f)+(1-\eta(x))\phi(-f)-\eta(x)\log\frac{1}{\eta(x)} -(1-\eta(x))\log\frac{1}{1-\eta(x)}.\]

The desired inequalities then follow immediately by applying Lemma A.6. 

**Lemma A.8**.: _Let \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) be the logistic loss, \(d\in\mathbb{N}\), \(f:[0,1]^{d}\to\mathbb{R}\) be a measurable function, and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the conditional probability function \([0,1]^{d}\ni z\mapsto P(\{1\}\,|z)\in[0,1]\) is denoted by \(\eta\). Assume that there exist constants \((a,b)\in\mathbb{R}^{2}\), \(\delta\in(0,1/2)\), and a measurable function \(\hat{\eta}:[0,1]^{d}\to\mathbb{R}\), such that \(\hat{\eta}=\eta\), \(P_{X}\)-a.s.,_

\[\log\frac{\delta}{1-\delta}\leq f(x)\leq-a,\;\forall\;x\in[0,1]^{d}\text{ satisfying }0\leq\hat{\eta}(x)=\eta(x)<\delta,\]

_and_

\[b\leq f(x)\leq\log\frac{1-\delta}{\delta},\;\forall\;x\in[0,1]^{d}\text{ satisfying }1-\delta<\hat{\eta}(x)=\eta(x)\leq 1.\]

_Then_

\[\mathcal{E}_{P}^{\phi}\left(f\right)-\phi(a)P_{X}(\Omega_{2})- \phi(b)P_{X}(\Omega_{3})\] \[\leq\int_{\Omega_{1}}\sup\left\{\frac{\left|f(x)-\log\frac{\eta(x )}{1-\eta(x)}\right|^{2}}{2(2+\mathrm{e}^{t}+\mathrm{e}^{-t})}\Bigg{|}t\in \left[f(x)\wedge\log\frac{\eta(x)}{1-\eta(x)},f(x)\vee\log\frac{\eta(x)}{1- \eta(x)}\right]\right\}\mathrm{d}P_{X}(x)\] \[\leq\int_{\Omega_{1}}\left|f(x)-\log\frac{\eta(x)}{1-\eta(x)} \right|^{2}\mathrm{d}P_{X}(x),\]

_where_

\[\Omega_{1} :=\left\{\left.x\in[0,1]^{d}\right|\delta\leq\hat{\eta}(x)=\eta(x )\leq 1-\delta\right\},\] (A.39) \[\Omega_{2} :=\left\{\left.x\in[0,1]^{d}\right|0\leq\hat{\eta}(x)=\eta(x)< \delta\right\},\] \[\Omega_{3} :=\left\{\left.x\in[0,1]^{d}\right|1-\delta<\hat{\eta}(x)=\eta(x) \leq 1\right\}.\]

Proof.: Define

\[\psi:[0,1]^{d}\times\{-1,1\} \to[0,\infty),\] \[(x,y) \mapsto\begin{cases}\phi\left(y\log\frac{\eta(x)}{1-\eta(x)} \right),&\text{if }\eta(x)\in[\delta,1-\delta],\\ 0,&\text{if }\eta(x)\in\{0,1\},\\ \eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)},&\text{if } \eta(x)\in(0,\delta)\cup(1-\delta,1).\end{cases}\]

Since \(\hat{\eta}=\eta\in[0,1]\), \(P_{X}\)-a.s., we have that \(P_{X}([0,1]^{d}\setminus(\Omega_{1}\cup\Omega_{2}\cup\Omega_{3}))=0\). Then it follows from lemma A.3 that

\[\mathcal{E}_{P}^{\phi}(f)=\mathcal{R}_{P}^{\phi}(f)-\inf\left\{ \left.\mathcal{R}_{P}^{\phi}(g)\right|g:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\right\}\] (A.40) \[=\int_{[0,1]^{d}\times\{-1,1\}}\phi(yf(x))\mathrm{d}P(x,y)-\int_{[ 0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)=I_{1}+I_{2}+I_{3},\]
where

\[I_{i}:=\int_{\Omega_{i}\times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi(x,y) \right)\mathrm{d}P(x,y),\;i=1,2,3.\]

According to Lemma A.7, we have

\[\begin{split} I_{1}&=\int_{\Omega_{1}}\int_{\{-1,1 \}}\left(\phi\left(yf(x)\right)-\phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right) \right)\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\\ &\leq\int_{\Omega_{1}}\sup\left\{\left.\frac{\left|f(x)-\log \frac{\eta(x)}{1-\eta(x)}\right|^{2}}{2(2+\mathrm{e}^{t}+\mathrm{e}^{-t})} \right|\begin{aligned} & t\in\left[f(x)\wedge\log\frac{\eta(x)}{1- \eta(x)},\infty\right)\text{ and }\\ & t\in\left(-\infty,f(x)\vee\log\frac{\eta(x)}{1-\eta(x)}\right] \end{aligned}\right\}\mathrm{d}P_{X}(x).\end{split}\] (A.41)

Then it remains to bound \(I_{2}\) and \(I_{3}\).

Indeed, for any \(x\in\Omega_{2}\), if \(\eta(x)=0\), then

\[\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)=\phi(-f(x)) \leq\phi(a).\]

Otherwise, we have

\[\begin{split}&\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right) \mathrm{d}P(y|x)\\ &=\left(\phi(f(x))-\log\frac{1}{\eta(x)}\right)\eta(x)+\left( \phi(-f(x))-\log\frac{1}{1-\eta(x)}\right)(1-\eta(x))\\ &=\left(\phi\left(f(x)\right)-\phi\left(\log\frac{\eta(x)}{1- \eta(x)}\right)\right)\eta(x)+\left(\phi\left(-f(x)\right)-\phi\left(-\log \frac{\eta(x)}{1-\eta(x)}\right)\right)(1-\eta(x))\\ &\leq\left(\phi\left(\log\frac{\delta}{1-\delta}\right)-\phi \left(\log\frac{\eta(x)}{1-\eta(x)}\right)\right)\eta(x)+\phi(-f(x))(1-\eta(x) )\\ &\leq\phi(-f(x))(1-\eta(x))\leq\phi(-f(x))\leq\phi(a).\end{split}\]

Therefore, no matter whether \(\eta(x)=0\) or \(\eta(x)\neq 0\), there always holds

\[\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)\leq\phi(a),\]

which means that

\[\begin{split} I_{2}&=\int_{\Omega_{2}}\int_{\{-1,1 \}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\\ &\leq\int_{\Omega_{2}}\phi(a)\mathrm{d}P_{X}(x)=\phi(a)P_{X}( \Omega_{2}).\end{split}\] (A.42)

Similarly, for any \(x\in\Omega_{3}\), if \(\eta(x)=1\), then

\[\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)=\phi(f(x)) \leq\phi(b).\]

Otherwise, we have

\[\begin{split}&\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right) \mathrm{d}P(y|x)\\ &=\left(\phi(f(x))-\log\frac{1}{\eta(x)}\right)\eta(x)+\left( \phi(-f(x))-\log\frac{1}{1-\eta(x)}\right)(1-\eta(x))\\ &=\left(\phi\left(f(x)\right)-\phi\left(\log\frac{\eta(x)}{1- \eta(x)}\right)\right)\eta(x)+\left(\phi\left(-f(x)\right)-\phi\left(-\log \frac{\eta(x)}{1-\eta(x)}\right)\right)(1-\eta(x))\end{split}\]
\[\leq\phi(f(x))\eta(x)+\left(\phi\left(\log\frac{\delta}{1-\delta} \right)-\phi\left(\log\frac{1-\eta(x)}{\eta(x)}\right)\right)(1-\eta(x))\] \[\leq\phi(f(x))\eta(x)\leq\phi(f(x))\leq\phi(b).\]

Therefore, no matter whether \(\eta(x)=1\) or \(\eta(x)\neq 1\), we have

\[\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)\leq\phi(b),\]

which means that

\[\begin{split} I_{3}&=\int_{\Omega_{3}}\int_{\{-1, 1\}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\\ &\leq\int_{\Omega_{3}}\phi(b)\mathrm{d}P_{X}(x)=\phi(b)P_{X}( \Omega_{3}).\end{split}\] (A.43)

The desired inequality then follows immediately from (A.41), (A.42), (A.43) and (A.40). Thus we complete the proof. 

**Lemma A.9**.: _Let \(\delta\in(0,1/2)\), \(a\in[\delta,1-\delta]\), \(f\in\left[-\log\frac{1-\delta}{\delta},\log\frac{1-\delta}{\delta}\right]\), and \(\phi(t)=\log(1+\mathrm{e}^{-t})\) be the logistic loss. Then there hold_

\[H(a,f)\leq\Gamma\cdot G(a,f)\]

_with \(\Gamma=5000\left|\log\delta\right|^{2}\),_

\[H(a,f):=a\cdot\left|\phi(f)-\phi\left(\log\frac{a}{1-a}\right)\right|^{2}+(1-a )\cdot\left|\phi(-f)-\phi\left(-\log\frac{a}{1-a}\right)\right|^{2},\]

_and_

\[G(a,f) :=a\phi(f)+(1-a)\phi(-f)-a\phi\left(\log\frac{a}{1-a}\right)-(1-a) \phi\left(-\log\frac{a}{1-a}\right)\] \[=a\phi(f)+(1-a)\phi(-f)-a\log\frac{1}{a}-(1-a)\log\frac{1}{1-a}.\]

Proof.: In this proof, we will frequently use elementary inequalities

\[x\log\frac{1}{x}\leq\min\left\{1-x,(1-x)\cdot\log\frac{1}{1-x}\right\},\;\forall \;x\in[1/2,1),\] (A.44)

and

\[\begin{split}&-\log\frac{1}{1-x}-2<-\log 7\leq-\log\left(\exp \left(\frac{3-3x}{x}\log\frac{1}{1-x}\right)-1\right)\\ &<\log\frac{x}{1-x}<2+\log\frac{1}{1-x},\;\forall\;x\in[1/2,1). \end{split}\] (A.45)

We first show that

\[\begin{split} G(a,f)&\geq\frac{a\phi(f)}{3}\\ &\text{provided }\frac{1}{2}\leq a\leq 1-\delta\text{ and }f\leq-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1 \right).\end{split}\] (A.46)

Indeed, if \(1/2\leq a\leq 1-\delta\) and \(f\leq-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1\right)\), then

\[\begin{split}&\frac{2}{3}\cdot a\phi(f)\geq\frac{2}{3}\cdot a \phi\left(-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1 \right)\right)=(2-2a)\cdot\log\frac{1}{1-a}\\ &\geq a\log\frac{1}{a}+(1-a)\log\frac{1}{1-a},\end{split}\]
which means that

\[G(a,f)\geq a\phi(f)-a\log\frac{1}{a}-(1-a)\log\frac{1}{1-a}\geq\frac{a\phi(f)}{3}.\]

This proves (A.46).

We next show that

\[\begin{split}& G(a,f)\geq\frac{1-a}{18}\left|f-\log\frac{a}{1-a} \right|^{2}\\ &\text{provided }\frac{1}{2}\leq a\leq 1-\delta\text{ and }-2-\log\frac{1}{1-a}\leq f\leq 2+\log\frac{1}{1-a}.\end{split}\] (A.47)

Indeed, if \(1/2\leq a\leq 1-\delta\) and \(-2-\log\frac{1}{1-a}\leq f\leq 2+\log\frac{1}{1-a}\), then it follows from Lemma A.6 that

\[\begin{split}& G(a,f)\geq\frac{\left|f-\log\frac{a}{1-a}\right|^{2 }}{4+2\exp\left(2+\log\frac{1}{1-a}\right)+2\exp\left(-2-\log\frac{1}{1-a} \right)}\\ &\geq\frac{\left|f-\log\frac{a}{1-a}\right|^{2}}{5+15\cdot\frac{1 }{1-a}}\geq\frac{(1-a)\cdot\left|f-\log\frac{a}{1-a}\right|^{2}}{5-5a+15}\geq \frac{(1-a)\cdot\left|f-\log\frac{a}{1-a}\right|^{2}}{18},\end{split}\]

which proves (A.47).

We then show

\[H(a,f)\leq\Gamma\cdot G(a,f)\text{ provided }1/2\leq a\leq 1-\delta\text{ and }- \log\frac{1-\delta}{\delta}\leq f\leq\log\frac{1-\delta}{\delta}\] (A.48)

by considering the following four cases.

**Case I.**\(1/2\leq a\leq 1-\delta\) and \(2+\log\frac{1}{1-a}\leq f\leq\log\frac{1-\delta}{\delta}\). In this case we have

\[\begin{split}&\log\frac{1}{\delta}=\phi\left(\log\frac{\delta}{1- \delta}\right)\geq\phi(-f)=\log(1+\mathrm{e}^{f})\geq f\geq 2+\log\frac{1}{1-a} \\ &>\phi\left(-\log\frac{a}{1-a}\right)=\log\frac{1}{1-a}\geq\log \frac{1}{a}>0,\end{split}\] (A.49)

which, together with (A.44), yields

\[a\log\frac{1}{a}+(1-a)\log\frac{1}{1-a}\leq(1-a)\cdot\left(1+\log\frac{1}{1- a}\right)\leq(1-a)\cdot\frac{1+\log\frac{1}{1-a}}{2+\log\frac{1}{1-a}}\cdot \phi(-f).\]

Consequently,

\[\begin{split}& G(a,f)\geq(1-a)\cdot\phi(-f)-a\log\frac{1}{a}-(1-a )\log\frac{1}{1-a}\\ &\geq(1-a)\cdot\phi(-f)-(1-a)\cdot\frac{1+\log\frac{1}{1-a}}{2+ \log\frac{1}{1-a}}\cdot\phi(-f)\\ &=\frac{(1-a)\cdot\phi(-f)}{2+\log\frac{1}{1-a}}\geq\frac{(1-a) \cdot\phi(-f)}{4\log\frac{1}{\delta}}.\end{split}\] (A.50)

On the other hand, it follows from \(f\geq 2+\log\frac{1}{1-a}>\log\frac{a}{1-a}\) that

\[0\leq\phi\left(\log\frac{a}{1-a}\right)-\phi(f)<\phi\left(\log\frac{a}{1-a} \right),\]

which, together with (A.44) and (A.49), yields

\[\begin{split}& a\cdot\left|\phi(f)-\phi\left(\log\frac{a}{1-a} \right)\right|^{2}\leq a\cdot\left|\phi\left(\log\frac{a}{1-a}\right)\right|^ {2}\\ &=a\cdot\left|\log\frac{1}{a}\right|^{2}\leq(1-a)\cdot\log\frac{1 }{a}\leq(1-a)\cdot\phi(-f).\end{split}\] (A.51)
Besides, it follows from (A.50) that \(0\leq\phi(-f)-\phi\left(-\log\frac{a}{1-a}\right)\leq\phi(-f)\). Consequently,

\[(1-a)\cdot\left|\phi(-f)-\phi\left(-\log\frac{a}{1-a}\right)\right|^{2}\leq(1- a)\cdot\phi(-f)^{2}\leq(1-a)\cdot\phi(-f)\cdot\log\frac{1}{\delta}.\] (A.52)

Combining (A.50), (A.51) and (A.52), we deduce that

\[H(a,f)\leq(1-a)\cdot\phi(-f)\cdot\left|1+\log\frac{1}{\delta}\right|\leq(1-a) \cdot\phi(-f)\cdot\frac{\Gamma}{4\log\frac{1}{\delta}}\leq\Gamma\cdot G(a,f),\]

which proves the desired inequality.

**Case II.**\(1/2\leq a\leq 1-\delta\) and \(-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1\right)\leq f<2+ \log\frac{1}{1-a}\). In this case, we have \(-2-\log\frac{1}{1-a}\leq f\leq 2+\log\frac{1}{1-a}\), where we have used (A.45). Therefore, it follows from (A.47) that \(G(a,f)\geq\frac{1-a}{18}\left|f-\log\frac{a}{1-a}\right|^{2}\). On the other hand, it follow from (A.45) and Taylor's Theorem that there exists

\[-\log 7\leq-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a} \right)-1\right)\] \[\leq f\wedge\log\frac{a}{1-a}\leq\xi\leq f\vee\log\frac{a}{1-a} \leq 2+\log\frac{1}{1-a},\]

such that

\[a\cdot\left|\phi(f)-\phi\left(\log\frac{a}{1-a}\right)\right|^{2}\] (A.53) \[=a\cdot\left|\phi^{\prime}(\xi)\right|^{2}\cdot\left|f-\log\frac{ a}{1-a}\right|^{2}\leq a\cdot\mathrm{e}^{-2\xi}\cdot\left|f-\log\frac{a}{1-a} \right|^{2}\] \[\leq a\cdot\exp(\log 7)\cdot\exp\left(\log\left(\exp\left(\frac{3-3 a}{a}\log\frac{1}{1-a}\right)-1\right)\right)\cdot\left|f-\log\frac{a}{1-a} \right|^{2}\] \[=7a\cdot\int_{0}^{\frac{3-3a}{a}\log\frac{1}{1-a}}\mathrm{e}^{t} \mathrm{d}t\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\] \[\leq 7a\cdot\left|\frac{3-3a}{a}\log\frac{1}{1-a}\right|\cdot \exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)\cdot\left|f-\log\frac{a}{1-a }\right|^{2}\] \[\leq 7a\cdot\left|\frac{3-3a}{a}\log\frac{1}{1-a}\right|\cdot (1+\exp\left(\log 7\right))\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\] \[\leq 168\cdot\left|(1-a)\cdot\log\frac{1}{1-a}\right|\cdot \left|f-\log\frac{a}{1-a}\right|^{2}\] \[\leq 168\cdot\left|(1-a)\cdot\log\frac{1}{\delta}\right|\cdot \left|f-\log\frac{a}{1-a}\right|^{2}.\]

Besides, we have

\[(1-a)\cdot\left|\phi(-f)-\phi\left(-\log\frac{a}{1-a}\right) \right|^{2}\] (A.54) \[\leq\left|1-a\right|\cdot\left|\phi^{\prime}\right|_{\mathbb{R}} \cdot\left|f-\log\frac{a}{1-a}\right|^{2}\leq\left|1-a\right|\cdot\left|f- \log\frac{a}{1-a}\right|^{2}.\]

Combining (A.53), (A.54) and the fact that \(G(a,f)\geq\frac{1-a}{18}\left|f-\log\frac{a}{1-a}\right|^{2}\), we deduce that

\[H(a,f)\leq 168\cdot\left|(1-a)\cdot\log\frac{1}{\delta}\right| \cdot\left|f-\log\frac{a}{1-a}\right|^{2}+\left|1-a\right|\cdot\left|f-\log \frac{a}{1-a}\right|^{2}\] \[\leq 170\cdot\left|(1-a)\cdot\log\frac{1}{\delta}\right|\cdot \left|f-\log\frac{a}{1-a}\right|^{2}\leq\Gamma\cdot\frac{1-a}{18}\cdot\left|f- \log\frac{a}{1-a}\right|^{2}\leq\Gamma\cdot G(a,f),\]
which proves the desired inequality.

**Case III.**\(1/2\leq a\leq 1-\delta\) and \(-\log\frac{a}{1-a}\leq f<-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a} \right)-1\right)\). In this case, we still have (A.54). Besides, it follows from (A.46) that \(G(a,f)\geq\frac{a\phi(f)}{3}\). Moreover, by (A.45) we obtain \(-2-\log\frac{1}{1-a}<f<2+\log\frac{1}{1-a}\), which, together with (A.47), yields \(G(a,f)\geq\frac{1-a}{18}\left|f-\log\frac{a}{1-a}\right|^{2}\). In addition, since \(f<-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1\right)\leq\log \frac{a}{1-a}\), we have that \(0<\phi(f)-\phi\left(\log\frac{a}{1-a}\right)<\phi(f)\), which means that

\[\begin{split}& a\cdot\left|\phi(f)-\phi\left(\log\frac{a}{1-a} \right)\right|^{2}\leq a\cdot\left|\phi(f)\right|^{2}\\ &\leq a\phi(f)\phi\left(-\log\frac{a}{1-a}\right)=a\phi(f)\log \frac{1}{1-a}\leq a\phi(f)\log\frac{1}{\delta}.\end{split}\] (A.55)

Combining all these inequalities, we obtain

\[\begin{split}& H(a,f)\leq a\phi(f)\cdot\left|\log\frac{1}{ \delta}\right|+\left|1-a\right|\cdot\left|f-\log\frac{a}{1-a}\right|^{2}\\ &\leq\frac{\Gamma a\phi(f)}{6}+\Gamma\cdot\frac{1-a}{36}\cdot \left|f-\log\frac{a}{1-a}\right|^{2}\\ &\leq\frac{\Gamma\cdot G(a,f)}{2}+\frac{\Gamma\cdot G(a,f)}{2}= \Gamma\cdot G(a,f),\end{split}\]

which proves the desired inequality.

**Case IV.**\(-\log\frac{1-\delta}{\delta}\leq f<\min\left\{-\log\frac{a}{1-a},-\log \left(\exp\left(\frac{3-3a}{a}\log\frac{1}{1-a}\right)-1\right)\right\}\) and \(1/2\leq a\leq 1-\delta\). In this case, we still have \(G(a,f)\geq\frac{a\phi(f)}{3}\) according to (A.46). Besides, it follows from

\[f<\min\left\{-\log\frac{a}{1-a},-\log\left(\exp\left(\frac{3-3a}{a}\log\frac{ 1}{1-a}\right)-1\right)\right\}\leq-\log\frac{a}{1-a}\leq\log\frac{a}{1-a}\]

that

\[\begin{split} 0&\leq\min\left\{\phi\left(-\log\frac{a}{1-a} \right)-\phi(-f),\phi(f)-\phi\left(\log\frac{a}{1-a}\right)\right\}\\ &\leq\max\left\{\phi\left(-\log\frac{a}{1-a}\right)-\phi(-f),\phi (f)-\phi\left(\log\frac{a}{1-a}\right)\right\}\\ &\leq\max\left\{\phi\left(-\log\frac{a}{1-a}\right),\phi(f) \right\}=\phi(f).\end{split}\] (A.56)

Combining (A.56) and the fact that \(G(a,f)\geq\frac{a\phi(f)}{3}\), we deduce that

\[\begin{split}& H(a,f)\leq a\cdot\left|\phi(f)\right|^{2}+(1-a) \cdot\left|\phi(f)\right|^{2}\leq\phi(f)\phi\left(-\log\frac{1-\delta}{\delta }\right)\\ &=\phi(f)\log\frac{1}{\delta}\leq\frac{\Gamma a\phi(f)}{3}\leq \Gamma\cdot G(a,f),\end{split}\]

which proves the desired inequality.

Combining all these four cases, we conclude that (A.48) has been proved. Furthermore, (A.48) yields that

\[H(a,f)=H(1-a,-f)\leq\Gamma\cdot G(1-a,-f)=\Gamma\cdot G(a,f)\]

provided \(\delta\leq a\leq 1/2\) and \(-\log\frac{1-\delta}{\delta}\leq f\leq\log\frac{1-\delta}{\delta}\), which, together with (A.48), proves this lemma. 

**Lemma A.10**.: _Let \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) be the logistic loss, \(\delta_{0}\in(0,1/3),\)\(d\in\mathbb{N}\) and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the conditional probability function \([0,1]^{d}\ni z\mapsto P(\{1\}\left|z\right)\in[0,1]\) is denoted by \(\eta\). Then there exists a measurable function_

\[\psi:[0,1]^{d}\times\{-1,1\}\rightarrow\left[0,\log\frac{10\log(1/\delta_{0})} {\delta_{0}}\right]\]
_such that_

\[\int_{[0,1]^{d}\times\{-1,1\}}\psi\left(x,y\right)\mathrm{d}P(x,y)=\inf\Big{\{} \,\mathcal{R}_{P}^{\phi}(g)\Big{|}\,g:[0,1]^{d}\to\mathbb{R}\text{ is measurable}\Big{\}}\] (A.57)

_and_

\[\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi(x, y)\right)^{2}\mathrm{d}P(x,y)\] (A.58) \[\leq 125000\left|\log\delta_{0}\right|^{2}\cdot\int_{[0,1]^{d} \times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi(x,y)\right)\mathrm{d}P(x,y)\]

_for any measurable \(f:[0,1]^{d}\to\left[\log\frac{\delta_{0}}{1-\delta_{0}},\log\frac{1-\delta_{0} }{\delta_{0}}\right]\)._

Proof.: Let

\[H:[0,1]\to[0,\infty),\quad t\mapsto\begin{cases}t\log\left(\frac{1}{t}\right) +(1-t)\log\left(\frac{1}{1-t}\right),&\text{ if }\in(0,1),\\ 0,&\text{ if }t\in\{0,1\}.\end{cases}\]

Then it is easy to show that \(H\left(\frac{\delta_{0}}{10\log(1/\delta_{0})}\right)\leq\frac{4}{5}\log \left(\frac{1}{1-\delta_{0}}\right)\leq H\left(\frac{\delta_{0}}{\log(1/\delta _{0})}\right)\). Thus there exists \(\delta_{1}\in\left(0,\frac{1}{3}\right)\) such that

\[H(\delta_{1})\leq\frac{4}{5}\log\left(\frac{1}{1-\delta_{0}}\right)\]

and

\[0<\frac{\delta_{0}}{10\log\left(1/\delta_{0}\right)}\leq\delta_{1}\leq\frac{ \delta_{0}}{\log(1/\delta_{0})}\leq\delta_{0}<1/3.\]

Take

\[\psi:[0,1]^{d}\times\{-1,1\}\to\mathbb{R},\quad(x,y)\mapsto\begin{cases}\phi \left(y\log\frac{\eta(x)}{1-\eta(x)}\right),&\text{if }\eta(x)\in[\delta_{1},1-\delta_{1}],\\ H(\eta(x)),&\text{if }\eta(x)\notin[\delta_{1},1-\delta_{1}],\end{cases}\]

which can be further expressed as

\[\psi:[0,1]^{d}\times\{-1,1\}\to\mathbb{R},\] \[(x,y)\mapsto\begin{cases}\phi\left(y\log\frac{\eta(x)}{1-\eta(x) }\right),&\text{if }\eta(x)\in[\delta_{1},1-\delta_{1}],\\ 0,&\text{if }\eta(x)\in\{0,1\},\\ \eta(x)\log\frac{1}{\eta(x)}+(1-\eta(x))\log\frac{1}{1-\eta(x)},&\text{if } \eta(x)\in(0,\delta_{1})\cup(1-\delta_{1},1).\end{cases}\]

Obviously, \(\psi\) is a measurable function such that

\[0\leq\psi(x,y)\leq\log\frac{1}{\delta_{1}}\leq\log\frac{10\log(1/\delta_{0})}{ \delta_{0}},\quad\forall\;(x,y)\in[0,1]^{d}\times\{-1,1\},\]

and it follows immediately from Lemma A.3 that (A.57) holds. We next show (A.58).

For any measurable function \(f:[0,1]^{d}\to\left[\log\frac{\delta_{0}}{1-\delta_{0}},\log\frac{1-\delta_{0 }}{\delta_{0}}\right]\) and any \(x\in[0,1]^{d}\), if \(\eta(x)\notin[\delta_{1},1-\delta_{1}]\), then we have

\[0 \leq\psi(x,y)=H(\eta(x))\leq H(\delta_{1})\leq\frac{4}{5}\log \frac{1}{1-\delta_{0}}\] \[=\frac{4}{5}\phi\left(\log\frac{1-\delta_{0}}{\delta_{0}}\right) \leq\frac{4}{5}\phi(yf(x))\leq\phi(yf(x)),\quad\forall\;y\in\{-1,1\}.\]
Hence \(0\leq\frac{1}{5}\phi(yf(x))\leq\phi(yf(x))-\psi(x,y)\leq\phi(yf(x)),\ \forall\ y\in \{-1,1\}\), which means that

\[(\phi(yf(x))-\psi(x,y))^{2}\leq\phi(yf(x))^{2}\leq\phi(yf(x))\phi \left(-\log\frac{1-\delta_{0}}{\delta_{0}}\right)\] \[=\frac{1}{5}\phi(yf(x))\cdot 5\log\frac{1}{\delta_{0}}\leq(\phi(yf(x ))-\psi(x,y))\cdot 5000\left|\log\delta_{1}\right|^{2},\ \ \ \ \forall\ y\in\{-1,1\}.\]

Integrating both sides with respect to \(y\), we obtain

\[\begin{split}&\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)^{2} \mathrm{d}P(y|x)\\ &\leq 5000\left|\log\delta_{1}\right|^{2}\cdot\int_{\{-1,1\}}\left( \phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(y|x).\end{split}\] (A.59)

If \(\eta(x)\in[\delta_{1},1-\delta_{1}]\), then it follows from Lemma A.9 that

\[\begin{split}&\int_{\{-1,1\}}\left(\phi(yf(x))-\psi(x,y)\right)^{ 2}\mathrm{d}P(y|x)\\ &=\eta(x)\left|\phi(f(x))-\phi\left(\log\frac{\eta(x)}{1-\eta(x) }\right)\right|^{2}+(1-\eta(x))\left|\phi(-f(x))-\phi\left(-\log\frac{\eta(x) }{1-\eta(x)}\right)\right|^{2}\\ &\leq 5000\left|\log\delta_{1}\right|^{2}\cdot\left(\eta(x)\phi(f(x)) +(1-\eta(x))\phi(-f(x))\right.\\ &\hskip 113.811024pt-\eta(x)\phi\Big{(}\log\frac{\eta(x)}{1-\eta( x)}\Big{)}-(1-\eta(x))\phi\Big{(}-\log\frac{\eta(x)}{1-\eta(x)}\Big{)}\Bigg{)}\\ &=5000\left|\log\delta_{1}\right|^{2}\int_{\{-1,1\}}\left(\phi(yf (x))-\psi(x,y)\right)\mathrm{d}P(y|x),\end{split}\]

which means that (A.59) still holds. Therefore, (A.59) holds for all \(x\in[0,1]^{d}\). We then integrate both sides of (A.59) with respect to \(x\) and obtain

\[\begin{split}&\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(yf(x))- \psi(x,y)\right)^{2}\mathrm{d}P(x,y)\\ &\leq 5000\left|\log\delta_{1}\right|^{2}\int_{[0,1]^{d}\times\{-1,1\} }\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(x,y)\\ &\leq 125000\left|\log\delta_{0}\right|^{2}\int_{[0,1]^{d}\times\{-1,1 \}}\left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(x,y),\end{split}\]

which yields (A.58). In conclusion, the function \(\psi\) defined above has all the desired properties. Thus we complete the proof. 

The following Lemma A.11 is similar to Lemma 3 of [37].

**Lemma A.11**.: _Let \((d,d_{\star},d_{\star},K)\in\mathbb{N}^{4}\), \(\beta\in(0,\infty)\), \(r\in[1,\infty)\), and \(q\in\mathbb{N}\cup\{0\}\). Suppose \(h_{0},h_{1},\ldots,h_{q},\tilde{h}_{0},\tilde{h}_{1},\ldots,\tilde{h}_{q}\) are functions satisfying that_

* \(\mathbf{dom}(h_{i})=\mathbf{dom}(\tilde{h}_{i})=[0,1]^{K}\) _for_ \(0<i\leq q\) _and_ \(\mathbf{dom}(h_{0})=\)__ \(\mathbf{dom}(\tilde{h}_{0})=[0,1]^{d}\)_;_
* \(\mathbf{ran}(h_{i})\cup\mathbf{ran}(\tilde{h}_{i})\subset[0,1]^{K}\) _for_ \(0\leq i<q\) _and_ \(\mathbf{ran}(h_{q})\cup\mathbf{ran}(\tilde{h}_{q})\subset\mathbb{R}\)_;_
* \(h_{q}\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{\star},\beta,r)\cup\mathcal{G}_ {\infty}^{\mathbf{M}}(d_{\star})\)_;_
* _For_ \(0\leq i<q\) _and_ \(1\leq j\leq K\)_, the_ \(j\)_-th coordinate function of_ \(h_{i}\) _given by_ \(\mathbf{dom}(h_{i})\ni x\mapsto(h_{i}(x))_{j}\in\mathbb{R}\) _belongs to_ \(\mathcal{G}_{\infty}^{\mathbf{H}}(d_{\star},\beta,r)\cup\mathcal{G}_{\infty}^{ \mathbf{M}}(d_{\star})\)_._
_Then there holds_

\[\begin{split}&\left\|h_{q}\circ h_{q-1}\circ\cdots\circ h_{1}\circ h _{0}-\tilde{h}_{q}\circ\tilde{h}_{q-1}\circ\cdots\circ\tilde{h}_{1}\circ\tilde {h}_{0}\right\|_{[0,1]^{d}}\\ &\leq\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-1}(1 \wedge\beta)^{k}}\cdot\sum_{k=0}^{q}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-k}}.\end{split}\] (A.60)

Proof.: We will prove this lemma by induction on \(q\). The case \(q=0\) is trivial. Now assume that \(q>0\) and that the desired result holds for \(q-1\). Consider the case \(q\). For each \(0\leq i<q\) and \(1\leq j\leq K\), denote

\[\tilde{h}_{i,j}:\mathbf{dom}(\tilde{h}_{i})\rightarrow\mathbb{R},\;\;x\mapsto \big{(}\tilde{h}_{i}(x)\big{)}_{j},\]

and

\[h_{i,j}:\mathbf{dom}(h_{i})\rightarrow\mathbb{R},\;\;x\mapsto\big{(}h_{i}(x) \big{)}_{j}.\]

Obviously, \(\mathbf{ran}(\tilde{h}_{i,j})\cup\mathbf{ran}(h_{i,j})\subset[0,1]\). By induction hypothesis (that is, the case \(q-1\) of this lemma), we have that

\[\begin{split}&\left\|h_{q-1,j}\circ h_{q-2}\circ h_{q-3}\circ \cdots\circ h_{0}-\tilde{h}_{q-1,j}\circ\tilde{h}_{q-2}\circ\tilde{h}_{q-3} \circ\cdots\circ\tilde{h}_{0}\right\|_{[0,1]^{d}}\\ &\leq\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-2}(1 \wedge\beta)^{k}}\cdot\left(\left\|\tilde{h}_{q-1,j}-h_{q-1,j}\right\|_{ \mathbf{dom}(h_{q-1,j})}+\sum_{k=0}^{q-2}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1-k}}\right)\\ &\leq\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-2}(1 \wedge\beta)^{k}}\cdot\sum_{k=0}^{q-1}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1-k}},\;\forall\;j\in\mathbb{Z}\cap(0, K].\end{split}\]

Therefore,

\[\begin{split}&\left\|h_{q-1}\circ h_{q-2}\circ h_{q-3}\circ \cdots\circ h_{0}-\tilde{h}_{q-1}\circ\tilde{h}_{q-2}\circ\tilde{h}_{q-3} \circ\cdots\circ\tilde{h}_{0}\right\|_{[0,1]^{d}}\\ &=\sup_{j\in\mathbb{Z}\cap(0,K]}\left\|h_{q-1,j}\circ h_{q-2} \circ h_{q-3}\circ\cdots\circ h_{0}-\tilde{h}_{q-1,j}\circ\tilde{h}_{q-2} \circ\tilde{h}_{q-3}\circ\cdots\circ\tilde{h}_{0}\right\|_{[0,1]^{d}}\\ &\leq\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-2}(1 \wedge\beta)^{k}}\cdot\sum_{k=0}^{q-1}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1-k}}.\end{split}\] (A.61)

We next show that

\[\left|h_{q}(x)-h_{q}(x^{\prime})\right|\leq r\cdot d_{*}^{1\wedge\beta}\cdot \left\|x-x^{\prime}\right\|_{\infty}^{1\wedge\beta},\;\forall\;x,x^{\prime} \in[0,1]^{K}\] (A.62)

by considering three cases.

**Case I:**\(h_{q}\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{*},\beta,r)\) and \(\beta>1\). In this case, we must have that \(h_{q}\in\mathcal{G}_{K}^{\mathbf{H}}(d_{*},\beta,r)\) since \(\mathbf{dom}(h_{q})=[0,1]^{K}\). Therefore, there exist \(I\subset\{1,2,\ldots,K\}\) and \(g\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d_{*}}\right)\) such that \(\#(I)=d_{*}\) and \(h_{q}(x)=g((x)_{I})\) for all \(x\in[0,1]^{K}\). Denote \(\lambda:=\beta+1-\lceil\beta\rceil\). We then use Taylor's formula to deduce that

\[\begin{split}&\left|h_{q}(x)-h_{q}(x^{\prime})\right|=\left|g((x)_ {I})-g((x^{\prime})_{I})\right|\stackrel{{\exists\in[0,1]^{d_{*} }}}{{=}}\left|\nabla g(\xi)\cdot((x)_{I}-(x^{\prime})_{I})\right|\\ &\leq\left\|\nabla g(\xi)\right\|_{\infty}\cdot\left\|(x)_{I}-(x^ {\prime})_{I}\right\|_{1}\leq\left\|\nabla g\right\|_{[0,1]^{d}}\cdot d_{*} \cdot\left\|(x)_{I}-(x^{\prime})_{I}\right\|_{\infty}\\ &\leq\left\|g\right\|_{\mathcal{C}^{\beta-\lambda,\lambda}([0,1]^ {d})}\cdot d_{*}\cdot\left\|(x)_{I}-(x^{\prime})_{I}\right\|_{\infty}\leq r \cdot d_{*}\cdot\left\|(x)_{I}-(x^{\prime})_{I}\right\|_{\infty}\\ &\leq r\cdot d_{*}^{1\wedge\beta}\cdot\left\|x-x^{\prime}\right\|_ {\infty}^{1\wedge\beta},\;\forall\;x,x^{\prime}\in[0,1]^{K},\end{split}\]

which yields (A.62).

**Case II:**\(h_{q}\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{*},\beta,r)\) and \(\beta\leq 1\). In this case, we still have that \(h_{q}\in\mathcal{G}_{K}^{\mathbf{H}}(d_{*},\beta,r)\). Therefore, there exist \(I\subset\{1,2,\ldots,K\}\) and \(g\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d_{*}}\right)\) such that \(\#(I)=d_{*}\) and \(h_{q}(x)=g((x)_{I})\) for all \(x\in[0,1]^{K}\). Consequently,

\[\left|h_{q}(x)-h_{q}(x^{\prime})\right|=\left|g((x)_{I})-g((x^{\prime})_{I}) \right|\leq\left\|(x)_{I}-(x^{\prime})_{I}\right\|_{2}^{\beta}\cdot\sup_{[0,1]^{d_ {*}}\ni z\neq z^{\prime}\in[0,1]^{d_{*}}}\frac{\left|g(z)-g(z^{\prime})\right|}{ \left\|z-z^{\prime}\right\|_{2}^{\beta}}\]
\[\leq r\cdot d_{*}^{1\wedge\beta}\cdot\left\|r\cdot d_{*}^{1\wedge \beta}\right|^{\sum_{k=0}^{q-2}(1\wedge\beta)^{k+1}}\cdot\sum_{k=0}^{q-1}\left\| \tilde{h}_{k}-h_{k}\right\|_{\mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1-k}}+ \left\|h_{q}-\tilde{h}_{q}\right\|_{\mathbf{dom}(h_{q})}\] \[=r\cdot d_{*}^{1\wedge\beta}\cdot\left|r\cdot d_{*}^{1\wedge \beta}\right|^{\sum_{k=0}^{q-2}(1\wedge\beta)^{k+1}}\cdot\sum_{k=0}^{q-1} \left\|\tilde{h}_{k}-h_{k}\right\|_{\mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1- k}}+\left\|h_{q}-\tilde{h}_{q}\right\|_{\mathbf{dom}(h_{q})}\] \[\leq r\cdot d_{*}^{1\wedge\beta}\cdot\left|r\cdot d_{*}^{1\wedge \beta}\right|^{\sum_{k=0}^{q-2}(1\wedge\beta)^{k+1}}\cdot\sum_{k=0}^{q-1} \left\|\tilde{h}_{k}-h_{k}\right\|_{\mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1- k}}+\left\|h_{q}-\tilde{h}_{q}\right\|_{\mathbf{dom}(h_{q})}\] \[\leq r\cdot d_{*}^{1\wedge\beta}\cdot\left|r\cdot d_{*}^{1\wedge \beta}\right|^{\sum_{k=0}^{q-2}(1\wedge\beta)^{k+1}}\cdot\sum_{k=0}^{q-1} \left\|\tilde{h}_{k}-h_{k}\right\|_{\mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-1- k}}+\left\|h_{q}-\tilde{h}_{q}\right\|_{\mathbf{dom}(h_{q})}\] \[=\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-1}(1 \wedge\beta)^{k}}\cdot\sum_{k=0}^{q}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-k}},\ \forall\ x\in[0,1]^{d}.\]

Therefore,

\[\left\|h_{q}\circ h_{q-1}\circ\cdots\circ h_{1}\circ h_{0}-\tilde {h}_{q}\circ\tilde{h}_{q-1}\circ\cdots\circ\tilde{h}_{1}\circ\tilde{h}_{0} \right\|_{[0,1]^{d}}\] \[=\sup_{x\in[0,1]^{d}}\left|h_{q}\circ h_{q-1}\circ\cdots\circ h_{ 1}\circ h_{0}(x)-\tilde{h}_{q}\circ\tilde{h}_{q-1}\circ\cdots\circ\tilde{h}_{ 1}\circ\tilde{h}_{0}(x)\right|\] \[\leq\left|r\cdot d_{*}^{1\wedge\beta}\right|^{\sum_{k=0}^{q-1}(1 \wedge\beta)^{k}}\cdot\sum_{k=0}^{q}\left\|\tilde{h}_{k}-h_{k}\right\|_{ \mathbf{dom}(h_{k})}^{(1\wedge\beta)^{q-k}},\]

meaning that the desired result holds for \(q\).

In conclusion, according to mathematical induction, we have that the desired result holds for all \(q\in\mathbb{N}\cup\{0\}\). This completes the proof.
**Lemma A.12**.: _Let \(k\) be an positive integer. Then there exists a neural network_

\[\tilde{f}\in\mathcal{F}_{k}^{\mathbf{FNN}}\left(1+2\cdot\left\lceil\frac{\log k}{ \log 2}\right\rceil,2k,26\cdot 2^{\left\lceil\frac{\log k}{\log 2}\right\rceil}-20-2 \cdot\left\lceil\frac{\log k}{\log 2}\right\rceil,1,1\right)\]

_such that_

\[\tilde{f}(x)=\left\|x\right\|_{\infty},\;\forall\;x\in\mathbb{R}^{k}.\]

Proof.: We argue by induction.

Firstly, consider the case \(k=1\). Define

\[\tilde{f}_{1}:\mathbb{R}\rightarrow\mathbb{R},x\mapsto\sigma(x)+\sigma(-x).\]

Obviously,

\[\tilde{f}_{1} \in\mathcal{F}_{1}^{\mathbf{FNN}}(1,2,6,1,1)\] \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(1+2\cdot\left\lceil \frac{\log 1}{\log 2}\right\rceil,2\cdot 1,26\cdot 2^{\left\lceil\frac{\log 1 }{\log 2}\right\rceil}-20-2\cdot\left\lceil\frac{\log 1}{\log 2}\right\rceil,1,1\right)\]

and \(\tilde{f}(x)=\sigma(x)+\sigma(-x)=\left|x\right|=\left\|x\right\|_{\infty}\) for all \(x\in\mathbb{R}=\mathbb{R}^{1}\). This proves the \(k=1\) case.

Now assume that the desired result holds for \(k=1,2,3,\ldots,m-1\) (\(m\geq 2\)), and consider the case \(k=m\). Define

\[\tilde{g}_{1}:\mathbb{R}^{m} \rightarrow\mathbb{R}^{\left\lfloor\frac{m}{2}\right\rfloor},\] \[x \mapsto\left((x)_{1},(x)_{2},\cdots,(x)_{\left\lfloor\frac{m}{2} \right\rfloor-^{1}},(x)_{\left\lfloor\frac{m}{2}\right\rfloor}\right),\]

\[\tilde{g}_{2}:\mathbb{R}^{m} \rightarrow\mathbb{R}^{\left\lceil\frac{m}{2}\right\rceil},\] \[x \mapsto\left((x)_{\left\lfloor\frac{m}{2}\right\rfloor+1},(x)_{ \left\lfloor\frac{m}{2}\right\rfloor+2},\cdots,(x)_{m-1},(x)_{m}\right),\]

and

\[\tilde{f}_{m}:\mathbb{R}^{m} \rightarrow\mathbb{R},\] (A.63) \[x \mapsto\sigma\left(\frac{1}{2}\cdot\sigma\Big{(}\tilde{f}_{\left \lfloor\frac{m}{2}\right\rfloor}(\tilde{g}_{1}(x))\Big{)}-\frac{1}{2}\cdot \sigma\Big{(}\tilde{f}_{\left\lfloor\frac{m}{2}\right\rfloor}(\tilde{g}_{2}(x) )\Big{)}\right)\] \[\qquad+\frac{1}{2}\cdot\sigma\Big{(}\tilde{f}_{\left\lfloor\frac{ m}{2}\right\rfloor}(\tilde{g}_{1}(x))\Big{)}+\frac{1}{2}\cdot\sigma\Big{(} \tilde{f}_{\left\lfloor\frac{m}{2}\right\rfloor}(\tilde{g}_{2}(x))\Big{)}.\]

It follows from the induction hypothesis that

\[\tilde{f}_{\left\lceil\frac{m}{2}\right\rceil}\circ\tilde{g}_{2} \in\mathcal{F}_{m}^{\mathbf{FNN}}\left(1+2\left\lceil\frac{\log \left\lceil\frac{m}{2}\right\rceil}{\log 2}\right\rceil,2\left\lceil\frac{m}{2} \right\rceil,26\cdot 2^{\left\lceil\frac{\log\left\lceil\frac{m}{2}\right\rceil}{\log 2} \right\rceil}-20-2\left\lceil\frac{\log\left\lceil\frac{m}{2}\right\rceil}{ \log 2}\right\rceil,1,1\right)\] \[=\mathcal{F}_{m}^{\mathbf{FNN}}\left(-1+2\left\lceil\frac{\log m}{ \log 2}\right\rceil,2\left\lceil\frac{m}{2}\right\rceil,13\cdot 2^{\left\lceil \frac{\log m}{\log 2}\right\rceil}-18-2\left\lceil\frac{\log m}{\log 2}\right\rceil,1,1\right)\]

and

\[\tilde{f}_{\left\lfloor\frac{m}{2}\right\rfloor}\circ\tilde{g}_{1}\in\mathcal{ F}_{m}^{\mathbf{FNN}}\left(1+2\left\lceil\frac{\log\left\lfloor\frac{m}{2} \right\rfloor}{\log 2}\right\rceil,2\left\lfloor\frac{m}{2}\right\rfloor,26\cdot 2^{ \left\lceil\frac{\log\left\lfloor\frac{m}{2}\right\rfloor}{\log 2}\right\rceil}-20-2\left\lceil \frac{\log\left\lfloor\frac{m}{2}\right\rfloor}{\log 2}\right\rceil,1,1\right)\]
\[\subset\mathcal{F}_{m}^{\mathbf{FNN}}\left(-1+2\left\lceil\frac{\log m}{\log 2} \right\rceil,2\left\lfloor\frac{m}{2}\right\rfloor,13\cdot 2^{\left\lceil\frac{\log m }{\log 2}\right\rceil}-18-2\left\lceil\frac{\log m}{\log 2}\right\rceil,1,1\right),\]

which, together with (A.63), yield

\[\tilde{f}_{m}\in\mathcal{F}_{m}^{\mathbf{FNN}}\left(2-1+2\left\lceil \frac{\log m}{\log 2}\right\rceil,2\left\lceil\frac{m}{2}\right\rceil+2 \left\lfloor\frac{m}{2}\right\rfloor,\right.\] (A.64) \[\left.\qquad\qquad\qquad 2\cdot\left|13\cdot 2^{\left\lceil\frac{\log m }{\log 2}\right\rceil}-18-2\left\lceil\frac{\log m}{\log 2}\right\rceil \right|+2\left\lceil\frac{\log m}{\log 2}\right\rceil+16,1,\infty\right)\] \[=\mathcal{F}_{m}^{\mathbf{FNN}}\left(1+\left\lceil\frac{\log m}{ \log 2}\right\rceil,2m,26\cdot 2^{\left\lceil\frac{\log m}{\log 2}\right\rceil}-20-2 \left\lceil\frac{\log m}{\log 2}\right\rceil,1,\infty\right)\]

(cf. Figure A.6). Besides, it is easy to verify that

Figure A.6: The network \(\tilde{f}_{m}\).
\[\begin{split}&\tilde{f}_{m}(x)=\max\left\{\sigma\Big{(}\tilde{f}_{\big{\lfloor} \frac{n}{2}\big{\rfloor}}(\tilde{g}_{1}(x))\Big{)},\sigma\Big{(}\tilde{f}_{ \big{\lceil}\frac{n}{2}\big{\rceil}}(\tilde{g}_{2}(x))\Big{)}\right\}\\ &=\max\left\{\sigma\Bigg{(}\left\|\Big{(}(x)_{1},\ldots,(x)_{ \big{\lfloor}\frac{n}{2}\big{\rfloor}}\Big{)}\right\|_{\infty}\right),\sigma \Bigg{(}\left\|\Big{(}(x)_{\big{\lfloor}\frac{n}{2}\big{\rfloor}+1},\ldots,(x) _{m}\Big{)}\right\|_{\infty}\Bigg{)}\right\}\\ &=\max\left\{\left\|\Big{(}(x)_{1},\ldots,(x)_{\big{\lfloor} \frac{n}{2}\big{\rfloor}}\Big{)}\right\|_{\infty},\left\|\Big{(}(x)_{\big{ \lfloor}\frac{n}{2}\big{\rfloor}+1},\ldots,(x)_{m}\Big{)}\right\|_{\infty} \right\}\\ &=\max\left\{\max_{1\leq i\leq\lfloor\frac{n}{2}\rfloor}|(x)_{i }|\,,\max_{\lfloor\frac{n}{2}\rfloor+1\leq i\leq m}|(x)_{i}|\right\}=\max_{1 \leq i\leq m}|(x)_{i}|=\|x\|_{\infty}\,,\;\forall\;x\in\mathbb{R}^{m}.\end{split}\] (A.65)

Combining (A.64) and (A.65), we deduce that the desired result holds for \(k=m\). Therefore, according to mathematical induction, we have that the desired result hold for all positive integer \(k\). This completes the proof. 

**Lemma A.13**.: _Let \((\varepsilon,d,d_{\star},d_{\star},\beta,r)\in(0,1/2]\times\mathbb{N}\times \mathbb{N}\times(0,\infty)\times(0,\infty)\) and \(f\) be a function from \([0,1]^{d}\) to \(\mathbb{R}\). Suppose \(f\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{\star},\beta,r\lor 1)\cup\mathcal{G}_{ \infty}^{\mathbf{M}}(d_{\star})\). Then there exist constants \(E_{1},E_{2},E_{3}\in(0,\infty)\) only depending on \((d_{\star},\beta,r)\) and a neural network_

\[\tilde{f}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(3\log d_{\star}+E_{1}\log \frac{1}{\varepsilon},2d_{\star}+E_{2}\varepsilon^{-\frac{d_{\star}}{\beta}},52d_{\star}+E_{3}\varepsilon^{-\frac{d_{\star}}{\beta}}\log\frac{1}{ \varepsilon},1,\infty\right)\]

_such that_

\[\sup_{x\in[0,1]^{d}}\Big{|}\tilde{f}(x)-f(x)\Big{|}<2\varepsilon.\]

Proof.: According to Corollary A.2, there exist constants \(E_{1},E_{2},E_{3}\in(6,\infty)\) only depending on \((d_{\star},\beta,r)\), such that

\[\begin{split}&\inf\left\{\sup_{x\in[0,1]^{d_{\star}}}|g(x)- \tilde{g}(x)|\,\left|\tilde{g}\in\mathcal{F}_{d_{\star}}^{\mathbf{FNN}}\left(E _{1}\log\frac{1}{t},E_{2}t^{-\frac{d_{\star}}{\beta}},E_{3}t^{-\frac{d_{\star }}{\beta}}\log\frac{1}{t},1,\infty\right)\right.\right\}\\ &\leq t,\;\forall\;g\in\mathcal{G}_{r\lor 1}^{\beta} \left([0,1]^{d_{\star}}\right),\;\forall\;t\in(0,1/2].\end{split}\] (A.66)

We next consider two cases.

**Case I:**\(f\in\mathcal{G}_{\infty}^{\mathbf{M}}(d_{\star})\). In this case, we must have \(f\in\mathcal{G}_{d}^{\mathbf{M}}(d_{\star})\), since \(\mathbf{dom}(f)=[0,1]^{d}\). Therefore, there exists \(I\subset\{1,2,\ldots,d\}\), such that \(1\leq\#(I)\leq d_{\star}\) and

\[f(x)=\max\left\{(x)_{i}\big{|}i\in I\right\},\;\forall\;x\in[0,1]^{d}.\]

According to Lemma A.12, there exists

\[\begin{split}\tilde{g}&\in\mathcal{F}_{\#(I)}^{ \mathbf{FNN}}\left(1+2\cdot\left\lceil\frac{\log\#(I)}{\log 2}\right\rceil,2\cdot\#(I),26\cdot 2^{\left\lceil\frac{ \log\#(I)}{\log 2}\right\rceil}-20-2\cdot\left\lceil\frac{\log\#(I)}{\log 2} \right\rceil,1,1\right)\\ &\subset\mathcal{F}_{\#(I)}^{\mathbf{FNN}}\left(1+2\cdot\left\lceil \frac{\log d_{\star}}{\log 2}\right\rceil,2d_{\star},26\cdot 2^{\left\lceil\frac{ \log d_{\star}}{\log 2}\right\rceil},1,1\right)\\ &\subset\mathcal{F}_{\#(I)}^{\mathbf{FNN}}\left(3+3\log d_{\star},2 d_{\star},52d_{\star},1,1\right)\end{split}\] (A.67)

such that

\[\tilde{g}(x)=\left\|x\right\|_{\infty},\;\forall\;x\in\mathbb{R}^{\#(I)}.\]

Define \(\tilde{f}:\mathbb{R}^{d}\to\mathbb{R},\;x\mapsto\tilde{g}((x)_{I})\). Then it follows from (A.67) that

\[\begin{split}\tilde{f}&\in\mathcal{F}_{d}^{\mathbf{FNN }}\left(3+3\log d_{\star},2d_{\star},52d_{\star},1,1\right)\\ &\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left(3\log d_{\star}+E_{1} \log\frac{1}{\varepsilon},2d_{\star}+E_{2}\varepsilon^{-\frac{d_{\star}}{ \beta}},52d_{\star}+E_{3}\varepsilon^{-\frac{d_{\star}}{\beta}}\log\frac{1}{ \varepsilon},1,\infty\right)\end{split}\]
and

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|=\sup_{x\in[0,1]^{d} }\left|\max\big{\{}(x)_{i}\big{|}i\in I\big{\}}-\tilde{g}((x)_{I})\right|\] \[=\sup_{x\in[0,1]^{d}}\left|\max\big{\{}|(x)_{i}\big{|}\left|i\in I \right\}-\left\|(x)_{I}\right\|_{\infty}\right|=0<2\varepsilon,\]

which yield the desired result.

**Case II:**\(f\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{*},\beta,r\lor 1)\). In this case, we must have \(f\in\mathcal{G}_{d}^{\mathbf{H}}(d_{*},\beta,r\lor 1)\), since \(\mathbf{dom}(f)=[0,1]^{d}\). By definition, there exist \(I\subset\{1,2,\ldots,d\}\) and \(g\in\mathcal{B}_{r\lor 1}^{\beta}\left([0,1]^{d_{*}}\right)\) such that \(\#(I)=d_{*}\) and \(f(x)=g\left((x)_{I}\right)\) for all \(x\in[0,1]^{d}\). Then it follows from (A.66) that there exists \(\tilde{g}\in\mathcal{F}_{d_{*}}^{\mathbf{FNN}}\left(E_{1}\log\frac{1}{ \varepsilon},E_{2}\varepsilon^{-\frac{d_{*}}{\beta}},E_{3}\varepsilon^{- \frac{d_{*}}{\beta}}\log\frac{1}{\varepsilon},1\right)\) such that

\[\sup_{x\in[0,1]^{d_{*}}}|g(x)-\tilde{g}(x)|<2\varepsilon.\]

Define \(\tilde{f}:\mathbb{R}^{d}\to\mathbb{R},\ x\mapsto\tilde{g}((x)_{I})\). Then we have that

\[\tilde{f} \in\mathcal{F}_{d}^{\mathbf{FNN}}\left(E_{1}\log\frac{1}{ \varepsilon},E_{2}\varepsilon^{-\frac{d_{*}}{\beta}},E_{3}\varepsilon^{- \frac{d_{*}}{\beta}}\log\frac{1}{\varepsilon},1,\infty\right)\] \[\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left(3\log d_{*}+E_{1}\log \frac{1}{\varepsilon},2d_{*}+E_{2}\varepsilon^{-\frac{d_{*}}{\beta}},52d_{*}+ E_{3}\varepsilon^{-\frac{d_{*}}{\beta}}\log\frac{1}{\varepsilon},1,\infty\right)\]

and

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|=\sup_{x\in[0,1]^{d}}|g((x) _{I})-\tilde{g}((x)_{I})|=\sup_{x\in[0,1]^{d_{*}}}|g(x)-\tilde{g}(x)|<2\varepsilon.\]

These yield the desired result again.

In conclusion, the desired result always holds. Thus we completes the proof of this lemma. 

**Lemma A.14**.: _Let \(\beta\in(0,\infty)\), \(r\in(0,\infty)\), \(q\in\mathbb{N}\cup\{0\}\), and \((d,d_{*},d_{*},K)\in\mathbb{N}^{4}\) with \(d_{*}\leq\min\big{\{}d,K+\mathbb{1}_{\{0\}}(q)\cdot(d-K)\big{\}}\). Suppose \(f\in\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\) and \(\varepsilon\in(0,1/2]\). Then there exist \(E_{7}\in(0,\infty)\) only depending on \((d_{*},\beta,r,q)\) and_

\[\tilde{f}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left((q+1)\cdot\left|3 \log d_{*}+E_{7}\log\frac{1}{\varepsilon}\right|,2Kd_{*}+KE_{7}\varepsilon^{- \frac{d_{*}}{\beta\cdot(1\wedge\beta)t}},\right.\] (A.68) \[\left.(Kq+1)\cdot\left|63d_{*}+E_{7}\varepsilon^{-\frac{d_{*}}{ \beta\cdot(1\wedge\beta)t}}\log\frac{1}{\varepsilon}\right|,1,\infty\right)\]

_such that_

\[\sup_{x\in[0,1]^{d}}\left|f(x)-\tilde{f}(x)\right|\leq\frac{\varepsilon}{8}.\] (A.69)

Proof.: By the definition of \(\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{*},d_{*},\beta,r)\), there exist functions \(h_{0},h_{1},\ldots,h_{q}\) such that

* \(\mathbf{dom}(h_{i})=[0,1]^{K}\) for \(0<i\leq q\) and \(\mathbf{dom}(h_{0})=[0,1]^{d}\);
* \(\mathbf{ran}(h_{i})\subset[0,1]^{K}\) for \(0\leq i<q\) and \(\mathbf{ran}(h_{q})\subset\mathbb{R}\);
* \(h_{q}\in\mathcal{G}_{\infty}^{\mathbf{H}}(d_{*},\beta,r\lor 1)\cup\mathcal{G}_{ \infty}^{\mathbf{M}}(d_{*})\);
* For \(0\leq i<q\) and \(1\leq j\leq K\), the \(j\)-th coordinate function of \(h_{i}\) given by \(\mathbf{dom}(h_{i})\ni x\mapsto(h_{i}(x))_{j}\in\mathbb{R}\) belongs to \(\mathcal{G}_{\infty}^{\mathbf{H}}(d_{*},\beta,r\lor 1)\cup\mathcal{G}_{\infty}^{ \mathbf{M}}(d_{*})\);
* \(f=h_{q}\circ h_{q-1}\circ\cdots\circ h_{2}\circ h_{1}\circ h_{0}\).

Define \(\Omega:=\big{\{}(i,j)\in\mathbb{Z}^{2}\big{|}0\leq i\leq q,1\leq j\leq K, \mathbb{1}_{\{q\}}(i)\leq\mathbb{1}_{\{1\}}(j)\big{\}}\). For each \((i,j)\in\Omega\), denote \(d_{i,j}:=K+\mathbb{1}_{\{0\}}(i)\cdot(d-K)\) and

\[h_{i,j}:\mathbf{dom}(h_{i})\to\mathbb{R},\ \ x\mapsto\big{(}h_{i}(x)\big{)}_{j}.\]
Then it is easy to verify that,

\[\mathbf{dom}(h_{i,j})=[0,1]^{d_{i,j}}\text{ and }h_{i,j}\in\mathcal{G}_{\infty}^{ \mathbf{H}}(d_{*},\beta,r\lor 1)\cup\mathcal{G}_{\infty}^{\mathbf{M}}(d_{*}),\; \forall\;(i,j)\in\Omega,\] (A.70)

and

\[\mathbf{ran}\left(h_{i,j}\right)\subset[0,1],\;\forall\;(i,j)\in\Omega\setminus \left\{(q,1)\right\}.\] (A.71)

Fix \(\varepsilon\in(0,1/2]\). Take

\[\delta:=\frac{1}{2}\cdot\left|\frac{\varepsilon}{8\cdot\left|(1\lor r)\cdot d _{*}\right|^{q}\cdot(q+1)}\right|^{\frac{1}{(1\wedge j)q}}\leq\frac{ \varepsilon/2}{8\cdot\left|(1\lor r)\cdot d_{*}\right|^{q}\cdot(q+1)}\leq \frac{\varepsilon}{8}\leq\frac{1}{16}.\]

According to (A.70) and Lemma A.13, there exists a constant \(E_{1}\in(6,\infty)\) only depending on \((d_{*},\beta,r)\) and a set of functions \(\left\{\tilde{g}_{i,j}:\mathbb{R}^{d_{i,j}}\rightarrow\mathbb{R}\right\}_{(i,j)\in\Omega}\), such that

\[\begin{split}\tilde{g}_{i,j}\in\mathcal{F}_{d_{i,j}}^{\mathbf{FNN }}\left(3\log d_{\star}+E_{1}\log\frac{1}{\delta},2d_{\star}+E_{1}\delta^{- \frac{d_{*}}{\beta}},\right.\\ \left.52d_{\star}+E_{1}\delta^{-\frac{d_{*}}{\beta}}\log\frac{1}{ \delta},1,\infty\right),\forall(i,j)\in\Omega\end{split}\] (A.72)

and

\[\sup\left\{\left|\tilde{g}_{i,j}(x)-h_{i,j}(x)\right|\left|x\in[0,1]^{d_{i,j}} \right.\right\}\leq 2\delta,\;\forall\;(i,j)\in\Omega.\] (A.73)

Define

\[E_{4} :=8\cdot|(1\lor r)\cdot d_{*}|^{q}\cdot(q+1),\] \[E_{5} :=2^{\frac{d_{*}}{\beta}}\cdot E_{4}^{\frac{d_{*}}{9(1\wedge j)^{ 7}}},\] \[E_{6} :=\frac{1}{(1\wedge\beta)^{q}}+\frac{2\log E_{4}}{(1\wedge\beta)^ {q}}+2\log 2,\] \[E_{7} :=E_{1}E_{6}+E_{1}E_{5}+2E_{1}E_{5}E_{6}+6,\]

Obviously, \(E_{4},E_{5},E_{6},E_{7}\) are constants only depending on \((d_{*},\beta,r,q)\). Next, define

\[\tilde{h}_{i,j}:\mathbb{R}^{d_{i,j}}\rightarrow\mathbb{R},\;x\mapsto\sigma \big{(}\sigma\left(\tilde{g}_{i,j}(x)\right)\big{)}-\sigma\big{(}\sigma\left( \tilde{g}_{i,j}(x)\right)-1\big{)}\]

for each \((i,j)\in\Omega\setminus\left\{(q,1)\right\}\), and define \(\tilde{h}_{q,1}:=\tilde{g}_{q,1}\). It follows from the fact

\[\sigma\big{(}\sigma\left(z\right)\big{)}-\sigma\big{(}\sigma\left(z\right)-1 \big{)}\in[0,1],\;\forall\;z\in\mathbb{R}\]

and (A.72) that

\[\mathbf{ran}(\tilde{h}_{i,j})\subset[0,1],\;\forall\;(i,j)\in\Omega\setminus (q,1)\] (A.74)

and

\[\begin{split}\tilde{h}_{i,j}\in\mathcal{F}_{d_{i,j}}^{\mathbf{FNN }}\left(2+3\log d_{\star}+E_{1}\log\frac{1}{\delta},2d_{\star}+E_{1}\delta^{- \frac{d_{*}}{\beta}},\right.\\ \left.58d_{\star}+E_{1}\delta^{-\frac{d_{*}}{\beta}}\log\frac{1}{ \delta},1,\infty\right),\forall(i,j)\in\Omega.\end{split}\] (A.75)

Besides, it follows from the fact that

\[\left|\sigma\big{(}\sigma\left(z\right)\big{)}-\sigma\big{(}\sigma\left(z \right)-1\big{)}-w\right|\leq|w-z|\,,\;\forall\;z\in\mathbb{R},\;\forall\;w\in[0,1]\]

and (A.73) that

\[\begin{split}&\sup\left\{\left|\tilde{h}_{i,j}(x)-h_{i,j}(x) \right|\left|x\in[0,1]^{d_{i,j}}\right.\right\}\\ &\leq\sup\left\{\left|\tilde{g}_{i,j}(x)-h_{i,j}(x)\right|\left|x \in[0,1]^{d_{i,j}}\right.\right\}\leq 2\delta.\end{split}\] (A.76)
We then define

\[\tilde{h}_{i}:\mathbb{R}^{d_{i,1}}\to\mathbb{R}^{K},x\mapsto\left(\tilde{h}_{i,1}( x),\tilde{h}_{i,2}(x),\ldots,\tilde{h}_{i,K}(x)\right)^{\top}\]

for each \(i\in\{0,1,\ldots,q-1\}\), and \(\tilde{h}_{q}:=\tilde{h}_{q,1}\). From (A.74) we obtain

\[\mathbf{ran}(\tilde{h}_{i})\subset[0,1]^{K}\subset\mathbf{dom}(\tilde{h}_{i+1 }),\ \forall\ i\in\{0,1,\ldots,q-1\}\,.\] (A.77)

Thus we can well define the function \(\tilde{f}:=\tilde{h}_{q}\circ\tilde{h}_{q-1}\circ\cdots\circ\tilde{h}_{1} \circ\tilde{h}_{0}\), which is from \(\mathbb{R}^{d}\) to \(\mathbb{R}\). Since all the functions \(\tilde{h}_{i,j}\) (\((i,j)\in\Omega\)) are neural networks satisfying (A.75), we deduce that \(\tilde{f}\) is also a neural network, which is comprised of all those networks \(\tilde{h}_{i,j}\) through series and parallel connection. Obviously, the depth of \(\tilde{f}\) is less than or equal to

\[\sum_{i}\left(1+\max_{j}\Big{(}\text{the depth of }\tilde{h}_{i,j}\Big{)} \right),\]

the width of \(\tilde{f}\) is less than or equal to

\[\max_{i}\sum_{j}\Big{(}\text{the width of }\tilde{h}_{i,j}\Big{)},\]

the number of nonzero parameters of \(\tilde{f}\) is less than or equal to

\[\sum_{i,j}\left(\Big{(}\text{the number of nonzero parameters }\tilde{h}_{i,j} \Big{)}+\max_{k}\Big{(}\text{the depth of }\tilde{h}_{i,k}\Big{)}\right),\]

and the parameters of \(\tilde{f}\) is bounded by \(1\) in absolute value. Thus we have that

\[\tilde{f} \in\mathcal{F}_{d}^{\mathbf{FNN}}\left((q+1)\cdot\left|3+3\log d_{ \star}+E_{1}\log\frac{1}{\delta}\right|,2Kd_{\star}+KE_{1}\delta^{-\frac{d_{ \star}}{\beta}},\right.\] \[\qquad\qquad\qquad\left.(Kq+1)\cdot\left|63d_{\star}+2E_{1}\delta ^{-\frac{d_{\star}}{\beta}}\log\frac{1}{\delta}\right|,1,\infty\right)\] \[=\mathcal{F}_{d}^{\mathbf{FNN}}\left((q+1)\cdot\left|3+3\log d_{ \star}+E_{1}\cdot\Big{(}\log 2+\frac{\log\frac{E_{4}}{\varepsilon}}{(1\wedge \beta)^{q}}\Big{)}\right|,2Kd_{\star}+KE_{1}E_{5}\varepsilon^{-\frac{d_{ \star}}{\beta(1\wedge\beta)^{q}}},\right.\] \[\qquad\qquad\qquad\left.(Kq+1)\cdot\left|63d_{\star}+2E_{1}E_{5} \varepsilon^{-\frac{d_{\star}}{\beta(1\wedge\beta)^{q}}}\cdot\Big{(}\log 2+ \frac{\log\frac{E_{4}}{\varepsilon}}{(1\wedge\beta)^{q}}\Big{)}\right|,1,\infty\right)\] \[\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left((q+1)\cdot\left|3+3 \log d_{\star}+E_{1}E_{6}\log\frac{1}{\varepsilon}\right|,2Kd_{\star}+KE_{1}E_ {5}\varepsilon^{-\frac{d_{\star}}{\beta(1\wedge\beta)^{q}}},\right.\] \[\qquad\qquad\qquad\left.(Kq+1)\cdot\left|63d_{\star}+2E_{1}E_{5} \varepsilon^{-\frac{d_{\star}}{\beta(1\wedge\beta)^{q}}}E_{6}\log\frac{1}{ \varepsilon}\right|,1,\infty\right)\] \[\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left((q+1)\cdot\left|3\log d _{\star}+E_{7}\log\frac{1}{\varepsilon}\right|,2Kd_{\star}+KE_{7}\varepsilon^ {-\frac{d_{\star}}{\beta(1\wedge\beta)^{q}}},\right.\] \[\qquad\qquad\qquad\left.(Kq+1)\cdot\left|63d_{\star}+E_{7} \varepsilon^{-\frac{d_{\star}}{\beta(1\wedge\beta)^{q}}}\log\frac{1}{ \varepsilon}\right|,1,\infty\right),\]

leading to (A.68). Moreover, it follows from (A.76) and Lemma A.11 that

\[\sup_{x\in[0,1]^{d}}\left|\tilde{f}(x)-f(x)\right|=\sup_{x\in[0,1 ]^{d}}\left|\tilde{h}_{q}\circ\cdots\circ\tilde{h}_{0}(x)-h_{q}\circ\cdots \circ h_{0}(x)\right|\] \[\leq\left|(1\lor r)\cdot d_{\star}^{1\wedge\beta}\right|^{\sum_{i= 0}^{q-1}(1\wedge\beta)^{i}}\cdot\sum_{i=0}^{q}\left|\sup_{x\in[0,1]^{d_{i,1}}} \left\|\tilde{h}_{i}(x)-h_{i}(x)\right\|_{\infty}\right|^{(1\wedge\beta)^{q-i}}\]
\[\leq|(1\lor r)\cdot d_{*}|^{q}\cdot\sum_{i=0}^{q}|2\delta|^{(1\wedge\beta)^{q-i}} \leq|(1\lor r)\cdot d_{*}|^{q}\cdot\sum_{i=0}^{q}|2\delta|^{(1\wedge\beta)^{q}}= \frac{\varepsilon}{8},\]

which yields (A.69).

In conclusion, the constant \(E_{7}\) and the neural network \(\tilde{f}\) have all the desired properties. The proof of this lemma is then completed.

The next lemma aims to estimate the approximation error.

**Lemma A.15**.: _Let \(\phi(t)=\log\left(1+\mathrm{e}^{-t}\right)\) be the logistic loss, \(q\in\mathbb{N}\cup\{0\}\), \((\beta,r)\in(0,\infty)^{2}\), \((d,d_{*},d_{*},K)\in\mathbb{N}^{4}\) with \(d_{*}\leq\min\left\{d,K+1_{\{0\}}(q)\cdot(d-K)\right\}\), and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\). Suppose that there exists an \(\hat{\eta}\in\mathcal{G}_{d}^{\mathbf{CHON}}(q,K,d_{*},d_{*},\beta,r)\) such that \(P_{X}(\left\{x\in[0,1]^{d}\right|\hat{\eta}(x)=P(\left\{1\right\}|x))\right\}=1\). Then there exist constants \(D_{1},D_{2},D_{3}\) only depending on \((d_{*},d_{*},\beta,r,q)\) such that for any \(\delta\in(0,1/3)\),_

\[\inf\left\{\mathcal{E}_{P}^{\phi}\left(f\right)\left|f\in\mathcal{ F}_{d}^{\mathbf{FNN}}\left(D_{1}\log\frac{1}{\delta},KD_{2}\delta^{\frac{-d_{*} /\delta}{(1\wedge\beta)^{q}}},KD_{3}\delta^{\frac{-d_{*}/\delta}{(1\wedge\beta )^{q}}}\cdot\log\frac{1}{\delta},1,\log\frac{1-\delta}{\delta}\right)\right\}\] (A.78) \[\leq 8\delta.\]

Proof.: Denote by \(\eta\) the conditional probability function \([0,1]^{d}\ni x\mapsto P(\left\{1\right\}|x)\in[0,1]\). Fix \(\delta\in(0,1/3)\). Then it follows from Lemma A.14 that there exists

\[\tilde{\eta}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(C_{d_{*},d_{* },\beta,r,q}\log\frac{1}{\delta},KC_{d_{*},d_{*},\beta,r,q}\delta^{-\frac{d_{* }}{\beta\cdot(1\wedge\beta)^{q}}},\right.\] (A.79) \[\left.KC_{d_{*},d_{*},\beta,r,q}\delta^{-\frac{d_{*}}{\beta\cdot (1\wedge\beta)^{q}}}\log\frac{1}{\delta},1,\infty\right)\]

such that

\[\sup_{x\in[0,1]^{d}}|\tilde{\eta}(x)-\hat{\eta}(x)|\leq\delta/8.\] (A.80)

Also, by Theorem 2.4 with \(a=\varepsilon=\delta\), \(b=1-\delta\) and \(\alpha=\frac{2\beta}{d_{*}}\), there exists

\[\tilde{l}\in\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{d_{*},\beta} \log\frac{1}{\delta}+139\log\frac{1}{\delta},C_{d_{*},\beta}\cdot\left(\frac{ 1}{\delta}\right)^{\frac{1}{2\beta/d_{*}}}\log\frac{1}{\delta},\right.\] (A.81) \[\left.C_{d_{*},\beta}\cdot\left(\frac{1}{\delta}\right)^{\frac{ 1}{2\beta/d_{*}}}\cdot\left(\log\frac{1}{\delta}\right)\cdot\left(\log\frac{ 1}{\delta}\right)+65440\left(\log\delta\right)^{2},1,\infty\right)\] \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{d_{*},\beta}\log \frac{1}{\delta},C_{d_{*},\beta}\delta^{-\frac{d_{*}}{\beta\cdot(1\wedge\beta )^{q}}},C_{d_{*},\beta}\delta^{-\frac{d_{*}}{\beta\cdot(1\wedge\beta)^{q}}} \log\frac{1}{\delta},1,\infty\right)\] \[\subset\mathcal{F}_{1}^{\mathbf{FNN}}\left(C_{d_{*},\beta}\log \frac{1}{\delta},C_{d_{*},\beta}\delta^{-\frac{d_{*}}{\beta\cdot(1\wedge\beta) ^{q}}},C_{d_{*},\beta}\delta^{-\frac{d_{*}}{\beta\cdot(1\wedge\beta)^{q}}} \log\frac{1}{\delta},1,\infty\right)\]

such that

\[\sup_{t\in[\delta,1-\delta]}\left|\tilde{l}(t)-\log t\right|\leq\delta\] (A.82)

and

\[\log\delta\leq\tilde{l}(t)\leq\log\left(1-\delta\right)<0,\;\forall\;t\in \mathbb{R}.\] (A.83)

Recall that the clipping function \(\Pi_{\delta}\) is given by

\[\Pi_{\delta}:\mathbb{R}\rightarrow[\delta,1-\delta],\quad t\mapsto\begin{cases}1- \delta,&\text{ if }t>1-\delta,\\ \delta,&\text{ if }t<\delta,\\ t,&\text{ otherwise.}\end{cases}\]
Define \(\tilde{f}:\mathbb{R}\rightarrow\mathbb{R},x\mapsto\tilde{l}\left(\Pi_{\delta}\left( \tilde{\eta}(x)\right)\right)-\tilde{l}\left(1-\Pi_{\delta}\left(\tilde{\eta}( x)\right)\right)\). Consequently, we know from (A.79), (A.81) and (A.83) that (cf. Figure A.7)

\[\tilde{f}\in\mathcal{F}_{d}^{\mathbf{FNN}}\left(C_{d_{*},d_{*}, \beta,r,q}\log\frac{1}{\delta},KC_{d_{*},d_{*},\beta,r,q}\delta^{-\frac{d_{*}} {\beta-(1\wedge\beta)^{\mathsf{T}}}},\right.\] \[KC_{d_{*},d_{*},\beta,r,q}\delta^{-\frac{d_{*}}{\beta-(1\wedge \beta)^{\mathsf{T}}}}\log\frac{1}{\delta},1,\log\frac{1-\delta}{\delta}\right).\]

Let \(\Omega_{1},\Omega_{2},\Omega_{3}\) be defined in (A.39). Then it follows from (A.80) that

\[|\Pi_{\delta}(\tilde{\eta}(x))-\eta(x)|=|\Pi_{\delta}(\tilde{ \eta}(x))-\Pi_{\delta}\left(\tilde{\eta}(x)\right)|\leq|\tilde{\eta}(x)- \tilde{\eta}(x)|\] \[\leq\frac{\delta}{8}\leq\frac{\min\left\{\eta(x),1-\eta(x) \right\}}{8},\;\forall\;x\in\Omega_{1},\]

which means that

\[\min\left\{\frac{\Pi_{\delta}\left(\tilde{\eta}(x)\right)}{\eta(x)},\frac{1- \Pi_{\delta}\left(\tilde{\eta}(x)\right)}{1-\eta(x)}\right\}\geq 7/8,\;\forall \;x\in\Omega_{1}.\] (A.84)

Combining (A.82) and (A.84), we obtain that

\[\left|\tilde{f}(x)-\log\frac{\eta(x)}{1-\eta(x)}\right|\] \[\leq\left|\tilde{l}\left(\Pi_{\delta}\left(\tilde{\eta}(x)\right) \right)-\log\left(\eta(x)\right)\right|+\left|\tilde{l}\left(1-\Pi_{\delta} \left(\tilde{\eta}(x)\right)\right)-\log\left(1-\eta(x)\right)\right|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x),1-\eta(x)\right\},\infty)]}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta}\left( \tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x),1-\eta(x)\right\},\infty)]}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x),1-\eta(x)\right\},\infty)]}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x) ),1-\eta(x)\right\},\infty)}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta}(\tilde{\eta}(x),1-\eta(x)\right\},\infty)]}\left|\log^{\prime}(t)\right|\cdot|\Pi_{\delta} \left(\tilde{\eta}(x)\right)-\eta(x)|\] \[\leq\delta+\sup_{t\in[\min\left\{1-\Pi_{\delta
Besides, note that

\[x\in\Omega_{2}\Rightarrow\tilde{\eta}(x)\in[-\xi_{1},\delta+\xi_{1}] \Rightarrow\Pi_{\delta}\left(\tilde{\eta}(x)\right)\in[\delta,\delta+\xi_{1}]\] \[\Rightarrow\tilde{l}\left(\Pi_{\delta}\left(\tilde{\eta}(x) \right)\right)\in[\log\delta,\delta+\log\left(\delta+\xi_{1}\right)]\] \[\quad\text{as well as }\tilde{l}\left(1-\Pi_{\delta}\left(\tilde{\eta}(x) \right)\right)\in[-\delta+\log(1-\delta-\xi_{1}),\log(1-\delta)]\] \[\Rightarrow\tilde{f}(x)\leq 2\delta+\log\frac{\xi_{1}+\delta}{1- \xi_{1}-\delta}\leq\log 2+\log\frac{2\delta}{1-2\delta}=\log\frac{4\delta}{1-2 \delta}.\]

Therefore, by (A.83) and the definition of \(\tilde{f}\), we have

\[\log\frac{\delta}{1-\delta}\leq\tilde{f}(x)\leq\log\frac{4\delta}{1-2\delta} =-\log\frac{1-2\delta}{4\delta},\ \forall\ x\in\Omega_{2}.\] (A.86)

Similarly, we can show that

\[\log\frac{1-2\delta}{4\delta}\leq\tilde{f}(x)\leq\log\frac{1-\delta}{\delta}, \ \forall\ x\in\Omega_{3}.\] (A.87)

Then it follows from (A.85), (A.86), (A.87) and Lemma A.8 that

\[\left.\inf\left\{\mathcal{E}_{P}^{\phi}\left(f\right)\left|f\in \mathcal{F}_{d}^{\mathbf{FNN}}\left(C_{d_{*},d_{*},\beta,r_{*}q}\log\frac{1}{ \delta},KC_{d_{*},d_{*},\beta,r_{*}q}\delta^{-\frac{d_{*}}{\beta\cdot(1+\beta) \eta}},\right.\right.\right\}\] \[\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \left.\left.
\[\leq\int_{\Omega_{1}}\sup\left\{\left.\frac{\left|\tilde{f}(x)-\log \frac{\eta(x)}{1-\eta(x)}\right|^{2}}{2(2+\mathrm{e}^{t}+\mathrm{e}^{-t})}\right| ^{2}\right|t\in\left[-1+\log\frac{\eta(x)}{1-\eta(x)},1+\log\frac{\eta(x)}{1- \eta(x)}\right]\right\}\mathrm{d}P_{X}(x)\] \[\qquad+P_{X}(\Omega_{2}\cup\Omega_{3})\cdot\log\frac{1+2\delta}{1 -2\delta}\] \[\leq\int_{\Omega_{1}}\left|\tilde{f}\left(x\right)-\log\frac{\eta (x)}{1-\eta(x)}\right|^{2}\cdot 2\cdot(1-\eta(x))\eta(x)\mathrm{d}P_{X}(x)+6\delta\] \[\leq\int_{\Omega_{1}}\left|2\delta+\frac{\delta}{7\eta(x)(1-\eta (x))}\right|^{2}\cdot 2\cdot(1-\eta(x))\eta(x)\mathrm{d}P_{X}(x)+6\delta\] \[\leq\int_{\Omega_{1}}\frac{\delta^{2}}{(1-\eta(x))\eta(x)} \mathrm{d}P_{X}(x)+6\delta\leq\frac{\delta^{2}}{\delta(1-\delta)}+6\delta<8\delta,\]

which proves this lemma. 

Now we are in the position to prove Theorem 2.2 and Theorem 2.3.

Proof of Theorem 2.2 and Theorem 2.3.: We first prove Theorem 2.3. According to Lemma A.15, there exist \((D_{1},D_{2},D_{3})\in(0,\infty)^{3}\) only depending on \((d_{\star},d_{\star},\beta,r,q)\) such that (A.78) holds for any \(\delta\in(0,1/3)\) and any \(P\in\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d,\beta,r}\). Take \(E_{1}=1+D_{1}\), then \(E_{1}>0\) only depends on \((d_{\star},d_{\star},\beta,r,q)\). We next show that for any constants \(\mathbf{a}:=(a_{2},a_{3})\in(0,\infty)^{2}\) and \(\mathbf{b}:=(b_{1},b_{2},b_{3},b_{4},b_{5})\in(0,\infty)^{5}\), there exist constants \(E_{2}\in(3,\infty)\) only depends on \((\mathbf{a},d_{\star},d_{\star},\beta,r,q,K)\) and \(E_{3}\in(0,\infty)\) only depending on \((\mathbf{a},\mathbf{b},\nu,d,d_{\star},d_{\star},\beta,r,q,K)\) such that when \(n\geq E_{2}\), the \(\phi\)-ERM \(\hat{f}_{n}^{\mathbf{FNN}}\) defined by (2.14) with

\[\begin{split}& E_{1}\cdot\log n\leq G\leq b_{1}\cdot\log n,\\ & a_{2}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star }}{d_{\star}+\beta\cdot(1\wedge\beta)^{\eta}}}\leq N\leq b_{2}\cdot\left(\frac{ (\log n)^{5}}{n}\right)^{\frac{-d_{\star}}{d_{\star}+\beta\cdot(1\wedge\beta)^ {\eta}}},\\ & a_{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star }}{d_{\star}+\beta\cdot(1\wedge\beta)^{\eta}}}\cdot\log n\leq S\leq b_{3} \cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star}}{d_{\star}+\beta \cdot(1\wedge\beta)^{\eta}}}\cdot\log n,\\ &\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{\star}+\beta\cdot(1 \wedge\beta)^{q}}\cdot\log n\leq F\leq b_{4}\log n,\text{ and }1\leq B\leq b_{5}\cdot n^{\nu}\end{split}\] (A.88)

must satisfy

\[\begin{split}&\sup_{P\in\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}^{ \mathbf{FNN}}\right)\right]\leq E_{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^ {\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{\star}+\beta\cdot(1\wedge\beta)^{ \eta}}}\\ &\text{ and }\sup_{P\in\mathcal{H}_{4,q,K,d_{\star},d_{\star}}^{d, \beta,r}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^{ \mathbf{FNN}}\right)\right]\leq E_{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^ {\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{\star}+2\beta\cdot(1\wedge\beta)^{ \eta}}},\end{split}\] (A.89)

which will lead to the results of Theorem 2.3.

Let \(\mathbf{a}:=(a_{2},a_{3})\in(0,\infty)^{2}\) and \(\mathbf{b}:=(b_{1},b_{2},b_{3},b_{4},b_{5})\in(0,\infty)^{5}\) be arbitrary and fixed. Take

\[D_{4}=1\vee\left(\frac{D_{2}K}{a_{2}}\right)^{\frac{\beta\cdot(1\wedge\beta)^{ q}}{d_{\star}}}\vee\left(\frac{D_{3}E_{1}K}{D_{1}a_{3}}\right)^{\frac{\beta\cdot(1 \wedge\beta)^{q}}{d_{\star}}},\]

then \(D_{4}>0\) only depends on \((\mathbf{a},d_{\star},d_{\star},\beta,r,q,K)\). Hence there exists \(E_{2}\in(3,\infty)\) only depending on \((\mathbf{a},d_{\star},d_{\star},\beta,r,q,K)\) such that

\[\begin{split} 0&<\frac{(\log t)^{5}}{t}<D_{4}\cdot\left(\frac{( \log t)^{5}}{t}\right)^{\frac{\beta\cdot(1\wedge\beta)^{q}}{d_{\star}+\beta \cdot(1\wedge\beta)^{\eta}}}<1/4\\ &<1<\log t,\;\forall\;t\in[E_{2},\infty).\end{split}\] (A.90)
From now on we assume that \(n\geq E_{2}\), and (A.88) holds. We have to show that there exists \(E_{3}\in(0,\infty)\) only depending on \((\mathbf{a},\mathbf{b},\nu,d,d_{\star},d_{\star},\beta,r,q,K)\) such that (A.89) holds.

Let \(P\) be an arbitrary probability in \(\mathcal{H}_{d,0,K,d_{\star},d_{\star},d_{\star}}^{d,\beta,r}\). Denote by \(\eta\) the conditional probability function \(x\mapsto P(\left\{1\right\}|x)\) of \(P\). Then there exists an \(\hat{\eta}\in\mathcal{G}_{d}^{\mathbf{CHOM}}(q,K,d_{\star},d_{\star},\beta,r)\) such that \(\hat{\eta}=\eta\), \(P_{X}\)-a.s.. Define

\[\zeta:=D_{4}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta \cdot(1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}.\] (A.91)

By (A.90), \(0<n^{\frac{-\beta\cdot(1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta) \eta}}\leq\zeta<\frac{1}{4}\) and there hold inequalities

\[\log 2<\log\frac{1-\zeta}{\zeta}\leq\log\frac{1}{\zeta}\leq\log \left(n^{\frac{\beta\cdot(1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta) \eta}}\right)\leq F,\] (A.92) \[D_{1}\log\frac{1}{\zeta}\leq D_{1}\log\left(n^{\frac{\beta\cdot (1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}\right)\] (A.93) \[\leq D_{1}\log n\leq\max\left\{1,D_{1}\log n\right\}\leq E_{1} \log n\leq G,\]

and

\[KD_{2}\zeta^{\frac{-d_{\star}/\beta}{(1\wedge\beta)\eta}}=KD_{2} \cdot D_{4}^{\frac{-d_{\star}/\beta}{(1\wedge\beta)\eta}}\cdot\left(\frac{( \log n)^{5}}{n}\right)^{\frac{-d_{\star}}{d_{\star}+\beta\cdot(1\wedge\beta) \eta}}\] (A.94) \[\leq KD_{2}\cdot\left|\left(\frac{D_{2}K}{a_{2}}\right)^{\frac{ \beta\cdot(1\wedge\beta)\eta}{d_{\star}}}\right|^{\frac{-d_{\star}/\beta}{(1 \wedge\beta)\eta}}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star}} {d_{\star}+\beta\cdot(1\wedge\beta)\eta}}\] \[=a_{2}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star} }{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}\leq N.\]

Consequently,

\[KD_{3}\zeta^{\frac{-d_{\star}/\beta}{(1\wedge\beta)\eta}}\cdot \log\frac{1}{\zeta}=KD_{3}\cdot D_{4}^{\frac{-d_{\star}/\beta}{(1\wedge\beta) \eta}}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star}}{d_{\star}+ \beta\cdot(1\wedge\beta)\eta}}\cdot\log\frac{1}{\zeta}\] (A.95) \[\leq KD_{3}\cdot\left|\left(\frac{D_{3}E_{1}K}{D_{1}a_{3}} \right)^{\frac{\beta\cdot(1\wedge\beta)\eta}{d_{\star}}}\right|^{\frac{-d_{ \star}/\beta}{(1\wedge\beta)\eta}}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{ \frac{-d_{\star}}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}\cdot\log\frac{1}{\zeta}\] \[=a_{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\star} +\beta\cdot(1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}\cdot \frac{D_{1}\cdot\log\frac{1}{\zeta}}{E_{1}}\leq a_{3}\cdot\left(\frac{(\log n )^{5}}{n}\right)^{\frac{-d_{\star}}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}} \cdot\log n\leq S.\]

Then it follows from (A.78), (A.91), (A.93), (A.92), (A.94), and (A.95) that

\[\inf\left\{\mathcal{E}_{P}^{\phi}\left(f\right)\left|f\in\mathcal{ F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right)\right.\right\}\] (A.96) \[\leq 8\zeta=8D_{4}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{ \beta\cdot(1\wedge\beta)\eta}{d_{\star}+\beta\cdot(1\wedge\beta)\eta}}.\]

Besides, from (A.92) we know \(\mathrm{e}^{F}>2\). Hence by taking \(\delta_{0}=\frac{1}{\mathrm{e}^{F}+1}\) in Lemma A.10, we obtain immediately that there exists

\[\psi:[0,1]^{d}\times\left\{-1,1\right\}\rightarrow\left[0,\log \left((10\mathrm{e}^{F}+10)\cdot\log\left(\mathrm{e}^{F}+1\right)\right)\right],\] (A.97)

such that

\[\int_{[0,1]^{d}\times\left\{-1,1\right\}}\psi\left(x,y\right) \mathrm{d}P(x,y)=\inf\left\{\mathcal{R}_{P}^{\phi}(f)\mid f:[0,1]^{d} \rightarrow\mathbb{R}\text{ is measurable}\right\},\] (A.98)
and for any measurable \(f:[0,1]^{d}\to[-F,F]\),

\[\begin{split}&\int_{[0,1]^{d}\times\{-1,1\}}(\phi\left(yf(x) \right)-\psi(x,y))^{2}\mathrm{d}P(x,y)\\ &\leq 125000\left|\log\left(1+\mathrm{e}^{F}\right)\right|^{2}\cdot \int_{[0,1]^{d}\times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi(x,y)\right) \mathrm{d}P(x,y)\\ &\leq 500000F^{2}\cdot\int_{[0,1]^{d}\times\{-1,1\}}(\phi\left(yf(x) \right)-\psi(x,y))\,\mathrm{d}P(x,y).\end{split}\] (A.99)

Moreover, it follows from Corollary A.1 with \(\gamma=\frac{1}{n}\) that

\[\begin{split}&\log W\leq(S+Gd+1)(2G+5)\log\left((\max\left\{N,d \right\}+1)(2nG+2n)B\right)\\ &\leq C_{\mathbf{b},d}\cdot(\log n)^{2}\cdot\left(\frac{(\log n)^{5} }{n}\right)^{\frac{-d_{\mathbf{s}}}{d_{\mathbf{s}}+\beta+(1\wedge\beta)^{d}}}\cdot \log\left((\max\left\{N,d\right\}+1)(2nG+2n)b_{5}n^{\nu}\right)\\ &\leq C_{\mathbf{b},d,\nu}\cdot(\log n)^{3}\cdot\left(\frac{(\log n )^{5}}{n}\right)^{\frac{-d_{\mathbf{s}}}{d_{\mathbf{s}}+\beta+(1\wedge\beta)^{d}}}=E_{ 4}\cdot(\log n)^{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\mathbf{s} }}{d_{\mathbf{s}}+\beta+(1\wedge\beta)^{d}}}\end{split}\] (A.100)

for some constant \(E_{4}\in(0,\infty)\) only depending on\((\mathbf{b},d,\nu)\), where

\[W=3\vee\mathcal{N}\left(\left\{\left.f\right|_{[0,1]^{d}}\right|f\in\mathcal{F }_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right)\right\},\frac{1}{n}\right).\]

Also, note that

\[\sup_{t\in[-F,F]}\phi(t)=\log\left(1+\mathrm{e}^{F}\right)\leq\log\left((10 \mathrm{e}^{F}+10)\cdot\log\left(\mathrm{e}^{F}+1\right)\right)\leq 7F.\] (A.101)

Therefore, by taking \(\epsilon=\frac{1}{2}\), \(\gamma=\frac{1}{n}\), \(\Gamma=500000F^{2}\), \(M=7F\), and

\[\mathcal{F}=\left\{\left.f\right|_{[0,1]^{d}}\right|f\in\mathcal{F}_{d}^{ \mathbf{FNN}}\left(G,N,S,B,F\right)\right\}\]

in Theorem 2.1 and combining (A.97), (A.98), (A.99), (A.100), (A.96), we obtain

\[\begin{split}&\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}^{ \phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]=\mathbf{E}_{P^{\otimes n}} \left[\mathcal{R}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)-\int_{[0,1] ^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right]\\ &\leq 360\cdot\frac{\Gamma\log W}{n}+\frac{4}{n}+\frac{30M\log W}{n}+3 0\cdot\sqrt{\frac{\Gamma\log W}{n^{2}}}+2\inf_{f\in\mathcal{F}}\left( \mathcal{R}_{P}^{\phi}(f)-\int\psi\mathrm{d}P\right)\\ &\leq\frac{360\Gamma\log W}{n}+\frac{\Gamma\log W}{n}+\frac{ \Gamma\log W}{n}+\frac{\Gamma\log W}{n}+\frac{2}{n}\inf_{f\in\mathcal{F}} \mathcal{E}_{P}^{\phi}(f)\\ &\leq\frac{2\cdot 10^{8}\cdot F^{2}\cdot\log W}{n}+16D_{4} \cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)^{d}} {d_{\mathbf{s}}+\beta\cdot(1\wedge\beta)^{d}}}\\ &\leq\frac{10^{9}\cdot\left|b_{4}\log n\right|^{2}\cdot E_{4} \cdot(\log n)^{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d_{\mathbf{s}}}{ d_{\mathbf{s}}+\beta\cdot(1\wedge\beta)^{d}}}}{n}+16D_{4}\cdot\left(\frac{(\log n)^{5}}{n} \right)^{\frac{\beta\cdot(1\wedge\beta)^{d}}{d_{\mathbf{s}}+\beta\cdot(1\wedge \beta)^{d}}}\\ &=\left(16D_{4}+10^{9}\cdot\left|b_{4}\right|^{2}\cdot E_{4} \right)\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge \beta)^{d}}{d_{\mathbf{s}}+\beta\cdot(1\wedge\beta)^{d}}}\leq E_{3}\cdot\left( \frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1\wedge\beta)^{d}}{d_{\mathbf{s}}+ \beta\cdot(1\wedge\beta)^{d}}}\end{split}\]

with

\[E_{3}:=4\cdot\left(16D_{4}+10^{9}\cdot\left|b_{4}\right|^{2}\cdot E_{4}\right)+4\]
only depending on \((\mathbf{a},\mathbf{b},\nu,d,d_{*},d_{*},\beta,r,q,K)\). We then apply the calibration inequality (2.21) and conclude that

\[\begin{split}&\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left( \hat{f}_{n}^{\mathbf{FNN}}\right)\right]\leq 2\sqrt{2}\cdot\mathbf{E}_{P^{\otimes n}} \left[\sqrt{\mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)} \right]\leq 4\cdot\sqrt{\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}^{\phi} \left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]}\\ &\leq 4\cdot\sqrt{\left(16D_{4}+10^{9}\cdot\left|b_{4}\right|^{2} \cdot E_{4}\right)\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1 \wedge\beta)^{q}}{d_{*}+\beta\cdot(1\wedge\beta)^{q}}}}\\ &\leq E_{3}\cdot\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta \cdot(1\wedge\beta)^{q}}{2d_{*}+2\cdot(1\wedge\beta)^{q}}}.\end{split}\] (A.102)

Since \(P\) is arbitrary, the desired bound (A.89) follows. Setting \(\mathrm{c}=E_{1}\) completes the proof of Theorem 2.3.

Now it remains to show Theorem 2.2. Indeed, it follows from (2.33) that

\[\mathcal{H}_{1}^{d,\beta,r}\subset\mathcal{H}_{4,0,1,1,d}^{d,\beta,r}.\]

Then by taking \(q=0\), \(d_{*}=d\) and \(d_{*}=K=1\) in Theorem 2.3, we obtain that there exists a constant \(\mathrm{c}\in(0,\infty)\) only depending on \((d,\beta,r)\) such that the estimator \(\hat{f}_{n}^{\mathbf{FNN}}\) defined by (2.14) with

\[\begin{split}&\mathrm{c}\log n\leq G\lesssim\log n,\ N\times\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d}{d+\beta \cdot(1\wedge\beta)^{q}}}=\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d}{d+ \beta}},\\ & S\asymp\left(\frac{(\log n)^{5}}{n}\right)^{\frac{-d}{d+\beta \cdot(1\wedge\beta)^{0}}}\cdot\log n=\left(\frac{(\log n)^{5}}{n}\right)^{ \frac{-d}{d+\beta}}\cdot\log n,\\ & 1\leq B\lesssim n^{\nu},\ \text{and}\ \ \frac{\beta}{d+\beta}\cdot\log n=\frac{\beta\cdot(1\wedge\beta)^{0}}{d+\beta \cdot(1\wedge\beta)^{0}}\cdot\log n\leq F\lesssim\log n\end{split}\]

must satisfy

\[\begin{split}&\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{ \otimes n}}\left[\mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right) \right]\leq\sup_{P\in\mathcal{H}_{4,0,1,d}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n}} \left[\mathcal{E}_{P}^{\phi}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\\ &\lesssim\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1 \wedge\beta)^{0}}{d+\beta\cdot(1\wedge\beta)^{0}}}=\left(\frac{(\log n)^{5}}{ n}\right)^{\frac{\beta}{d+\beta}}\end{split}\]

and

\[\begin{split}&\sup_{P\in\mathcal{H}_{1}^{d,\beta,r}}\mathbf{E}_{P^{ \otimes n}}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right) \right]\leq\sup_{P\in\mathcal{H}_{4,0,1,1,d}^{d,\beta,r}}\mathbf{E}_{P^{\otimes n }}\left[\mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\\ &\lesssim\left(\frac{(\log n)^{5}}{n}\right)^{\frac{\beta\cdot(1 \wedge\beta)^{0}}{2d+2\cdot(1\wedge\beta)^{0}}}=\left(\frac{(\log n)^{5}}{n} \right)^{\frac{\beta\cdot(1\wedge\beta)^{0}}{2d+2\beta}}.\end{split}\]

This completes the proof of Theorem 2.2. 

#### a.3.5 Proof of Theorem 2.5

This subsection is devoted to the proof of Theorem 2.5. To this end, we need the following lemmas. Note that the logistic loss is given by \(\phi(t)=\log(1+\mathrm{e}^{-t})\) with \(\phi^{\prime}(t)=-\frac{1}{1+\mathrm{e}^{t}}\in(-1,0)\) and \(\phi^{\prime\prime}(t)=\frac{\mathrm{e}^{t}}{(1+\mathrm{e}^{t})^{2}}=\frac{1} {\mathrm{e}^{t}+\mathrm{e}^{-t}+2}\in(0,\frac{1}{4}]\) for all \(t\in\mathbb{R}\).

**Lemma A.16**.: _Let \(\eta_{0}\in(0,1)\), \(F_{0}\in\left(0,\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)\), \(a\in[-F_{0},F_{0}]\), \(\phi(t)=\log(1+\mathrm{e}^{-t})\) be the logistic loss, \(d\in\mathbb{N}\), and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the conditional probability
function \([0,1]^{d}\ni z\mapsto P(\{1\}\,|z)\in[0,1]\) is denoted by \(\eta\). Then for any \(x\in[0,1]^{d}\) such that \(|2\eta(x)-1|>\eta_{0}\), there holds_

\[0 \leq|a-F_{0}\mathrm{sgn}(2\eta(x)-1)|\cdot\left(\frac{1-\eta_{0}}{ 2}\phi^{\prime}(-F_{0})-\frac{\eta_{0}+1}{2}\phi^{\prime}(F_{0})\right)\] (A.103) \[\leq|a-F_{0}\mathrm{sgn}(2\eta(x)-1)|\cdot\left(\frac{1-\eta_{0}} {2}\phi^{\prime}(-F_{0})-\frac{\eta_{0}+1}{2}\phi^{\prime}(F_{0})\right)\] \[\qquad+\frac{1}{2\left(\mathrm{e}^{-F_{0}}+\mathrm{e}^{F_{0}}+2 \right)}\left|a-F_{0}\mathrm{sgn}(2\eta(x)-1)\right|^{2}\] \[\leq\int_{\{-1,1\}}\left(\phi(ya)-\phi(yF_{0}\mathrm{sgn}(2\eta( x)-1))\right)\mathrm{d}P(y|x)\] \[\leq|a-F_{0}\mathrm{sgn}(2\eta(x)-1)|+F_{0}^{2}.\]

Proof.: Given \(x\in[0,1]^{d}\), recall the function \(V_{x}\) defined in the proof of Lemma A.7. By Taylor expansion, there exists \(\xi\) between \(a\) and \(F_{0}\mathrm{sgn}(2\eta(x)-1)\) such that

\[\int_{\{-1,1\}}\left(\phi(ya)-\phi(yF_{0}\mathrm{sgn}(2\eta(x)-1) )\right)\mathrm{d}P(y|x)\] (A.104) \[=V_{x}(a)-V_{x}(F_{0}\mathrm{sgn}(2\eta(x)-1))\] \[=(a-F_{0}\mathrm{sgn}(2\eta(x)-1))\cdot V_{x}^{\prime}(F_{0} \mathrm{sgn}(2\eta(x)-1))+\frac{1}{2}\left|a-F_{0}\mathrm{sgn}(2\eta(x)-1) \right|^{2}\cdot V_{x}^{\prime\prime}(\xi).\]

Since \(\xi\in[-F_{0},F_{0}]\), we have

\[0 \leq\frac{1}{\mathrm{e}^{-F_{0}}+\mathrm{e}^{F_{0}}+2}=\inf\left\{ \phi^{\prime\prime}(t)\mid t\in[-F_{0},F_{0}]\right\}\] \[\leq V_{x}^{\prime\prime}(\xi)=\eta(x)\phi^{\prime\prime}(\xi)+( 1-\eta(x))\phi^{\prime\prime}(-\xi)\leq\frac{1}{4}\]

and then

\[0 \leq\frac{1}{2}\left|a-F_{0}\mathrm{sgn}(2\eta(x)-1)\right|^{2} \cdot\frac{1}{\mathrm{e}^{-F_{0}}+\mathrm{e}^{F_{0}}+2}\] (A.105) \[\leq\frac{1}{2}\left|a-F_{0}\mathrm{sgn}(2\eta(x)-1)\right|^{2} \cdot V_{x}^{\prime\prime}(\xi)\] \[\leq\frac{1}{2}\left(|a|+F_{0}\right)^{2}\cdot\frac{1}{4}\leq \frac{1}{2}F_{0}^{2}.\]

On the other hand, if \(2\eta(x)-1>\eta_{0}\), then

\[(a-F_{0}\mathrm{sgn}(2\eta(x)-1))\cdot V_{x}^{\prime}(F_{0} \mathrm{sgn}(2\eta(x)-1))\] \[=(a-F_{0})\left(\eta(x)\phi^{\prime}(F_{0})-(1-\eta(x))\phi^{ \prime}(-F_{0})\right)\] \[=|a-F_{0}|\left((1-\eta(x))\phi^{\prime}(-F_{0})-\eta(x)\phi^{ \prime}(F_{0})\right)\] \[\geq|a-F_{0}|\left(\left(1-\frac{1+\eta_{0}}{2}\right)\phi^{ \prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F_{0})\right)\] \[=|a-F_{0}\mathrm{sgn}(2\eta(x)-1)|\cdot\left(\frac{1-\eta_{0}}{2} \phi^{\prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F_{0})\right).\]

Similarly, if \(2\eta(x)-1<-\eta_{0}\), then

\[(a-F_{0}\mathrm{sgn}(2\eta(x)-1))\cdot V_{x}^{\prime}(F_{0} \mathrm{sgn}(2\eta(x)-1))\] \[=(a+F_{0})\left(\eta(x)\phi^{\prime}(-F_{0})-(1-\eta(x))\phi^{ \prime}(F_{0})\right)\] \[=|a+F_{0}|\left(\eta(x)\phi^{\prime}(-F_{0})-(1-\eta(x))\phi^{ \prime}(F_{0})\right)\] \[\geq|a+F_{0}|\left(\frac{1-\eta_{0}}{2}\phi^{\prime}(-F_{0})- \left(1-\frac{1-\eta_{0}}{2}\right)\phi^{\prime}(F_{0})\right)\]
\[=|a-F_{0}\text{sgn}(2\eta(x)-1)|\cdot\left(\frac{1-\eta_{0}}{2}\phi^{ \prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F_{0})\right).\]

Therefore, for given \(x\in[0,1]^{d}\) satisfying \(|2\eta(x)-1|>\eta_{0}\), there always holds

\[(a-F_{0}\text{sgn}(2\eta(x)-1))\cdot V_{x}^{\prime}(F_{0}\text{ sgn}(2\eta(x)-1))\] (A.106) \[\geq|a-F_{0}\text{sgn}(2\eta(x)-1)|\cdot\left(\frac{1-\eta_{0}}{ 2}\phi^{\prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F_{0})\right).\]

We next show that \(\frac{1-\eta_{0}}{2}\phi^{\prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F_ {0})>0\). Indeed, let \(g(t)=\frac{1-\eta_{0}}{2}\phi^{\prime}(-t)-\frac{1+\eta_{0}}{2}\phi^{\prime}(t)\). Then \(g^{\prime}(t)=-\frac{1-\eta_{0}}{2}\phi^{\prime\prime}(-t)-\frac{1+\eta_{0}} {2}\phi^{\prime\prime}(t)<0\), i.e., \(g\) is strictly decreasing, and thus

\[\frac{1-\eta_{0}}{2}\phi^{\prime}(-F_{0})-\frac{1+\eta_{0}}{2}\phi^{\prime}(F _{0})=g(F_{0})>g\left(\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)=0.\] (A.107)

Moreover, we also have

\[(a-F_{0}\text{sgn}(2\eta(x)-1))\cdot V_{x}^{\prime}(F_{0}\text{ sgn}(2\eta(x)-1))\] (A.108) \[\leq|a-F_{0}\text{sgn}(2\eta(x)-1)|\cdot|V_{x}^{\prime}(F_{0} \text{sgn}(2\eta(x)-1))|\] \[=|a-F_{0}\text{sgn}(2\eta(x)-1)|\cdot|\eta(x)\phi^{\prime}(F_{0} \text{sgn}(2\eta(x)-1))-(1-\eta(x))\phi^{\prime}(-F_{0}\text{sgn}(2\eta(x)-1))|\] \[\leq|a-F_{0}\text{sgn}(2\eta(x)-1)|\,|\eta(x)+(1-\eta(x))|=|a-F_{ 0}\text{sgn}(2\eta(x)-1)|\,.\]

Then the first inequality of (A.103) is from (A.107), the third inequality of (A.103) is due to (A.104), (A.105) and (A.106), and the last inequality of (A.103) is from (A.104), (A.105) and (A.108). Thus we complete the proof. 

**Lemma A.17**.: _Let \(\eta_{0}\in(0,1)\), \(F_{0}\in\left(0,\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)\), \(d\in\mathbb{N}\), and \(P\) be a Borel probability measure on \([0,1]^{d}\times\{-1,1\}\) of which the conditional probability function \([0,1]^{d}\ni z\mapsto P(\{1\}\,|z)\in[0,1]\) is denoted by \(\eta\). Define_

\[\psi:[0,1]^{d}\times\{-1,1\} \to\mathbb{R},\] (A.109) \[(x,y) \mapsto\begin{cases}\phi\left(yF_{0}\text{sgn}(2\eta(x)-1)\right), &\text{ if }\,|2\eta(x)-1|>\eta_{0},\\ \phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right),&\text{ if }\,|2\eta(x)-1|\leq \eta_{0}.\end{cases}\]

_Then there hold_

\[\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi( x,y)\right)^{2}\!\mathrm{d}P(x,y)\] (A.110) \[\leq\frac{8}{1-\eta_{0}^{2}}\cdot\int_{[0,1]^{d}\times\{-1,1\}} \left(\phi\left(yf(x)\right)-\psi(x,y)\right)\mathrm{d}P(x,y)\]

_for any measurable \(f:[0,1]^{d}\to[-F_{0},F_{0}]\,,\) and_

\[0\leq\psi(x,y)\leq\log\frac{2}{1-\eta_{0}},\quad\forall(x,y)\in[0,1]^{d}\times \{-1,1\}\,.\] (A.111)

Proof.: Recall that given \(x\in[0,1]^{d}\), \(V_{x}(t)=\eta(x)\phi(t)+(1-\eta(x))\phi(-t),\forall t\in\mathbb{R}\). Due to inequality (A.103) and Lemma A.7, for any measurable \(f:[0,1]^{d}\to[-F_{0},F_{0}]\), we have

\[\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi\left(yf(x)\right)-\psi( x,y)\right)\mathrm{d}P(x,y)\] (A.112) \[=\int_{|2\eta(x)-1|>\eta_{0}}\int_{\{-1,1\}}\phi\left(yf(x)\right) -\phi\left(yF_{0}\text{sgn}(2\eta(x)-1)\right)\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\] \[\quad+\int_{|2\eta(x)-1|\leq\eta_{0}}\int_{\{-1,1\}}\phi\left(yf(x )\right)-\phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right)\mathrm{d}P(y|x) \mathrm{d}P_{X}(x)\]
\[\geq\int_{|2\eta(x)-1|>\eta_{0}}\frac{1}{2\left(\mathrm{e}^{F_{0}}+ \mathrm{e}^{-F_{0}}+2\right)}\left|f(x)-F_{0}\mathrm{sgn}(2\eta(x)-1)\right|^{2 }\mathrm{d}P_{X}(x)\] \[\quad+\int_{|2\eta(x)-1|\leq\eta_{0}}\left[\inf_{t\in\left[\log \frac{1-\eta_{0}}{1+\eta_{0}},\log\frac{1+\eta_{0}}{1-\eta_{0}}\right]}\frac{1} {2(\mathrm{e}^{t}+\mathrm{e}^{-t}+2)}\right]\left|f(x)-\log\frac{\eta(x)}{1- \eta(x)}\right|^{2}\mathrm{d}P_{X}(x)\] \[\geq\frac{1}{2}\frac{1}{\frac{1+\eta_{0}}{1-\eta_{0}}+\frac{1- \eta_{0}}{1+\eta_{0}}+2}\int_{\{|2\eta(x)-1|>\eta_{0}\}\times\{-1,1\}}\left| \phi\left(yf(x)\right)-\phi\left(yF_{0}\mathrm{sgn}(2\eta(x)-1)\right)\right|^ {2}\mathrm{d}P(x,y)\] \[\quad+\frac{1}{2}\frac{1+\eta_{0}}{1-\eta_{0}}+\frac{1-\eta_{0}} {1+\eta_{0}}+2\int_{\{|2\eta(x)-1|\leq\eta_{0}\}\times\{-1,1\}}\left|\phi \left(yf(x)\right)-\phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right)\right|^{2} \mathrm{d}P(x,y)\] \[=\frac{1-\eta_{0}^{2}}{8}\cdot\int_{[0,1]^{d}\times\{-1,1\}} \left(\phi\left(yf(x)\right)-\psi(x,y)\right)^{2}\mathrm{d}P(x,y),\]

where the second inequality is from (A.20) and the fact that \(F_{0}\in\left(0,\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)\). Thus we have proved the inequality (A.110).

On the other hand, from the definition of \(\psi\) as well as \(F_{0}\in\left(0,\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)\), we also have

\[0\leq\psi(x,y)\leq\max\left\{\phi(-F_{0}),\phi\left(-\log\frac{1+\eta_{0}}{1- \eta_{0}}\right)\right\}\leq\phi\left(-\log\frac{1+\eta_{0}}{1-\eta_{0}} \right)=\log\frac{2}{1-\eta_{0}},\]

which gives the inequality (A.111). The proof is completed. 

Now we are in the position to prove Theorem 2.5.

Proof of Theorem 2.5.: Let \(\eta_{0}\in(0,1)\cap[0,t_{1}]\), \(F_{0}\in(0,\log\frac{1+\eta_{0}}{1-\eta_{0}})\cap[0,1]\), \(\xi\in(0,\frac{1}{2}\wedge t_{2}]\) and \(P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_{2}}\) be arbitrary. Denote by \(\eta\) the conditional probability function \(P(\{1\}\left|\cdot\right)\) of \(P\). By definition, there exists a classifier \(\mathtt{C}\in\mathcal{C}^{d,\beta,r,I,\Theta}\) such that (2.24), (2.50) and (2.51) hold. According to Proposition A.4 and the proof of Theorem 3.4 in [22], there exist positive constants \(G_{0},N_{0},S_{0},B_{0}\) only depending on \(d,\beta,r,I,\Theta\) and \(\tilde{f}_{0}\in\mathcal{F}_{d}^{\mathbf{FNN}}(G_{\xi},N_{\xi},S_{\xi},B_{\xi},1)\) such that \(\tilde{f}_{0}(x)=\mathtt{C}(x)\) for \(x\in[0,1]^{d}\) with \(\Delta_{\mathtt{C}}(x)>\xi\), where

\[G_{\xi}=G_{0}\log\frac{1}{\xi},\ N_{\xi}=N_{0}\left(\frac{1}{\xi}\right)^{\frac{ d-1}{\beta}},\ S_{\xi}=S_{0}\left(\frac{1}{\xi}\right)^{\frac{d-1}{\beta}}\log \left(\frac{1}{\xi}\right),\ B_{\xi}=\left(\frac{B_{0}}{\xi}\right).\] (A.112)

Define \(\psi:[0,1]^{d}\times\{-1,1\}\to\mathbb{R}\) by (A.109). Then for any measurable function \(f:[0,1]^{d}\to[-F_{0},F_{0}]\), there holds

\[\mathcal{E}_{P}(f)=\mathcal{E}_{P}\left(\frac{f}{F_{0}}\right)\leq \int_{[0,1]^{d}}\left|\frac{f(x)}{F_{0}}-\mathrm{sgn}(2\eta(x)-1)\right|\left| 2\eta(x)-1\right|\mathrm{d}P_{X}(x)\] \[\leq 2P\left(|2\eta(x)-1|\leq\eta_{0}\right)+\int_{|2\eta(x)-1|> \eta_{0}}\left|\frac{f(x)}{F_{0}}-\mathrm{sgn}(2\eta(x)-1)\right|\mathrm{d}P_{ X}(x)\] \[\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{1}{F_{0}}\int_{|2\eta(x)-1|> \eta_{0}}|f(x)-F_{0}\mathrm{sgn}(2\eta(x)-1)|\mathrm{d}P_{X}(x)\] (A.113) \[\leq 2c_{1}\eta_{0}^{s_{1}}+\int_{|2\eta(x)-1|>\eta_{0}}\frac{\int (\phi(yf(x))-\phi(yF_{0}\mathrm{sgn}(2\eta(x)-1)))\,\mathrm{d}P(y|x)}{F_{0}\cdot \left(\frac{1-\eta_{0}}{2}\phi^{\prime}(-F_{0})-\frac{\eta_{0}+1}{2}\phi^{ \prime}(F_{0})\right)}\mathrm{d}P_{X}(x)\] \[\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{\int_{[0,1]^{d}\times\{-1,1\}} \left(\phi(yf(x))-\psi(x,y)\right)\mathrm{d}P(x,y)}{F_{0}\cdot\left(\frac{1- \eta_{0}}{2}\phi^{\prime}(-F_{0})-\frac{\eta_{0}+1}{2}\phi^{\prime}(F_{0}) \right)},\]

where the first inequality is from Theorem 2.31 of [40], the third inequality is due to the noise condition (2.24), and the fourth inequality is from (A.103) in Lemma A.16.
Let \(\mathcal{F}=\mathcal{F}_{d}^{\mathbf{FNN}}(G_{\xi},N_{\xi},S_{\xi},B_{\xi},F_{0})\) with \((G_{\xi},N_{\xi},S_{\xi},B_{\xi})\) given by (A.112), \(\Gamma=\frac{8}{1-\eta_{0}^{2}}\) and \(M=\frac{2}{1-\eta_{0}}\) in Theorem 2.1. Then we will use this theorem to derive the desired generalization bounds for the \(\phi\)-\(\operatorname{ERM}\hat{f}_{n}:=\hat{f}_{n}^{\mathbf{FNN}}\) over \(\mathcal{F}_{d}^{\mathbf{FNN}}(G_{\xi},N_{\xi},S_{\xi},B_{\xi},F_{0})\). Indeed, Lemma A.17 guarantees that the conditions (2.3), (2.4) and (2.5) of Theorem 2.1 are satisfied. Moreover, take \(\gamma=\frac{1}{n}\). Then \(W=\max\left\{3,\;\mathcal{N}\left(\mathcal{F},\gamma\right)\right\}\) satisfies

\[\log W\leq C_{d,\beta,r,I,\Theta}\xi^{-\frac{d-1}{\beta}}\left(\log\frac{1}{ \xi}\right)^{2}\left(\log\frac{1}{\xi}+\log n\right).\]

Thus the expectation of \(\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(y\hat{f}_{n}(x))-\psi(x,y)\right) \mathrm{d}P(x,y)\) can be bounded by inequality (2.6) in Theorem 2.1 as

\[\begin{split}&\boldsymbol{E}_{P^{\otimes n}}\left[\int_{[0,1]^{d} \times\{-1,1\}}\left(\phi(y\hat{f}_{n}(x))-\psi(x,y)\right)\mathrm{d}P(x,y) \right]\\ &\leq\frac{4000C_{d,\beta,r,I,\Theta}\xi^{-\frac{d-1}{\beta}} \left(\log\frac{1}{\xi}\right)^{2}\left(\log\frac{1}{\xi}+\log n\right)}{n(1- \eta_{0}^{2})}\\ &\qquad\qquad+2\inf_{f\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi }(f)-\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right).\end{split}\] (A.114)

We next estimate the approximation error, i.e., the second term on the right hand side of (A.114). Take \(f_{0}=F_{0}\tilde{f}_{0}\in\mathcal{F}\) where \(\tilde{f}_{0}\in\mathcal{F}_{d}^{\mathbf{FNN}}(G_{\xi},N_{\xi},S_{\xi},B_{\xi},1)\) satisfying \(\tilde{f}_{0}(x)=\mathtt{C}(x)\) for \(x\in[0,1]^{d}\) with \(\Delta_{\mathtt{C}}(x)>\xi\). Then one can bound the approximation error as

\[\begin{split}&\inf_{f\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi}(f)- \int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right)\\ &\leq\mathcal{R}_{P}^{\phi}(f_{0})-\int_{[0,1]^{d}\times\{-1,1\}} \psi(x,y)\mathrm{d}P(x,y)=I_{1}+I_{2}+I_{3},\end{split}\] (A.115)

where

\[\begin{split}& I_{1}:=\int_{\{[2\eta(x)-1]>\eta_{0},\Delta_{ \mathtt{C}}(x)>\xi\}\times\{-1,1\}}\phi(yf_{0}(x))-\phi(yF_{0}\mathrm{sgn}(2 \eta(x)-1))\mathrm{d}P(x,y),\\ & I_{2}:=\int_{\{[2\eta(x)-1]\leq\eta_{0}\}\times\{-1,1\}}\phi(yf _{0}(x))-\phi\left(y\log\frac{\eta(x)}{1-\eta(x)}\right)\mathrm{d}P(x,y),\\ & I_{3}:=\int_{\{[2\eta(x)-1]>\eta_{0},\Delta_{\mathtt{C}}(x)\leq \xi\}\times\{-1,1\}}\phi(yf_{0}(x))-\phi(yF_{0}\mathrm{sgn}(2\eta(x)-1)) \mathrm{d}P(x,y).\end{split}\]

Note that \(f_{0}(x)=F_{0}\tilde{f}_{0}(x)=F_{0}\mathtt{C}(x)=F_{0}\mathrm{sgn}(2\eta(x)-1)\) for \(P_{X}\)-almost all \(x\in[0,1]^{d}\) with \(\Delta_{\mathtt{C}}(x)>\xi\). Thus it follows that \(I_{1}=0\). On the other hand, from Lemma A.7 and the noise condition (2.24), we see that

\[\begin{split}& I_{2}\leq\int_{\{[2\eta(x)-1]\leq\eta_{0}\}\times\{- 1,1\}}\left|f_{0}(x)-\log\frac{\eta(x)}{1-\eta(x)}\right|^{2}\mathrm{d}P(x,y) \\ &\leq\int_{\{\left|2\eta_{0}(x)-1\right|\leq\eta_{0}\}\times\{-1,1 \}}\left(F_{0}+\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)^{2}\mathrm{d}P(x,y) \leq 4\left(\log\frac{1+\eta_{0}}{1-\eta_{0}}\right)^{2}c_{1}\cdot\eta_{0} ^{s_{1}}.\end{split}\] (A.116)

Moreover, due to Lemma A.16 and the margin condition (2.51), we have

\[\begin{split}& I_{3}\leq\int_{\{[2\eta(x)-1]>\eta_{0},\Delta_{ \mathtt{C}}(x)\leq\xi\}}\left(2F_{0}+F_{0}^{2}\right)\mathrm{d}P_{X}(x)\\ &\leq 3F_{0}\cdot P_{X}(\left\{x\in[0,1]^{d}|\,\Delta_{\mathtt{C}}(x) \leq\xi\right\})\leq 3F_{0}\cdot c_{2}\cdot\xi^{s_{2}}.\end{split}\] (A.117)
The estimates above together with (A.113) and (A.114) give

\[\begin{split}&\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}(\hat{f}_{ n})\right]\\ &\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{1}{F_{0}}\cdot\frac{\mathbf{E}_{P^{ \otimes n}}\left[\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(y\hat{f}_{n}(x))- \psi(x,y)\right)\mathrm{d}P(x,y)\right]}{\frac{1-\eta_{0}}{2}\phi^{\prime}(-F_ {0})-\frac{\eta_{0}+1}{2}\phi^{\prime}(F_{0})}\\ &\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{8\left|\log\frac{1+\eta_{0}}{1- \eta_{0}}\right|^{2}c_{1}\eta_{0}^{s_{1}}+6F_{0}c_{2}\xi^{s_{2}}+\frac{4000C_{ d,\beta,r,I,\Theta}\xi^{\frac{d-1}{\beta}}\left(\log\frac{1}{\xi}\right)^{2} \left(\log\frac{1}{\xi}+\log n\right)}{n(1-\eta_{0}^{s})}}.\end{split}\] (A.118)

Since \(P\) is arbitrary, we can take the supremum over all \(P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_{2}}\) to obtain from (A.118) that

\[\begin{split}&\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d, \beta,r,I,\Theta,s_{1},s_{2}}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}(f_{ n}^{\mathbf{FNN}})\right]\\ &\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{8\left|\log\frac{1+\eta_{0}}{1- \eta_{0}}\right|^{2}c_{1}\eta_{0}^{s_{1}}+6F_{0}c_{2}\xi^{s_{2}}+\frac{4000C_{ d,\beta,r,I,\Theta}\xi^{\frac{d-1}{\beta}}\left(\log\frac{1}{\xi}\right)^{2} \left(\log\frac{1}{\xi}+\log n\right)}{n(1-\eta_{0}^{s})}}.\end{split}\] (A.119)

(A.119) holds for all \(\eta_{0}\in(0,1)\cap[0,t_{1}]\), \(F_{0}\in(0,\log\frac{1+\eta_{0}}{1-\eta_{0}})\cap[0,1]\), \(\xi\in(0,\frac{1}{2}\wedge t_{2}]\). We then take suitable \(\eta_{0}\), \(F_{0}\), and \(\xi\) in (A.119) to derive the convergence rates stated in Theorem 2.5.

\[\begin{split}&\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d, \beta,r,I,\Theta,s_{1},s_{2}}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}(f_{ n}^{\mathbf{FNN}})\right]\\ &\leq 2c_{1}\eta_{0}^{s_{1}}+\frac{8\left|\log\frac{1+\eta_{0}}{1- \eta_{0}}\right|^{2}c_{1}\cdot\eta_{0}^{s_{1}}+6F_{0}c_{2}\xi^{s_{2}}+\frac{4000 C_{d,\beta,r,I,\Theta}\xi^{\frac{d-1}{\beta}}\left(\log\frac{1}{\xi}\right)^{2} \left(\log\frac{1}{\xi}+\log n\right)}{n(1-\eta_{0}^{s})}}.\end{split}\] (A.120)

**Case I.** When \(s_{1}=s_{2}=\infty\), taking \(\eta_{0}=F_{0}=t_{1}\wedge\frac{1}{2}\) and \(\xi=t_{2}\wedge\frac{1}{2}\) in (A.119) yields

\[\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_ {2}}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(f_{n}^{\mathbf{FNN}} \right)\right]\lesssim\frac{\log n}{n}.\]

**Case II.** When \(s_{1}=\infty\) and \(s_{2}<\infty\), taking \(\eta_{0}=F_{0}=t_{1}\wedge\frac{1}{2}\) and \(\xi\simeq\left(\frac{\log n}{n}\right)^{\frac{1}{s_{2}+\frac{d-1}{\beta}}}\) in (A.119) yields

\[\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_ {2}}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\tilde{f}_{n}^{\mathbf{FNN}} \right)\right]\lesssim\left(\frac{\left(\log n\right)^{3}}{n}\right)^{\frac{1 }{1+\frac{1}{\beta\beta\beta\beta}}}.\]

**Case III.** When \(s_{1}<\infty\) and \(s_{2}=\infty\), take \(\eta_{0}=F_{0}\asymp\left(\frac{\log n}{n}\right)^{\frac{1}{s_{1}+2}}\) and \(\xi=t_{2}\wedge\frac{1}{2}\) in (A.119). From the fact that \(\frac{\eta_{0}}{4}\leq\frac{1-\eta_{0}}{2}\phi^{\prime}(-\eta_{0})-\frac{\eta_{ 0}+1}{2}\phi^{\prime}(\eta_{0})\leq\eta_{0},\forall 0\leq\eta_{0}\leq 1\), the item in the denominator of the second term on the right hand side of (A.119) is larger than \(\frac{1}{4}\eta_{0}^{2}\). Then we have

\[\sup_{P\in\mathcal{H}_{6,t_{1},c_{1},t_{2},c_{2}}^{d,\beta,r,I,\Theta,s_{1},s_ {2}}}\mathbf{E}_{P^{\otimes n}}\left[\mathcal{E}_{P}\left(\tilde{f}_{n}^{\mathbf{FNN}} \right)\right]\lesssim\left(\frac{\log n}{n}\right)^{\frac{s_{1}}{s_{1}+2}}.\]

**Case IV.** When \(s_{1}<\infty\) and \(s_{2}<\infty\), taking

\[\eta_{0}=F_{0}\asymp\left(\frac{\left(\log n\right)^{3}}{n}\right)^{\frac{s_{2} }{s_{2}+(s_{1}+1)\left(s_{2}+\frac{d-1}{\beta}\right)}}\text{ and }\xi\asymp\left(\frac{\left(\log n\right)^{3}}{n}\right)^{\frac{s_{1}+1}{s_{2}+(s_{ 1}+1)\left(s_{2}+\frac{d-1}{\beta}\right)}}\]
in (A.119) yields

\[\sup_{P\in\mathcal{H}_{0,t_{1},\cdot\cdot\cdot_{1},\cdot\cdot_{2},\cdot\cdot_{2}} ^{d,\beta,r,I,\varrho,\epsilon_{1},\cdot\cdot_{2}}}\mathbf{E}_{P^{\otimes n}}\left[ \mathcal{E}_{P}\left(\hat{f}_{n}^{\mathbf{FNN}}\right)\right]\lesssim\left( \frac{\left(\log n\right)^{3}}{n}\right)^{\frac{\varepsilon_{1}}{1+\left( \varepsilon_{1}+1\right)\left(1+\frac{d+1}{\beta\sigma_{2}}\right)}}.\]

Combining above cases, we obtain the desired results. The proof of Theorem 2.5 is completed. 

#### a.3.6 Proof of Theorem 2.6 and Corollary 2.1

Hereinafter, for \(a\in\mathbb{R}^{d}\) and \(R\in\mathbb{R}\), we define \(\mathscr{B}(a,R):=\big{\{}\,x\in\mathbb{R}^{d}|\,\|x-a\|_{2}\leq R\big{\}}\).

**Lemma A.18**.: _Let \(d\in\mathbb{N}\), \(\beta\in(0,\infty)\), \(r\in(0,\infty)\), \(Q\in\mathbb{N}\cap(1,\infty)\),_

\[G_{Q,d}:=\bigg{\{}\,(\frac{k_{1}}{2Q},\dots,\frac{k_{d}}{2Q})^{\top}\bigg{|}k_ {1},\dots,k_{d}\text{ are odd integers}\bigg{\}}\cap[0,1]^{d},\]

_and \(T:G_{Q,d}\to\{-1,1\}\) be a map. Then there exist a constant \(\mathrm{c}_{1}\in(0,\frac{1}{9999})\) only depending on \((d,\beta,r)\), and an \(f\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\) depending on \((d,\beta,r,Q,T)\), such that \(\|f\|_{[0,1]^{d}}=\frac{\mathrm{c}_{1}}{Q^{d}}\), and_

\[f(x)=\|f\|_{[0,1]^{d}}\cdot T(a)=\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot T(a), \;\forall\;a\in G_{Q,d},\;x\in\mathscr{B}(a,\frac{1}{5Q})\cap[0,1]^{d}.\]

Proof.: Let

\[\kappa:\mathbb{R}\to[0,1],t\mapsto\frac{\int_{t}^{\infty}\exp\left(-1/(x-1/9) \right)\cdot\exp\left(-1/(1/8-x)\right)\cdot\mathbb{1}_{(1/9,1/8)}(x)\mathrm{ d}x}{\int_{1/9}^{1/8}\exp\left(-1/(x-1/9)\right)\cdot\exp\left(-1/(1/8-x) \right)\mathrm{d}x}\]

be a well defined infinitely differentiable decreasing function on \(\mathbb{R}\) with \(\kappa(t)=1\) for \(t\leq 1/9\) and \(\kappa(t)=0\) for \(t\geq 1/8\). Then define \(b:=\lceil\beta\rceil-1\), \(\lambda:=\beta-b\),

\[u:\mathbb{R}^{d}\to[0,1],x\mapsto\kappa(\|x\|_{2}^{2}),\]

and \(\mathrm{c}_{2}:=\big{\|}u|_{[-2,2]^{d}}\big{\|}_{\mathcal{C}^{b,\lambda}([-2, 2]^{d})}\). Obviously, \(u\) only depends on \(d\), and \(\mathrm{c}_{2}\) only depends on \((d,\beta)\). Since \(u\) is infinitely differentiable and supported in \(\mathscr{B}(\mathbf{0},\sqrt{\frac{1}{8}})\), we have \(0<\mathrm{c}_{2}<\infty\). Take \(\mathrm{c}_{1}:=\frac{r}{4\mathrm{c}_{2}}\wedge\frac{1}{10000}\). Then \(\mathrm{c}_{1}\) only depends on \((d,\beta,r)\), and \(0<\mathrm{c}_{1}<\frac{1}{9999}\). Define

\[f:[0,1]^{d}\to\mathbb{R},x\mapsto\sum_{a\in G_{Q,d}}T(a)\cdot\frac{\mathrm{c}_ {1}}{Q^{\beta}}\cdot u(Q\cdot(x-a)).\]

We then show that these \(\mathrm{c}_{1}\) and \(f\) defined above have the desired properties.

For any \(\mathbf{m}\in(\mathbb{N}\cup\{0\})^{d}\), we write \(u_{\mathbf{m}}\) for \(\mathrm{D}^{\mathbf{m}}u\), i.e., the partial derivative of \(u\) with respect to the multi-index \(\mathbf{m}\). An elementary calculation yields

\[\mathrm{D}^{\mathbf{m}}f(x)=\sum_{a\in G_{Q,d}}T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{ \beta-\|\mathbf{m}\|_{1}}}\cdot u_{\mathbf{m}}(Q\cdot(x-a)),\;\forall\;\mathbf{m}\in( \mathbb{N}\cup\{0\})^{d},\;x\in[0,1]^{d}.\] (A.121)

Note that the supports of the functions \(T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta-\|\mathbf{m}\|_{1}}}\cdot u_{\mathbf{m}}(Q \cdot(x-a))\) (\(a\in G_{Q,d}\)) in (A.121) are disjoint. Indeed, we have

\[\begin{split}&\bigg{\{}x\in\mathbb{R}^{d}\left|T(a)\cdot\frac{ \mathrm{c}_{1}}{Q^{\beta-\|\mathbf{m}\|_{1}}}\cdot u_{\mathbf{m}}(Q\cdot(x-a))\neq 0 \right\}\\ &\subset\mathscr{B}(a,\frac{\sqrt{1/8}}{Q})\subset\bigg{\{}a+v \left|v\in(\frac{-1}{2Q},\frac{1}{2Q})^{d}\,\right\}\\ &\subset[0,1]^{d}\setminus\bigg{\{}z+v\left|v\in[\frac{-1}{2Q}, \frac{1}{2Q}]^{d}\right.\!\bigg{\}}\,,\,\forall\,\mathbf{m}\in(\mathbb{N}\cup\{0\} )^{d},\,a\in G_{Q,d},\,z\in G_{Q,d}\setminus\{a\}\,,\end{split}\] (A.122)
and sets \(\mathscr{B}(a,\frac{\sqrt{1/8}}{Q})\) (\(a\in G_{Q,d}\)) are disjoint. Therefore,

\[\begin{split}&\|\mathrm{D}^{\boldsymbol{m}}f\|_{[0,1]^{d}}=\sup_{a \in G_{Q,d}}\sup_{x\in[0,1]^{d}}\left|T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta- \left\|\boldsymbol{m}\right\|_{1}}}\cdot u_{\boldsymbol{m}}(Q\cdot(x-a))\right| \\ &=\sup_{a\in G_{Q,d}}\sup_{x\in\mathscr{B}(\boldsymbol{0},\frac{ \sqrt{1/8}}{Q})}\left|T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta-\left\| \boldsymbol{m}\right\|_{1}}}\cdot u_{\boldsymbol{m}}(Q\cdot(x-a))\right|\\ &=\sup_{a\in G_{Q,d}}\sup_{x\in\mathscr{B}(\boldsymbol{0}, \sqrt{1/8})}\left|\frac{\mathrm{c}_{1}}{Q^{\beta-\left\|\boldsymbol{m}\right\| _{1}}}\cdot u_{\boldsymbol{m}}(x)\right|\leq\sup_{x\in[-2,2]^{d}}\left|\frac{ \mathrm{c}_{1}}{Q^{\beta-\left\|\boldsymbol{m}\right\|_{1}}}\cdot u_{ \boldsymbol{m}}(x)\right|\\ &\leq\sup_{x\in[-2,2]^{d}}\left|\mathrm{c}_{1}\cdot u_{ \boldsymbol{m}}(x)\right|\leq\mathrm{c}_{1}\mathrm{c}_{2},\;\forall\; \boldsymbol{m}\in(\mathbb{N}\cup\{0\})^{d}\text{ with }\left\|\boldsymbol{m}\right\|_{1}\leq b.\end{split}\] (A.123)

In particular, we have that

\[\left\|f\right\|_{[0,1]^{d}}=\sup_{a\in G_{Q,d}}\sup_{x\in\mathscr{B}( \boldsymbol{0},\sqrt{1/8})}\left|\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot u(x) \right|=\frac{\mathrm{c}_{1}}{Q^{\beta}}.\] (A.124)

Besides, for any \(a\in G_{Q,d}\), any \(x\in\mathscr{B}(a,\frac{1}{5Q})\cap[0,1]^{d}\), and any \(z\in G_{Q,d}\setminus\{a\}\), we have

\[\left\|Q\cdot(x-z)\right\|_{2}\geq Q\left\|a-z\right\|_{2}-Q\left\|x-a\right\| _{2}\geq 1-\frac{1}{5}>\sqrt{1/8}>\sqrt{1/9}>\left\|Q\cdot(x-a)\right\|_{2},\]

which means that \(u(Q\cdot(x-z))=0\) and \(u(Q\cdot(x-a))=1\). Thus

\[f(x)=T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot u(Q\cdot(x-a ))+\sum_{z\in G_{Q,d}\setminus\{a\}}T(z)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta}} \cdot u(Q\cdot(x-z))\] \[=T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot 1+\sum_{z\in G_{Q,d} \setminus\{a\}}T(z)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot 0\] (A.125) \[=T(a)\cdot\frac{\mathrm{c}_{1}}{Q^{\beta}},\;\forall\;a\in G_{Q,d },\;x\in\mathscr{B}(a,\frac{1}{5Q})\cap[0,1]^{d}.\]

Now it remains to show that \(f\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d}\right)\). Let \(\boldsymbol{m}\in(\mathbb{N}\cup\{0\})^{d}\) be an arbitrary multi-index with \(\left\|\boldsymbol{m}\right\|_{1}=b\), and \(x,y\) be arbitrary points in \(\bigcup_{a\in G_{Q,d}}\left\{a+v\left|v\in(-\frac{1}{2Q},\frac{1}{2Q})^{d} \right.\right\}\). Then there exist \(a_{x},a_{y}\in G_{Q,d}\), such that \(x-a_{x}\in(-\frac{1}{2Q},\frac{1}{2Q})^{d}\) and \(y-a_{y}\in(-\frac{1}{2Q},\frac{1}{2Q})^{d}\). If \(a_{x}=a_{y}\), then it follows from (A.122) that

\[u_{\boldsymbol{m}}(Q\cdot(x-z))=u_{\boldsymbol{m}}(Q\cdot(y-z))=0,\;\forall\;z \in G_{Q,d}\setminus\{a_{x}\}\,,\]

which, together with the fact that \(\{Q\cdot(x-a_{x}),Q\cdot(y-a_{y})\}\subset(-\frac{1}{2},\frac{1}{2})^{d}\), yields

\[\begin{split}&|\mathrm{D}^{\boldsymbol{m}}f(x)-\mathrm{D}^{ \boldsymbol{m}}f(y)|\\ &=\left|T(a_{x})\cdot\frac{\mathrm{c}_{1}}{Q^{\beta-\left\| \boldsymbol{m}\right\|_{1}}}\cdot u_{\boldsymbol{m}}(Q\cdot(x-a_{x}))-T(a_{ y})\cdot\frac{\mathrm{c}_{1}}{Q^{\beta-\left\|\boldsymbol{m}\right\|_{1}}} \cdot u_{\boldsymbol{m}}(Q\cdot(y-a_{y}))\right|\\ &=\mathrm{c}_{1}\cdot\left|\frac{u_{\boldsymbol{m}}(Q\cdot(x-a_{ x}))-u_{\boldsymbol{m}}(Q\cdot(y-a_{y}))}{Q^{\lambda}}\right|\\ &\leq\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot\left\|Q\cdot(x-a_{ x})-Q\cdot(y-a_{y})\right\|_{2}^{\lambda}\cdot\sup_{z,z^{\prime}\in(-\frac{1}{2}, \frac{1}{2})^{d},z\neq z^{\prime},}\left|\frac{u_{\boldsymbol{m}}(z)-u_{ \boldsymbol{m}}(z^{\prime})}{\left\|z-z^{\prime}\right\|_{2}^{\lambda}}\right| \\ &\leq\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot\left\|Q\cdot(x-a_{ x})-Q\cdot(y-a_{y})\right\|_{2}^{\lambda}\cdot\mathrm{c}_{2}=\mathrm{c}_{1}\mathrm{c}_{2} \cdot\left\|x-y\right\|_{2}^{\lambda}.\end{split}\]

If, otherwise, \(a_{x}\neq a_{y}\), then it is easy to show that

\[\{t\cdot x+(1-t)\cdot y|t\in[0,1]\}\cap\left\{a_{x}+v\left|v\in[-\frac{1}{2Q}, \frac{1}{2Q}]^{d}\setminus(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\neq\varnothing,\]
\[\{t\cdot x+(1-t)\cdot y|t\in[0,1]\}\cap\left\{a_{y}+v\left|v\in[-\frac{1}{2Q}, \frac{1}{2Q}]^{d}\setminus(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\neq\varnothing.\]

In other words, the line segment joining points \(x\) and \(y\) intersects boundaries of rectangles \(\left\{a_{x}+v\left|v\in(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\) and \(\left\{a_{y}+v\left|v\in(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\). Take

\[x^{\prime}\in\{t\cdot x+(1-t)\cdot y|t\in[0,1]\}\cap\left\{a_{x}+v\left|v\in[- \frac{1}{2Q},\frac{1}{2Q}]^{d}\setminus(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\]

and

\[y^{\prime}\in\{t\cdot x+(1-t)\cdot y|t\in[0,1]\}\cap\left\{a_{y}+v\left|v\in[- \frac{1}{2Q},\frac{1}{2Q}]^{d}\setminus(-\frac{1}{2Q},\frac{1}{2Q})^{d}\right.\right\}\]

(cf. Figure A.8).

Obviously, we have that

\[\{Q\cdot(x-a_{x}),Q\cdot(x^{\prime}-a_{x}),Q\cdot(y-a_{y}),Q\cdot(y^{\prime}- a_{y})\}\subset[-\frac{1}{2},\frac{1}{2}]^{d}.\]

By (A.122), we have that

\[u_{\mathbf{m}}(Q\cdot(x-z))\cdot(1-\mathbb{1}_{\{a_{x}\}}(z))=u_{ \mathbf{m}}(Q\cdot(x^{\prime}-z))\] \[=u_{\mathbf{m}}(Q\cdot(y^{\prime}-z))=u_{\mathbf{m}}(Q\cdot(y-z))\cdot(1- \mathbb{1}_{\{a_{y}\}}(z))=0,\;\forall\;z\in G_{Q,d}.\]

Consequently,

\[|\mathrm{D}^{\mathbf{m}}f(x)-\mathrm{D}^{\mathbf{m}}f(y)|\leq|\mathrm{D}^ {\mathbf{m}}f(x)|+|\mathrm{D}^{\mathbf{m}}f(y)|\] \[=\left|T(a_{x})\cdot\frac{\mathrm{c}_{1}}{Q^{\beta-\|\mathbf{m}\|_{1 }}}\cdot u_{\mathbf{m}}(Q\cdot(x-a_{x}))\right|+\left|T(a_{y})\cdot\frac{\mathrm{ c}_{1}}{Q^{\beta-\|\mathbf{m}\|_{1}}}\cdot u_{\mathbf{m}}(Q\cdot(y-a_{y}))\right|\] \[=\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot|u_{\mathbf{m}}(Q\cdot(x-a_{ x}))|+\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot|u_{\mathbf{m}}(Q\cdot(y-a_{y}))|\] \[=\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot|u_{\mathbf{m}}(Q\cdot(x-a_{x }))-u_{\mathbf{m}}(Q\cdot(x^{\prime}-a_{x}))|\] \[\qquad+\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot|u_{\mathbf{m}}(Q\cdot (y-a_{y}))-u_{\mathbf{m}}(Q\cdot(y^{\prime}-a_{y}))|\] \[\leq\frac{\mathrm{c}_{1}}{Q^{\lambda}}\cdot\left\|Q\cdot(x-a_{x} )-Q\cdot(x^{\prime}-a_{x})\right\|_{2}^{\lambda}\cdot\sup_{z,z^{\prime}\in[- \frac{1}{2},\frac{1}{2}]^{d},z\neq z^{\prime},}\left|\frac{u_{\mathbf{m}}(z)-u_{ \mathbf{m}}(z^{\prime})}{\|z-z^{\prime}\|_{2}^{\lambda}}\right|\]

Figure A.8: Illustration of the points \(x\), \(y\), \(a_{x}\), \(a_{y}\), \(x^{\prime}\), \(y^{\prime}\) when \(Q=3\) and \(d=2\).
\[\frac{\mathrm{d}P_{\eta_{1},\mathscr{Q}}}{\mathrm{d}P_{\eta_{2}, \mathscr{Q}}}(x,y)=\begin{cases}\frac{\eta_{1}(x)}{\eta_{2}(x)},&\text{if $y=1$},\\ \frac{1-\eta_{1}(x)}{1-\eta_{2}(x)},&\text{if $y=-1$}.\end{cases}\]

Proof.: Let \(f:[0,1]^{d}\times\{-1,1\}\to[0,\infty),\ (x,y)\mapsto\begin{cases}\frac{\eta_{1}(x)}{ \eta_{2}(x)},&\text{if $y=1$},\\ \frac{1-\eta_{1}(x)}{1-\eta_{2}(x)},&\text{if $y=-1$}.\end{cases}\) Then we have that \(f\) is well defined and measurable. For any Borel subset \(S\) of \([0,1]^{d}\times\{-1,1\}\), let \(S_{1}:=\big{\{}\,x\in[0,1]^{d}\big{|}\,(x,1)\in S\big{\}}\), and \(S_{2}:=\big{\{}\,x\in[0,1]^{d}\big{|}\,(x,-1)\in S\big{\}}\). Obviouls, \(S_{1}\times\{1\}\) and \(S_{2}\times\{-1\}\) are measurable and disjoint. Besides, it is easy to verify that \(S=(S_{1}\times\{1\})\cup(S_{2}\times\{-1\})\). Therefore,

\[\int_{S}f(x,y)\mathrm{d}P_{\eta_{2},\mathscr{Q}}(x,y)\] \[=\int_{S_{1}}\int_{\{1\}}f(x,y)\mathrm{d}\mathscr{M}_{\eta_{2}(x )}(y)\mathrm{d}\mathscr{Q}(x)+\int_{S_{2}}\int_{\{-1\}}f(x,y)\mathrm{d} \mathscr{M}_{\eta_{2}(x)}(y)\mathrm{d}\mathscr{Q}(x)\] \[=\int_{S_{1}}\eta_{2}(x)f(x,1)\mathrm{d}\mathscr{Q}(x)+\int_{S_{ 2}}(1-\eta_{2}(x))f(x,-1)\mathrm{d}\mathscr{Q}(x)\] \[=\int_{S_{1}}\eta_{1}(x)\mathrm{d}\mathscr{Q}(x)+\int_{S_{2}}(1 -\eta_{1}(x))\mathrm{d}\mathscr{Q}(x)\]
\[=\int_{S_{1}}\int_{\{1\}}\mathrm{d}\mathscr{M}_{\eta_{1}(x)}(y)\mathrm{d} \mathscr{Q}(x)+\int_{S_{2}}\int_{\{-1\}}\mathrm{d}\mathscr{M}_{\eta_{1}(x)}(y) \mathrm{d}\mathscr{Q}(x)\] \[=P_{\eta_{1},\mathscr{Q}}(S_{1}\times\{1\})+P_{\eta_{1},\mathscr{Q }}(S_{2}\times\{-1\})=P_{\eta_{1},\mathscr{Q}}(S).\]

Since \(S\) is arbitrary, we deduce that \(P_{\eta_{1},\mathscr{Q}}<<P_{\eta_{2},\mathscr{Q}}\), and \(\frac{\mathrm{d}P_{\eta_{1},\mathscr{Q}}}{\mathrm{d}P_{\eta_{2},\mathscr{Q}}}=f\). This completes the proof. 

**Lemma A.20**.: _Let \(\varepsilon\in(0,\frac{1}{5}]\), \(\mathscr{Q}\) be a Borel probability on \([0,1]^{d}\), and \(\eta_{1}:[0,1]^{d}\to[\varepsilon,3\varepsilon]\), \(\eta_{2}:[0,1]^{d}\to[\varepsilon,3\varepsilon]\) be two measurable functions. Then_

\[\mathrm{KL}(P_{\eta_{1},\mathscr{Q}}||P_{\eta_{2},\mathscr{Q}})\leq 9\varepsilon.\]

Proof.: By Lemma A.19,

\[\begin{split}&\mathrm{KL}(P_{\eta_{1},\mathscr{Q}}||P_{\eta_{2}, \mathscr{Q}})\\ &=\int_{[0,1]^{d}\times\{-1,1\}}\log\left(\frac{\eta_{1}(x)}{ \eta_{2}(x)}\cdot\mathbb{1}_{\{1\}}(y)+\frac{1-\eta_{1}(x)}{1-\eta_{2}(x)} \cdot\mathbb{1}_{\{-1\}}(y)\right)\mathrm{d}P_{\eta_{1},\mathscr{Q}}(x,y)\\ &=\int_{[0,1]^{d}}\left(\eta_{1}(x)\log\left(\frac{\eta_{1}(x)}{ \eta_{2}(x)}\right)+(1-\eta_{1}(x))\log\left(\frac{1-\eta_{1}(x)}{1-\eta_{2}(x )}\right)\right)\mathrm{d}\mathscr{Q}(x)\\ &\leq\int_{[0,1]^{d}}\left(3\varepsilon\cdot\left|\log\left(\frac {\eta_{1}(x)}{\eta_{2}(x)}\right)\right|+\left|\log\left(\frac{1-\eta_{1}(x)}{ 1-\eta_{2}(x)}\right)\right|\right)\mathrm{d}\mathscr{Q}(x)\\ &\leq\int_{[0,1]^{d}}\left(3\varepsilon\cdot\log\left(\frac{3 \varepsilon}{\varepsilon}\right)+\log\left(\frac{1-\varepsilon}{1-3\varepsilon }\right)\right)\mathrm{d}\mathscr{Q}(x)\\ &=\log\left(1+\frac{2\varepsilon}{1-3\varepsilon}\right)+3 \varepsilon\cdot\log 3\leq\frac{2\varepsilon}{1-3\varepsilon}+4\varepsilon\leq 9 \varepsilon.\end{split}\]

**Lemma A.21**.: _Let \(m\in\mathbb{N}\cap(1,\infty)\), \(\Omega\) be a set with \(\#(\Omega)=m\), and \(\left\{0,1\right\}^{\Omega}\) be the set of all functions mapping from \(\Omega\) to \(\left\{0,1\right\}\). Then there exists a subset \(E\) of \(\left\{0,1\right\}^{\Omega}\), such that \(\#(E)\geq 1+2^{m/8}\), and_

\[\#\left(\left\{x\in\Omega\right|f(x)\neq g(x)\right\}\right)\geq\frac{m}{8}, \;\forall\;f\in E,\;\forall\;g\in E\setminus\left\{f\right\}.\]

Proof.: If \(m\leq 8\), then \(E=\left\{0,1\right\}^{\Omega}\) have the desired properties. The proof for the case \(m>8\) can be found in Lemma 2.9 of [44]. 

**Lemma A.22**.: _Let \(\phi\) be the logistic loss,_

\[\begin{split}\mathcal{J}:(0,1)^{2}&\to\mathbb{R}\\ (x,y)&\mapsto(x+y)\log\frac{2}{x+y}+(2-x-y)\log\frac{ 2}{2-x-y}\\ &\qquad-\left(x\log\frac{1}{x}+(1-x)\log\frac{1}{1-x}+y\log\frac{ 1}{y}+(1-y)\log\frac{1}{1-y}\right),\end{split}\] (A.127)

\(\mathscr{Q}\) _be a Borel probability measure on \([0,1]^{d}\), and \(\eta_{1}:[0,1]^{d}\to(0,1)\), \(\eta_{2}:[0,1]^{d}\to(0,1)\) be two measurable functions. Then there hold_

\[\mathcal{J}(x,y)=\mathcal{J}(y,x)\geq 0,\;\forall\;x\in(0,1),\;y\in(0,1),\] (A.128)

\[\frac{\varepsilon}{4}<\mathcal{J}(\varepsilon,3\varepsilon)=\mathcal{J}(3 \varepsilon,\varepsilon)<\varepsilon,\;\forall\;\varepsilon\in(0,\frac{1}{6}],\] (A.129)

_and_

\[\int_{[0,1]^{d}}\mathcal{J}(\eta_{1}(x),\eta_{2}(x))\mathrm{d}\mathscr{Q}(x) \leq\inf_{f\in\mathcal{F}_{d}}\left|\mathcal{E}_{\eta_{1},\mathscr{Q}}^{\phi}( f)+\mathcal{E}_{\eta_{2},\mathscr{Q}}^{\phi}(f)\right|.\] (A.130)
Proof.: Let \(g:(0,1)\to(0,\infty),x\mapsto x\log\frac{1}{x}+(1-x)\log\frac{1}{1-x}\). Then it is easy to verify that \(g\) is concave (i.e., \(-g\) is convex), and

\[\mathcal{J}(x,y)=2g(\frac{x+y}{2})-g(x)-g(y),\;\forall\;x\in(0,1),\;y\in(0,1).\]

This yields (A.128).

An elementary calculation gives

\[\mathcal{J}(\varepsilon,3\varepsilon)=\mathcal{J}(3\varepsilon,\varepsilon)\] \[=\varepsilon\log\frac{27}{16}-\log\left(\frac{(1-2\varepsilon)^{ 2}}{(1-\varepsilon)(1-3\varepsilon)}\right)+4\varepsilon\log(1-2\varepsilon) -\varepsilon\log(1-\varepsilon)-3\varepsilon\log(1-3\varepsilon)\] \[\xlongetimes\text{Taylor expansion}\;\varepsilon\log\frac{27}{16} +\sum_{k=2}^{\infty}\frac{3^{k}+1-2\cdot 2^{k}}{k\cdot(k-1)}\cdot\varepsilon^{k},\; \forall\;\varepsilon\in(0,1/3).\]

Therefore,

\[\frac{\varepsilon}{4}<\varepsilon\log\frac{27}{16}\leq\varepsilon \log\frac{27}{16}+\sum_{k=2}^{\infty}\frac{1+\left(\left(\frac{3}{2}\right)^{ k}-2\right)\cdot 2^{k}}{k\cdot(k-1)}\cdot\varepsilon^{k}=\varepsilon\log\frac{27}{16}+\sum_{k=2}^{ \infty}\frac{3^{k}+1-2\cdot 2^{k}}{k\cdot(k-1)}\cdot\varepsilon^{k}\] \[=\mathcal{J}(\varepsilon,3\varepsilon)=\mathcal{J}(3\varepsilon, \varepsilon)=\varepsilon\log\frac{27}{16}+\sum_{k=2}^{\infty}\frac{3^{k}+1-2 \cdot 2^{k}}{k\cdot(k-1)}\cdot\varepsilon^{k}\leq\varepsilon\log\frac{27}{16}+ \sum_{k=2}^{\infty}\frac{3^{k}-7}{k\cdot(k-1)}\cdot\varepsilon^{k}\] \[=\varepsilon\log\frac{27}{16}+\varepsilon^{2}+\varepsilon\cdot \sum_{k=3}^{\infty}\frac{3^{k}-7}{k\cdot(k-1)}\cdot\varepsilon^{k-1}\leq \varepsilon\log\frac{27}{16}+\varepsilon/6+\varepsilon\cdot\sum_{k=3}^{ \infty}\frac{3^{k}}{3\cdot(3-1)}\cdot\left(\frac{1}{6}\right)^{k-1}\] \[=\left(\frac{1}{6}+\frac{1}{4}+\log\frac{27}{16}\right)\cdot \varepsilon<\varepsilon,\;\forall\;\varepsilon\in(0,1/6],\]

which proves (A.129).

Define \(f_{1}:[0,1]^{d}\to\mathbb{R},x\mapsto\log\frac{\eta_{1}(x)}{1-\eta_{1}(x)}\) and \(f_{2}:[0,1]^{d}\to\mathbb{R},x\mapsto\log\frac{\eta_{2}(x)}{1-\eta_{2}(x)}\). Then it is easy to verify that

\[\mathcal{R}^{\phi}_{P_{\eta_{1},2}}(f_{i})=\int_{[0,1]^{d}}g(\eta_{i}(x)) \mathrm{d}\mathscr{Q}(x)\in(0,\infty),\;\forall\;i\in\left\{1,2\right\},\]

and

\[\inf\left\{a\phi(t)+(1-a)\phi(-t)\big{|}t\in\mathbb{R}\right\}=g(a),\;\forall \;a\in(0,1).\]

Consequently, for any measurable function \(f:[0,1]^{d}\to\mathbb{R}\), there holds

\[\mathcal{E}^{\phi}_{P_{\eta_{1},2}}(f)+\mathcal{E}^{\phi}_{P_{ \eta_{2},2}}(f)\geq\mathcal{R}^{\phi}_{P_{\eta_{1},2}}(f)-\mathcal{R}^{\phi}_ {P_{\eta_{1},2}}(f)+\mathcal{R}^{\phi}_{P_{\eta_{2},2}}(f)-\mathcal{R}^{\phi} _{P_{\eta_{2},2}}(f_{2})\] \[=\int_{[0,1]^{d}}\left((\eta_{1}(x)+\eta_{2}(x))\phi(f(x))+(2- \eta_{1}(x)-\eta_{2}(x))\phi(-f(x))\,\mathrm{d}\mathscr{Q}(x)\right.\] \[\qquad\qquad\qquad-\mathcal{R}^{\phi}_{P_{\eta_{1},2}}(f_{1})- \mathcal{R}^{\phi}_{P_{\eta_{2},2}}(f_{2})\] \[\geq\int_{[0,1]^{d}}2\cdot\inf\left\{\frac{\eta_{1}(x)+\eta_{2}(x )}{2}\phi(t)+(1-\frac{\eta_{1}(x)+\eta_{2}(x)}{2})\phi(-t)\bigg{|}\,t\in \mathbb{R}\right\}\mathrm{d}\mathscr{Q}(x)\] \[\qquad\qquad\qquad-\mathcal{R}^{\phi}_{P_{\eta_{1},2}}(f_{1})- \mathcal{R}^{\phi}_{P_{\eta_{2},2}}(f_{2})\] \[=\int_{[0,1]^{d}}2g(\frac{\eta_{1}(x)+\eta_{2}(x)}{2})\mathrm{d} \mathscr{Q}(x)-\mathcal{R}^{\phi}_{P_{\eta_{1},2}}(f_{1})-\mathcal{R}^{\phi} _{P_{\eta_{2},2}}(f_{2})\] \[=\int_{[0,1]^{d}}\left(2g(\frac{\eta_{1}(x)+\eta_{2}(x)}{2})-g( \eta_{1}(x))-g(\eta_{2}(x))\right)\mathrm{d}\mathscr{Q}(x)\] \[=\int_{[0,1]^{d}}\mathcal{J}(\eta_{1}(x),\eta_{2}(x))\mathrm{d} \mathscr{Q}(x).\]

This proves (A.130).
Proof of Theorem 2.6 and Corollary 2.1.: We first prove Theorem 2.6. Let \(n\) be an arbitrary integer greater than \(\left\lfloor\frac{7}{1-A}\right\rfloor^{\frac{d_{*}+\beta+(1-\beta)\beta}{-(1, \beta)^{d_{*}}}}\). Take \(b:=\lceil\beta\rceil-1\), \(\lambda:=\beta+1-\lceil\beta\rceil\), \(Q:=\left\lfloor n^{\frac{1}{d_{*}+\beta+(1\wedge\beta)\vartheta}}\right\rfloor+1,\mathtt{M}:=\left\lceil 2^{Q^{d_{*}}/8}\right\rceil\),

\[G_{Q,d_{*}}:=\left\{(\frac{k_{1}}{2Q},\ldots,\frac{k_{d_{*}}}{2Q})^{\top} \right|k_{1},\ldots,k_{d_{*}}\text{ are odd integers}\right\}\cap[0,1]^{d_{*}},\]

and \(\mathcal{J}\) to be the function defined in (A.127). Note that \(\#\left(G_{Q,d_{*}}\right)=Q^{d_{*}}\). Thus it follows from Lemma A.21 that there exist functions \(T_{j}:G_{Q,d_{*}}\to\{-1,1\}\), \(j=0,1,2,\ldots,\mathtt{M}\), such that

\[\#\left(\left\{a\in G_{Q,d_{*}}|\,T_{i}(a)\neq T_{j}(a)\right\}\right)\geq \frac{Q^{d_{*}}}{8},\ \forall\ 0\leq i<j\leq\mathtt{M}.\] (A.131)

According to Lemma A.18, for each \(j\in\{0,1,\ldots,\mathtt{M}\}\), there exists an \(f_{j}\in\mathcal{B}_{\frac{r\wedge 1}{777}}^{\beta}\left([0,1]^{d_{*}}\right)\), such that

\[\frac{\mathrm{c}_{1}}{Q^{\beta}}=\|f_{j}\|_{[0,1]^{d_{*}}}\leq\|f_{j}\|_{ \mathcal{C}^{b,\lambda}([0,1]^{d_{*}})}\leq\frac{1\wedge r}{777},\] (A.132)

and

\[f_{j}(x)=\frac{\mathrm{c}_{1}}{Q^{\beta}}\cdot T_{j}(a),\ \forall\ a\in G_{Q,d_{*}}, \ x\in\mathscr{B}(a,\frac{1}{5Q})\cap[0,1]^{d_{*}},\] (A.133)

where \(\mathrm{c}_{1}\in(0,\frac{1}{9999})\) only depends on \((d_{*},\beta,r)\). Define

\[g_{j}:[0,1]^{d_{*}}\to\mathbb{R},x\mapsto\frac{\mathrm{c}_{1}}{Q^{\beta}}+f_{ j}(x).\]

It follows from (A.132) that

\[\mathbf{ran}(g_{j})\subset\left[0,\frac{2\mathrm{c}_{1}}{Q^{\beta}}\right] \subset\left[0,2\cdot\frac{1\wedge r}{777}\right]\subset[0,1]\] (A.134)

and

\[\frac{\mathrm{c}_{1}}{Q^{\beta}}+\|g_{j}\|_{\mathcal{C}^{b,\lambda}([0,1]^{d_ {*}})}\leq\frac{2\mathrm{c}_{1}}{Q^{\beta}}+\|f_{j}\|_{\mathcal{C}^{b,\lambda }([0,1]^{d_{*}})}\leq 2\cdot\frac{1\wedge r}{777}+\frac{1\wedge r}{777}<r,\] (A.135)

meaning that

\[g_{j}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d_{*}}\right)\text{ and }g_{j}+ \frac{\mathrm{c}_{1}}{Q^{\beta}}\in\mathcal{B}_{r}^{\beta}\left([0,1]^{d_{*}} \right).\] (A.136)

We then define

\[h_{0,j}:[0,1]^{d}\to[0,1],\ (x_{1},\ldots,x_{d})^{\top}\mapsto g_{j}(x_{1}, \ldots,x_{d_{*}})\]

if \(q=0\), and define

\[h_{0,j}:[0,1]^{d}\to[0,1]^{K},\ (x_{1},\ldots,x_{d})^{\top}\mapsto(g_{j}(x_{1}, \ldots,x_{d_{*}}),0,0,\ldots,0)^{\top}\]

if \(q>0\). Note that \(h_{0,j}\) is well defined because \(d_{*}\leq d\) and \(\mathbf{ran}(g_{j})\subset[0,1]\). Take

\[\varepsilon=\frac{1}{2}\cdot\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{ \beta-1}(1\wedge\beta)^{k}}\cdot\left|\frac{2\mathrm{c}_{1}}{Q^{\beta}} \right|^{(1\wedge\beta)^{q}}.\]

From (A.132) we see that

\[0<\varepsilon\leq\frac{1\wedge r}{777}.\] (A.137)

For all real number \(t\), define the function

\[u_{t}:[0,1]^{d_{*}}\to\mathbb{R},\ (x_{1},\ldots,x_{d_{*}})^{\top}\mapsto t+ \frac{1\wedge r}{777}\cdot|x_{1}|^{(1\wedge\beta)}.\]
Then it follows from (A.137) and the elementary inequality

\[\left|\left|z_{1}\right|^{w}-\left|z_{2}\right|^{w}\right|\leq\left|z_{1}-z_{2} \right|^{w},\;\forall\;z_{1}\in\mathbb{R},z_{2}\in\mathbb{R},w\in(0,1]\]

that

\[\begin{split}&\max\left\{\left\|u_{e}\right\|_{[0,1]^{d_{*}}}, \left\|u_{0}\right\|_{[0,1]^{d_{*}}}\right\}\leq\max\left\{\left\|u_{e}\right\| _{\mathcal{C}^{\mathsf{R},\lambda}([0,1]^{d_{*}})},\left\|u_{0}\right\|_{ \mathcal{C}^{\mathsf{R},\lambda}([0,1]^{d_{*}})}\right\}\\ &\leq\left\|u_{0}\right\|_{\mathcal{C}^{\mathsf{R},\lambda}([0,1] ^{d_{*}})}+\varepsilon\leq\frac{1\wedge r}{777}\cdot 2+\varepsilon\leq\frac{1 \wedge r}{777}\cdot 2+\frac{1\wedge r}{777}<r\wedge 1,\end{split}\] (A.138)

which means that

\[\mathbf{ran}(u_{0})\cup\mathbf{ran}(u_{\varepsilon})\subset[0,1],\] (A.139)

and

\[\left\{u_{0},u_{\varepsilon}\right\}\subset\mathcal{B}_{r}^{\beta}\left([0,1] ^{d_{*}}\right).\] (A.140)

Next, for each \(i\in\mathbb{N}\), define

\[h_{i}:[0,1]^{K} \to\mathbb{R},\] \[(x_{1},\ldots,x_{K})^{\top} \mapsto u_{0}(x_{1},\ldots,x_{d_{*}})\]

if \(i=q>0\), and define

\[h_{i}:[0,1]^{K}\to\mathbb{R}^{K},\;(x_{1},\ldots,x_{K})^{\top}\mapsto(u_{0}(x _{1},\ldots,x_{d_{*}}),0,0,\ldots,0)^{\top}\]

otherwise. It follows from (A.139) that \(\mathbf{ran}(h_{i})\subset[0,1]\) if \(i=q>0\), and \(\mathbf{ran}(h_{i})\subset[0,1]^{K}\) otherwise. Thus, for each \(j\in\{0,1,\ldots,\mathsf{M}\}\), we can well define

\[\eta_{j}:[0,1]^{d}\to\mathbb{R},\;x\mapsto\varepsilon+h_{q}\circ h_{q-1}\circ \cdots\circ h_{3}\circ h_{2}\circ h_{1}\circ h_{0,j}(x).\]

We then deduce from (A.136) and (A.140) that

\[\eta_{j}\in\mathcal{G}_{d}^{\mathbf{CH}}(q,K,d_{*},\beta,r),\;\forall\;j\in\{ 0,1,\ldots,\mathsf{M}\}\,.\] (A.141)

Moreover, an elementary calculation gives

\[\begin{split}&\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{q- 1}(1\wedge\beta)^{k}}\cdot\left|g_{j}(x_{1},\ldots,x_{d_{*}})\right|^{(1\wedge \beta)^{q}}+\varepsilon\\ &=\eta_{j}(x_{1},\ldots,x_{d}),\;\forall\;(x_{1},\ldots,x_{d}) \in[0,1]^{d},\;\forall\;j\in\{0,1,\ldots,\mathsf{M}\}\,,\end{split}\] (A.142)

which, together with (A.134), yields

\[\begin{split}& 0<\varepsilon\leq\eta_{j}(x_{1},\ldots,x_{d})\leq \left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{q-1}(1\wedge\beta)^{k}}\cdot \left|\frac{2c_{1}}{Q^{\beta}}\right|^{(1\wedge\beta)^{q}}+\varepsilon=2 \varepsilon+\varepsilon\\ &=3\varepsilon\leq\left|\frac{3c_{1}}{Q^{\beta}}\right|^{(1 \wedge\beta)^{q}}<\frac{1}{Q^{\beta-(1\wedge\beta)^{q}}}\leq\frac{1}{n^{\frac{ \beta-(1\wedge\beta)^{q}}{4+\beta^{q}(1\wedge\beta)^{q}}}}\leq\frac{1-A}{7}< \frac{1-A}{2}\\ &<1,\;\forall\;(x_{1},\ldots,x_{d})\in[0,1]^{d},\;\forall\;j\in\{ 0,1,\ldots,\mathsf{M}\}\,.\end{split}\]

Consequently,

\[\mathbf{ran}(\eta_{j})\subset[\varepsilon,3\varepsilon]\subset(0,1),\; \forall\;j\in\{0,1,\ldots,\mathsf{M}\}\,,\] (A.143)

and

\[\left\{\left.x\in[0,1]^{d}\right|\left|2\eta_{j}(x)-1\right|\leq A\right\}= \varnothing,\;\forall\;j\in\{0,1,\ldots,\mathsf{M}\}\,.\] (A.144)

Combining (A.141), (A.143), and (A.144), we obtain

\[P_{j}:=P_{\eta_{j}}\in\mathcal{H}_{5,A,q,K,d_{*}}^{d,\beta,r},\;\forall\;j\in \{0,1,2,\ldots,\mathsf{M}\}\,.\] (A.145)
By (A.133) and (A.142), for any \(0\leq i<j\leq\mathtt{M}\), any \(a\in G_{Q,d_{*}}\) with \(T_{i}(a)\neq T_{j}(a)\), and any \(x\in[0,1]^{d}\) with \((x)_{\{1,2,\ldots,d_{*}\}}\in\mathscr{B}(a,\frac{1}{5Q})\), there holds

\[\mathcal{J}(\eta_{i}(x),\eta_{j}(x))\] \[=\mathcal{J}\left(\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0} ^{-1}(1\wedge\beta)^{k}}\cdot\left|\frac{c_{1}}{Q^{\beta}}+T_{i}(a)\cdot\frac{ c_{1}}{Q^{\beta}}\right|^{(1\wedge\beta)^{q}}+\varepsilon,\right.\] \[\left.\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{-1}(1\wedge \beta)^{k}}\cdot\left|\frac{c_{1}}{Q^{\beta}}+T_{j}(a)\cdot\frac{c_{1}}{Q^{ \beta}}\right|^{(1\wedge\beta)^{q}}+\varepsilon\right)\] \[=\mathcal{J}\left(\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^ {-1}(1\wedge\beta)^{k}}\cdot\left|\frac{2c_{1}}{Q^{\beta}}\right|^{(1\wedge \beta)^{q}}+\varepsilon,\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{-1}(1 \wedge\beta)^{k}}\cdot|0|^{(1\wedge\beta)^{q}}+\varepsilon\right)\] \[=\mathcal{J}(2\varepsilon+\varepsilon,\varepsilon)=\mathcal{J}( \varepsilon,3\varepsilon).\]

Thus it follows from Lemma A.22 and (A.131) that

\[\inf_{f\in\mathcal{F}_{d}}\left(\mathcal{E}_{P_{j}}^{\phi}(f)+ \mathcal{E}_{P_{1}}^{\phi}(f)\right)\geq\int_{[0,1]^{d}}\mathcal{J}(\eta_{i}( x),\eta_{j}(x))\mathrm{d}x\] \[\geq\sum_{a\in G_{Q,d_{*}}:\,T_{j}(a)\neq T_{i}(a)}\int_{[0,1]^{d} }\mathcal{J}(\eta_{i}(x),\eta_{j}(x))\cdot\mathbb{1}_{\mathscr{B}(a,\frac{1} {5Q})}\big{(}(x)_{\{1,\ldots,d_{*}\}}\big{)}\mathrm{d}x\] \[=\sum_{a\in G_{Q,d_{*}}:\,T_{j}(a)\neq T_{i}(a)}\int_{[0,1]^{d}} \mathcal{J}(\varepsilon,3\varepsilon)\cdot\mathbb{1}_{\mathscr{B}(a,\frac{1} {5Q})}\big{(}(x)_{\{1,\ldots,d_{*}\}}\big{)}\mathrm{d}x\] \[\geq\sum_{a\in G_{Q,d_{*}}:\,T_{j}(a)\neq T_{i}(a)}\int_{[0,1]^{d} }\frac{\varepsilon}{4}\cdot\mathbb{1}_{\mathscr{B}(a,\frac{1}{5Q})}\big{(}(x )_{\{1,\ldots,d_{*}\}}\big{)}\mathrm{d}x\] (A.146) \[=\frac{\#\big{(}\big{\{}a\in G_{Q,d_{*}}|T_{j}(a)\neq T_{i}(a) \big{\}}\big{)}}{Q^{d_{*}}}\cdot\int_{\mathscr{B}(\mathbf{0},\frac{1}{5})} \frac{\varepsilon}{4}\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{d_{*}}\] \[\geq\frac{1}{8}\cdot\int_{\mathscr{B}(\mathbf{0},\frac{1}{5})} \frac{\varepsilon}{4}\mathrm{d}x_{1}\mathrm{d}x_{2}\cdots\mathrm{d}x_{d_{*}} \geq\frac{1}{8}\cdot\int_{[-\frac{1}{\sqrt{254}\varepsilon},\frac{1}{\sqrt{25 4}\varepsilon}]^{d_{*}}}\frac{\varepsilon}{4}\mathrm{d}x_{1}\mathrm{d}x_{2} \cdots\mathrm{d}x_{d_{*}}\] \[\geq\left|\frac{2}{\sqrt{25d_{*}}}\right|^{d_{*}}\cdot\frac{ \varepsilon}{32}=:s,\;\forall\;0\leq i<j\leq\mathtt{M}.\]

Let \(\hat{f}_{n}\) be an arbitrary \(\mathcal{F}_{d}\)-valued statistic on \(([0,1]^{d}\times\{-1,1\})^{n}\) from the sample \(\{(X_{i},Y_{i})\}_{i=1}^{n}\), and let \(\mathcal{T}:([0,1]^{d}\times\{-1,1\})^{n}\to\mathcal{F}_{d}\) be the map associated with \(\hat{f}_{n}\), i.e., \(\hat{f}_{n}=\mathcal{T}(X_{1},Y_{1},\ldots,X_{n},Y_{n})\). Take

\[\mathcal{T}_{0}:\mathcal{F}_{d}\to\{0,1,\ldots,\mathtt{M}\}\,,f\mapsto\inf\mathop {\arg\min}_{j\in\{0,1,\ldots,\mathtt{M}\}}\mathcal{E}_{P_{j}}^{\phi}(f),\]

i.e., \(\mathcal{T}_{0}(f)\) is the smallest integer \(j\in\{0,\ldots,\mathtt{M}\}\) such that \(\mathcal{E}_{P_{j}}^{\phi}(f)\leq\mathcal{E}_{P_{i}}^{\phi}(f)\) for any \(i\in\{0,\ldots,\mathtt{M}\}\). Define \(g_{*}=\mathcal{T}_{0}\circ\mathcal{T}\). Note that, for any \(j\in\{0,1,\ldots,\mathtt{M}\}\) and any \(f\in\mathcal{F}_{d}\) there holds

\[\mathcal{T}_{0}(f)\neq j\stackrel{{(\ref{eq:146})}}{{\Rightarrow}} \mathcal{E}_{P_{7_{0}(f)}}^{\phi}(f)+\mathcal{E}_{P_{j}}^{\phi}(f)\geq s \Rightarrow\mathcal{E}_{P_{j}}^{\phi}(f)+\mathcal{E}_{P_{j}}^{\phi}(f)\geq s \Rightarrow\mathcal{E}_{P_{j}}^{\phi}(f)\geq\frac{s}{2},\]

which, together with the fact that the range of \(\mathcal{T}\) is contained in \(\mathcal{F}_{d}\), yields

\[\mathbb{1}_{\mathbb{R}\setminus\{j\}}(g_{*}(z))=\mathbb{1}_{ \mathbb{R}\setminus\{j\}}(\mathcal{T}_{0}(\mathcal{T}((z))))\] (A.147) \[\leq\mathbb{1}_{[\frac{z}{2},\infty]}(\mathcal{E}_{P_{j}}^{\phi}( \mathcal{T}(z))),\;\forall\;z\in([0,1]^{d}\times\{-1,1\})^{n},\;\forall\;j\in\{0,1, \ldots,\mathtt{M}\}\,.\]
Consequently,

\[\sup_{P\in\mathcal{H}^{d,\beta,r}_{5,A,q,K,d_{*}}}\mathbf{E}_{P^{\otimes n }}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\geq\sup_{j\in\{0,1,\dots, \mathsf{M}\}}\mathbf{E}_{P_{j}^{\otimes n}}\left[\mathcal{E}^{\phi}_{P_{j}}(\hat{f} _{n})\right]\] (A.148) \[=\sup_{j\in\{0,1,\dots,\mathsf{M}\}}\int\mathcal{E}^{\phi}_{P_{j} }(\mathcal{T}(z))\mathrm{d}P_{j}^{\otimes n}(z)\geq\sup_{j\in\{0,1,\dots, \mathsf{M}\}}\int\frac{\mathbb{1}_{[\frac{z}{2},\infty]}(\mathcal{E}^{\phi}_{P _{j}}(\mathcal{T}(z)))}{2/s}\mathrm{d}P_{j}^{\otimes n}(z)\] \[\geq\sup_{j\in\{0,1,\dots,\mathsf{M}\}}\int\frac{\mathbb{1}_{ \mathbb{R}\setminus\{j\}}(g_{*}(z))}{2/s}\mathrm{d}P_{j}^{\otimes n}(z)=\sup_ {j\in\{0,1,\dots,\mathsf{M}\}}\frac{P_{j}^{\otimes n}\left(g_{*}\neq j\right)} {2/s}\] \[\geq\frac{s}{2}\cdot\inf\left\{\sup_{j\in\{0,1,\dots,\mathsf{M}\}} P_{j}^{\otimes n}\left(g\neq j\right)\left|g\text{ is a measurable function from}\right.\right.\]

where the first inequality follows from (A.145) and the third inequality follows from (A.147).

We then use Proposition 2.3 of [44] to bound the right hand side of (A.148). By Lemma A.20, we have that

\[\frac{1}{\mathsf{M}}\cdot\sum_{j=1}^{\mathsf{M}}\mathrm{KL}(P_{j}^{\otimes n }||P_{0}^{\otimes n})=\frac{n}{\mathsf{M}}\cdot\sum_{j=1}^{\mathsf{M}} \mathrm{KL}(P_{j}||P_{0})\leq\frac{n}{\mathsf{M}}\cdot\sum_{j=1}^{\mathsf{M}} 9\varepsilon=9n\varepsilon,\]

which, together with Proposition 2.3 of [44], yields

\[\inf\Bigg{\{}\sup_{j\in\{0,1,\dots,\mathsf{M}\}}P_{j}^{\otimes n }\left(g\neq j\right)\left|g\text{ is a measurable function from}\right.\] \[\geq\sup_{\tau\in(0,1)}\left(\frac{\tau\mathsf{M}}{1+\tau\mathsf{ M}}\cdot\left(1+\frac{9n\varepsilon+\sqrt{\frac{9n\varepsilon}{2}}}{\log \tau}\right)\right)\geq\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}}\cdot \left(1+\frac{9n\varepsilon+\sqrt{\frac{9n\varepsilon}{2}}}{\log\frac{1}{ \sqrt{\mathsf{M}}}}\right)\] \[\geq\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}}\cdot\left(1- \left|\frac{9n\varepsilon+\sqrt{\frac{9n\varepsilon}{2}}}{\log\frac{1}{\sqrt{ \mathsf{M}}}}\right|\right)\geq\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}} \cdot\left(1-\left|\frac{9n\varepsilon+\frac{1}{10}+12n\varepsilon}{\log\sqrt{ \mathsf{M}}}\right|\right)\] \[\geq\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}}\cdot\left(1- \left|\frac{21n\varepsilon}{\frac{1}{2}\log\left(2^{Q^{d_{*}}/8}\right)} \right|-\frac{1/10}{\log\sqrt{2}}\right)\] \[=\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}}\cdot\left(1-\left| \frac{336n}{Q^{d_{*}}\log 2}\cdot\frac{1}{2}\cdot\frac{1}{777}\right|-\frac{1/10}{\log\sqrt{2}} \right)\geq\frac{\sqrt{\mathsf{M}}}{1+\sqrt{\mathsf{M}}}\cdot\frac{1}{3}\geq \frac{1}{6}.\]

Combining this with (A.148), we obtain that

\[\sup_{P\in\mathcal{H}^{d,\beta,r}_{5,A,d_{*}}}\mathbf{E}_{P^{\otimes n }}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\geq\frac{s}{2}\cdot\frac{1} {6}\] \[=\left|\frac{2}{\sqrt{25d_{*}}}\right|^{d_{*}}\cdot\frac{|2c_{1}|^ {(1\wedge\beta)^{q}}}{768}\cdot\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0}^{ \beta-1}(1\wedge\beta)^{k}}\cdot\left|\frac{1}{Q^{\beta}}\right|^{(1\wedge \beta)^{q}}\] \[\geq\left|\frac{2}{\sqrt{25d_{*}}}\right|^{d_{*}}\cdot\frac{|2c_{ 1}|^{(1\wedge\beta)^{q}}}{768}\cdot\left|\frac{1\wedge r}{777}\right|^{\sum_{k=0 }^{\beta-1}(1\wedge\beta)^{q}}\cdot\frac{1}{2^{\beta\cdot(1\wedge\beta)^{q}}}.\]

Since \(\hat{f}_{n}\) is arbitrary, we deduce that

\[\inf_{\hat{f}_{n}}\sup_{P\in\mathcal{H}^{d,\beta,r}_{5,A,q,K,d_{*}}}\mathbf{E}_{P^{ \otimes n}}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\geq c_{0}n^{-\frac {\beta\cdot(1\wedge\beta)^{q}}{d_{*}+\beta\cdot(1\wedge\beta)^{q}}}\]
with \(\mathrm{c}_{0}:=\left|\frac{2}{\sqrt{25d_{*}}}\right|^{d_{*}}\cdot\frac{|2\mathrm{c} _{1}|^{(1\wedge\beta)^{q}}}{768}\cdot\left|\frac{1\wedge r}{777}\right|^{\sum_ {k=0}^{q-1}(1\wedge\beta)^{k}}\cdot\frac{1}{2^{\beta\cdot(1\wedge\beta)^{k}}}\) only depending on \((d_{*},\beta,r,q)\). Thus we complete the proof of Theorem 2.1.

Now it remains to prove Corollary 2.1. Indeed, it follows from (2.33) that

\[\mathcal{H}^{d,\beta,r}_{3,A}=\mathcal{H}^{d,\beta,r}_{5,A,0,1,d}.\]

Taking \(q=0\), \(K=1\) and \(d_{*}=d\) in Theorem 2.6, we obtain that there exists an constant \(\mathrm{c}_{0}\in(0,\infty)\) only depending on \((d,\beta,r)\), such that

\[\inf_{f_{n}}\sup_{P\in\mathcal{H}^{d,\beta,r}_{3,A}}\mathbf{E}_{P^{ \otimes n}}\left[\mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]=\inf_{f_{n}}\sup_ {P\in\mathcal{H}^{d,\beta,r}_{5,A,0,1,d}}\mathbf{E}_{P^{\otimes n}}\left[ \mathcal{E}^{\phi}_{P}(\hat{f}_{n})\right]\geq\mathrm{c}_{0}n^{-\frac{\beta \cdot(1\wedge\beta)^{0}}{2+\beta\cdot(1\wedge\beta)^{0}}}\] \[=\mathrm{c}_{0}n^{-\frac{\beta}{4+\beta}}\text{ provided that }n> \left|\frac{7}{1-A}\right|^{\frac{4+\beta\cdot(1\wedge\beta)^{0}}{\beta \cdot(1\wedge\beta)^{0}}}=\left|\frac{7}{1-A}\right|^{\frac{4+\beta}{\beta}}.\]

This proves Corollary 2.1. 

#### a.3.7 Proof of (3.6)

This subsection is devoted to the proof of the bound (3.6).

Proof of (3.6).: Fix \(\nu\in[0,\infty)\) and \(\mu\in[1,\infty)\). Let \(P\) be an arbitrary probability in \(\mathcal{H}^{d,\beta}_{7}\). Denote by \(\eta\) the conditional probability function \(P(\{1\}\left|\cdot\right.)\) of \(P\). According to Lemma A.2 and the definition of \(\mathcal{H}^{d,\beta}_{7}\), there exists a function \(f^{*}\in\mathcal{B}^{\beta}_{1}\left([0,1]^{d}\right)\) such that

\[f^{*}_{\phi,P}\xrightarrow{P_{X}\text{-a.s.}}\log\frac{\eta}{1-\eta} \xrightarrow{P_{X}\text{-a.s.}}f^{*}.\] (A.149)

Thus there exists a measurable set \(\Omega\) contained in \([0,1]^{d}\) such that \(P_{X}(\Omega)=1\) and

\[\log\frac{\eta(x)}{1-\eta(x)}=f^{*}(x),\;\forall\;x\in\Omega.\] (A.150)

Let \(\delta\) be an arbitrary number in \((0,1/3)\). Then it follows from Corollary A.2 that there exists

\[\tilde{g}\in\mathcal{F}^{\mathbf{FNN}}_{d}\left(C_{d,\beta}\log\frac{1}{ \delta},C_{d,\beta}\delta^{-d/\beta},C_{d,\beta}\delta^{-d/\beta}\log\frac{1}{ \delta},1,\infty\right)\] (A.151)

such that \(\sup_{x\in[0,1]^{d}}|f^{*}(x)-\tilde{g}(x)|\leq\delta\). Let \(T:\mathbb{R}\to[-1,1],z\mapsto\min\left\{\max\left\{z,-1\right\},1\right\}\) and

\[\tilde{f}:\mathbb{R}\to[-1,1],\;x\mapsto T(\tilde{g}(x))=\begin{cases}-1,& \text{if }\tilde{g}(x)<-1,\\ \tilde{g}(x),&\text{if }-1\leq\tilde{g}(x)\leq 1,\\ 1,&\text{if }\tilde{g}(x)>1.\end{cases}\]

Obviously, \(|T(z)-T(w)|\leq|z-w|\) for any real numbers \(z\) and \(w\), and

\[\sup_{x\in[0,1]^{d}}\left|f^{*}(x)-\tilde{f}(x)\right|\xrightarrow{ \cdot\cdot\|f^{*}\|_{[0,1]^{d}}\leq 1}\sup_{x\in[0,1]^{d}}|T(f^{*}(x))-T(\tilde{g}(x))|\] (A.152) \[\leq\sup_{x\in[0,1]^{d}}|f^{*}(x)-\tilde{g}(x)|\leq\delta.\]

Besides, it is easy to verify that

\[\tilde{f}(x)=\sigma(\tilde{g}(x)+1)-\sigma(\tilde{g}(x)-1)-1,\;\forall\;x\in \mathbb{R}^{d},\]
which, together with (A.151), yields

\[\begin{split}\tilde{f}&\in\mathcal{F}_{d}^{\mathbf{FNN} }\left(1+C_{d,\beta}\log\frac{1}{\delta},1+C_{d,\beta}\delta^{-d/\beta},4+C_{d,\beta}\delta^{-d/\beta}\log\frac{1}{\delta},1,1\right)\\ &\subset\mathcal{F}_{d}^{\mathbf{FNN}}\left(C_{d,\beta}\log\frac{ 1}{\delta},C_{d,\beta}\delta^{-d/\beta},C_{d,\beta}\delta^{-d/\beta}\log\frac{ 1}{\delta},1,1\right).\end{split}\]

In addition, it follows from Lemma A.7 that

\[\begin{split}&\frac{1}{2(\mathrm{e}^{\mu}+\mathrm{e}^{-\mu}+2)} \left|f(x)-f^{*}(x)\right|^{2}\leq\int_{\{-1,1\}}\left(\phi(yf(x))-\phi(yf^{*}( x))\right)\mathrm{d}P(y|x)\\ &\leq\frac{1}{4}\left|f(x)-f^{*}(x)\right|^{2},\ \forall\text{ measurable }f:[0,1]^{d}\to[-\mu,\mu],\ \forall\ x\in\Omega.\end{split}\] (A.153)

Take \(\widetilde{C}:=2(\mathrm{e}^{\mu}+\mathrm{e}^{-\mu}+2)\). Integrating both side with respect to \(x\) in (A.153) and using (A.152), we obtain

\[\begin{split}&\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(yf(x))- \phi(yf^{*}(x))\right)^{2}\mathrm{d}P(x,y)\\ &\leq\int_{[0,1]^{d}\times\{-1,1\}}\left(f(x)-f^{*}(x)\right)^{2 }\mathrm{d}P(x,y)=\int_{[0,1]^{d}}\left|f(x)-f^{*}(x)\right|^{2}\mathrm{d}P_{X} (x)\\ &\xlongeqed{\underbrace{\ddots P_{X}(\Omega)=1}_{\Omega}\int_{ \Omega}\frac{\widetilde{C}}{2(\mathrm{e}^{\mu}+\mathrm{e}^{-\mu}+2)}\left|f(x )-f^{*}(x)\right|^{2}\mathrm{d}P_{X}(x)}\\ &\leq\widetilde{C}\int_{\Omega}\int_{\{-1,1\}}\left(\phi(yf(x))- \phi(yf^{*}(x))\right)\mathrm{d}P(y|x)\mathrm{d}P_{X}(x)\\ &\xlongeqed{\underbrace{\ddots P_{X}(\Omega)=1}_{\Omega}\widetilde {C}\int_{[0,1]^{d}\times\{-1,1\}}\left(\phi(yf(x))-\phi(yf^{*}(x))\right) \mathrm{d}P(x,y)}\\ &\xlongeqed{\underbrace{\text{Lemma A.3}}_{\ref{lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lemlem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lemlem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lemlem:lem:lem:lemlem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lemlem:lem:lem:lem:lemlem:lem:lem:lem:lem:lem:lem:lem:lemlem:lemlem:lemlem:lem:lemlem:lem:lemlem:lem:lemlem:lem:lemlem:lemlem:lem:lemlem:lemlem:lemlem:lemlem:lem:lemlem:lem:lem:lemlem:lem:lemlem:lemlem:lem:lem:lem:lemlem:lem:lemlem:lem:lemlem:lem:lemlem:lemlem:lem:lemlem:lem:lem:lemlem:lem:lem:lem:lemlem:lem:lem:lemlem:lemlem:lemlem:lem:lem:lemlem:lemlem:lem:lemlem:lem:lem:lem:lem:lemlem:lem:lem:lem:lemlem:lemlem:lem:lemlem:lemlem:lem:lemlem:lemlem:lemlem:lem:lem:lemlem:lem:lemlem:lem:lemlem:lemlem:lem:lem:lemlem:lemlem:lemlem:lemlem:lemlemlem:lem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lem:lemlem:lemlem:lemlem:lem:lemlem:lemlem:lem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlem:lemlemlem:lem:lemlemlem:lemlem:lemlemlem:lemlem:lemlem:lemlemlem:lemlemlem:lemlem:lemlemlem:lemlemlem:lemlemlem:lemlem:lemlemlem:lemlemlem:lem
that \(\frac{1}{n}\leq\delta_{n}<1/3\) for \(n>C_{l,c,d,\beta}\). We then deduce from (A.155) that

\[\begin{split}&\inf\left\{\mathcal{E}_{P}^{\phi}(f)\left|f\in \mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F\right)\right.\right\}\\ &\leq\inf\left\{\mathcal{E}_{P}^{\phi}(f)\left|f\in\mathcal{F}_{ d}^{\mathbf{FNN}}\left(c\log n,l\left|\frac{(\log n)^{3}}{n}\right|^{\frac{-d}{2 \beta+\beta}},l\left|\frac{(\log n)^{3}}{n}\right|^{\frac{-d}{2\beta+\beta}} \log n,B,F\right)\right.\right\}\\ &=\inf\left\{\mathcal{E}_{P}^{\phi}(f)\left|f\in\mathcal{F}_{d}^{ \mathbf{FNN}}\left(c\log n,c\delta_{n}^{-\frac{d}{\beta}},c\delta_{n}^{-\frac{ d}{\beta}}\log n,B,F\right)\right.\right\}\\ &\leq\inf\left\{\mathcal{E}_{P}^{\phi}(f)\left|f\in\mathcal{F}_{ d}^{\mathbf{FNN}}\left(c\log\frac{1}{\delta_{n}},c\delta_{n}^{-\frac{d}{ \beta}},c\delta_{n}^{-\frac{d}{\beta}}\log\frac{1}{\delta_{n}},B,F\right) \right.\right\}\\ &\leq\inf\left\{\mathcal{E}_{P}^{\phi}(f)\left|f\in\mathcal{F}_{ d}^{\mathbf{FNN}}\left(C_{d,\beta}\log\frac{1}{\delta_{n}},C_{d,\beta} \delta_{n}^{-\frac{d}{\beta}},C_{d,\beta}\delta_{n}^{-\frac{d}{\beta}}\log \frac{1}{\delta_{n}},1,1\right)\right.\right\}\\ &\leq\delta_{n}^{2},\;\forall\;n>C_{l,c,d,\beta},\end{split}\] (A.156)

where we use the fact the infimum taken over a larger set is smaller. Define \(W=3\sqrt{\mathcal{N}}\left(\mathcal{F}_{d}^{\mathbf{FNN}}\left(G,N,S,B,F \right),\frac{1}{n}\right)\). Then by taking \(\mathcal{F}=\left\{\left.f\right|_{[0,1]^{d}}\right|f\in\mathcal{F}_{d}^{ \mathbf{FNN}}\left(G,N,S,B,F\right)\right\}\), \(\psi(x,y)=\phi(yf^{*})\), \(\Gamma=\widetilde{C}\), \(M=2\), \(\gamma=\frac{1}{n}\) in Theorem 2.1, and using (A.149), (A.154), (A.156), we deduce that

\[\begin{split}&\boldsymbol{E}_{P\ni\infty}\left[\left\|\tilde{f }_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{E}_{\mathcal{P}_{\mathcal{ N}}}^{2}}^{2}\right]=\boldsymbol{E}_{P\ni\infty}\left[\left\|\tilde{f}_{n}^{ \mathbf{FNN}}-f^{*}\right\|_{\mathcal{L}_{\mathcal{P}_{\mathcal{N}}}^{2}}^{2} \right]\leq\widetilde{C}\boldsymbol{E}_{P\ni\infty}\left[\mathcal{E}_{P}^{ \phi}(\tilde{f}_{n}^{\mathbf{FNN}})\right]\\ &\xlongeqed{\text{by Lemma A.3}}{\text{by Lemma A.3}}\; \widetilde{C}\boldsymbol{E}_{P\ni\infty}\left[\mathcal{R}_{P}^{\phi}\left( \tilde{f}_{n}^{\mathbf{FNN}}\right)-\int_{[0,1]^{d}\times\{-1,1\}}\psi(x,y) \mathrm{d}P(x,y)\right]\\ &\leq\frac{500\cdot\left|\widetilde{C}\right|^{2}\cdot\log W}{n} +2\widetilde{C}\inf_{f\in\mathcal{F}}\left(\mathcal{R}_{P}^{\phi}(f)-\int_{[ 1]^{d}\times\{-1,1\}}\psi(x,y)\mathrm{d}P(x,y)\right)\\ &\xlongeqed{\text{by Lemma A.3}}{\text{by Lemma A.3}}\; \frac{500\cdot\left|\widetilde{C}\right|^{2}\cdot\log W}{n}+2\widetilde{C}\inf _{f\in\mathcal{F}}\mathcal{E}_{P}^{\phi}(f)\leq\frac{500\cdot\left|\widetilde{C }\right|^{2}\cdot\log W}{n}+2\widetilde{C}\delta_{n}^{2}\end{split}\]

for \(n>C_{l,c,d,\beta}\). Taking the supremum, we obtain,

\[\sup_{P\in\mathcal{H}_{\tau}^{d,\beta}}\boldsymbol{E}_{P\ni\infty}\left[ \left\|\tilde{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{\mathcal{L}_{ \mathcal{P}_{\mathcal{N}}}^{2}}^{2}\right]\leq\frac{500\cdot\left|\widetilde{C} \right|^{2}\cdot\log W}{n}+2\widetilde{C}\delta_{n}^{2},\;\forall\;n>C_{l,c,d, \beta}.\] (A.157)

Moreover, it follows from (3.7) and Corollary A.1 that

\[\begin{split}&\log W\leq(S+Gd+1)(2G+5)\log\left((\max\left\{N,d \right\}+1)\cdot B\cdot(2nG+2n)\right)\lesssim(G+S)G\log n\\ &\lesssim\left(\left(\frac{n}{\log^{3}n}\right)^{\frac{d}{2\beta+ 2\beta}}\log n+\log n\right)\cdot(\log n)\cdot(\log n)\lesssim n\cdot\left( \frac{(\log n)^{3}}{n}\right)^{\frac{2\beta}{2+2\beta}}.\end{split}\]

Plugging this into (A.157), we obtain

\[\begin{split}&\sup_{P\in\mathcal{H}_{\tau}^{d,\beta}}\boldsymbol{E}_ {P\ni\infty}\left[\left\|\tilde{f}_{n}^{\mathbf{FNN}}-f_{\phi,P}^{*}\right\|_{ \mathcal{L}_{\mathcal{P}_{\mathcal{N}}}^{2}}^{2}\right]\lesssim\frac{\log W}{n }+\delta_{n}^{2}\\ &\lesssim\left|\frac{(\log n)^{3}}{n}\right|^{\frac{2\beta}{2+2 \beta}}+\left|\left(\frac{(\log n)^{3}}{n}\right)^{\frac{2\beta}{2+2\beta}} \right|^{2}\lesssim\left|\frac{(\log n)^{3}}{n}\right|^{\frac{2\beta}{2+2\beta}},\end{split}\]

which proves the desired result.