# Estimating a regression function in Exponential Families by Model Selection

Juntong CHEN  Department of Mathematics, University of Luxembourg Maison du
nombre 6 avenue de la Fonte L-4364 Esch-sur-Alzette Grand Duchy of Luxembourg
<EMAIL_ADDRESS>
(Date: August 27, 2024)

###### Abstract.

Let $X_{1}=(W_{1},Y_{1}),\ldots,X_{n}=(W_{n},Y_{n})$ be $n$ pairs of
independent random variables. We assume that, for each $i\in\\{1,\ldots,n\\}$,
the conditional distribution of $Y_{i}$ given $W_{i}$ belongs to a one-
parameter exponential family with parameter
${\bm{\gamma}}^{\star}(W_{i})\in{\mathbb{R}}$, or at least, is close enough to
a distribution of this form. The objective of the present paper is to estimate
these conditional distributions on the basis of the observation
${\bm{X}}=(X_{1},\ldots,X_{n})$ and to do so, we propose a model selection
procedure together with a non-asymptotic risk bound for the resulted estimator
with respect to a Hellinger-type distance. When ${\bm{\gamma}}^{\star}$ does
exist, the procedure allows to obtain an estimator $\widehat{\bm{\gamma}}$ of
${\bm{\gamma}}^{\star}$ adapted to a wide range of the anisotropic Besov
spaces. When ${\bm{\gamma}}^{\star}$ has a general additive or multiple index
structure, we construct suitable models and show the resulted estimators by
our procedure based on such models can circumvent the curse of dimensionality.
Moreover, we consider model selection problems for ReLU neural networks and
provide an example where estimation based on neural networks enjoys a much
faster converge rate than the classical models. Finally, we apply this
procedure to solve variable selection problem in exponential families. The
proofs in the paper rely on bounding the VC dimensions of several collections
of functions, which can be of independent interest.

###### Key words and phrases:

Model selection, adaptive estimation, generalized additive models, multiple
index models, ReLU neural networks, variable selection.

###### 2020 Mathematics Subject Classification:

Primary 62G05, 62G35; Secondary 62J12

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 811017

##  1\. Introduction

We observe $n$ pairs of independent random variables $X_{i}=(W_{i},Y_{i})$,
for $i\in\\{1,\ldots,n\\}$ with values in a measurable product space
$({\mathscr{W}}\times{\mathscr{Y}},{\mathcal{W}}\otimes{\mathcal{Y}})$ and
assume (even if this may not be true) that there exists an unknown function
${\bm{\gamma}}^{\star}$ on ${\mathscr{W}}$ such that for each
$i\in\\{1,\ldots,n\\}$, the conditional distribution of $Y_{i}$ given $W_{i}$
belongs to a one parameter exponential family with parameter
${\bm{\gamma}}^{\star}(W_{i})\in{\mathbb{R}}$. From the observation
${\bm{X}}=(X_{1},\ldots,X_{n})$, our aim is to estimate the conditional
distributions of $Y_{i}$ given $W_{i}$. Our approach is based on an estimation
of the potential ${\bm{\gamma}}^{\star}$ (may not exist). When such a
${\bm{\gamma}}^{\star}$ does exist, the above statistical setting includes as
particular cases those of binary, Gaussian and Poisson regressions as well as
exponential multiplicative regression.

We are not aware of many results in the literature that tackle these
regression problems and establish a risk bound for the proposed estimator
$\widehat{\bm{\gamma}}$ of ${\bm{\gamma}}^{\star}$. When the exponential
family is given under its canonical form and
${\mathscr{W}}=\left[{0,1}\right]$, Kolaczyk and Nowak, (2005) proposed a
piecewise polynomial estimator $\widehat{\bm{\gamma}}$ of
${\bm{\gamma}}^{\star}$ under the assumption that the natural parameter is a
smooth function of the mean. They showed that their estimator achieves the
usual rate $n^{-2\alpha/(1+2\alpha)}$ over the Besov spaces with regularity
$\alpha>0$ up to a logarithmic factor with respect to the squared Hellinger
loss. When the family is a univariate natural exponential family where the
variances of the distributions are quadratic functions of the means,
Antoniadis et al., (2001) proposed a wavelet shrinkage method to estimate
${\bm{\gamma}}^{\star}$ while Brown et al., (2010) parametrized the
exponential family by its mean and transformed the problem of estimating
${\bm{\gamma}}^{\star}$ into homoscedastic Gaussian regression estimation by
stabilizing the variance. Under the assumption that ${\bm{\gamma}}^{\star}$
belongs to the Besov spaces and is bounded from both below and above, Brown et
al., (2010) showed their estimator achieves the converge rate
$n^{-2\alpha/(1+2\alpha)}$ with respect to the squared
${\mathbb{L}}_{2}$-loss. All of the literatures mentioned above make strong
assumptions on the distributions of the covariates $W_{i}$ where they require
them to be known (for example deterministic) or partly known. Moreover, none
of them consider the problem of estimation under a possible misspecification
framework.

Recently, Baraud and Chen, (2020) proposed a robust procedure based on the
$\rho$-estimation to estimate ${\bm{\gamma}}^{\star}$. Their approach is
restricted to the case of a single model. Up to a numerical constant, the risk
of their estimator $\widehat{\bm{\gamma}}$ on the model is bounded by the sum
of an approximation term and a complexity term. Such an estimation procedure
is satisfactory if we know in advance a suitable model for
${\bm{\gamma}}^{\star}$, i.e. a model which is not too complex and provides a
good enough approximation of ${\bm{\gamma}}^{\star}$. However, such a model
may not be easy to design without any prior information and a safer approach
is to consider a family of candidate models instead and let the data decide
which is the most appropriate one for estimating ${\bm{\gamma}}^{\star}$.

###  1.1. Our contributions

In this paper, we consider estimating the conditional distributions
$R_{i}^{\star}(W_{i})$ of $Y_{i}$ given $W_{i}$ by model selection. Our main
contributions are as follows.

  1. (i)

We propose a model selection procedure to estimate the conditional
distributions and establish a non-asymptotic risk bound for the resulted
estimator. Our approach is based on the presumption that there exists an
unknown ${\bm{\gamma}}^{\star}$ on ${\mathscr{W}}$ belonging to some of our
models such that $R_{i}^{\star}(W_{i})$ is of the form
$R_{{\bm{\gamma}}^{\star}(W_{i})}$ for all $i\in\\{1,\ldots,n\\}$. However,
our approach is not restricted to this assumption. This is to say, we allow
our statistical models to be slightly misspecified: $R_{i}^{\star}(W_{i})$ may
not be exactly of the form $R_{{\bm{\gamma}}^{\star}(W_{i})}$ and even if they
were, ${\bm{\gamma}}^{\star}$ may not belong to any of our models. What we
really assume is the form $R_{{\bm{\gamma}}(W_{i})}$ with some ${\bm{\gamma}}$
belonging to our models provides a suitable approximation of the conditional
distributions.

  2. (ii)

When $X_{1},\ldots,X_{n}$ are i.i.d., this model selection procedure solves
adaptation and variable selection problems in exponential families.

  3. (iii)

In i.i.d. case, when the dimensionality $d$ of covariate $W$ is large, the
converge rate of estimating ${\bm{\gamma}}^{\star}$ can be extremely slow
which is, as a well-known phenomenon, called the curse of dimensionality. When
${\bm{\gamma}}^{\star}$ has some particular structures or at least close to
some function with such features, we consider model selection problems based
on the composite piecewise polynomials and ReLU neural networks and show that
the resulted estimators by our procedure based on such models can circumvent
the curse of dimensionality. The structures discussed in the paper includes
genaralized additive structure, multiple index structure and multiple
composition structure.

  4. (iv)

In i.i.d. case, when ${\bm{\gamma}}^{\star}$ belongs to the Takagi class we
provide an example where estimation based on ReLU neural networks results in
an estimator converging to ${\bm{\gamma}}^{\star}$ with parametric rate
although ${\bm{\gamma}}^{\star}$ has very little smoothness. At least for such
an example, neural networks outperform all the other traditional approximation
methods, e.g. piecewise polynomials and wavelets.

  5. (v)

We construct models to approximate general additive and multiple index
functions and derive VC dimension bounds for them. Besides, we adapt the VC
dimension result of ReLU neural networks to the sparse setting. These results
can be of independent interest for readers.

The paper is organized as follows. We introduce the specific statistical
framework and set notations in Section 2. An estimation procedure based on
model selection is proposed in Section 3 together with the non-asymptotic
exponential deviation inequalities. We then discuss the adaptive estimation
problem in exponential families when the regression function belongs to
anisotropic Besov spaces as an application in Section 4. We show that under a
suitable parametrization of exponential families, our estimator is adaptive
over a wide range of the anisotropic Besov spaces with the risk bound
independent of choice of the exponential family. In Section 5, we consider the
applications of our procedure to two examples of the structural assumptions,
general additive functions and multiple index functions, to circumvent the
curse of dimensionality. Estimation by model selection based on ReLU neural
networks is discussed in Section 6 and variable selection problem in
generalized linear models is considered in Section 7. Finally, all the proofs
of this paper can be found in the appendix.

##  2\. The statistical setting

As already mentioned, we observe $n$ independent pairs of random variables
$X_{1}=(W_{1},Y_{1}),\ldots,X_{n}=(W_{n},Y_{n})$ with values in a measurable
product space
$({\mathscr{X}},{\mathcal{X}})=({\mathscr{W}}\times{\mathscr{Y}},{\mathcal{W}}\otimes{\mathcal{Y}})$.
We assume that for each $i\in\\{1,\ldots,n\\}$, the conditional distribution
of $Y_{i}$ given $W_{i}$ is given by $R_{i}^{\star}(W_{i})$, where
$R_{i}^{\star}$ is a measurable function from $({\mathscr{W}},{\mathcal{W}})$
to the set of all probabilities on $({\mathscr{Y}},{\mathcal{Y}})$ equipped
with the Borel $\sigma$-algebra ${\mathscr{T}}$ associated to the Hellinger
distance. We denote ${\mathbf{R}}^{\star}$ by $n$-tuple
$(R_{1}^{\star},\ldots,R_{n}^{\star})$.

Let $I$ be a non-trivial interval of ${\mathbb{R}}$, i.e. the interior of $I$
is not empty and
$\widetilde{\mathscr{Q}}=\\{R_{\gamma}=\overline{r}_{\gamma}\cdot\mu,\;\gamma\in
I\\}$ be an exponential family under its general form on the measured space
$({\mathscr{Y}},{\mathcal{Y}},\mu)$. More precisely, $\widetilde{\mathscr{Q}}$
is a family of probabilities on $({\mathscr{Y}},{\mathcal{Y}})$ admitting
densities $\overline{r}_{\gamma}$ with respect to $\mu$ of the form, for all
$y\in{\mathscr{Y}}$ and $\gamma\in I$

(1) $\overline{r}_{\gamma}(y)=e^{u(\gamma)T(y)-B(\gamma)}a(y)\text{\ where\
}B(\gamma)=\log\left[{\int_{{\mathscr{Y}}}e^{u(\gamma)T(y)}a(y)d\mu(y)}\right],$

$T$ is a real-valued measurable function on $({\mathscr{Y}},{\mathcal{Y}})$
which does not coincide with a constant $\nu=a\cdot\mu$-a.e., $u$ is a
continuous, strictly monotone function on $I$ and $a$ is a nonnegative
function on ${\mathscr{Y}}$. For convenience, we denote

(2) $r_{\gamma}(y)=e^{u(\gamma)T(y)-B(\gamma)},\text{\quad for all\
}y\in{\mathscr{Y}}\text{\ and\ }\gamma\in I,$

and rewrite
$\widetilde{\mathscr{Q}}=\\{R_{\gamma}=r_{\gamma}\cdot\nu,\;\gamma\in I\\}$.

Our estimator takes the form of a mapping
${\mathbf{R}}_{\widehat{\bm{\gamma}}}:{\mathbf{w}}=({\bm{w}}_{1},\ldots,{\bm{w}}_{n})\in{\mathscr{W}}^{n}\mapsto(R_{\widehat{\bm{\gamma}}({\bm{w}}_{1})},\ldots,R_{\widehat{\bm{\gamma}}({\bm{w}}_{n})})$
with values in $\widetilde{\mathscr{Q}}^{n}$, where $\widehat{\bm{\gamma}}$ is
a (random) function from ${\mathscr{W}}$ into $I$. When the mapping
${\mathbf{R}}^{\star}$ is also of this form, i.e.
${\mathbf{R}}^{\star}={\mathbf{R}}_{{\bm{\gamma}}^{\star}}$ for some
(deterministic) function ${\bm{\gamma}}^{\star}:{\mathscr{W}}\to I$,
$\widehat{\bm{\gamma}}$ provides an estimator of the so called regression
function.

In order to evaluate the performance of our estimator
${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ of ${\mathbf{R}}^{\star}$, we introduce
a loss function based on the Hellinger distance. Given $P$ and $Q$ two
probabilities dominated by some reference measure $\mu$ on a measurable space
$(A,{\mathcal{A}})$, the Hellinger distance between $P=p\cdot\mu$ and
$Q=q\cdot\mu$ is given by

(3)
$h(P,Q)=\left[{\frac{1}{2}\int_{A}\left({\sqrt{p}-\sqrt{q}}\right)^{2}d\mu}\right]^{1/2}.$

We remark that (3) does not depend on the choice of $\mu$. We denote
${\mathscr{Q}}_{{\mathscr{W}}}$ as the set of all measurable mappings from
$({\mathscr{W}},{\mathcal{W}})$ to the space of probabilities on
$({\mathscr{Y}},{\mathcal{Y}})$ equipped with the topology ${\mathscr{T}}$ and
define ${\mathscrbf{Q}}_{{\mathscr{W}}}={\mathscr{Q}}_{{\mathscr{W}}}^{n}$.
Therefore, both ${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ and
${\mathbf{R}}^{\star}=(R_{1}^{\star},\ldots,R_{n}^{\star})$ belong to
${\mathscrbf{Q}}_{{\mathscr{W}}}$. We endow the space
${\mathscrbf{Q}}_{{\mathscr{W}}}$ with the pseudo Hellinger distance
${\mathbf{h}}$ defined for ${\mathbf{R}}=(R_{1},\ldots,R_{n})$ and
${\mathbf{R}}^{\prime}=(R^{\prime}_{1},\ldots,R^{\prime}_{n})$ in
${\mathscrbf{Q}}_{{\mathscr{W}}}$ by

(4) $\displaystyle{\mathbf{h}}^{2}({\mathbf{R}},{\mathbf{R}}^{\prime})$
$\displaystyle={\mathbb{E}}\left[{\sum_{i=1}^{n}h^{2}\left({R_{i}(W_{i}),R^{\prime}_{i}(W_{i})}\right)}\right]$
$\displaystyle=\sum_{i=1}^{n}\int_{{\mathscr{W}}}h^{2}\left({R_{i}(w),R^{\prime}_{i}(w)}\right)dP_{W_{i}}(w),$

where $h$ is Hellinger distance defined in (3). We evaluate the performance of
the estimator ${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ of ${\mathbf{R}}^{\star}$
by the quantity
${\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})$.
In the favourable situation where ${\bm{\gamma}}^{\star}$ does exist, we
automatically deduce a performance of $\widehat{\bm{\gamma}}$ with respect to
${\bm{\gamma}}^{\star}$ by the distance
$d({\bm{\gamma}}^{\star},\widehat{\bm{\gamma}})={\mathbf{h}}({\mathbf{R}}_{{\bm{\gamma}}^{\star}},{\mathbf{R}}_{\widehat{\bm{\gamma}}})$.

When $W_{i}$ are i.i.d. with the common distribution $P_{W}$ and
$R_{i}^{\star}=R^{\star}$ for all $i\in\\{1,\ldots,n\\}$, we slightly abuse
the notation $h^{2}(R^{\star},R_{\widehat{\bm{\gamma}}})$ to measure the
distance between $R^{\star}$ and $R_{\widehat{\bm{\gamma}}}$ which is defined
as

$h^{2}(R^{\star},R_{\widehat{\bm{\gamma}}})=\frac{1}{n}{\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})=\int_{{\mathscr{W}}}h^{2}(R^{\star}(w),R_{\widehat{\bm{\gamma}}(w)})dP_{W}(w).$

We end this section by introducing some notations for later use. We denote
${\mathbb{N}}^{*}$ the set of all positive natural numbers, ${\mathbb{R}}_{+}$
the set of all non-negative real numbers and ${\mathbb{R}}_{+}^{*}$ the set of
all positive real numbers. For a set $m$, we use $|m|$ to denote its
cardinality. By $(x)_{+}$, we mean the function $\max\\{0,x\\}$. We denote
$x\vee y$ the largest value among $\left\\{x,y\right\\}$ while $x\wedge y$ is
the smallest. We use the notation $\lfloor x\rfloor$ for any
$x\in{\mathbb{R}}$ to denote the largest integer strictly smaller than $x$.
For a ${\mathbf{R}}\in{\mathscrbf{Q}}_{{\mathscr{W}}}$ and a set
${\mathbf{A}}\subset{\mathscrbf{Q}}_{{\mathscr{W}}}$, we define
${\mathbf{h}}^{2}({\mathbf{R}},{\mathbf{A}})=\inf_{{\mathbf{R}}^{\prime}\in{\mathbf{A}}}{\mathbf{h}}^{2}({\mathbf{R}},{\mathbf{R}}^{\prime})$.
Unless otherwise specified, $\log$ denotes the logarithm function with base
$e$. Let $(A,{\mathcal{A}})$ be a measurable space and $\mu$ be a
$\sigma$-finite measure on $(A,{\mathcal{A}})$. For
$k\in\left[{1,+\infty}\right]$, we define ${\mathcal{L}}_{k}(A,\mu)$ the
collection of all the measurable functions $f$ on $(A,{\mathcal{A}},\mu)$ such
that $\|f\|_{k,\mu}<+\infty$, where

$\|f\|_{k,\mu}=\left(\int_{A}|f|^{k}d\mu\right)^{\frac{1}{k}},\mbox{\quad for\
}k\in[1,+\infty),$ $\|f\|_{\infty,\mu}=\inf\\{K>0,\;|f|\leq
K\;\mu-\mbox{a.e.}\\},\mbox{\quad for\ }k=\infty.$

We denote the associated equivalent classes as ${\mathbb{L}}_{k}(A,\mu)$ where
any two functions coincide for $\mu$-a.e. can not be distinguished. In
particular, we write the norm $\|\cdot\|_{k}$ with
$k\in\left[{1,+\infty}\right]$ when $\mu=\lambda$ is the Lebesgue measure.
Throughout the paper, $C$ denotes positive numerical constant which may vary
from line to line.

##  3\. Estimation based on model selection

Our approach is based on $\rho$-estimation. For basic ideas that underline the
construction of the $\rho$-estimator, we refer Baraud and Birgé, (2018) and
Baraud et al., (2017).

###  3.1. Main assumption

Let ${\mathcal{M}}$ be a finite or countable set. For each
$m\in{\mathcal{M}}$, ${\overline{\bm{\Gamma}}}_{m}$ stands for a class of
measurable functions from ${\mathscr{W}}$ into $I$, which we call it a model.
We begin with an at most countable family
$\\{{\overline{\bm{\Gamma}}}_{m},\;m\in{\mathcal{M}}\\}$ of classes and assume
the following.

######  Assumption 1.

For any $m\in{\mathcal{M}}$, ${\overline{\bm{\Gamma}}}_{m}$ is VC-subgraph on
${\mathscr{W}}$ with dimension not larger than $V_{m}\geq 1$.

For completeness, we recall the definition of VC-subgraph. An (open) subgraph
of a function ${\bm{\gamma}}$ in ${\overline{\bm{\Gamma}}}_{m}$ is the subset
of ${\mathscr{W}}\times{\mathbb{R}}$ given by

${\mathscr{S}}_{{\bm{\gamma}}}=\left\\{(w,t)\in{\mathscr{W}}\times{\mathbb{R}},\;t<{\bm{\gamma}}(w)\right\\}.$

A collection ${\overline{\bm{\Gamma}}}_{m}$ of measurable functions on
${\mathscr{W}}$ is VC-subgraph with dimension not larger than $V_{m}$ if, for
any finite subset ${\mathcal{S}}\subset{\mathscr{W}}\times{\mathbb{R}}$ with
$|{\mathcal{S}}|=V_{m}+1$, there exists at least one subset $S$ of
${\mathcal{S}}$ such that for any
${\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}$, $S$ is not the intersection of
${\mathcal{S}}$ with ${\mathscr{S}}_{{\bm{\gamma}}}$, i.e.

$S\neq{\mathcal{S}}\cap{\mathscr{S}}_{{\bm{\gamma}}}\quad\text{whatever
${\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}$.}$

In particular, when ${\overline{\bm{\Gamma}}}_{m}$ is contained in a linear
space with finite dimension $d_{m}\geq 1$ Assumption 1 is fulfilled with
$V_{m}=d_{m}+1$. Another property which can be derived from Lemma 2.6.18 of
van der Vaart and Wellner, (1996) is that if ${\overline{\bm{\Gamma}}}$ is VC-
subgraph on a set ${\mathscr{W}}$ with dimension $V$ and $a,b\in{\mathbb{R}}$
are fixed numbers, then the classes of functions
${\overline{\bm{\Gamma}}}_{a}=\left\\{{\bm{\gamma}}\vee
a,\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}\right\\}$ and
${\overline{\bm{\Gamma}}}^{b}=\left\\{{\bm{\gamma}}\wedge
b,\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}\right\\}$ are also VC-subgraphs
on ${\mathscr{W}}$ with dimension not larger than $V$. For more properties of
the VC-subgraph class of functions, we refer the reader to Section 2.6.2 of
van der Vaart and Wellner, (1996) and Section 8 of Baraud et al., (2017).

###  3.2. Model selection procedure

We consider
$\left\\{{\overline{\bm{\Gamma}}}_{m},\;m\in{\mathcal{M}}\right\\}$ an at most
countable family of models satisfying Assumption 1. To avoid measurability
issues, for any $m\in{\mathcal{M}}$, we take $\bm{\Gamma}_{m}$ a finite or
countable subset of ${\overline{\bm{\Gamma}}}_{m}$ and denote
$\bm{\Gamma}=\cup_{m\in{\mathcal{M}}}\bm{\Gamma}_{m}$. Let $\psi$ be the map
defined on $[0,+\infty]$ as

(5) $\psi(x)=\left\\{\begin{aligned}
&\frac{x-1}{x+1}&,&\quad\mbox{$x\in[0,+\infty)$,}\\\
&1&,&\quad\mbox{$x=+\infty$.}\end{aligned}\right.$

For any ${\bm{\gamma}},{\bm{\gamma}}^{\prime}\in\bm{\Gamma}$, we define the
${\mathbf{T}}$-statistic as

(6)
${\mathbf{T}}({\bm{X}},{\bm{\gamma}},{\bm{\gamma}}^{\prime})=\sum_{i=1}^{n}\psi\left({\sqrt{{r_{{\bm{\gamma}}^{\prime}(W_{i})}(Y_{i})}\over{r_{{\bm{\gamma}}(W_{i})}(Y_{i})}}}\right)$

with the conventions $0/0=1$ and $c/0=+\infty$ for all $c>0$.

Let $\Delta$ be a map from ${\mathcal{M}}$ to ${\mathbb{R}}_{+}$. For each
$m\in{\mathcal{M}}$, we associate it with a nonnegative weight $\Delta(m)$
which satisfies

(7) $\Sigma=\sum_{m\in{\mathcal{M}}}e^{-\Delta(m)}<+\infty.$

In particular, when $\Sigma=1$, this gives a Bayesian flavour to our procedure
by regarding $\Delta(m)$ as a prior distribution on the family
$\\{\bm{\Gamma}_{m},\;m\in{\mathcal{M}}\\}$.

Let $D_{n}$ be a map from ${\mathcal{M}}$ to ${\mathbb{R}}_{+}$ defined as,
for any $m\in{\mathcal{M}}$,

$D_{n}(m)=10^{3}V_{m}\left[{9.11+\log_{+}\left(\frac{n}{V_{m}}\right)}\right],$

where $V_{m}$ stands for the VC dimension of the class
${\overline{\bm{\Gamma}}}_{m}$. We define the penalty function from
$\bm{\Gamma}$ to ${\mathbb{R}}_{+}$ as

(8) $\mathop{\rm\bf
pen}\nolimits({\bm{\gamma}})=10^{2}\inf_{\left\\{{m\in{\mathcal{M}}|{\bm{\gamma}}\in\bm{\Gamma}_{m}}\right\\}}\left[{D_{n}(m)+4.7\Delta(m)}\right],\mbox{\quad
for all\ }{\bm{\gamma}}\in\bm{\Gamma}.$

For all ${\bm{\gamma}}\in\bm{\Gamma}$, we set

(9)
$\bm{\upsilon}({\bm{X}},{\bm{\gamma}})=\sup_{{\bm{\gamma}}^{\prime}\in\bm{\Gamma}}\left[{{\mathbf{T}}({\bm{X}},{\bm{\gamma}},{\bm{\gamma}}^{\prime})-\mathop{\rm\bf
pen}\nolimits({\bm{\gamma}}^{\prime})}\right]+\mathop{\rm\bf
pen}\nolimits({\bm{\gamma}}).$

We define $\widehat{\bm{\gamma}}=\widehat{\bm{\gamma}}({\bm{X}})$ as any
measurable element of the random (and non-void) set

(10) ${\mathcal{E}}({\bm{X}})=\left\\{{{\bm{\gamma}}\in\bm{\Gamma}\;\mbox{
such that
}\;\bm{\upsilon}({\bm{X}},{\bm{\gamma}})\leq\inf_{{\bm{\gamma}}^{\prime}\in\bm{\Gamma}}\bm{\upsilon}({\bm{X}},{\bm{\gamma}}^{\prime})+1}\right\\}.$

Finally,
${\mathbf{R}}_{\widehat{\bm{\gamma}}}=(R_{\widehat{\bm{\gamma}}},\ldots,R_{\widehat{\bm{\gamma}}})$
is our estimator for ${\mathbf{R}}^{\star}$.

We comment that the number 1 in (10) does not play any role. Any small number
$\delta>0$ can work for defining our estimator. We hereby choose $\delta=1$
just for presenting our results in a simple way.

As one can observe from the construction procedure, our estimator depends on
the choice of the exponential family in (6), the countable subsets
$\bm{\Gamma}_{m}$ of ${\overline{\bm{\Gamma}}}_{m}$ and the weights
$\Delta(m)$ we choose. However, we do not require any information for the
distributions of covariates $W_{i}$ which, therefore, could be unknown. This
is one of the feature distinguishing our procedure with the ones Antoniadis et
al., (2001), Brown et al., (2010) and Kolaczyk and Nowak, (2005) in the
literature.

###  3.3. The performance of the estimator

######  Theorem 1.

Let
${\mathscrbf{Q}}_{m}=\\{{\mathbf{R}}_{{\bm{\gamma}}},\;{\bm{\gamma}}\in\bm{\Gamma}_{m}\\}$
and $\Xi(m)=D_{n}(m)/4.7+\Delta(m)$, for all $m\in{\mathcal{M}}$. Under
Assumption 1, whatever the conditional probabilities
${\mathbf{R}}^{\star}=(R_{1}^{\star},\ldots,R_{n}^{\star})$ of $Y_{i}$ given
$W_{i}$ and the distributions of $W_{i}$, the estimator
${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ obtained by our model selection
procedure in Section 3.2 satisfies for any $\xi>0$, with a probability at
least $1-\Sigma e^{-\xi}$

(11)
${\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})\leq\inf_{m\in{\mathcal{M}}}\left[{c_{1}{\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})+c_{2}\left(\Xi(m)+1.49+\xi\right)}\right],$

where $c_{1}=149.8$ and $c_{2}=5013.2$.

The proof of Theorem 1 is postponed to Appendix A. We shall use (11) in the
forthcoming sections to solve many model selection problems simultaneously. We
need to point out that the result we present here is different with the work
in Section 8 of Baraud and Birgé, (2018), where they assumed the pairs of
random variables $X_{i}=(W_{i},Y_{i})$, $i\in\\{1,\ldots,n\\}$ are i.i.d..
Besides, we study the regression problem in exponential families so that it is
more natural to put assumption on the models ${\overline{\bm{\Gamma}}}_{m}$ of
the potential regression function ${\bm{\gamma}}^{\star}$. In this statistical
setting, additional work has to be done to understand the performance of the
$\rho$-estimator.

We give some comments on our result here. The numerical constants $c_{1}$ and
$c_{2}$ are independent of the choice of the exponential family. For all
$m\in{\mathcal{M}}$, let us set
$\overline{\mathscrbf{Q}}_{m}=\\{{\mathbf{R}}_{{\bm{\gamma}}},\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}\\}$.
If for all $m\in{\mathcal{M}}$, ${\mathscrbf{Q}}_{m}$ is dense in
$\overline{\mathscrbf{Q}}_{m}$ with respect to the pseudo Hellinger distance
${\mathbf{h}}$, i.e.
${\mathbf{h}}({\mathbf{R}}^{\star},\overline{\mathscrbf{Q}}_{m})={\mathbf{h}}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})$,
(11) is equivalent to

${\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})\leq\inf_{m\in{\mathcal{M}}}\left[{c_{1}{\mathbf{h}}^{2}({\mathbf{R}}^{\star},\overline{\mathscrbf{Q}}_{m})+c_{2}\left(\Xi(m)+1.49+\xi\right)}\right],$

where we involve the models ${\overline{\bm{\Gamma}}}_{m}$ into the risk bound
of our estimator but not its countable subset $\bm{\Gamma}_{m}$ as we derived
in (11). As it was discussed in Section 4.2 of Baraud and Birgé, (2018), this
is exact the case when $\bm{\Gamma}_{m}$ is a dense subset of
${\overline{\bm{\Gamma}}}_{m}$ for the topology of pointwise convergence for
all $m\in{\mathcal{M}}$.

An integration of (11) with respect to $\xi$ leads to

(12)
${\mathbb{E}}\left[{{\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})}\right]\leq\inf_{m\in{\mathcal{M}}}\left[{c_{1}{\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})+c_{2}\left(\Xi(m)+\Sigma+1.49\right)}\right].$

We note from (12) that the risk of the estimator
${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ is bounded, up to a constant depending
on $\Sigma$, by the infimum over the whole family ${\mathcal{M}}$ of the
quantity summing up the distance from each ${\mathscrbf{Q}}_{m}$ to
${\mathbf{R}}^{\star}$, the complexity of each ${\overline{\bm{\Gamma}}}_{m}$
(up to a logarithmic factor) and the associated weight $\Delta(m)$. The
magnitude of the bias term and the complexity term is of the optimal order so
that if for all $m\in{\mathcal{M}}$, the weight function $\Delta(m)$ is chosen
to be not larger than $V_{m}$ (up to a logarithmic factor), we are able to
select the model achieving the best trade-off between approximation and
model’s complexity among the collection ${\mathcal{M}}$.

Moreover, the bias term
${\mathbf{h}}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})$ in (12) accounts for
the robustness property of our estimator with respect to the possible model
misspecification and data contamination. To illustrate it simply, let us focus
on each single $\bm{\Gamma}_{m}$ and assume the weight $\Delta(m)$ has been
assigned such that $\Delta(m)\lesssim D_{n}(m)$. If $\bm{\Gamma}_{m}$ is
exact, i.e. ${\mathbf{R}}^{\star}={\mathbf{R}}_{{\bm{\gamma}}^{\star}}$ with
${\bm{\gamma}}^{\star}\in\bm{\Gamma}_{m}$, up to a constant, the risk of the
estimator ${\mathbf{R}}_{\widehat{\bm{\gamma}}}$ will be smaller than
$V_{m}\left[{1+\log_{+}\left(n/V_{m}\right)}\right]$. If it is not the case,
the risk involves an additional bias term
${\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})$ due to a
potential model misspecification or data contamination. However, as long as
this bias term remains small compared to
$V_{m}\left[{1+\log_{+}\left(n/V_{m}\right)}\right]$, the performance of our
estimator will not deteriorate much as the case when $\bm{\Gamma}_{m}$ is
exact.

In the situation where the covariates $W_{i}$ are truly i.i.d. with a common
distribution $P_{W}$ and $R^{\star}_{i}=R^{\star}$ for all
$i\in\\{1,\ldots,n\\}$, we deduce from (12) that for any $R^{\star}$ and
$P_{W}$, our estimator $R_{\widehat{\bm{\gamma}}}$ satisfies

(13)
${\mathbb{E}}\left[{h^{2}(R^{\star},R_{\widehat{\bm{\gamma}}})}\right]\leq
c_{2}\left(c_{3}+\Sigma\right)\inf_{m\in{\mathcal{M}}}\left[{h^{2}(R^{\star},{\mathscr{Q}}_{m})+\frac{\Delta(m)}{n}+\frac{V_{m}}{n}L_{n}(m)}\right],$

where $c_{3}=1939.8$,
${\mathscr{Q}}_{m}=\\{R_{{\bm{\gamma}}},\;{\bm{\gamma}}\in\bm{\Gamma}_{m}\\}$
and $L_{n}(m)=1+\log_{+}\left(n/V_{m}\right)$.

##  4\. Adaptation to anisotropic Besov spaces

In this section, we assume the covariates $W_{i}$ are truly i.i.d. on
${\mathscr{W}}=\left[{0,1}\right]^{d}$, $d\geq 1$ with a common distribution
$P_{W}$ and $R_{i}^{\star}=R^{\star}$ for all $i\in\\{1,\ldots,n\\}$ and
consider adaptive estimation in exponential families. The problem is stated as
follows.

Let $0<p,q\leq\infty$,
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{d})\in({\mathbb{R}}_{+}^{*})^{d}$
and $R\in{\mathbb{R}}_{+}^{*}$. We denote
$B_{p,q}^{{\bm{\alpha}}}(\left[{0,1}\right]^{d},R)$ as the anisotropic Besov
ball which gathers all the functions $f$ in the anisotropic Besov space
$B_{p,q}^{{\bm{\alpha}}}(\left[{0,1}\right]^{d})$ with (quasi-) semi-norm
$|f|_{{\bm{\alpha}},p,q}<R$. Including Hölder and Sobolev spaces, Besov space
is a considerable general function space. It can also capture the spatial
inhomogeneity of the smoothness property as discussed by Suzuki and Nitanda,
(2021). For readers who concern the definitions, we refer Chapter 5 of
Triebel, (2006) and Hochmuch, (2002) which gives a detailed introduction
restricted to $d=2$ but can be generalized easily. Similarly to the isotropic
case, the $d$-dimensional parameter ${\bm{\alpha}}$ indicates the smooth
property in each direction $j\in\\{1,\ldots,d\\}$. More precisely, for all
functions $f\in B_{p,q}^{{\bm{\alpha}}}(\left[{0,1}\right]^{d})$, if
$\alpha_{j}$ is large, then $f$ is smooth to the $j$-th direction.

For a given interval $\left[{v_{-},v_{+}}\right]\subset I$ with $v_{-}<v_{+}$,
the notation $B_{p,q}^{{\bm{\alpha}}}(R,v_{-},v_{+})$ stands for the
collection of functions $f\in
B_{p,q}^{{\bm{\alpha}}}(\left[{0,1}\right]^{d},R)$ with
$f({\bm{w}})\in\left[{v_{-},v_{+}}\right]$ for all
${\bm{w}}\in\left[{0,1}\right]^{d}$. We assume that the regression function
${\bm{\gamma}}^{\star}\in B_{p,q}^{{\bm{\alpha}}}(R,v_{-},v_{+})$. Our aim, in
this section, is to design a specific procedure for estimating this
${\bm{\gamma}}^{\star}$ without assuming the parameters ${\bm{\alpha}}$, $p$
and $R$ to be known.

###  4.1. Models construction

We begin with introducing the conception of hyperrectangle. Given
$s_{j}\in{\mathbb{N}}$, $1\leq j\leq d$, for any
$k_{j}\in\Psi(s_{j})=\\{0,\ldots,2^{s_{j}}-1\\}$, we set

(14) $I_{j}(k_{j})=\left\\{\begin{aligned}
&\left[{0,2^{-s_{j}}}\right]&,&\quad k_{j}=0,\\\ &\
(k_{j}2^{-s_{j}},(k_{j}+1)2^{-s_{j}}]&,&\quad
k_{j}=1,\ldots,2^{s_{j}}-1.\end{aligned}\right.$

We call a hyperrectangle by any subset of $\left[{0,1}\right]^{d}$ of the form
$\prod_{j=1}^{d}I_{j}(k_{j})$. Given a vector
${\bm{s}}=(s_{1},\ldots,s_{d})\in{\mathbb{N}}^{d}$, we denote
$M^{{\mathcal{B}},d}_{\bm{s}}$ the resulted partition of
$\left[{0,1}\right]^{d}$ into the union of hyperrectangles
$\cup_{(k_{1},\ldots,k_{d})\in\Psi(s_{1})\times\cdots\times\Psi(s_{d})}\prod_{j=1}^{d}I_{j}(k_{j})$.

We take ${\mathcal{M}}={\mathbb{N}}^{d}\times{\mathbb{N}}$. Given
$({\bm{s}},r)\in{\mathcal{M}}$, we define
$\overline{\bm{S}}^{{\mathcal{B}},d}_{({\bm{s}},r)}$ as the space of piecewise
polynomial functions on $\left[{0,1}\right]^{d}$, where on each hyperrectangle
$\prod_{j=1}^{d}I_{j}(k_{j})$,
${\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{({\bm{s}},r)}$ is a
polynomial in $d$ variables of degree at most $r$ for each variable. This is
to say given $({\bm{s}},r)\in{\mathcal{M}}$, for any
$(\overline{k}_{1},\ldots,\overline{k}_{d})\in\Psi(s_{1})\times\cdots\times\Psi(s_{d})$,
any ${\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{({\bm{s}},r)}$ is of
the form for all
${\bm{w}}=(w_{1},\ldots,w_{d})\in\prod_{j=1}^{d}I_{j}(\overline{k}_{j})$

(15)
${\bm{\gamma}}({\bm{w}})=\sum_{(r_{1},\ldots,r_{d})\in\\{0,\ldots,r\\}^{d}}\gamma_{(r_{1},\ldots,r_{d})}\prod_{j=1}^{d}{w_{j}}^{r_{j}},$

where $\gamma_{(r_{1},\ldots,r_{d})}\in{\mathbb{R}}$, for all $0\leq r_{j}\leq
r$, $1\leq j\leq d$.

Recall that in our setting ${\bm{\gamma}}^{\star}$ takes values in some non-
trivial interval $I$ which may vary from the choice of the exponential family
and the choice of parametrization. A few examples are given in Section 2.1 of
Baraud and Chen, (2020) when the exponential family is parametrized in its
natural form. To estimate ${\bm{\gamma}}^{\star}$, we assume that we have a
prior information of $v_{-},v_{+}\in{\mathbb{R}}$ such that the regression
function ${\bm{\gamma}}^{\star}$ with values in
$\left[{v_{-},v_{+}}\right]\subset I$. For each
$({\bm{s}},r)\in{\mathcal{M}}$, we define
${\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{({\bm{s}},r)}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{({\bm{s}},r)}\right\\}$
and the family of models is given by
$\left\\{{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$.
For each $\overline{\bm{S}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$, we take its
countable subset ${\bm{S}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ as the collection
of functions of the same form in (15) apart from restricting
$\gamma_{(r_{1},\ldots,r_{d})}\in{\mathbb{Q}}$, for all $0\leq r_{j}\leq r$,
$1\leq j\leq d$ and define
$\bm{\Gamma}^{{\mathcal{B}},d}_{({\bm{s}},r)}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}^{{\mathcal{B}},d}_{({\bm{s}},r)}\right\\}$.

######  Lemma 1.

For any $d\in{\mathbb{N}}^{*}$, $r\in{\mathbb{N}}$ and
${\bm{s}}\in{\mathbb{N}}^{d}$, ${\bm{S}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ is
dense in $\overline{\bm{S}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ and
$\bm{\Gamma}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ is dense in
${\overline{\bm{\Gamma}}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ with respect to
the supremum norm $\|\cdot\|_{\infty}$.

For any $({\bm{s}},r)\in{\mathcal{M}}$, since $M^{{\mathcal{B}},d}_{\bm{s}}$
is a partition of $\left[{0,1}\right]^{d}$ with $\prod_{j=1}^{d}2^{s_{j}}$
hyperrectangles and on each hyperrectangle the space of functions is spanned
by $(r+1)^{d}$ basis, $\overline{\bm{S}}_{({\bm{s}},r)}^{{\mathcal{B}},d}$ is
a $(r+1)^{d}\prod_{j=1}^{d}2^{s_{j}}$ dimensional vector space. By the
properties of VC-subgraph we introduced in Section 3, for any
$({\bm{s}},r)\in{\mathcal{M}}$,
${\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{({\bm{s}},r)}$ is a VC-subgraph
on ${\mathscr{W}}$ with dimension not lager than
$(r+1)^{d}\prod_{j=1}^{d}2^{s_{j}}+1$ which fulfills Assumption 1 with

(16) $V_{({\bm{s}},r)}=(r+1)^{d}\prod_{j=1}^{d}2^{s_{j}}+1.$

For each $({\bm{s}},r)\in{\mathcal{M}}$, we associate it with the weight

(17) $\Delta({\bm{s}},r)=\log(8d)\prod_{j=1}^{d}2^{s_{j}}+r.$

We have the following result which shows inequality (7) is satisfied with the
weights defined by (17).

######  Lemma 2.

For each $({\bm{s}},r)\in{\mathcal{M}}$, let the weight be assigned by (17).
Then

$\sum_{({\bm{s}},r)\in{\mathcal{M}}}e^{-\Delta({\bm{s}},r)}\leq\frac{e}{e-1}.$

We denote
$M^{{\mathcal{B}},d}=\cup_{{\bm{s}}\in{\mathbb{N}}^{d}}M^{{\mathcal{B}},d}_{\bm{s}}$.
Given a partition $\pi\in M^{{\mathcal{B}},d}$ without knowing the specific
values of $(s_{1},\ldots,s_{d})$, sometimes it is useful to introduce an
alternative notation
${\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{(\pi,r)}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{(\pi,r)}\right\\}$
for ${\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{({\bm{s}},r)}$, where
$\overline{\bm{S}}^{{\mathcal{B}},d}_{(\pi,r)}$ characterises the space of
piecewise polynomial functions on $\left[{0,1}\right]^{d}$ such that on each
hyperrectangle of $\pi$, any
${\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{(\pi,r)}$ is a
polynomial in $d$ variables of degree not larger than $r$ for each variable.
Similarly, the VC dimension bound for the class of functions
${\overline{\bm{\Gamma}}}_{(\pi,r)}^{{\mathcal{B}},d}$ on ${\mathscr{W}}$ is
given by

(18) $V_{(\pi,r)}=(r+1)^{d}|\pi|+1,$

where $|\pi|$ denotes the cardinality of hyperrectangles given by the
partition $\pi$ of $\left[{0,1}\right]^{d}$. Under this new notation, the
weight associated to each $(\pi,r)\in M^{{\mathcal{B}},d}\times{\mathbb{N}}$
can be deduced from (17) as

(19) $\Delta(\pi,r)=\log(8d)|\pi|+r.$

###  4.2. Adaptivity result

Before deriving the risk bound for our estimator based on the constructed
family
$\left\\{\bm{\Gamma}_{({\bm{s}},r)}^{{\mathcal{B}},d},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$,
we first discuss the parametrization issue of the exponential family. As it
has been explained in Section 4.1 and 4.2 of Baraud and Chen, (2020), the
parametrization of the exponential family influences the converge rate of
$\widehat{\bm{\gamma}}$ to ${\bm{\gamma}}^{\star}$. For example, when $d=1$
one can see from Section 4.1 of Baraud and Chen, (2020) that if we parametrize
exponential families by their mean, Poisson regression achieves much slower
rate than the Gaussian case under the same $\alpha$-Hölder smoothness
assumption on ${\bm{\gamma}}^{\star}$ with $\alpha\in(0,1]$. However, there do
exist ways of parametrization such that the same rate of convergence can be
achieved uniformly regardless the choice of the exponential family. We assume
the following holds.

######  Assumption 2.

The exponential family $\widetilde{\mathscr{Q}}=\\{R_{\gamma},\;\gamma\in
I\\}$ has been parametrized in the way that there exists a constant $\kappa>0$
such that

$h(R_{\gamma},R_{\gamma^{\prime}})\leq\kappa|\gamma-\gamma^{\prime}|\text{\quad
for all\quad}\gamma,\gamma^{\prime}\in I.$

Let us remark that, by Proposition 2 of Baraud and Chen, (2020), Assumption 2
is fulfilled with $\kappa=1$ when the exponential family is parametrized by
$\gamma=\upsilon(\theta)$, where $\theta$ is the natural parameter and
$\upsilon$ satisfies
$\upsilon^{\prime}(\theta)=\sqrt{A^{\prime\prime}(\theta)/8}$ with the
function $A$ defined as

$A(\theta)=\log\left[{\int_{{\mathscr{Y}}}e^{\theta T(y)}d\nu(y)}\right].$

For any
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{d})\in({\mathbb{R}}_{+}^{*})^{d}$,
we denote $\alpha_{\min}=\min_{1\leq j\leq d}\alpha_{j}$ and
$\overline{\alpha}$ the harmonic mean of $\alpha_{1},\ldots,\alpha_{d}$, i.e.

$\overline{\alpha}=\left(\frac{1}{d}\sum_{j=1}^{d}\frac{1}{\alpha_{j}}\right)^{-1}.$

With the family of models
$\left\\{{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$
defined in Section 4.1, the associated countable subsets
$\left\\{\bm{\Gamma}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$
and the weights defined by (17), we are now able to apply the model selection
procedure introduced in Section 3.2 to estimate ${\bm{\gamma}}^{\star}$. The
following result shows that under Assumption 2, the resulted estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ based on
$\left\\{\bm{\Gamma}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$
is adapted to the possible anisotropy over a wide range of the anisotropic
Besov spaces with a risk bound of order
$n^{-2\overline{\alpha}/(2\overline{\alpha}+d)}$ up to a logarithmic factor
with respect to the distance
$d({\bm{\gamma}}^{\star},\widehat{\bm{\gamma}})=h^{2}\left({R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}}}\right)$.
One nice feature is that this risk bound is independent of the choice of the
exponential family.

######  Corollary 1.

Under Assumption 2, whatever the distribution of $W$, the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ given by the model selection procedure in
Section 3.2 over the family
$\left\\{\bm{\Gamma}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathbb{N}}^{d}\times{\mathbb{N}}\right\\}$
with the weights defined by (17) satisfies for all $R>0$, $p>0$ and
${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\overline{\alpha}/d>1/p$,

$\sup_{{\bm{\gamma}}^{\star}\in
B_{p,q}^{{\bm{\alpha}}}\left(R,v_{-},v_{+}\right)}{\mathbb{E}}\left[{h^{2}\left({R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}}}\right)}\right]\leq
C_{\kappa,d,{\bm{\alpha}},p}\left(R^{\frac{2d}{d+2\overline{\alpha}}}n^{-\frac{2\overline{\alpha}}{d+2\overline{\alpha}}}+\frac{1}{n}\right)\left(1+\log
n\right),$

where $q=\infty$ if $0<p\leq 1$ or $p\geq 2$ and $q=p$ if $1<p<2$,
$C_{\kappa,d,{\bm{\alpha}},p}$ is a constant depending on
$\kappa,d,{\bm{\alpha}},p$ only.

The proof of Corollary 1 is postponed to Appendix A. We hereby give some
comments on this result. First, Corollary 1 in fact holds for any
$0<q\leq\infty$, if $0<p\leq 1$ or $p\geq 2$ and $0<q\leq p$, if $1<p<2$ as a
consequence of embedding the anisotropic Besov spaces to some bigger spaces.
We will not discuss too much on this direction but refer the reader to Section
2.3 of Akakpo, (2012). Second, as it has been discussed by Section 2.1 of
Suzuki and Nitanda, (2021), the parameter $p$ plays a role of controlling the
spatial inhomogeneity of the smoothness. In particular, when $p=\infty$, the
smoothness is ensured uniformly. Our result, therefore, is also adapted to
${\bm{\gamma}}^{\star}$ with potentially inhomogeneous smoothness. Third, the
rate is optimal up to a logarithmic factor in the minimax sense at least when
$d=1$ as it has been shown in Proposition 4 of Baraud and Chen, (2020).
Finally, the condition $\overline{\alpha}/d>1/p$ appearing in the result is
more strict than the usual one which only requires
$\overline{\alpha}/d>\left(1/p-1/2\right)_{+}$. This is because we do not make
any assumption on the distribution of the covariate $W$. Therefore, we bound
the approximation bias with respect to the sup-norm $\|\cdot\|_{\infty}$. As
one can see from the proof of Corollary 1, this bias bound can be reconsidered
if the specific distribution of the covariate $W$ is given. In the particular
case when the probability measure $P_{W}$ admits a density
$P_{W}=p_{W}\cdot\lambda$ with respect to the Lebesgue measure $\lambda$ and
$\|p_{W}\|_{\infty}\leq K$ (i.e. the probability measure $P_{W}$ is equivalent
to the Lebesgue probability on ${\mathscr{W}}=\left[{0,1}\right]^{d}$), we
only need to require the usual condition
$\overline{\alpha}/d>\left(1/p-1/2\right)_{+}$ to obtain the same rate in
Corollary 1, where the numerical constant depends on $K$, $\kappa$, $d$,
${\bm{\alpha}}$ and $p$.

##  5\. Model selection under structural assumptions

In the last section, we have seen that when the covariates $W_{i}$ are truly
i.i.d. on $\left[{0,1}\right]^{d}$ and
$R_{i}^{\star}=R_{{\bm{\gamma}}^{\star}}$ for all $i\in\\{1,\ldots,n\\}$ with
${\bm{\gamma}}^{\star}\in B_{p,q}^{{\bm{\alpha}}}\left(R,v_{-},v_{+}\right)$,
the estimator $\widehat{\bm{\gamma}}({\bm{X}})$ obtained from our model
selection procedure based on
$\left\\{\bm{\Gamma}^{{\mathcal{B}},d}_{({\bm{s}},r)},\;({\bm{s}},r)\in{\mathcal{M}}\right\\}$
achieves the converge rate $n^{-2\overline{\alpha}/(d+2\overline{\alpha})}$
adaptively. When the value of $d$ is large, this rate becomes slow, which is,
as a well-known phenomenon, called the curse of dimensionality. To circumvent
it, we impose structural assumptions on ${\bm{\gamma}}^{\star}$ in this
section and consider additional models to implement our procedure. We mainly
discuss two examples of the structural assumptions: generalized additive
structure and multiple index structure.

We begin with setting some notations. Let $k\in{\mathbb{N}}^{*}$ and
${\bm{w}}=(w_{1},\ldots,w_{k})\in\left[{0,1}\right]^{k}$. For a vector
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{k})\in({\mathbb{R}}_{+}^{*})^{k}$
with $\alpha_{j}=r_{j}+\alpha_{j}^{\prime}$, $r_{j}\in{\mathbb{N}}$ and
$\alpha^{\prime}_{j}\in(0,1]$ for $j\in\\{1,\ldots,k\\}$, Hölder space
${\mathcal{H}}^{\bm{\alpha}}(\left[{0,1}\right]^{k})$ denotes the collection
of functions $f$ on $\left[{0,1}\right]^{k}$ satisfying for any
$(w_{1},\ldots,w_{j-1},w_{j+1},\ldots,w_{k})\in\left[{0,1}\right]^{k-1}$ and
all $x,y\in\left[{0,1}\right]$

$\left|\partial_{j}^{r_{j}}f(w_{1},\ldots,x,\ldots,w_{k})-\partial_{j}^{r_{j}}f(w_{1},\ldots,y,\ldots,w_{k})\right|\leq
L(f)|x-y|^{\alpha^{\prime}_{j}},$

where $\partial_{j}^{r_{j}}f$ denotes the $r_{j}$-th order partial derivative
of the function $f$ on the $j$-th component. We define the anisotropic Hölder
class ${\mathcal{H}}^{{\bm{\alpha}}}(\left[{0,1}\right]^{k},L)$ as the
collection of all the functions
$f\in{\mathcal{H}}^{{\bm{\alpha}}}(\left[{0,1}\right]^{k})$ with
$L(f)+\inf\overline{L}\leq L$, where the infimum runs among all $\overline{L}$
such that

$|f({\bm{w}})-f({\bm{w}}^{\prime})|\leq\overline{L}\sum_{j=1}^{k}|w_{j}-w^{\prime}_{j}|^{\alpha_{j}\wedge
1},\text{\ for all\ }{\bm{w}},{\bm{w}}^{\prime}\in\left[{0,1}\right]^{k}$

and define ${\mathcal{H}}^{{\bm{\alpha}}}(L,v_{-},v_{+})$ as the collection of
all functions $f\in{\mathcal{H}}^{{\bm{\alpha}}}(\left[{0,1}\right]^{k},L)$
taking values in $\left[{v_{-},v_{+}}\right]\subset I$ with $v_{-}<v_{+}$.

Given $t_{j}\in{\mathbb{N}}^{*}$, $1\leq j\leq k$, for any
$h_{j}\in\Phi(t_{j})=\\{0,\ldots,t_{j}-1\\}$, we define

(20) $I^{\prime}_{j}(h_{j})=\left\\{\begin{aligned}
&\left[{0,1/t_{j}}\right]&,&\quad h_{j}=0,\\\ &\
(h_{j}/t_{j},(h_{j}+1)/t_{j}]&,&\quad
h_{j}=1,\ldots,t_{j}-1.\end{aligned}\right.$

For a given $k\in{\mathbb{N}}^{*}$ and
${\bm{t}}=(t_{1},\ldots,t_{k})\in({\mathbb{N}}^{*})^{k}$, we denote
$M_{\bm{t}}^{{\mathcal{H}},k}$ the resulted partition of
$\left[{0,1}\right]^{k}$ into the union of $\prod_{j=1}^{k}t_{j}$
hyperrectangles

$\cup_{(h_{1},\ldots,h_{k})\in\Phi(t_{1})\times\cdots\times\Phi(t_{k})}\prod_{j=1}^{k}I^{\prime}_{j}(h_{j}),$

where on $j$-th direction the interval $\left[{0,1}\right]$ is divided into
$t_{j}$ regular subintervals, for $j\in\\{1,\ldots,k\\}$. For any
$k\in{\mathbb{N}}^{*}$, ${\bm{t}}\in({\mathbb{N}}^{*})^{k}$ and
$r\in{\mathbb{N}}$, we denote
$\overline{\bm{S}}^{{\mathcal{H}},k}_{({\bm{t}},r)}$ the space of piecewise
polynomial functions $f$ on $\left[{0,1}\right]^{k}$ such that the restriction
of $f$ to each hyperrectangle is a polynomial in $k$ variables of degree not
larger than $r$ for each variable and
${\bm{S}}^{{\mathcal{H}},k}_{({\bm{t}},r)}$ the collection of functions with
the same form as the ones belonging to
$\overline{\bm{S}}^{{\mathcal{H}},k}_{({\bm{t}},r)}$ apart from restricting
the coefficients in front of the polynomial basis to be rational numbers. With
a similar argument as the proof of Lemma 1, the following result is easy to
obtain.

######  Lemma 3.

For any $k\in{\mathbb{N}}^{*}$, ${\bm{t}}\in({\mathbb{N}}^{*})^{k}$ and
$r\in{\mathbb{N}}$, ${\bm{S}}_{({\bm{t}},r)}^{{\mathcal{H}},k}$ is dense in
$\overline{\bm{S}}_{({\bm{t}},r)}^{{\mathcal{H}},k}$ with respect to the
supremum norm $\|\cdot\|_{\infty}$.

###  5.1. Generalized additive structure

Generalized additive functions, as a classical structural assumption, have
been considered in many statistical literatures. Let
$\alpha,L\in{\mathbb{R}}_{+}^{*}$,
${\bm{\beta}}=(\beta_{1},\ldots,\beta_{d})\in({\mathbb{R}}_{+}^{*})^{d}$,
${\bf{p}}=(p_{1},\ldots,p_{d})\in({\mathbb{R}}_{+}^{*})^{d}$ and
${\bf{R}}=(R_{1},\ldots,R_{d})\in({\mathbb{R}}_{+}^{*})^{d}$. We denote
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$
the collection of functions
${\bm{\gamma}}:\left[{0,1}\right]^{d}\rightarrow\left[{v_{-},v_{+}}\right]\subset
I$ of the following form

${\bm{\gamma}}({\bm{w}})=f\left(\sum_{j=1}^{d}g_{j}(w_{j})\right),\mbox{\quad
for all\ }{\bm{w}}=(w_{1},\ldots,w_{d})\in\left[{0,1}\right]^{d},$

where $f\in{\mathcal{H}}^{\alpha}(L,v_{-},v_{+})$ and $g_{j}\in
B^{\beta_{j}}_{p_{j},p_{j}}(\left[{0,1}\right],R_{j})$ taking values in
$\left[{0,1/d}\right]$, for $j\in\\{1,\ldots,d\\}$.

We assume the regression function
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$
but without the knowledge of $\alpha$, ${\bm{\beta}}$, ${\bf{p}}$, $L$ and
${\bf{R}}$. To estimate ${\bm{\gamma}}^{\star}$ by our model selection
procedure, we need to first build suitable approximation models for the class
of functions
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$.

To approximate the Besov class of functions
$B^{\beta_{j}}_{p_{j},p_{j}}(\left[{0,1}\right],R_{j})$, we consider the
family
$\left\\{\overline{\bm{S}}_{(s,r)}^{{\mathcal{B}},1},\;(s,r)\in{\mathbb{N}}\times{\mathbb{N}}\right\\}$
introduced in Section 4.1 taking $d=1$. We recall that the functions belonging
to the above family are built based on the collection of particular partitions
$M^{{\mathcal{B}},1}=\cup_{s\in{\mathbb{N}}}M_{s}^{{\mathcal{B}},1}$.
Therefore, we can rewrite the family in an alternative way
$\left\\{\overline{\bm{S}}_{(\pi,r)}^{{\mathcal{B}},1},\;(\pi,r)\in
M^{{\mathcal{B}},1}\times{\mathbb{N}}\right\\}$. To approximate the Hölder
class of functions ${\mathcal{H}}^{\alpha}(\left[{0,1}\right],L)$ with values
in $\left[{v_{-},v_{+}}\right]$, we consider the family
$\left\\{{\overline{\bm{\Gamma}}}^{{\mathcal{H}},1}_{(t,r)},\;(t,r)\in{\mathbb{N}}^{*}\times{\mathbb{N}}\right\\}$,
where
${\overline{\bm{\Gamma}}}^{{\mathcal{H}},1}_{(t,r)}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}^{{\mathcal{H}},1}_{(t,r)}\right\\}$.

For any $r\in{\mathbb{N}}$, $t\in{\mathbb{N}}^{*}$ and
${\bm{\pi}}=(\pi_{1},\ldots,\pi_{d})\in(M^{{\mathcal{B}},1})^{d}$, we define
${\overline{\bm{\Gamma}}}^{A}_{({\bm{\pi}},t,r)}$ the collection of all the
functions ${\bm{\gamma}}$ on ${\mathscr{W}}=\left[{0,1}\right]^{d}$ of the
form

(21) ${\bm{\gamma}}({\bm{w}})=f\left[{(g({\bm{w}})\vee 0)\wedge
1}\right],\mbox{\quad for all\
}{\bm{w}}=(w_{1},\ldots,w_{d})\in\left[{0,1}\right]^{d},$

where $g({\bm{w}})=\sum_{j=1}^{d}g_{j}(w_{j})$ with
$g_{j}\in\overline{\bm{S}}^{{\mathcal{B}},1}_{(\pi_{j},r)}$, for
$j\in\\{1,\ldots,d\\}$ and
$f\in{\overline{\bm{\Gamma}}}^{{\mathcal{H}},1}_{(t,r)}$. The following result
reveals the upper bound of the VC dimension for the class of functions
${\overline{\bm{\Gamma}}}^{A}_{({\bm{\pi}},t,r)}$.

######  Proposition 1.

Given $r\in{\mathbb{N}}$, $t\in{\mathbb{N}}^{*}$ and
${\bm{\pi}}\in(M^{{\mathcal{B}},1})^{d}$, the class of functions
${\overline{\bm{\Gamma}}}^{A}_{({\bm{\pi}},t,r)}$ is a VC-subgraph on
$\left[{0,1}\right]^{d}$ with dimension

$V^{A}_{({\bm{\pi}},t,r)}\leq
2+\left[{t(r+1)+2\sum_{j=1}^{d}|\pi_{j}|(r+1)}\right]\log_{2}\left[{4eU\log_{2}\left(2eU\right)}\right],$

where $U=t+r+2$.

The proof is postponed to Appendix C. For each $r\in{\mathbb{N}}$,
$t\in{\mathbb{N}}^{*}$ and
${\bm{\pi}}=(\pi_{1},\ldots,\pi_{d})\in(M^{{\mathcal{B}},1})^{d}$, we take the
countable subset $\bm{\Gamma}^{A}_{({\bm{\pi}},t,r)}$ of
${\overline{\bm{\Gamma}}}^{A}_{({\bm{\pi}},t,r)}$ defined as

$\bm{\Gamma}^{A}_{({\bm{\pi}},t,r)}=\left\\{f\left[{(g\vee 0)\wedge
1}\right],\;f\in\bm{\Gamma}^{{\mathcal{H}},1}_{(t,r)},\;g_{j}\in{\bm{S}}^{{\mathcal{B}},1}_{(\pi_{j},r)},\;j=1,\ldots,d\right\\},$

where ${\bm{S}}^{{\mathcal{B}},1}_{(\pi_{j},r)}$ is the rational version of
$\overline{\bm{S}}^{{\mathcal{B}},1}_{(\pi_{j},r)}$ which has been introduced
in Section 4.1,
$\bm{\Gamma}^{{\mathcal{H}},1}_{(t,r)}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{S}}^{{\mathcal{H}},1}_{(t,r)}\right\\}$ and
$g({\bm{w}})=\sum_{j=1}^{d}g_{j}(w_{j})$, for all
${\bm{w}}=(w_{1},\ldots,w_{d})\in\left[{0,1}\right]^{d}$.

Let
${\mathcal{M}}=(M^{{\mathcal{B}},1})^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}$.
For any
$({\bm{\pi}},t,r)\in(M^{{\mathcal{B}},1})^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}$,
we associate it with the weight

(22) $\Delta({\bm{\pi}},t,r)=3\log 2\left(\sum_{j=1}^{d}|\pi_{j}|\right)+r+t.$

The following result shows inequality (7) is satisfied with the weights
defined by (22).

######  Lemma 4.

With the weights defined by (22), we have

$\sum_{({\bm{\pi}},t,r)\in(M^{{\mathcal{B}},1})^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}}e^{-\Delta({\bm{\pi}},t,r)}\leq\frac{e}{e-1}.$

With Proposition 1 and Lemma 4, we can apply the model selection procedure
introduced in Section 3.2 and obtain the following.

######  Corollary 2.

Under Assumption 2, no matter what the distribution of $W$ is, the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ given by the model selection procedure in
Section 3.2 over
$\left\\{\bm{\Gamma}^{A}_{({\bm{\pi}},t,r)},\;({\bm{\pi}},t,r)\in(M^{{\mathcal{B}},1})^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}\right\\}$
with the weights defined by (22) satisfies for all
$\alpha,L\in{\mathbb{R}}_{+}^{*}$ and
${\bm{\beta}},{\bf{p}},{\bf{R}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\beta_{j}>1/p_{j}$

$\displaystyle\sup_{{\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})}C^{\prime}_{\kappa,d,\alpha,{\bm{\beta}},{\bf{p}}}{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
(23) $\displaystyle\leq\left\\{\left[{\sum_{j=1}^{d}\left(LR_{j}^{\alpha\wedge
1}\right)^{\frac{2}{2(\alpha\wedge 1)\beta_{j}+1}}n^{-\frac{2(\alpha\wedge
1)\beta_{j}}{2(\alpha\wedge
1)\beta_{j}+1}}}\right]+L^{\frac{2}{2\alpha+1}}n^{-\frac{2\alpha}{2\alpha+1}}+\frac{1}{n}\right\\}{\mathcal{L}}_{n}^{2},$

where ${\mathcal{L}}_{n}=\log n\vee\log L^{2}\vee 1$ and
$C^{\prime}_{\kappa,d,\alpha,{\bm{\beta}},{\bf{p}}}$ is a constant depending
on $\kappa$, $d$, $\alpha$, ${\bm{\beta}}$ and ${\bf{p}}$.

Corollary 2 tells that in the ideal situation
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$
for some $\alpha$, ${\bm{\beta}}$, ${\bf{p}}$, $L$ and ${\mathbf{R}}$, the
converge rate of the estimator is independent of $d$ which entails the
procedure does not suffer from the curse of dimensionality. When
$R^{\star}\not=R_{{\bm{\gamma}}^{\star}}$ or ${\bm{\gamma}}^{\star}$ exists
but does not belong to any
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$,
a bias term will be added into the risk bound in Corollary 2. However, as long
as the bias term is not too large compared to the quantity on the right hand
side of (23), the accuracy of the resulted estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ remains the same magnitude as the ideal case
which confirms the robustness of our estimator.

###  5.2. Multiple index structure

Let ${\mathcal{C}}_{d}$ be the unit ball for the $\ell_{1}$-norm, i.e.

${\mathcal{C}}_{d}=\left\\{(c_{1},\ldots,c_{d})\in{\mathbb{R}}^{d},\;\sum_{j=1}^{d}|c_{j}|\leq
1\right\\}.$

For some known $l\in{\mathbb{N}}^{*}$ (typically $l\leq d$), we denote
${\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$ the collection
of all the functions ${\bm{\gamma}}$ of the following form

(24) ${\bm{\gamma}}({\bm{w}})=f\circ g({\bm{w}}),\mbox{\quad for all\
}{\bm{w}}=(w_{1},\ldots,w_{d})\in\left[{0,1}\right]^{d},$

where $g:\left[{0,1}\right]^{d}\rightarrow\left[{0,1}\right]^{l}$ defined as
$g({\bm{w}})=(g_{1}({\bm{w}}),\ldots,g_{l}({\bm{w}}))$ with

$g_{j}({\bm{w}})=\frac{1}{2}\left[\langle
a_{j},{\bm{w}}\rangle+1\right],\;a_{j}\in{\mathcal{C}}_{d}\mbox{\quad for all
\ }j\in\\{1,\ldots,l\\}$

and $f\in{\mathcal{H}}^{\bm{\alpha}}\left(L,v_{-},v_{+}\right)$ mapping
$\left[{0,1}\right]^{l}$ to $\left[{v_{-},v_{+}}\right]\subset I$ with
$L\in{\mathbb{R}}^{*}_{+}$,
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{l})\in({\mathbb{R}}_{+}^{*})^{l}$
and $v_{-}<v_{+}$. We assume
${\bm{\gamma}}^{\star}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$
but without knowing the values of ${\bm{\alpha}}$ and $L$.

To approximate the Hölder classes on $\left[{0,1}\right]^{l}$ with values in
$\left[{v_{-},v_{+}}\right]$, we adopt the same strategy by considering the
family
$\left\\{{\overline{\bm{\Gamma}}}_{({\bm{t}},r)}^{{\mathcal{H}},l},\;({\bm{t}},r)\in({\mathbb{N}}^{*})^{l}\times{\mathbb{N}}\right\\}$,
where
${\overline{\bm{\Gamma}}}_{({\bm{t}},r)}^{{\mathcal{H}},l}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}_{({\bm{t}},r)}^{{\mathcal{H}},l}\right\\}$.
Let $\left[{l}\right]=\\{1,\ldots,l\\}$. For any $r\in{\mathbb{N}}$ and
${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$, we define the class
of functions ${\overline{\bm{\Gamma}}}^{M}_{({\bm{t}},r)}$ on
${\mathscr{W}}=\left[{0,1}\right]^{d}$ as

${\overline{\bm{\Gamma}}}^{M}_{({\bm{t}},r)}=\left\\{f\left(g_{1}(\cdot),\ldots,g_{l}(\cdot)\right),\;f\in{\overline{\bm{\Gamma}}}^{{\mathcal{H}},l}_{({\bm{t}},r)},\;g_{j}=\frac{1}{2}\left[\langle
a_{j},\cdot\rangle+1\right],\;a_{j}\in{\mathcal{C}}_{d},\;j\in\left[{l}\right]\right\\}.$

The following result entails that
${\overline{\bm{\Gamma}}}^{M}_{({\bm{t}},r)}$ is VC-subgraph on
${\mathscr{W}}$.

######  Proposition 2.

For any $r\in{\mathbb{N}}$ and
${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$, the class of
functions ${\overline{\bm{\Gamma}}}^{M}_{({\bm{t}},r)}$ is a VC-subgraph on
${\mathscr{W}}=\left[{0,1}\right]^{d}$ with dimension

(25) $V^{M}_{({\bm{t}},r)}\leq
2+\left[{2ld+\left(\prod_{j=1}^{l}t_{j}\right)(r+1)^{l}}\right]\log_{2}\left[{4eU\log_{2}\left(2eU\right)}\right],$

where $U=\sum_{j=1}^{l}t_{j}+lr+l+1$.

The proof is postponed to Appendix C. For any $r\in{\mathbb{N}}$ and
${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$, we take the
countable subset $\bm{\Gamma}^{M}_{({\bm{t}},r)}$ of
${\overline{\bm{\Gamma}}}^{M}_{({\bm{t}},r)}$ defined as

$\bm{\Gamma}^{M}_{({\bm{t}},r)}=\left\\{f\left(g_{1}(\cdot),\ldots,g_{l}(\cdot)\right),\;f\in\bm{\Gamma}^{{\mathcal{H}},l}_{({\bm{t}},r)},\;g_{j}=\frac{\left[\langle
a_{j},\cdot\rangle+1\right]}{2},\;a_{j}\in{\mathcal{C}}_{d}\cap{\mathbb{Q}}^{d},\;j\in\left[{l}\right]\right\\},$

where $\bm{\Gamma}^{{\mathcal{H}},l}_{({\bm{t}},r)}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}\right\\}$
with ${\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}$ the countable subset of
$\overline{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}$ as we introduced in the
beginning of this section.

Let ${\mathcal{M}}=({\mathbb{N}}^{*})^{l}\times{\mathbb{N}}$. For any
$r\in{\mathbb{N}}$ and ${\bm{t}}\in({\mathbb{N}}^{*})^{l}$, we associate it
with the weight

(26) $\Delta({\bm{t}},r)=\sum_{j=1}^{l}t_{j}+r.$

The following result shows inequality (7) is satisfied with the weights
defined by (26).

######  Lemma 5.

With the weights defined by (26), we have

$\sum_{({\bm{t}},r)\in({\mathbb{N}}^{*})^{l}\times{\mathbb{N}}}e^{-\Delta({\bm{t}},r)}\leq\frac{e}{e-1}.$

The proof is postponed to Appendix B. With Proposition 2 and Lemma 5, we are
able to apply the model selection procedure introduced in Section 3.2 and
obtain the following.

######  Corollary 3.

Under Assumption 2, no matter what the distribution of $W$ is, the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ given by the model selection procedure in
Section 3.2 over
$\left\\{\bm{\Gamma}^{M}_{({\bm{t}},r)},\;({\bm{t}},r)\in({\mathbb{N}}^{*})^{l}\times{\mathbb{N}}\right\\}$
with the weights defined by (26) satisfies for all
${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$ and $L>0$,

$\displaystyle\sup_{{\bm{\gamma}}^{\star}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)}{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq
C_{\kappa,l,{\bm{\alpha}}}\left(L^{\frac{2l}{2\overline{\alpha}+l}}n^{-\frac{2\overline{\alpha}}{2\overline{\alpha}+l}}+\frac{d}{n}\right){\mathcal{L}}_{n}^{2},$

where ${\mathcal{L}}_{n}=\log n\vee\log L^{2}\vee 1$ and
$C_{\kappa,l,{\bm{\alpha}}}$ is a constant depending only on $\kappa$, $l$ and
${\bm{\alpha}}$.

The result tells that if for some ${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$
and $L>0$,
${\bm{\gamma}}^{\star}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$
where the value of $l$ is smaller than $d$, we mitigate the curse of
dimensionality by taking the information that ${\bm{\gamma}}^{\star}$ is a
multiple index function. If it is not the case, a bias term will be added into
the risk bound in Corollary 3. But as long as the conditional distribution
$R^{\star}$ is not far away from some set of conditional distributions
$\left\\{R_{{\bm{\gamma}}},\;{\bm{\gamma}}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)\right\\}$,
the performance of our estimator will not deteriorate too much.

When the value of $l$ is large ($l>d$), the multiple index model (24) does not
help to circumvent the curse of dimensionality. In this situation, we could
assume ${\bm{\gamma}}^{\star}$ has an additive structure, i.e.

${\bm{\gamma}}^{\star}({\bm{w}})=\sum_{j=1}^{l}{\bm{\gamma}}_{j}\left(\frac{\langle
a_{j},{\bm{w}}\rangle+1}{2}\right),\mbox{\quad for all\
}{\bm{w}}\in\left[{0,1}\right]^{d},$

where $a_{j}\in{\mathcal{C}}_{d}$. Imposing some smoothness on
${\bm{\gamma}}_{j}$, we can construct models and perform our model selection
procedure to mitigate the curse of dimensionality. The construction is similar
to a combination of what we have done in Section 5.1 and 5.2.

##  6\. Model selection for neural networks

Throughout this section, we assume the covariates $W_{i}$ are i.i.d. on
$\left[{0,1}\right]^{d}$ with the common distribution $P_{W}$ and
$R_{i}^{\star}=R_{{\bm{\gamma}}^{\star}}$ for all $i\in\\{1,\ldots,n\\}$. The
idea in this section is to estimate the regression function
${\bm{\gamma}}^{\star}$ by our model selection procedure based on ReLU neural
networks.

We start with setting some notations. We recall the Rectifier Linear Unit
(ReLU) activation function $\sigma:{\mathbb{R}}\rightarrow{\mathbb{R}}$
defined as

$\sigma(x)=\max(0,x).$

For any vector ${\bm{x}}=(x_{1},\ldots,x_{p})^{\top}\in{\mathbb{R}}^{p}$ with
some $p\in{\mathbb{N}}^{*}$, by writing $\sigma({\bm{x}})$ we mean the
activation function operating component-wise, i.e.

$\sigma({\bm{x}})=(\max\\{0,x_{1}\\},\ldots,\max\\{0,x_{d}\\})^{\top}.$

We formulate $\overline{\bm{S}}_{(L,p)}$ the Multi-Layer Perception (MLP) with
width $p\in{\mathbb{N}}^{*}$ and depth $L\in{\mathbb{N}}^{*}$, which is a
collection of functions of the form

(27) $f:{\mathbb{R}}^{d}\rightarrow{\mathbb{R}},\quad{\bm{w}}\mapsto
f({\bm{w}})=M_{L}\circ\sigma\circ M_{L-1}\circ\cdots\circ\sigma\circ
M_{0}({\bm{w}}),$

where

$M_{l}({\bm{y}})=A_{l}({\bm{y}})+b_{l},\mbox{\quad for\ }l=0,\ldots,L,$

$A_{l}$ is a $p\times p$ weight matrix for $l\in\\{1,\ldots,L-1\\}$, $A_{0}$
has size $p\times d$, $A_{L}$ has size $1\times p$ and the shift vectors
$b_{l}$ is of size $p$ if $l\in\\{0,\ldots,L-1\\}$, a scalar if $l=L$. All the
parameters in weight matrices and shift vectors vary in ${\mathbb{R}}$. We
denote the MLP as ${\bm{S}}_{(L,p)}$ when it has the same architecture as
$\overline{\bm{S}}_{(L,p)}$ but all the parameters in weight matrices and
shift vectors vary in ${\mathbb{Q}}$.

Besides learning all the parameters in weight matrices and shift vectors,
people also enforce their algorithm on some sparse neural networks depending
on the problem they want to solve. Some examples can be found in Section 7.10
of Goodfellow et al., (2016). Another more tuitive example for the sparse
setting is the convolutional neural network (CNN) which has been widely used
in computer vision, sequence analysis in bioinformatics and natural language
processing.

We formulate the sparse ReLU neural networks as follows. For
$l\in\\{0,\ldots,L\\}$, we define ${\bm{s}}_{l}$ the indicator vector in which
the component is either 0 or 1. The size of the vector ${\bm{s}}_{l}$ equals
to the total number of parameters in weight matrix $A_{l}$ and shift vector
$b_{l}$. For $l=0$, ${\bm{s}}_{0}$ is of size $p(d+1)$, for
$l\in\\{1,\ldots,L-1\\}$, ${\bm{s}}_{l}$ is of size $p(p+1)$ and for $l=L$,
${\bm{s}}_{L}$ is of size $p+1$. Essentially, indicator vectors
${\bm{s}}_{l}$, $l\in\\{0,\ldots,L\\}$ represent collections of functions
based on the structure of neural networks. The last $p$ components in
${\bm{s}}_{l}$, $l\in\\{0,\ldots,L-1\\}$ and the last one in ${\bm{s}}_{L}$
address to the collection of shift vectors $b_{l}$. More precisely, for any
component in $b_{l}$ if the corresponding position in ${\bm{s}}_{l}$ is 1, we
allow this component in $b_{l}$ varies in ${\mathbb{R}}$ otherwise the value
of it is fixed at 0. The other components in ${\bm{s}}_{l}$ address to the
collection of weight matrices $A_{l}$ with the same way as we have introduced
to $b_{l}$ after reshaping the matrices one row after another into vectors. To
illustrate, we take $p=2$, $L=3$ and $l=1$ as an example. Let
${\bm{s}}_{1}=(1,0,0,1,1,0)^{\top}$ which is a vector of size $6$. As
mentioned before, $A_{1}$ is a $2\times 2$ matrix which we write as

$A_{1}=\begin{pmatrix}a_{1}&a_{3}\\\ a_{4}&a_{2}\end{pmatrix}$

and $b_{1}$ is of size 2. The last 2 components in ${\bm{s}}_{1}$ is
$(1,0)^{\top}$ which entails that the first component in $b_{1}$ varies in
${\mathbb{R}}$ and the second is fixed at 0. We then reshape $A_{1}$ one row
after another into a vector, namely $(a_{1},a_{3},a_{4},a_{2})^{\top}$. The
remaining components of ${\bm{s}}_{l}$ is $(1,0,0,1)^{\top}$ which entails
$a_{1}$ and $a_{2}$ are allowed varying in ${\mathbb{R}}$ while
$a_{3},a_{4}=0$. To conclude, such an indicator vector ${\bm{s}}_{1}$
corresponds to the collection of weight matrices

$A_{1}=\begin{pmatrix}a_{1}&0\\\ 0&a_{2}\end{pmatrix},\mbox{\quad with\
}a_{1},a_{2}\in{\mathbb{R}}$

and shift vectors $b_{1}=(b,0)^{\top}$ with $b\in{\mathbb{R}}$.

Given $p,L\in{\mathbb{N}}^{*}$ and a joint indicator vector
${\bm{s}}=({\bm{s}}_{0}^{\top},\ldots,{\bm{s}}_{L}^{\top})^{\top}$, we denote
$\overline{\bm{S}}_{(L,p,{\bm{s}})}$ as the corresponding collection of
functions on $\left[{0,1}\right]^{d}$. Similarly, ${\bm{S}}_{(L,p,{\bm{s}})}$
denotes the class of functions with the same architecture as
$\overline{\bm{S}}_{(L,p,{\bm{s}})}$, where the non-zero parameters vary in
${\mathbb{Q}}$ but not ${\mathbb{R}}$. Let us remark that given
$L\in{\mathbb{N}}^{*}$, $p\in{\mathbb{N}}^{*}$, ${\bm{s}}$ is of size

$\overline{p}=p^{2}(L-1)+p(L+d+1)+1.$

The following result gives an upper bound of the VC dimension for the class of
the functions $\overline{\bm{S}}_{(L,p,{\bm{s}})}$ on
${\mathscr{W}}=\left[{0,1}\right]^{d}$.

######  Proposition 3.

For any $L\in{\mathbb{N}}^{*}$, $p\in{\mathbb{N}}^{*}$ and
${\bm{s}}\in\\{0,1\\}^{\overline{p}}$, a fixed designed neural network
$\overline{\bm{S}}_{(L,p,{\bm{s}})}$ is a VC-subgraph on ${\mathscr{W}}$ with
dimension

$V_{(L,p,{\bm{s}})}\leq(L+1)(\|{\bm{s}}\|_{0}+1)\log_{2}\left[{2\left(2e(L+1)\left(\frac{pL}{2}+1\right)\right)^{2}}\right],$

where $\|{\bm{s}}\|_{0}$ denotes the number of non-zero components in
${\bm{s}}$.

The proof is postponed to Appendix C. In particular, when all the components
in ${\bm{s}}$ are 1, $\overline{\bm{S}}_{(L,p,{\bm{s}})}$ is the Multi-Layer
Perception $\overline{\bm{S}}_{(L,p)}$ and Proposition 3 entails the VC
dimension of $\overline{\bm{S}}_{(L,p)}$ is, up to a constant, bounded by
$\overline{p}L\log\left[{(L+1)\left(pL/2+1\right)}\right]$.

###  6.1. The Takagi class of functions

We provide an example in this subsection where estimation based on ReLU neural
networks enjoys a significant advantage.

Let $v_{-},v_{+}\in{\mathbb{R}}$ such that $v_{-}<v_{+}$ and
$\left[{v_{-},v_{+}}\right]\subset I$. For any $t\in(-1,1)$,
$l\in{\mathbb{N}}^{*}$,
${\bf{p}}=(p_{1},p_{2})\in{\mathbb{N}}^{*}\times{\mathbb{N}}^{*}$ and $K\geq
0$, we denote ${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$ the
collection of functions where for all
$f\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$, it takes
values in $\left[{v_{-},v_{+}}\right]\subset I$ and is of the form

(28) $f(w)=\sum_{k\in{\mathbb{N}}^{*}}t^{k}g\left(h^{\circ
k}(w)\right),\;\mbox{\quad for all\ }w\in\left[{0,1}\right],$

where $g\in{\bm{S}}_{(l,p_{1})}$ defined on $\left[{0,1}\right]$,
$\|g\|_{\infty}\leq K$, $h\in{\bm{S}}_{(l,p_{2})}$ maps $\left[{0,1}\right]$
to $\left[{0,1}\right]$ and $h^{\circ k}=h\circ\cdots\circ h$ denotes the
resulted function when $h$ is composed with itself $k$ times. We assume the
regression function
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$
but without the knowledge of $t$, $l$, ${\bf{p}}$ and $K$. This type of
setting provides elementary examples of self similar functions and dynamical
systems (see Yamaguti and Hata, (1983) for example). It also includes a number
of interesting functions belonging to the Takagi class (Daubechies et al.,
(2019) p.28), which is defined as the collection of all the functions of the
form

$f=\sum_{k\in{\mathbb{N}}^{*}}c_{k}h^{\circ k},$

where $(c_{k})_{k\in{\mathbb{N}}^{*}}$ is an absolutely summable sequence of
real numbers and $h$ is the hat function defined on $\left[{0,1}\right]$ as

(29) $h(w)=\left\\{\begin{aligned} &2w&,&\quad 0\leq w\leq\frac{1}{2},\\\
&2(1-w)&,&\quad\frac{1}{2}<w\leq 1.\end{aligned}\right.$

Let ${\mathcal{M}}={\mathbb{N}}^{*}\times{\mathbb{N}}^{*}$. The family of
models we consider here is given by

$\left\\{\bm{\Gamma}_{(L,p)},\;(L,p)\in{\mathbb{N}}^{*}\times{\mathbb{N}}^{*}\right\\},$

where $\bm{\Gamma}_{(L,p)}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}_{(L,p)}\right\\}$. We note that for each
$\bm{\Gamma}_{(L,p)}$, it is a countable collection of functions on
$\left[{0,1}\right]$ and satisfies Assumption 1 with $V_{(L,p)}$, up to a
constant, bounded by
$\overline{p}L\log\left[{(L+1)\left(pL/2+1\right)}\right]$. For any
$(L,p)\in{\mathcal{M}}$, we associate it with the weight

(30) $\Delta(L,p)=L+p.$

As an immediate consequence, we have
$\Sigma=\sum_{(L,p)\in({\mathbb{N}}^{*})^{2}}e^{-\Delta(L,p)}\leq 1$ which
satisfies the inequality (7). Therefore, we are able to apply the model
selection procedure introduced in Section 3.2 and obtain the following result.

######  Corollary 4.

Under Assumption 2, whatever the distribution of $W$, the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ given by the model selection procedure in
Section 3.2 over the family
$\left\\{\bm{\Gamma}_{(L,p)},\;(L,p)\in{\mathbb{N}}^{*}\times{\mathbb{N}}^{*}\right\\}$
with the weights defined by (30) satisfies for all $t\in(-1,1)$,
$l\in{\mathbb{N}}^{*}$, ${\bf{p}}\in({\mathbb{N}}^{*})^{2}$ and $K\geq 0$

$\sup_{{\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)}{\mathbb{E}}\left[{h^{2}\left({R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}}}\right)}\right]\leq
C_{\kappa,t,l,{\bf{p}},K}\frac{1}{n}(1+\log n)^{4},$

where $C_{\kappa,t,l,{\bf{p}},K}$ is a constant depending on
$\kappa,t,l,{\bf{p}},K$ only.

The risk bound is optimal up to the logarithmic factors since any two
probabilities with a Hellinger distance smaller than
${\mathcal{O}}(1/\sqrt{n})$ are indistinguishable. To comment upon this result
further, we consider a specific example of ${\bm{\gamma}}^{\star}$ in Gaussian
regression problem with a known variance $\sigma>0$. We parametrize the
exponential family $\widetilde{\mathscr{Q}}=\left\\{R_{\gamma},\;\gamma\in
I\right\\}$ by taking $\gamma=\theta/(2\sqrt{2}\sigma)$, where $\theta$ is the
mean so that by Proposition 2 of Baraud and Chen, (2020), Assumption 2 is
satisfied with $\kappa=1$ and $I={\mathbb{R}}$. We therefore can take $v_{-}$
the smallest integer in computer and $v_{+}$ the largest so that
$\left[{v_{-},v_{+}}\right]\subset I$. Let
${\bm{\gamma}}^{\star}=\sum_{k\in{\mathbb{N}}^{*}}2^{-k}h^{\circ k}$ with $h$
defined by (29) be a function belonging to the Takagi class. This corresponds
to the situation where $g$ is the identity function on $\left[{0,1}\right]$ so
that $K=1$ and $t=1/2$ in the general formalization (28). We also observe that
$g\in{\bm{S}}_{(1,1)}$ and $h\in{\bm{S}}_{(1,2)}$ by rewriting them into the
following forms

$g(x)=\sigma\left(x+0\right),\mbox{\quad for all\ }x\in\left[{0,1}\right]$

and

$h(w)=\left(\begin{array}[]{cc}2&-4\\\
\end{array}\right)\sigma\left\\{\left(\begin{array}[]{c}1\\\
1\end{array}\right)w+\left(\begin{array}[]{c}0\\\
-\frac{1}{2}\end{array}\right)\right\\},\mbox{\quad for all\
}w\in\left[{0,1}\right].$

Therefore, we have
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(1/2,1,(1,2),1)$.
According to Corollary 4, the estimator $\widehat{\bm{\gamma}}({\bm{X}})$
obtained by the model selection procedure introduced in Section 3.2 based on
the fully connected ReLU neural networks converges to ${\bm{\gamma}}^{\star}$
with a rate of order $1/n$ up to logarithmic factors. However,
${\bm{\gamma}}^{\star}$ is nowhere differentiable hence it has very little
smoothness in the classical sense. Estimation based on the traditional models
will result a miserably slow rate considering the minimax converge rate for an
$\alpha$-smooth function is of order $n^{-2\alpha/(2\alpha+1)}$.

###  6.2. Composite Hölder class of functions

We have seen in the last subsection that the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ based on MLPs converges to the truth with an
optimal rate for some class of functions. In this subsection, we continue to
consider the problem of circumventing the curse of dimensionality based on
deep ReLU neural networks. A natural structure of the regression function
${\bm{\gamma}}^{\star}$ for neural networks to exhibit advantages could be a
composition of several functions which has been considered by Schmidt-Hieber,
(2020) for Gaussian regression. We shall reconsider it from another point of
view where we perform our model selection procedure based on the result of
controlling the VC dimension of sparse ReLU neural networks.

Let us introduce notations first. Given $t\in{\mathbb{N}}^{*}$ and
$\alpha\in{\mathbb{R}}_{+}^{*}$, we define ${\mathcal{C}}^{\alpha}_{t}(D,K)$
an $\alpha$-Hölder ball with radius $K$ as the collection of functions
$f:D\subset{\mathbb{R}}^{t}\rightarrow{\mathbb{R}}$ such that

$\sum_{\begin{subarray}{c}{\bm{\beta}}=(\beta_{1},\ldots,\beta_{t})\in{\mathbb{N}}^{t}\\\
\sum_{j=1}^{t}\beta_{j}<\alpha\end{subarray}}\|\partial^{\bm{\beta}}f\|_{\infty}+\sum_{\begin{subarray}{c}{\bm{\beta}}\in{\mathbb{N}}^{t}\\\
\sum_{j=1}^{t}\beta_{j}=\lfloor\alpha\rfloor\end{subarray}}\sup_{\begin{subarray}{c}{\bm{x}},{\bm{y}}\in
D\\\
{\bm{x}}\not={\bm{y}}\end{subarray}}\frac{\left|\partial^{\bm{\beta}}f({\bm{x}})-\partial^{\bm{\beta}}f({\bm{y}})\right|}{|{\bm{x}}-{\bm{y}}|_{\infty}^{\alpha-\lfloor\alpha\rfloor}}\leq
K,$

where for any ${\bm{\beta}}=(\beta_{1},\ldots,\beta_{t})\in{\mathbb{N}}^{t}$,
$\partial^{\bm{\beta}}=\partial^{\beta_{1}}\cdots\partial^{\beta_{t}}$ and for
any ${\bm{x}}=(x_{1},\ldots,x_{t})\in{\mathbb{R}}^{t}$,
$|{\bm{x}}|_{\infty}=\max_{i=1,\ldots,t}|x_{i}|$.

For any $k\in{\mathbb{N}}^{*}$,
${\bf{d}}=(d_{0},\ldots,d_{k})\in({\mathbb{N}}^{*})^{k+1}$,
${\bf{t}}=(t_{0},\ldots,t_{k})\in({\mathbb{N}}^{*})^{k+1}$,
${\bm{\alpha}}=(\alpha_{0},\ldots,\alpha_{k})\in({\mathbb{R}}_{+}^{*})^{k+1}$
and $K\geq 0$, we denote
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K)$
the class of functions with values in $\left[{v_{-},v_{+}}\right]\subset I$
as,

$\displaystyle{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K)=$
$\displaystyle\left\\{f_{k}\circ\cdots\circ
f_{0},\;f_{i}=(f_{ij})_{j}:\left[{a_{i},b_{i}}\right]^{d_{i}}\rightarrow\left[{a_{i+1},b_{i+1}}\right]^{d_{i+1}},\right.$
$\displaystyle\left.f_{ij}\in{\mathcal{C}}^{\alpha_{i}}_{t_{i}}(\left[{a_{i},b_{i}}\right]^{t_{i}},K),\;\mbox{for
some }|a_{i}|,|b_{i}|\leq K\right\\},$

where $d_{k+1}=1$. We assume the regression function
${\bm{\gamma}}^{\star}={\bm{\gamma}}_{k}\circ\cdots\circ{\bm{\gamma}}_{0}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K)$
but without the knowledge of $k$, ${\bf{d}}$, ${\bf{t}}$, ${\bm{\alpha}}$ and
$K$.

To approximate these classes of functions
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K)$,
we consider the sparse ReLU neural networks. Recall that for any
$(L,p)\in({\mathbb{N}}^{*})^{2}$,
${\bm{s}}=({\bm{s}}_{0}^{\top},\ldots,{\bm{s}}_{L}^{\top})^{\top}\in\\{0,1\\}^{\overline{p}}$
with $\overline{p}=p^{2}(L-1)+p(L+d+1)+1$ indicating the sparsity design of a
MLP with architecture $(L,p)$. More precisely, setting
${\mathcal{M}}=({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}$, we
consider the family of models based on sparse ReLU neural networks
$\left\\{{\overline{\bm{\Gamma}}}_{(L,p,{\bm{s}})},\;(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}\right\\}$,
where ${\overline{\bm{\Gamma}}}_{(L,p,{\bm{s}})}=\left\\{({\bm{\gamma}}\vee
v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}_{(L,p,{\bm{s}})}\right\\}$. The VC
dimension $V_{(L,p,{\bm{s}})}$ of each
${\overline{\bm{\Gamma}}}_{(L,p,{\bm{s}})}$ is bounded by Proposition 3. For
each $(L,p,{\bm{s}})\in{\mathcal{M}}$, we take the countable subset
$\bm{\Gamma}_{(L,p,{\bm{s}})}$ of ${\overline{\bm{\Gamma}}}_{(L,p,{\bm{s}})}$
as $\bm{\Gamma}_{(L,p,{\bm{s}})}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}_{(L,p,{\bm{s}})}\right\\}$.

For each
$(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}$, we
associate it with the weight

(31) $\Delta(L,p,{\bm{s}})=\left\\{\begin{aligned}
&\|{\bm{s}}\|_{0}\log\left(\frac{2e\overline{p}}{\|{\bm{s}}\|_{0}}\right)+p+L&,&\mbox{\quad$\|\bm{s}\|_{0}\not=0$,}\\\
&p+L&,&\mbox{\quad$\|\bm{s}\|_{0}=0$.}\end{aligned}\right.$

The following result shows (7) is satisfied with the associated weights
defined by (31).

######  Lemma 6.

For any $L\in{\mathbb{N}}^{*}$, $p\in{\mathbb{N}}^{*}$ and
${\bm{s}}=({\bm{s}}_{0}^{\top},\ldots,{\bm{s}}_{L}^{\top})^{\top}\in\\{0,1\\}^{\overline{p}}$,
we define $\Delta(L,p,{\bm{s}})$ by (31). Then,

$\sum_{(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}}e^{-\Delta(L,p,{\bm{s}})}\leq
2.$

For any
${\bm{\alpha}}=(\alpha_{0},\ldots,\alpha_{k})\in({\mathbb{R}}_{+}^{*})^{k+1}$,
we define the effective smoothness indices by
$\alpha^{\prime}_{i}=\alpha_{i}\prod_{l=i+1}^{k}\left(\alpha_{l}\wedge
1\right)$ for all $i\in\\{0,\ldots,k-1\\}$ and
$\alpha^{\prime}_{k}=\alpha_{k}$. We denote
$\phi_{n}=\max_{i=0,\ldots,k}n^{-2\alpha^{\prime}_{i}/(2\alpha^{\prime}_{i}+t_{i})}$.
Combining the result of Lemma 6 and Proposition 3, we are now able to apply
the model selection procedure in Section 3.2. The following result entails the
estimator $\widehat{\bm{\gamma}}({\bm{X}})$ converges to
${\bm{\gamma}}^{\star}$ with a rate of order $\phi_{n}$ up to logarithm
factors with respect to the distance
$d({\bm{\gamma}}^{\star},\widehat{\bm{\gamma}})=h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})$.

######  Corollary 5.

Under Assumption 2, whatever the distribution of $W$, the estimator
$\widehat{\bm{\gamma}}({\bm{X}})$ given by the model selection procedure in
Section 3.2 over the family
$\left\\{\bm{\Gamma}_{(L,p,{\bm{s}})},\;(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}\right\\}$
with the weights defined by (31) satisfies with a sufficiently large $n$, for
all $k\in{\mathbb{N}}^{*}$, $K\geq 0$, ${\bf{d}}\in({\mathbb{N}}^{*})^{k+1}$,
${\bf{t}}\in({\mathbb{N}}^{*})^{k+1}$ with $t_{j}\leq d_{j}$ for
$j\in\\{0,\ldots,k\\}$ and ${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{k+1}$,

(32)
$\sup_{{\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K)}{\mathbb{E}}\left[{h^{2}\left({R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}}}\right)}\right]\leq
C_{\kappa,k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K}\phi_{n}\log^{4}n,$

where $C_{\kappa,k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K}$ is a constant depending
on $\kappa,k,{\bf{d}},{\bf{t}},{\bm{\alpha}},K$ only.

By Corollary 5, we provide a theoretical guarantee for an alternative
estimation procedure based on sparse ReLU neural networks besides maximum
likelihood estimation (MLE) discussed in Schmidt-Hieber, (2020) for the
Gaussian regression. Our procedure is, however, designed to handle the
regression problems in exponential families and not only restricted to the
Gaussian case. It also endows the estimator an additional robust property
compared to MLE. When there is a misspecification or data contamination, as
long as the bias remains small compared to the right hand side of (32), the
behaviour of our estimator will be of the same order as the model is exact.

##  7\. Variable selection in exponential families

In this section, we propose to handle variable selection problem in
exponential families by model selection. The statistical setting is stated as
follows. Assuming that $W_{i}$ are i.i.d. on
${\mathscr{W}}\subset{\mathbb{R}}^{p}$ and for each $i\in\\{1,\ldots,n\\}$, we
observe $X_{i}=(W_{i}^{(1)},\ldots,W_{i}^{(p)},Y_{i})$ where $W_{i}^{(j)}$
represents the observation of the explanatory variable $W^{(j)}$ in the $i$-th
experiment. The value $p$ stands for the number of the explanatory variables.
This number may be large, possibly larger than $n$. The exponential family
$\widetilde{\mathscr{Q}}=\\{R_{\gamma}=r_{\gamma}\cdot\nu,\;\gamma\in I\\}$ is
parametrized in its natural form, i.e. for all $y\in{\mathscr{Y}}$, $\gamma\in
I$,

$r_{\gamma}(y)=e^{\gamma T(y)-B(\gamma)},$

which is the particular situation when taking $u$ as the identity function in
(2). We assume that there exists an unknown function ${\bm{\gamma}}^{\star}$
on ${\mathscr{W}}$ taking values in $\left[{v_{-},v_{+}}\right]\subset I$ with
$v_{-}<v_{+}$ as a linear combination of some subset of the $p$ explanatory
variables, namely

${\bm{\gamma}}^{\star}({\bm{w}})=\sum_{j=1}^{p}\gamma_{j}^{\star}w^{(j)}\text{\quad
for all\ }{\bm{w}}=(w^{(1)},\ldots,w^{(p)})\in{\mathscr{W}},$

with $\gamma_{j}^{\star}\in{\mathbb{R}}$, such that the conditional
distribution of $Y_{i}$ given $W_{i}$ belongs to a natural exponential family
with natural parameter ${\bm{\gamma}}^{\star}(W_{i})$, i.e.
$R_{{\bm{\gamma}}^{\star}(W_{i})}$. Variable selection problem attributes to
estimate this unknown ${\bm{\gamma}}^{\star}$ together with selecting the most
significant explanatory variables among the $p$ possible ones.

We set $\Omega=\\{1,\ldots,p\\}$ and ${\mathcal{M}}={\mathcal{P}}(\Omega)$.
For any subset $m\in{\mathcal{M}}$, we define $\overline{\bm{S}}_{m}$ as the
collection of functions ${\bm{\gamma}}$ on ${\mathscr{W}}$ of the form

(33) ${\bm{\gamma}}({\bm{w}})=\sum_{j=1}^{p}\gamma_{j}w^{(j)}\text{\quad for
all\ }{\bm{w}}\in{\mathscr{W}},$

where the coordinates of
$\widetilde{\gamma}=(\gamma_{1},\ldots,\gamma_{p})\in{\mathbb{R}}^{p}$ are all
zeros except for those indices $j\in m$. By convention,
$\overline{\bm{S}}_{m}=\\{0\\}$ if $m=\varnothing$. We define ${\bm{S}}_{m}$
as the collection of functions of the form given by (33) with a restriction to
the rational combinations, i.e. for any ${\bm{\gamma}}\in{\bm{S}}_{m}$,
$\widetilde{\gamma}=(\gamma_{1},\ldots,\gamma_{p})\in{\mathbb{Q}}^{p}$. For
each $m\in{\mathcal{M}}$, let us define
${\overline{\bm{\Gamma}}}_{m}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in\overline{\bm{S}}_{m}\right\\}$ and
$\bm{\Gamma}_{m}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}_{m}\right\\}$. With the fact that
${\mathbb{Q}}$ is dense in ${\mathbb{R}}$, ${\bm{S}}_{m}$ is dense in
$\overline{\bm{S}}_{m}$ for the topology of pointwise convergence. One can
observe that such dense property also holds for each $\bm{\Gamma}_{m}$ in
${\overline{\bm{\Gamma}}}_{m}$, $m\in{\mathcal{M}}$.

We define ${\mathcal{M}}_{o}=\left\\{m_{d}=\\{1,\ldots,d\\},\;1\leq d\leq
p\right\\}\cup\varnothing$. For each $m\in{\mathcal{M}}$, we associate it with
the weight

(34) $\Delta(m)=\left\\{\begin{aligned}
&2\log(1+|m|)&,&\mbox{\quad$m\in{\mathcal{M}}_{o}$,}\\\
&|m|\log\left(\frac{2ep}{|m|}\right)&,&\mbox{\quad$m\in{\mathcal{M}}\backslash{\mathcal{M}}_{o}$.}\end{aligned}\right.$

The following result shows with the weights defined by (34), inequality (7) is
satisfied.

######  Lemma 7.

Let ${\mathcal{M}}={\mathcal{P}}(\Omega)$. For any $m\in{\mathcal{M}}$, the
weight is defined by (34). Then
$\Sigma=\sum_{m\in{\mathcal{M}}}e^{-\Delta(m)}\leq 1+\pi^{2}/6$.

Moreover, for any $m\in{\mathcal{M}}$, $\overline{\bm{S}}_{m}$ defined by (33)
is a $\left|{m}\right|$-dimensional vector space. As an immediate consequence,
${\overline{\bm{\Gamma}}}_{m}$ is VC-subgraph on ${\mathscr{W}}$ with
dimension not larger than $\left|{m}\right|+1$ which satisfies the Assumption
1 with $V_{m}=\left|{m}\right|+1$. We are now able to apply the model
selection procedure presented in Section 3.2 and obtain the following result.

######  Corollary 6.

For all $m\in{\mathcal{M}}$, let
$\overline{\mathscr{Q}}_{m}=\left\\{R_{{\bm{\gamma}}},\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}\right\\}$.
Whatever the distribution of $W$, the estimator $R_{\widehat{\bm{\gamma}}}$
given by the model selection procedure in Section 3.2 over
$\left\\{\bm{\Gamma}_{m},\;m\in{\mathcal{M}}\right\\}$ associated with the
weight defined by (34) satisfies

(35)
${\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq
1.95\times 10^{7}({\mathscr{B}}_{o}\wedge{\mathscr{B}}_{c}),$

where

${\mathscr{B}}_{o}=\inf_{m\in{\mathcal{M}}_{o}}\left\\{h^{2}(R_{{\bm{\gamma}}^{\star}},\overline{\mathscr{Q}}_{m})+\frac{|m|+1}{n}\left[{1+\log_{+}\left(\frac{n}{|m|+1}\right)}\right]\right\\}$

and

${\mathscr{B}}_{c}=\inf_{m\in{\mathcal{M}}}\left\\{h^{2}(R_{{\bm{\gamma}}^{\star}},\overline{\mathscr{Q}}_{m})+\frac{|m|+1}{n}\left[{1+\log\left[{\frac{(2p)\vee
n}{|m|+1}}\right]}\right]\right\\}.$

The proof of Corollary 6 is postponed to Appendix A. Let us remark a little
bit here for the strategy of assigning weights which is different with the
typical choice, where for each $m\in{\mathcal{M}}$,

$\Delta(m)=\left\\{\begin{aligned}
&|m|\log\left(\frac{2ep}{|m|}\right)&,&\mbox{\quad$m\neq\varnothing$,}\\\
&0&,&\mbox{\quad$m=\varnothing$.}\end{aligned}\right.$

With the typical choice of the associated weights, one can derive a risk bound
${\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq
1.95\times 10^{7}{\mathscr{B}}_{c}$. Comparing this result with the one given
in (35), we note that (35) improves it by a $\log(p)$ term whenever the
minimizer $m^{\star}\in{\mathcal{M}}$ in the right hand side of (35) does
belong to ${\mathcal{M}}_{o}$.

##  Appendix A Proofs of the main theorem and its corollaries

### A.1 Proof of Theorem 1

Before starting to prove the main theorem, let us introduce some notations and
facts for later use. For all $i\in\\{1,\ldots,n\\}$, let $P_{i}^{\star}$ be
the true distribution of $X_{i}=(W_{i},Y_{i})$ and
${\mathbf{P}}^{\star}=\otimes_{i=1}^{n}P_{i}^{\star}$ be the true joint
distribution of the observed data ${\bm{X}}=(X_{1},\ldots,X_{n})$. We denote
${\mathbf{P}}_{{\bm{\gamma}}}=\otimes_{i=1}^{n}P_{i,{\bm{\gamma}}}$ as the
distribution of independent random variables
$(W_{1},Y_{1}),\ldots,(W_{n},Y_{n})$ for which the conditional distribution of
$Y_{i}$ given $W_{i}$ is given by
$R_{{\bm{\gamma}}(W_{i})}\in\widetilde{\mathscr{Q}}$ for each $i$. With the
equalities $P_{i}^{\star}=R_{i}^{\star}\cdot P_{W_{i}}$,
$P_{i,{\bm{\gamma}}}=R_{{\bm{\gamma}}}\cdot P_{W_{i}}$, we have

$h^{2}(P_{i}^{\star},P_{i,{\bm{\gamma}}})=\int_{{\mathscr{W}}}h^{2}\left({R_{i}^{\star}(w),R_{{\bm{\gamma}}(w)}}\right)dP_{W_{i}}(w).$

If we define the pseudo Hellinger distance ${\mathbf{h}}$ between two
probabilities ${\mathbf{P}}=\otimes_{i=1}^{n}P_{i}$ and
${\mathbf{P}}^{\prime}=\otimes_{i=1}^{n}P^{\prime}_{i}$ by

${\mathbf{h}}^{2}({\mathbf{P}},{\mathbf{P}}^{\prime})=\sum_{i=1}^{n}h^{2}\left({P_{i},P^{\prime}_{i}}\right),$

for any ${\bm{\gamma}}\in\bm{\Gamma}$, we have

$\displaystyle{\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{{\bm{\gamma}}})$
$\displaystyle=\sum_{i=1}^{n}\int_{{\mathscr{W}}}h^{2}\left({R_{i}^{\star}(w),R_{{\bm{\gamma}}(w)}}\right)dP_{W_{i}}(w)$
(36)
$\displaystyle=\sum_{i=1}^{n}h^{2}\left({P_{i}^{\star},P_{i,{\bm{\gamma}}}}\right)={\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathbf{P}}_{{\bm{\gamma}}}).$

We set ${\bm{\tau}}=\bigotimes_{i=1}^{n}\tau_{i}$ with
$\tau_{i}=P_{W_{i}}\otimes\nu$ for all $i\in\\{1,\ldots,n\\}$. For all
$m\in{\mathcal{M}}$, we denote by $\bm{\mathcal{R}}_{m}$ the following
families of densities (with respect to ${\bm{\tau}}$) on
${\mathscr{X}}^{n}=({\mathscr{W}}\times{\mathscr{Y}})^{n}$

$\bm{\mathcal{R}}_{m}=\\{{\mathbf{r}}_{{\bm{\gamma}}}:{\mathbf{x}}=(x_{1},\ldots,x_{n})\mapsto
r_{{\bm{\gamma}}(w_{1})}(y_{1})\ldots
r_{{\bm{\gamma}}(w_{n})}(y_{n}),\;{\bm{\gamma}}\in\bm{\Gamma}_{m}\\}$

and by ${\mathscrbf{P}}_{m}$ the corresponding $\rho$-model, i.e. the finite
or countable set of probabilities
$\left\\{{\mathbf{P}}={\mathbf{r}}_{{\bm{\gamma}}}\cdot{\bm{\tau}},\;{\bm{\gamma}}\in\bm{\Gamma}_{m}\right\\}$
with the representation $({\bm{\tau}},\bm{\mathcal{R}}_{m})$.

######  Proposition 4.

Under Assumption 1, for any $m\in{\mathcal{M}}$, the class of functions
${\mathcal{R}}_{m}=\\{r_{{\bm{\gamma}}}:\;(w,y)\mapsto
r_{{\bm{\gamma}}(w)}(y),\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}\\}$ on
${\mathscr{X}}={\mathscr{W}}\times{\mathscr{Y}}$ is VC-subgragh with dimension
not larger than $9.41V_{m}$.

###### Proof.

For any $m\in{\mathcal{M}}$, reparametrizing the exponential family in its
natural form, we obtain

${\mathcal{R}}_{m}=\\{q_{\bm{\theta}}:\;(w,y)\mapsto
e^{T(y)\bm{\theta}(w)-A(\bm{\theta}(w))},\;\bm{\theta}\in{\overline{\bm{\Theta}}}_{m}\\},$

where $A(\theta)=\log\left[{\int_{{\mathscr{Y}}}\exp(\theta
T(y))d\nu(y)}\right]$ and
${\overline{\bm{\Theta}}}_{m}=\\{\bm{\theta}=u\circ{\bm{\gamma}},\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{m}\\}$.
By Proposition 42-(ii) of Baraud et al., (2017), VC-subgraph is preserved by
composition with a monotone function. Therefore, under Assumption 1,
${\overline{\bm{\Theta}}}_{m}$ is also VC-subgraph on ${\mathscr{W}}$ with
dimension not larger than $V_{m}\geq 1$. Applying Proposition 5 of Baraud and
Chen, (2020) with ${\mathcal{P}}={\mathcal{R}}_{m}$ for each
$m\in{\mathcal{M}}$, we can conclude. ∎

Let us remark that the function $\psi$ defined by (5) in the present paper
satisfies the Assumption 2 in Baraud and Birgé, (2018) with $a_{0}=4$,
$a_{1}=3/8$, $a_{2}^{2}=3\sqrt{2}$ and for any $\rho$-model
${\mathscrbf{P}}_{m}$, we follow the definition of $\rho$-dimension function
$D^{{\mathscrbf{P}}_{m}}$ of ${\mathscrbf{P}}_{m}$ given by (15) in Baraud and
Birgé, (2018). The next result provides an upper bound for
$D^{{\mathscrbf{P}}_{m}}$.

######  Proposition 5.

Under Assumption 1, for any $m\in{\mathcal{M}}$, for all product probabilities
${\mathbf{P}}^{\star}$ and
$\overline{\mathbf{P}}=\otimes_{i=1}^{n}\overline{P}_{i}$ on
$({\mathscr{X}}^{n},{\mathcal{X}}^{n})$ with
$\overline{P}_{i}=\overline{p}\cdot\tau_{i}$ for all $i\in\\{1,\ldots,n\\}$,

$D^{{\mathscrbf{P}}_{m}}({\mathbf{P}}^{\star},\overline{\mathbf{P}})\leq
10^{3}V_{m}\left[{9.11+\log_{+}\left(\frac{n}{V_{m}}\right)}\right].$

###### Proof.

The proof is basically similar to the proof of Proposition 6 in Baraud and
Chen, (2020) except a modification of the class ${\mathscr{F}}_{y}$. More
precisely, for any $y>0$, we define

${\mathscr{F}}_{y}=\left\\{{\left.\psi\left({\sqrt{\frac{r_{{\bm{\gamma}}}}{\overline{p}}}}\right)\right|\;{\bm{\gamma}}\in\bm{\Gamma}_{m},\;{\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathbf{r}}_{{\bm{\gamma}}}\cdot{\bm{\tau}})+{\mathbf{h}}^{2}({\mathbf{P}}^{\star},\overline{\mathbf{P}})<y^{2}}\right\\}.$

Then combining Proposition 4, the conclusion is easy to obtain by following
the proof of Proposition 6 in Baraud and Chen, (2020). ∎

Now we turn to prove Theorem 1. It follows by Proposition 5 taking
$\overline{\mathbf{P}}={\mathbf{P}}_{{\bm{\gamma}}}$ that for any
${\bm{\gamma}}\in\bm{\Gamma}$,

$D^{{\mathscrbf{P}}_{m}}({\mathbf{P}}^{\star},{\mathbf{P}}_{{\bm{\gamma}}})\leq
10^{3}V_{m}\left[{9.11+\log_{+}\left(\frac{n}{V_{m}}\right)}\right]=D_{n}(m),$

which satisfies (22) of Baraud and Birgé, (2018) with $K=0$. Applying Theorem
2 of Baraud and Birgé, (2018) over the collection of $\rho$-models
$\\{{\mathscrbf{P}}_{m},\;m\in{\mathcal{M}}\\}$ with $\kappa_{1}=0$, we obtain
for any arbitrary ${\mathbf{P}}^{\star}$, the $\rho$-estimator
${\mathbf{P}}_{\widehat{\bm{\gamma}}}$ satisfies, for all $\xi>0$ with a
probability at least $1-\Sigma e^{-\xi}$,

(37)
${\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathbf{P}}_{\widehat{\bm{\gamma}}})\leq\inf_{m\in{\mathcal{M}}}\left[{c_{1}{\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathscrbf{P}}_{m})+c_{2}\left(\Xi(m)+1.49+\xi\right)}\right],$

where $c_{1}=149.8$ and $c_{2}=5013.2$. The constant $\Sigma$ in front of
$e^{-\xi}$ just due to in Theorem 2 of Baraud and Birgé, (2018) they assumed
$\Sigma\leq 1$ for the sake of simplicity. One can refer to their proof of
Theorem 2 for understanding the role $\Sigma$ plays. The conclusion finally
follows from the equalities
${\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathbf{P}}_{\widehat{\bm{\gamma}}})={\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathbf{R}}_{\widehat{\bm{\gamma}}})$
and
${\mathbf{h}}^{2}({\mathbf{P}}^{\star},{\mathscrbf{P}}_{m})={\mathbf{h}}^{2}({\mathbf{R}}^{\star},{\mathscrbf{Q}}_{m})$,
for all $m\in{\mathcal{M}}$.

#### A.2 Proof of Corollary 1

We first present the following approximation result which is an immediate
consequence combining Theorem 1 and Proposition 2 of Akakpo, (2012).

######  Proposition 6.

Let $r\in{\mathbb{N}}$, $R\in{\mathbb{R}}_{+}^{*}$,
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{d})\in\prod_{j=1}^{d}(0,r+1)$, $p>0$
and $1\leq p^{\prime}\leq\infty$ such that

$\frac{\overline{\alpha}}{d}>\left(\frac{1}{p}-\frac{1}{p^{\prime}}\right)_{+}.$

For all $f\in B_{p,q}^{{\bm{\alpha}}}\left(\left[{0,1}\right]^{d},R\right)$
and all $l\in{\mathbb{N}}$, there exists a partition
$\pi(l)\in\cup_{{\bm{s}}\in{\mathbb{N}}^{d}}M_{\bm{s}}^{{\mathcal{B}},d}$ of
$\left[{0,1}\right]^{d}$ containing only hyperrectangles such that

$|\pi(l)|\leq C_{d,{\bm{\alpha}},p}2^{ld}$

and

(38)
$\inf_{\widetilde{f}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\|f-\widetilde{f}\|_{p^{\prime}}\leq
C_{d,r,{\bm{\alpha}},p,p^{\prime}}R2^{-l\overline{\alpha}},$

where $q=\infty$ if $0<p\leq 1$ or $p\geq 2$ and $q=p$ if $1<p<2$,
$C_{d,r,{\bm{\alpha}},p,p^{\prime}}$ is a constant depending only on
$d,r,{\bm{\alpha}},p,p^{\prime}$.

Now we turn to prove Corollary 1. Under Assumption 2, applying (13), Lemma 1
and (2), we derive no matter what the distribution of $W$ is, for all
$R\in{\mathbb{R}}_{+}^{*}$, $p>0$ and
${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\overline{\alpha}/d>1/p$, any ${\bm{\gamma}}^{\star}\in
B_{p,q}^{{\bm{\alpha}}}\left(R,v_{-},v_{+}\right)$

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq$ $\displaystyle
c_{2}\left(c_{3}+\frac{e}{e-1}\right)\inf_{({\bm{s}},r)\in{\mathcal{M}}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},\overline{\mathscr{Q}}_{({\bm{s}},r)}^{d})+\frac{\Delta({\bm{s}},r)}{n}+\frac{V_{({\bm{s}},r)}}{n}(1+\log
n)}\right]$ (39) $\displaystyle\leq$ $\displaystyle
C_{\kappa}\inf_{({\bm{s}},r)\in{\mathcal{M}}}\left[{\inf_{\overline{{\bm{\gamma}}}\in{\overline{\bm{\Gamma}}}_{({\bm{s}},r)}^{{\mathcal{B}},d}}\left\|{\bm{\gamma}}^{\star}-\overline{{\bm{\gamma}}}\right\|_{2,P_{W}}^{2}+\frac{\Delta({\bm{s}},r)}{n}+\frac{V_{({\bm{s}},r)}}{n}(1+\log
n)}\right],$

where
$\overline{\mathscr{Q}}_{({\bm{s}},r)}^{d}=\left\\{R_{{\bm{\gamma}}},\;{\bm{\gamma}}\in{\overline{\bm{\Gamma}}}_{({\bm{s}},r)}^{{\mathcal{B}},d}\right\\}$
and $C_{\kappa}$ is a constant depending on $\kappa$ only. We then apply
Proposition 6 by taking
$r=\Big{\lfloor}\sup_{j=1,\ldots,d}\alpha_{j}\Big{\rfloor}\in{\mathbb{N}}$,
$p^{\prime}=\infty$ and obtain that for all $l\in{\mathbb{N}}$, there exists a
partition $\pi(l)\in M^{{\mathcal{B}},d}$ such that

(40) $\Delta(\pi(l),r)=\log(8d)|\pi(l)|+r\leq C_{d,{\bm{\alpha}},p}2^{ld},$
(41) $V_{(\pi(l),r)}=(r+1)^{d}|\pi(l)|+1\leq C_{d,{\bm{\alpha}},p}2^{ld},$
$\displaystyle\inf_{\overline{{\bm{\gamma}}}\in{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\left\|{\bm{\gamma}}^{\star}-\overline{{\bm{\gamma}}}\right\|_{2,P_{W}}^{2}$
$\displaystyle=\inf_{\overline{{\bm{\gamma}}}\in{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\int_{{\mathscr{W}}}|{\bm{\gamma}}^{\star}(w)-\overline{{\bm{\gamma}}}(w)|^{2}dP_{W}(w)$
$\displaystyle\leq\inf_{\overline{{\bm{\gamma}}}\in{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\left\|{\bm{\gamma}}^{\star}-\overline{{\bm{\gamma}}}\right\|_{\infty}^{2}$
$\displaystyle\leq\inf_{\overline{{\bm{\gamma}}}\in\overline{\bm{S}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\left\|{\bm{\gamma}}^{\star}-\overline{{\bm{\gamma}}}\right\|_{\infty}^{2}$
(42) $\displaystyle\leq C_{d,{\bm{\alpha}},p}R^{2}2^{-2l\overline{\alpha}}.$

Plugging (40), (41) and (42) into (39), we derive

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq$ $\displaystyle
C_{\kappa}\inf_{l\in{\mathbb{N}}}\left[{\inf_{\overline{{\bm{\gamma}}}\in{\overline{\bm{\Gamma}}}^{{\mathcal{B}},d}_{(\pi(l),r)}}\left\|{\bm{\gamma}}^{\star}-\overline{{\bm{\gamma}}}\right\|_{2,P_{W}}^{2}+\frac{\Delta(\pi(l),r)}{n}+\frac{V_{(\pi(l),r)}}{n}\left(1+\log
n\right)}\right]$ (43) $\displaystyle\leq$ $\displaystyle
C_{\kappa,d,{\bm{\alpha}},p}\inf_{l\in{\mathbb{N}}}\left(R^{2}2^{-2l\overline{\alpha}}+\frac{2^{ld}}{n}\right)(1+\log
n),$

where $C_{\kappa,d,{\bm{\alpha}},p}$ is a constant depending on
$\kappa,d,{\bm{\alpha}},p$ only. To conclude, we need to minimize the right
hand side of (43). If $nR^{2}<1$, we take $l=0$ so that

(44)
$R^{2}2^{-2l\overline{\alpha}}+\frac{2^{ld}}{n}=R^{2}+\frac{1}{n}<\frac{2}{n}.$

Otherwise, we take $l$ as the largest natural number such that $2^{ld}/n\leq
R^{2}2^{-2l\overline{\alpha}}$ which is well defined since $nR^{2}\geq 1$.
With this choice of $l$,

(45) $R^{2}2^{-2l\overline{\alpha}}+\frac{2^{ld}}{n}\leq
2R^{2}2^{-2l\overline{\alpha}}\leq
C_{{\bm{\alpha}}}R^{\frac{2d}{d+2\overline{\alpha}}}n^{-\frac{2\overline{\alpha}}{d+2\overline{\alpha}}},$

where $C_{{\bm{\alpha}}}$ is a constant depending only on ${\bm{\alpha}}$.
Combining (43), (44) and (45), we obtain

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq
C_{\kappa,d,{\bm{\alpha}},p}\left(R^{\frac{2d}{d+2\overline{\alpha}}}n^{-\frac{2\overline{\alpha}}{d+2\overline{\alpha}}}+\frac{1}{n}\right)\left(1+\log
n\right).$

We conclude by taking the supremum over the set
$B_{p,q}^{{\bm{\alpha}}}(R,v_{-},v_{+})$.

#### A.3 Proof of Corollary 2

######  Lemma 8.

For any $k\in{\mathbb{N}}^{*}$, $x_{1},\ldots,x_{k}\geq 0$ and
$\alpha\in(0,1]$,
$\left(\sum_{i=1}^{k}x_{i}\right)^{\alpha}\leq\sum_{i=1}^{k}x_{i}^{\alpha}$.

###### Proof.

In fact, it is enough to prove when $k=2$, i.e.
$\left(x_{1}+x_{2}\right)^{\alpha}\leq x_{1}^{\alpha}+x_{2}^{\alpha}$. If at
least one of $x_{1}$ and $x_{2}$ is equal to zero, then the conclusion is
trivial. So we suppose $x_{1},x_{2}>0$. The function $f(x)=x^{\alpha}$ is
concave on $(0,+\infty)$ since its second derivative
$f^{\prime\prime}(x)=\alpha(\alpha-1)x^{\alpha-2}$ is always negative for all
$x\in(0,+\infty)$. By the definition of the concave function, for any
$\lambda\in\left[{0,1}\right]$,

$(\lambda x)^{\alpha}=\left[{\lambda
x+(1-\lambda)0}\right]^{\alpha}\geq\lambda x^{\alpha}.$

Therefore, for any $x_{1},x_{2}>0$

$\displaystyle x_{1}^{\alpha}+x_{2}^{\alpha}$
$\displaystyle=\left[{\frac{x_{1}}{x_{1}+x_{2}}(x_{1}+x_{2})}\right]^{\alpha}+\left[{\frac{x_{2}}{x_{1}+x_{2}}(x_{1}+x_{2})}\right]^{\alpha}$
$\displaystyle\geq\frac{x_{1}}{x_{1}+x_{2}}(x_{1}+x_{2})^{\alpha}+\frac{x_{2}}{x_{1}+x_{2}}(x_{1}+x_{2})^{\alpha}$
$\displaystyle=(x_{1}+x_{2})^{\alpha}.$

∎

We then introduce a result given by Lemma 4 of Baraud and Birgé, (2014) which
we will use later in the proof.

######  Lemma 9.

Let $(A,{\mathcal{A}},\mu)$ be some probability space and $u$ some
nondecreasing and nonnegative concave function on $[0,+\infty)$ such that
$u(0)=0$. For all $k\in\left[{1,+\infty}\right]$ and
$h\in{\mathbb{L}}_{k}(A,\mu)$,

$\|u(|h|)\|_{k,\mu}\leq 2^{1/k}u(\|h\|_{k,\mu}),$

with the convention $2^{1/\infty}=1$.

Finally, we introduce the following approximation result which is obtained by
combining Corollary 3.1 of Dahmen et al., (1980) and Schumaker, (1981) (13.62
p.517). It also appeared in the proof of Proposition 5 in Barron et al.,
(1999) (4.25 p.347).

######  Proposition 7.

For a given $k\in{\mathbb{N}}^{*}$, let $r\in{\mathbb{N}}$ such that
${\bm{\alpha}}=(\alpha_{1},\ldots,\alpha_{k})\in\prod_{j=1}^{k}(0,r+1)$. For
all $f\in{\mathcal{H}}^{{\bm{\alpha}}}(\left[{0,1}\right]^{k},L)$ and all
${\bm{t}}=(t_{1},\ldots,t_{k})\in({\mathbb{N}}^{*})^{k}$, we have

(46)
$\inf_{\widetilde{f}\in\overline{\bm{S}}^{{\mathcal{H}},k}_{({\bm{t}},r)}}\|f-\widetilde{f}\|_{\infty}\leq
C_{k,r}L\sum_{j=1}^{k}{t_{j}}^{-\alpha_{j}},$

where $C_{k,r}$ is a constant depending on $k$ and $r$.

Now we turn to prove Corollary 2. First, we note that for any function
${\bm{\gamma}}^{\star}={\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$
and any $\left[{f\left[{\left(g\vee 0\right)\wedge 1}\right]\vee
v_{-}}\right]\wedge v_{+}$, where
$f\in\overline{\bm{S}}_{(t,r)}^{{\mathcal{H}},1}$,
$g({\bm{w}})=\sum_{j=1}^{d}g_{j}(w_{j})$,
$g_{j}\in\overline{\bm{S}}_{(\pi_{j},r)}$,
$({\bm{\pi}},t,r)\in(M^{{\mathcal{B}},1})^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}$,
with the fact that ${\bm{\gamma}}\in{\mathcal{H}}^{\alpha}(L,v_{-},v_{+})$ and
${\bm{\gamma}}^{\prime}_{j}$ taking values in $\left[{0,1/d}\right]$ for all
$j\in\\{1,\ldots,d\\}$, we have

$\displaystyle\sup_{{\bm{w}}\in\left[{0,1}\right]^{d}}\left|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}(w_{j})\right)-\left[{f\left[{\left(\left(\sum_{j=1}^{d}g_{j}(w_{j})\right)\vee
0\right)\wedge 1}\right]\vee v_{-}}\right]\wedge v_{+}\right|$
$\displaystyle\leq$
$\displaystyle\sup_{{\bm{w}}\in\left[{0,1}\right]^{d}}\left|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}(w_{j})\right)-f\left[{\left(\left(\sum_{j=1}^{d}g_{j}(w_{j})\right)\vee
0\right)\wedge 1}\right]\right|$ $\displaystyle\leq$
$\displaystyle\sup_{{\bm{w}}\in\left[{0,1}\right]^{d}}\left|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}(w_{j})\right)-{\bm{\gamma}}\left[{\left(\left(\sum_{j=1}^{d}g_{j}(w_{j})\right)\vee
0\right)\wedge 1}\right]\right|+\left\|{\bm{\gamma}}-f\right\|_{\infty}$
$\displaystyle\leq$ $\displaystyle
L\sup_{{\bm{w}}\in\left[{0,1}\right]^{d}}\left|\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}(w_{j})\right)-\left(\sum_{j=1}^{d}g_{j}(w_{j})\right)\right|^{\alpha\wedge
1}+\left\|{\bm{\gamma}}-f\right\|_{\infty}$ (47) $\displaystyle\leq$
$\displaystyle
L\left\|\left(\sum_{j=1}^{d}\left|{\bm{\gamma}}^{\prime}_{j}-g_{j}\right|\right)^{\alpha\wedge
1}\right\|_{\infty}+\left\|{\bm{\gamma}}-f\right\|_{\infty}.$

We then apply Lemma 8 and Lemma 9 with $k=\infty$, $\mu$ being the Lebesgue
measure (probability) and $u(z)=z^{\alpha\wedge 1}$ to (47) and obtain

$\displaystyle\left\|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)-\left[{f\left[{\left(g\vee
0\right)\wedge 1}\right]\vee v_{-}}\right]\wedge v_{+}\right\|_{\infty}$
$\displaystyle\leq$ $\displaystyle
L\sum_{j=1}^{d}\left\||{\bm{\gamma}}^{\prime}_{j}-g_{j}|^{\alpha\wedge
1}\right\|_{\infty}+\left\|{\bm{\gamma}}-f\right\|_{\infty}$ (48)
$\displaystyle\leq$ $\displaystyle
L\sum_{j=1}^{d}\left(\left\|{\bm{\gamma}}^{\prime}_{j}-g_{j}\right\|_{\infty}\right)^{\alpha\wedge
1}+\left\|{\bm{\gamma}}-f\right\|_{\infty}.$

We take

$r=r(\alpha,{\bm{\beta}})=\Bigl{\lfloor}\alpha\vee\max_{j=1,\ldots,d}\beta_{j}\Bigr{\rfloor}\in{\mathbb{N}}.$

By Proposition 6, 7 and Lemma 1, 3, for all $\alpha,L\in{\mathbb{R}}_{+}^{*}$,
${\bm{\beta}},{\bf{p}},{\bf{R}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\beta_{j}>1/p_{j}$, all
$({\bm{l}},t)=(l_{1},\ldots,l_{d},t)\in{\mathbb{N}}^{d}\times{\mathbb{N}}^{*}$
and any
${\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$,
we have

(49)
$\inf_{f\in{\bm{S}}^{{\mathcal{H}},1}_{(t,r)}}\|{\bm{\gamma}}-f\|_{\infty}=\inf_{f\in\overline{\bm{S}}^{{\mathcal{H}},1}_{(t,r)}}\|{\bm{\gamma}}-f\|_{\infty}\leq
C_{\alpha,{\bm{\beta}}}Lt^{-\alpha}$

and

(50)
$\inf_{g_{j}\in{\bm{S}}^{{\mathcal{B}},1}_{(\pi(l_{j}),r)}}\|{\bm{\gamma}}^{\prime}_{j}-g_{j}\|_{\infty}=\inf_{g_{j}\in\overline{\bm{S}}^{{\mathcal{B}},1}_{(\pi(l_{j}),r)}}\|{\bm{\gamma}}^{\prime}_{j}-g_{j}\|_{\infty}\leq
C_{\alpha,{\bm{\beta}},p_{j}}R_{j}2^{-l_{j}\beta_{j}}.$

Combining (48), (49) and (50), we have for all
$\alpha,L\in{\mathbb{R}}_{+}^{*}$,
${\bm{\beta}},{\bf{p}},{\bf{R}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\beta_{j}>1/p_{j}$, all
$({\bm{l}},t)=(l_{1},\ldots,l_{d},t)\in{\mathbb{N}}^{d}\times{\mathbb{N}}^{*}$
and any
${\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$,

$\displaystyle\inf_{f\in{\bm{S}}^{{\mathcal{H}},1}_{(t,r)},\;g_{j}\in{\bm{S}}^{{\mathcal{B}},1}_{(\pi(l_{j}),r)}}\left\|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)-\left[{f\left[{\left(\left(\sum_{j=1}^{d}g_{j}\right)\vee
0\right)\wedge 1}\right]\vee v_{-}}\right]\wedge v_{+}\right\|_{2,P_{W}}^{2}$
$\displaystyle\leq$
$\displaystyle\inf_{f\in{\bm{S}}^{{\mathcal{H}},1}_{(t,r)},\;g_{j}\in{\bm{S}}^{{\mathcal{B}},1}_{(\pi(l_{j}),r)}}\left\|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)-\left[{f\left[{\left(\left(\sum_{j=1}^{d}g_{j}\right)\vee
0\right)\wedge 1}\right]\vee v_{-}}\right]\wedge v_{+}\right\|_{\infty}^{2}$
$\displaystyle\leq$ $\displaystyle
C_{d}\left\\{L^{2}\sum_{j=1}^{d}\left[{\left(\inf_{g_{j}\in{\bm{S}}^{{\mathcal{B}},1}_{(\pi(l_{j}),r)}}\|{\bm{\gamma}}^{\prime}_{j}-g^{\prime}_{j}\|_{\infty}\right)^{\alpha\wedge
1}}\right]^{2}+\inf_{f\in{\bm{S}}^{{\mathcal{H}},1}_{(t,r)}}\|{\bm{\gamma}}-f\|_{\infty}^{2}\right\\}$
(51) $\displaystyle\leq$ $\displaystyle
C_{d,\alpha,{\bm{\beta}},{\bf{p}}}L^{2}\left[{\sum_{j=1}^{d}R_{j}^{2(\alpha\wedge
1)}2^{-2(\alpha\wedge 1)l_{j}\beta_{j}}+t^{-2\alpha}}\right].$

We denote ${\bm{\pi}}({\bm{l}})=(\pi(l_{1}),\ldots,\pi(l_{d}))$. For any
$(l_{1},\ldots,l_{d},t,r)\in{\mathbb{N}}^{d}\times{\mathbb{N}}^{*}\times{\mathbb{N}}$,
by Proposition 1 and 6, we have

$\displaystyle
V^{A}_{({\bm{\pi}}({\bm{l}}),t,r)}+\Delta({\bm{\pi}}({\bm{l}}),t,r)\leq$
$\displaystyle
C\left[{\left(t+\sum_{j=1}^{d}|\pi(l_{j})|\right)(r+1)}\right]\log(t+r+2)$
$\displaystyle+\left[{(3\log
2)\left(\sum_{j=1}^{d}|\pi(l_{j})|\right)+r+t}\right]$ $\displaystyle\leq$
$\displaystyle C_{r}\left(t+\sum_{j=1}^{d}|\pi({l_{j}})|\right)\log(t+r+2)$
(52) $\displaystyle\leq$ $\displaystyle
C_{\alpha,{\bm{\beta}},{\bf{p}}}\left(t+\sum_{j=1}^{d}2^{l_{j}}\right)\log(t+r+2),$

where $C$ is a numerical constant, $C_{r}$ is a numerical constant depending
only on $r$ and $C_{\alpha,{\bm{\beta}},{\bf{p}}}$ is a numerical constant
depending only on $\alpha$, ${\bm{\beta}}$, ${\bf{p}}$.

Under Assumption 2, applying (13) together with (51) and (52), we derive that
for all $\alpha,L\in{\mathbb{R}}_{+}^{*}$,
${\bm{\beta}},{\bf{p}},{\bf{R}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\beta_{j}>1/p_{j}$, all
$({\bm{l}},t)=(l_{1},\ldots,l_{d},t)\in{\mathbb{N}}^{d}\times{\mathbb{N}}^{*}$
and any
${\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$,

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq$
$\displaystyle
C_{\kappa}\inf_{({\bm{\pi}},t,r)\in(M^{{\mathcal{B}},1})^{d}\times({\mathbb{N}}^{*})^{2}}\left[\inf_{\widetilde{\bm{\gamma}}\in\bm{\Gamma}^{A}_{({\bm{\pi}},t,r)}}\left\|{\bm{\gamma}}\left(\sum_{j=1}^{d}{\bm{\gamma}}^{\prime}_{j}\right)-\widetilde{\bm{\gamma}}\right\|_{2,P_{W}}^{2}\right.$
$\displaystyle\left.+\frac{\Delta({\bm{\pi}},t,r)}{n}+\frac{V_{({\bm{\pi}},t,r)}}{n}\left(1+\log
n\right)\right]$ $\displaystyle\leq$ $\displaystyle
C_{\kappa,d,\alpha,{\bm{\beta}},{\bf{p}}}(1+\log
n)\inf_{(l_{1},\ldots,l_{d},t)\in{\mathbb{N}}^{d}\times{\mathbb{N}}^{*}}\left[\left(L^{2}t^{-2\alpha}+\frac{t}{n}\right)\right.$
(53) $\displaystyle\left.+\sum_{j=1}^{d}\left(L^{2}R_{j}^{2(\alpha\wedge
1)}2^{-2(\alpha\wedge
1)l_{j}\beta_{j}}+\frac{2^{l_{j}}}{n}\right)\right]\log(t+r+2).$

To conclude, we need to optimize the right hand side of (53). We choose $t\geq
1$ such that

$t-1<\left(nL^{2}\right)^{\frac{1}{1+2\alpha}}\leq t,$

therefore $L^{2}t^{-2\alpha}\leq t/n$ and
$t<1+\left(nL^{2}\right)^{\frac{1}{1+2\alpha}}$. As a consequence, we have

(54) $\displaystyle L^{2}t^{-2\alpha}+\frac{t}{n}$ $\displaystyle\leq
2\frac{t}{n}\leq\frac{2}{n}+2L^{\frac{2}{2\alpha+1}}n^{-\frac{2\alpha}{2\alpha+1}}.$

Moreover, we note that if $nL^{2}<1$, we choose $t=1$, then

(55) $\log(t+r+2)\leq\log\left(r+3\right)=C_{\alpha,{\bm{\beta}}}.$

Otherwise $nL^{2}\geq 1$,

$\displaystyle\log(t+r+2)$
$\displaystyle\leq\log\left[{\left(nL^{2}\right)^{\frac{1}{2\alpha+1}}+r+3}\right]$
$\displaystyle\leq\log\left[{C_{\alpha,{\bm{\beta}}}\left(nL^{2}\right)^{\frac{1}{2\alpha+1}}}\right]$
(56) $\displaystyle\leq C_{\alpha,{\bm{\beta}}}\left(\log n\vee\log L^{2}\vee
1\right).$

For any $j\in\left\\{1,\ldots,d\right\\}$, if $nL^{2}R_{j}^{2(\alpha\wedge
1)}<1$, we take $l_{j}=0$ so that

(57) $L^{2}R_{j}^{2(\alpha\wedge 1)}2^{-2(\alpha\wedge
1)l_{j}\beta_{j}}+\frac{2^{l_{j}}}{n}<\frac{2}{n}.$

Otherwise, we take $l_{j}$ as the largest natural number such that

$\frac{2^{l_{j}}}{n}\leq L^{2}R_{j}^{2(\alpha\wedge 1)}2^{-2(\alpha\wedge
1)l_{j}\beta_{j}},$

which yields

$\displaystyle L^{2}R_{j}^{2(\alpha\wedge 1)}2^{-2(\alpha\wedge
1)l_{j}\beta_{j}}+\frac{2^{l_{j}}}{n}$ $\displaystyle\leq
L^{2}R_{j}^{2(\alpha\wedge 1)}2^{1-2(\alpha\wedge 1)l_{j}\beta_{j}}$
$\displaystyle\leq C_{\alpha,{\bm{\beta}}}\left[{L^{2}R_{j}^{2(\alpha\wedge
1)}}\right]^{\frac{1}{2(\alpha\wedge 1)\beta_{j}+1}}n^{-\frac{2(\alpha\wedge
1)\beta_{j}}{2(\alpha\wedge 1)\beta_{j}+1}}$ (58) $\displaystyle\leq
C_{\alpha,{\bm{\beta}}}\left(LR_{j}^{\alpha\wedge
1}\right)^{\frac{2}{2(\alpha\wedge 1)\beta_{j}+1}}n^{-\frac{2(\alpha\wedge
1)\beta_{j}}{2(\alpha\wedge 1)\beta_{j}+1}}.$

Combining (53), (54), (55), (56), (57) and (58), we obtain whatever the
distribution of $W$, for all $\alpha,L\in{\mathbb{R}}_{+}^{*}$,
${\bm{\beta}},{\bf{p}},{\bf{R}}\in({\mathbb{R}}_{+}^{*})^{d}$ such that
$\beta_{j}>1/p_{j}$ and any
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$,

$\displaystyle
C^{\prime}_{\kappa,d,\alpha,{\bm{\beta}},{\bf{p}}}{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq$
$\displaystyle\left\\{\left[{\sum_{j=1}^{d}\left(LR_{j}^{\alpha\wedge
1}\right)^{\frac{2}{2(\alpha\wedge 1)\beta_{j}+1}}n^{-\frac{2(\alpha\wedge
1)\beta_{j}}{2(\alpha\wedge
1)\beta_{j}+1}}}\right]+L^{\frac{2}{2\alpha+1}}n^{-\frac{2\alpha}{2\alpha+1}}+\frac{1}{n}\right\\}{\mathcal{L}}^{2}_{n},$

where ${\mathcal{L}}_{n}=\log n\vee\log L^{2}\vee 1$. Finally, the conclusion
follows by taking the supremum over
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(\alpha,{\bm{\beta}},{\bf{p}},L,{\bf{R}})$.

#### A.4 Proof of Corollary 3

We first present the following result which can be proved by a similar
argument as the proof of Lemma 1.

######  Lemma 10.

Let
${\mathcal{C}}_{d}=\left\\{(c_{1},\ldots,c_{d})\in{\mathbb{R}}^{d},\;\sum_{j=1}^{d}|c_{j}|\leq
1\right\\}.$ We denote $\overline{\bm{S}}_{{\mathcal{C}}_{d}}$ the collection
of functions on $\left[{0,1}\right]^{d}$ of the form

(59) $f({\bm{w}})=\frac{1}{2}\left(\langle
c,{\bm{w}}\rangle+1\right),\mbox{\quad for
all\quad}{\bm{w}}\in\left[{0,1}\right]^{d},$

with $c\in{\mathcal{C}}_{d}$ and ${\bm{S}}_{{\mathcal{C}}_{d}}$ the collection
of functions of the form in (59) but with
$c\in{\mathcal{C}}_{d}\cap{\mathbb{Q}}^{d}$. Then
${\bm{S}}_{{\mathcal{C}}_{d}}$ is dense in
$\overline{\bm{S}}_{{\mathcal{C}}_{d}}$ with respect to the supremum norm.

Now let us turn to prove Corollary 3. For all
${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$, $L>0$, any
${\bm{\gamma}}^{\star}={\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$,
where
${\bm{\gamma}}^{\prime}({\bm{w}})=({\bm{\gamma}}^{\prime}_{1}({\bm{w}}),\ldots,{\bm{\gamma}}^{\prime}_{l}({\bm{w}}))$
with ${\bm{\gamma}}^{\prime}_{j}\in\overline{\bm{S}}_{{\mathcal{C}}_{d}}$ for
$j\in\\{1,\ldots,l\\}$,
${\bm{\gamma}}\in{\mathcal{H}}^{{\bm{\alpha}}}\left(L,v_{-},v_{+}\right)$ and
any $f\in{\bm{S}}_{({\bm{t}},r)}^{{\mathcal{H}},l}$,
$g:\left[{0,1}\right]^{d}\rightarrow\left[{0,1}\right]^{l}$ defined as
$g({\bm{w}})=(g_{1}({\bm{w}}),\ldots,g_{l}({\bm{w}}))$ with
$g_{j}\in{\bm{S}}_{{\mathcal{C}}_{d}}$ for $j\in\\{1,\ldots,l\\}$, we have

$\displaystyle\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-(f\circ g)\vee
v_{-})\wedge v_{+}\|_{\infty}$
$\displaystyle\leq\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-f\circ
g\|_{\infty}$
$\displaystyle\leq\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-{\bm{\gamma}}\circ
g\|_{\infty}+\|{\bm{\gamma}}\circ g-f\circ g\|_{\infty}$
$\displaystyle\leq\left\|L\sum_{j=1}^{l}\left|{\bm{\gamma}}^{\prime}_{j}-g_{j}\right|^{\alpha_{j}\wedge
1}\right\|_{\infty}+\|{\bm{\gamma}}-f\|_{\infty}$ (60) $\displaystyle\leq
L\sum_{j=1}^{l}\left\|\left|{\bm{\gamma}}^{\prime}_{j}-g_{j}\right|^{\alpha_{j}\wedge
1}\right\|_{\infty}+\|{\bm{\gamma}}-f\|_{\infty}.$

We apply Lemma 9 to (60) by taking $k=\infty$, $\mu$ the Lebesgue probability
and $u(z)=z^{\alpha_{j}\wedge 1}$ for each $j\in\\{1,\ldots,l\\}$ and obtain

(61) $\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-(f\circ g\vee v_{-})\wedge
v_{+}\|_{\infty}\leq
L\sum_{j=1}^{l}\left(\left\|{\bm{\gamma}}^{\prime}_{j}-g_{j}\right\|_{\infty}\right)^{\alpha_{j}\wedge
1}+\|{\bm{\gamma}}-f\|_{\infty}.$

We take $r=\max_{j=1,\ldots,l}\lfloor\alpha_{j}\rfloor\in{\mathbb{N}}.$ By
Proposition 7, for any
${\bm{\gamma}}^{\star}={\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$
and all ${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$, we have

$\inf_{f\in\overline{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}}\|{\bm{\gamma}}-f\|_{\infty}\leq
C_{l,{\bm{\alpha}}}L\sum_{j=1}^{l}t_{j}^{-\alpha_{j}},$

where $C_{l,{\bm{\alpha}}}$ is a constant depending on $l$ and ${\bm{\alpha}}$
only. Then by Lemma 3, ${\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}$ is dense in
$\overline{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}$ with respect to the
supremum norm $\|\cdot\|_{\infty}$, we obtain

(62)
$\inf_{f\in{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}}\|{\bm{\gamma}}-f\|_{\infty}=\inf_{f\in\overline{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)}}\|{\bm{\gamma}}-f\|_{\infty}\leq
C_{l,{\bm{\alpha}}}L\sum_{j=1}^{l}t_{j}^{-\alpha_{j}}.$

Therefore, following from (61), (62) and Lemma 10, for all
${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$, $L>0$, any
${\bm{\gamma}}^{\star}={\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$
and all ${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$,

$\displaystyle\inf_{f\in{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)},\;g_{j}\in{\bm{S}}_{{\mathcal{C}}_{d}}}\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-(f\circ
g\vee v_{-})\wedge v_{+}\|_{2,P_{W}}^{2}$ $\displaystyle\leq$
$\displaystyle\inf_{f\in{\bm{S}}^{{\mathcal{H}},l}_{({\bm{t}},r)},\;g_{j}\in{\bm{S}}_{{\mathcal{C}}_{d}}}\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-(f\circ
g\vee v_{-})\wedge v_{+}\|_{\infty}^{2}$ (63) $\displaystyle\leq$
$\displaystyle
C_{l,{\bm{\alpha}}}L^{2}\left(\sum_{j=1}^{l}t_{j}^{-\alpha_{j}}\right)^{2}.$

Moreover, for any $r\in{\mathbb{N}}$ and
${\bm{t}}=(t_{1},\ldots,t_{l})\in({\mathbb{N}}^{*})^{l}$, with the fact that

$\displaystyle\Delta({\bm{t}},r)=\sum_{j=1}^{l}t_{j}+r$ $\displaystyle\leq
l\prod_{j=1}^{l}t_{j}+r\leq l\left(\prod_{j=1}^{l}t_{j}\right)(r+1)^{l}$

and Proposition 2, we have

$\displaystyle V^{M}_{({\bm{t}},r)}+\Delta({\bm{t}},r)$ $\displaystyle\leq
C_{l}\left[{d+\left(\prod_{j=1}^{l}t_{j}\right)(r+1)^{l}}\right]\log\left[{\left(\sum_{j=1}^{l}t_{j}\right)+lr+l+1}\right]$
(64) $\displaystyle\leq
C_{l,{\bm{\alpha}}}\left[{d+\left(\prod_{j=1}^{l}t_{j}\right)}\right]\log\left[{\left(\sum_{j=1}^{l}t_{j}\right)+lr+l+1}\right],$

where $C_{l}$ is a numerical constant depending only on $l$ and
$C_{l,{\bm{\alpha}}}$ is a numerical constant depending on $l$,
${\bm{\alpha}}$ only

Under Assumption 2, applying (13) together with the inequalities (63) and
(64), we derive that for all ${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$ and
$L>0$, any
${\bm{\gamma}}^{\star}={\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$,
whatever the distribution of $W$,

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq$ $\displaystyle
C_{\kappa}\inf_{({\bm{t}},r)\in({\mathbb{N}}^{*})^{l}\times{\mathbb{N}}}\left[{\inf_{\widetilde{\bm{\gamma}}\in\bm{\Gamma}^{M}_{({\bm{t}},r)}}\|{\bm{\gamma}}\circ{\bm{\gamma}}^{\prime}-\widetilde{\bm{\gamma}}\|_{2,P_{W}}^{2}+\frac{\Delta({\bm{t}},r)}{n}+\frac{V^{M}_{({\bm{t}},r)}}{n}(1+\log
n)}\right]$ (65) $\displaystyle\leq$ $\displaystyle
C_{\kappa,l,{\bm{\alpha}}}(1+\log
n)\inf_{{\bm{t}}\in({\mathbb{N}}^{*})^{l}}\left[{L^{2}\left(\sum_{j=1}^{l}t_{j}^{-\alpha_{j}}\right)^{2}+\frac{\prod_{j=1}^{l}t_{j}}{n}+\frac{d}{n}}\right]\log(U),$

where $U=\sum_{j=1}^{l}t_{j}+lr+l+1$. We then optimize the risk bound given on
the right hand side of (65). For each $j\in\\{1,\ldots,l\\}$, we choose
$t_{j}\geq 1$ satisfying

$t_{j}-1<(nL^{2})^{\frac{\overline{\alpha}}{(2\overline{\alpha}+l)\alpha_{j}}}\leq
t_{j},$

where $\overline{\alpha}$ denotes the harmonic mean of
$\alpha_{1},\ldots,\alpha_{l}$. Therefore, we have

(66) $L^{2}\left(\sum_{j=1}^{l}t_{j}^{-\alpha_{j}}\right)^{2}\leq
l^{2}L^{2}(nL^{2})^{-\frac{2\overline{\alpha}}{2\overline{\alpha}+l}}=l^{2}L^{\frac{2l}{2\overline{\alpha}+l}}n^{-\frac{2\overline{\alpha}}{2\overline{\alpha}+l}}.$

If $nL^{2}\leq 1$, then $t_{j}=1$ for all $j\in\\{1,\ldots,l\\}$ hence

(67) $\frac{\prod_{j=1}^{l}t_{j}}{n}\leq\frac{1}{n}$

and for some numerical constant $C_{l,{\bm{\alpha}}}$ depending on $l$ and
${\bm{\alpha}}$ only

(68) $\log(U)=\log\left[{\left(\sum_{j=1}^{l}t_{j}\right)+lr+l+1}\right]\leq
C_{l,{\bm{\alpha}}}.$

Otherwise,

$\displaystyle\frac{\prod_{j=1}^{l}t_{j}}{n}$
$\displaystyle\leq\frac{\prod_{j=1}^{l}2(nL^{2})^{\frac{\overline{\alpha}}{(2\overline{\alpha}+l)\alpha_{j}}}}{n}=\frac{2^{l}(nL^{2})^{\frac{\overline{\alpha}}{2\overline{\alpha}+l}\sum_{j=1}^{l}\frac{1}{\alpha_{j}}}}{n}$
(69) $\displaystyle\leq
2^{l}\frac{(nL^{2})^{\frac{l}{2\overline{\alpha}+l}}}{n}\leq
2^{l}L^{\frac{2l}{2\overline{\alpha}+l}}n^{-\frac{2\overline{\alpha}}{2\overline{\alpha}+l}}$

and

$\displaystyle\log\left[{\left(\sum_{j=1}^{l}t_{j}\right)+lr+l+1}\right]$
$\displaystyle\leq\log\left[{l\left(\prod_{j=1}^{l}t_{j}\right)+lr+l+1}\right]$
$\displaystyle\leq\log\left[{C_{l}\left(nL^{2}\right)^{\frac{l}{2\overline{\alpha}+l}}+C_{l,{\bm{\alpha}}}}\right]$
(70) $\displaystyle\leq C_{l,{\bm{\alpha}}}\left(\log n\vee\log L^{2}\vee
1\right).$

Plugging (66), (67), (68), (69), (70) into (65), we have that whatever the
distribution of $W$, for all ${\bm{\alpha}}\in({\mathbb{R}}_{+}^{*})^{l}$ and
$L>0$, any
${\bm{\gamma}}^{\star}\in{\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$

(71)
$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]$
$\displaystyle\leq
C_{\kappa,l,{\bm{\alpha}}}\left(L^{\frac{2l}{2\overline{\alpha}+l}}n^{-\frac{2\overline{\alpha}}{2\overline{\alpha}+l}}+\frac{d}{n}\right)\left(\log
n\vee\log L^{2}\vee 1\right)^{2},$

where $C_{\kappa,l,{\bm{\alpha}}}$ is a constant depending only on $\kappa$,
$l$ and ${\bm{\alpha}}$. The conclusion finally follows by taking the supremum
over the set ${\mathcal{G}}_{\left[{v_{-},v_{+}}\right]}({\bm{\alpha}},L)$.

#### A.5 Proof of Corollary 4

For any
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$,
we first rewrite it as
${\bm{\gamma}}^{\star}=\sum_{k\in{\mathbb{N}}^{*}}t^{k}{\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}\right)$, where $t\in(-1,1)$, ${\bm{\gamma}}_{1}\in{\bm{S}}_{(l,p_{1})}$ and
${\bm{\gamma}}_{2}\in{\bm{S}}_{(l,p_{2})}$. For any $m\in{\mathbb{N}}^{*}$, we
denote
${\bm{\gamma}}^{\star}_{m}=\sum_{k=1}^{m}t^{k}{\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}\right)$ the m-partial sum of the function ${\bm{\gamma}}^{\star}$. We then
apply Proposition 4.4 of Daubechies et al., (2019) and obtain that
${\bm{\gamma}}^{\star}_{m}\in\overline{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}$. We
note that for any
${\bm{\gamma}}^{\star}_{m}=\sum_{k=1}^{m}t^{k}{\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}\right)$, there exists a sequence of functions
$\\{{\bm{\gamma}}_{i}\\}_{i\in{\mathbb{N}}}$ with
${\bm{\gamma}}_{i}=\sum_{k=1}^{m}t^{k}_{i}{\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}\right)\in{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}$ and
$t_{i}\in(-1,1)\cap{\mathbb{Q}}$ such that

$\displaystyle\lim_{i\rightarrow+\infty}\|{\bm{\gamma}}^{\star}_{m}-{\bm{\gamma}}_{i}\|_{\infty}$
$\displaystyle=\lim_{i\rightarrow+\infty}\left|\sum_{k=1}^{m}\left(t^{k}-t^{k}_{i}\right){\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}\right)(w)\right|$ $\displaystyle\leq
K\lim_{i\rightarrow+\infty}\sum_{k=1}^{m}\left|t^{k}-t^{k}_{i}\right|$ (72)
$\displaystyle\leq\frac{Km(m+1)}{2}\lim_{i\rightarrow+\infty}\left|t-t_{i}\right|=0,$

since ${\mathbb{Q}}$ is dense in ${\mathbb{R}}$. Therefore, with the fact that
${\bm{\gamma}}^{\star}$ taking values in $\left[{v_{-},v_{+}}\right]$ and
(72), we have

$\displaystyle\inf_{\overline{\bm{\gamma}}\in\bm{\Gamma}_{(l(m+1),p_{1}+p_{2}+2)}}\|{\bm{\gamma}}^{\star}-\overline{\bm{\gamma}}\|_{\infty}$
$\displaystyle\leq\inf_{\overline{\bm{\gamma}}\in{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}}\|{\bm{\gamma}}^{\star}-\overline{\bm{\gamma}}\|_{\infty}$
$\displaystyle\leq\|{\bm{\gamma}}^{\star}-{\bm{\gamma}}^{\star}_{m}\|_{\infty}+\inf_{\overline{\bm{\gamma}}\in{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}}\|{\bm{\gamma}}^{\star}_{m}-\overline{\bm{\gamma}}\|_{\infty}$
$\displaystyle\leq\sup_{w\in\left[{0,1}\right]}\left|\sum_{k=m+1}^{+\infty}t^{k}{\bm{\gamma}}_{1}\left({\bm{\gamma}}_{2}^{\circ
k}(w)\right)\right|$ (73) $\displaystyle\leq C_{t,K}|t|^{m+1},$

where $C_{t,K}$ stands for a numerical constant depending on $t$ and $K$ only.
We denote $V_{(l(m+1),p_{1}+p_{2}+2)}$ the VC dimension of
$\bm{\Gamma}_{(l(m+1),p_{1}+p_{2}+2)}$. With the fact that
${\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}\subset\overline{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}$,
Proposition 3 and

$\bm{\Gamma}_{(l(m+1),p_{1}+p_{2}+2)}=\left\\{({\bm{\gamma}}\vee v_{-})\wedge
v_{+},\;{\bm{\gamma}}\in{\bm{S}}_{(l(m+1),p_{1}+p_{2}+2)}\right\\},$

we derive that for some numerical constant $C$

$\displaystyle V_{(l(m+1),p_{1}+p_{2}+2)}\leq
Cl_{0}\left[{p_{0}^{2}(l_{0}-1)+p_{0}(l_{0}+2)+1}\right]\log\left[{(l_{0}+1)\left(\frac{p_{0}l_{0}}{2}+1\right)}\right],$

where $l_{0}=l(m+1)$ and $p_{0}=p_{1}+p_{2}+2$. Then it follows by a basic
computation that

(74) $V_{(l(m+1),p_{1}+p_{2}+2)}+\Delta(l(m+1),p_{1}+p_{2}+2)\leq
C_{l,{\bf{p}}}(m+1)^{3},$

where $C_{l,{\bf{p}}}$ is a numerical constant depending on $l$ and ${\bf{p}}$
only.

We take $L=l(m+1)$ and $p=p_{1}+p_{2}+2$. Under Assumption 2, applying (13)
together with the inequalities (73) and (74), we have no matter what the
distribution of $W$ is, for all $t\in(-1,1)$, $l\in{\mathbb{N}}^{*}$,
${\bf{p}}\in({\mathbb{N}}^{*})^{2}$ and $K\geq 0$, for any
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$,

$\displaystyle{\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq$
$\displaystyle
C_{\kappa,{\bf{p}},l}\inf_{m\in{\mathbb{N}}^{*}}\left[\inf_{\overline{\bm{\gamma}}\in\bm{\Gamma}_{(l(m+1),p_{1}+p_{2}+2)}}\|{\bm{\gamma}}^{\star}-\overline{\bm{\gamma}}\|_{2,P_{W}}^{2}\right.$
$\displaystyle\left.+\frac{(m+1)^{3}}{n}(1+\log n)\right]$ (75)
$\displaystyle\leq$ $\displaystyle
C_{\kappa,{\bf{p}},l,t,K}\inf_{m\in{\mathbb{N}}^{*}}\left[{|t|^{2(m+1)}+\frac{(m+1)^{3}}{n}(1+\log
n)}\right].$

Now we only need to optimize the right hand side of (75). If $|t|^{4}\leq
1/n$, we choose $m=1$ so that

(76)
${\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq
C_{\kappa,{\bf{p}},l,t,K}\frac{1}{n}(1+\log n).$

Otherwise, we choose $m\in{\mathbb{N}}^{*}$ such that

$m<\frac{\log n}{-2\log|t|}\leq m+1.$

With this choice, $|t|^{2(m+1)}\leq 1/n$ and we derive from (75),

(77)
${\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq
C_{\kappa,{\bf{p}},l,t,K}\frac{(1+\log n)^{4}}{n}.$

Combining the results in (76) and (77), whatever the distribution of $W$, for
all $t\in(-1,1)$, $l\in{\mathbb{N}}^{*}$, ${\bf{p}}\in({\mathbb{N}}^{*})^{2}$
and $K\geq 0$, any
${\bm{\gamma}}^{\star}\in{\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$,
we have

(78)
${\mathbb{E}}\left[{h^{2}(R_{{\bm{\gamma}}^{\star}},R_{\widehat{\bm{\gamma}}})}\right]\leq
C_{\kappa,{\bf{p}},l,t,K}\frac{1}{n}(1+\log n)^{4}.$

Then the conclusion follows by taking the supremum over
${\mathcal{F}}_{\left[{v_{-},v_{+}}\right]}(t,l,{\bf{p}},K)$ on both sides of
(78).

#### A.6 Proof of Corollary 5

For any
$(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}$, let
$\overline{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}\subset\overline{\bm{S}}_{(L,p,{\bm{s}})}$
be the collection of functions based on sparse neural network, where all the
non-zero parameters vary in $\left[{-1,1}\right]$ and
${\bm{S}}^{\prime}_{(L,p,{\bm{s}})}\subset\overline{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$
be the collection of functions, where all the non-zero parameters vary in
$\left[{-1,1}\right]\cap{\mathbb{Q}}$. We first show the following result.

######  Lemma 11.

For any
$(L,p,{\bm{s}})\in({\mathbb{N}}^{*})^{2}\times\\{0,1\\}^{\overline{p}}$,
${\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$ is dense in
$\overline{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$ with respect to the supremum
norm $\|\cdot\|_{\infty}$.

###### Proof.

By the definition of dense with respect to the supremum norm, we need to show
that for any function $f\in\overline{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$, there
is a sequence of functions $f_{i}\in{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$,
$i\in{\mathbb{N}}$ such that

$\lim_{i\rightarrow+\infty}\|f-f_{i}\|_{\infty}=0.$

The idea is inspired by the proof of Lemma 5 of Schmidt-Hieber, (2020). Recall
for any $f\in\overline{\bm{S}}_{(L,p)}$, it can be written as

$f({\bm{w}})=M_{L}\circ\sigma\circ M_{L-1}\circ\cdots\circ\sigma\circ
M_{0}({\bm{w}}),\mbox{\quad for all\ }{\bm{w}}\in\left[{0,1}\right]^{d}.$

For $l\in\\{0,\ldots,L+1\\}$, we define $p_{l}=p$ for $l\in\\{1,\ldots,L\\}$,
$p_{0}=d$ and $p_{L+1}=1$. For $l\in\\{1,\ldots,L\\}$, we define the function
$f_{l}^{+}:\left[{0,1}\right]^{d}\rightarrow{\mathbb{R}}^{p}$,

$f_{l}^{+}({\bm{w}})=\sigma\circ M_{l-1}\circ\cdots\circ\sigma\circ
M_{0}({\bm{w}})$

and for $l\in\\{1,\ldots,L+1\\}$, we define
$f_{l}^{-}:{\mathbb{R}}^{p_{l-1}}\rightarrow{\mathbb{R}}$

$f_{l}^{-}({\bm{x}})=M_{L}\circ\sigma\circ\cdots\circ\sigma\circ
M_{l-1}({\bm{x}}).$

We set the notations $f_{0}^{+}({\bm{w}})=f_{L+2}^{-}({\bm{w}})={\bm{w}}$.
Given a vector ${\bm{v}}=(v_{1},\ldots,v_{p})$ of any size
$p\in{\mathbb{N}}^{*}$, we denote
$|{\bm{v}}|_{\infty}=\max_{i=1,\ldots,p}|v_{i}|$.

For any $f\in\overline{\bm{S}}^{\prime}_{(L,p,{\bm{s}})}$, with the fact that