On the existence of minimizers in shallow residual ReLU neural network
  optimization landscapes
Many mathematical convergence results for gradient descent (GD) based
algorithms employ the assumption that the GD process is (almost surely) bounded
and, also in concrete numerical simulations, divergence of the GD process may
slow down, or even completely rule out, convergence of the error function. In
practical relevant learning problems, it thus seems to be advisable to design
the ANN architectures in a way so that GD optimization processes remain
bounded. The property of the boundedness of GD processes for a given learning
problem seems, however, to be closely related to the existence of minimizers in
the optimization landscape and, in particular, GD trajectories may escape to
infinity if the infimum of the error function (objective function) is not
attained in the optimization landscape. This naturally raises the question of
the existence of minimizers in the optimization landscape and, in the situation
of shallow residual ANNs with multi-dimensional input layers and
multi-dimensional hidden layers with the ReLU activation, the main result of
this work answers this question affirmatively for a general class of loss
functions and all continuous target functions. In our proof of this statement,
we propose a kind of closure of the search space, where the limits are called
generalized responses, and, thereafter, we provide sufficient criteria for the
loss function and the underlying probability distribution which ensure that all
additional artificial generalized responses are suboptimal which finally allows
us to conclude the existence of minimizers in the optimization landscape.
Steffen Dereich, Arnulf Jentzen, Sebastian Kassing
# On the existence of minimizers in shallow residual

###### Abstract.

Many mathematical convergence results for gradient descent (GD) based algorithms employ the assumption that the GD process is (almost surely) _bounded_ and, also in concrete numerical simulations, divergence of the GD process may _slow down_, or even completely rule out, convergence of the error function. In practical relevant learning problems, it thus seems to be advisable to design the ANN architectures in a way so that GD optimization processes remain bounded. The property of the boundedness of GD processes for a given learning problem seems, however, to be closely related to the _existence of minimizers in the optimization landscape_ and, in particular, GD trajectories may escape to infinity if the infimum of the error function (objective function) is not attained in the optimization landscape. This naturally raises the question of the existence of minimizers in the optimization landscape and, in the situation of shallow residual ANNs with multi-dimensional input layers and multi-dimensional hidden layers with the ReLU activation, the main result of this work answers this question affirmatively for a general class of loss functions and all continuous target functions. In our proof of this statement, we propose a kind of closure of the search space, where the limits are called _generalized responses_, and, thereafter, we provide sufficient criteria for the loss function and the underlying probability distribution which ensure that all additional artificial generalized responses are suboptimal which finally allows us to conclude the existence of minimizers in the optimization landscape.

Key words and phrases:Neural networks, shallow networks, best approximation, ReLU activation, approximatively compact 2020 Mathematics Subject Classification: Primary 68T07; Secondary 68T05, 41A50

## 1. Introduction

Machine learning methods - often consisting of _artificial neural networks (ANNs)_ trained through _gradient descent_ (GD) type optimization methods - are nowadays omnipresent computational methods which are heavily employed in many industrial applications as well as scientific research activities. Despite the mind-blowing success of such computational schemes, in general, it remains an open problem of research to rigorously prove (or disprove) the convergence of GD optimization methods in the training of ANNs. Even in the situation of shallow ANNs with just one hidden layer it remains an open research question whether the plain vanilla stochastic gradient descent (SGD) method does converge in the training of such ANNs.

In the literature regarding the training of ANNs there are, however, several partial error analysis results for GD type optimization methods (by which we mean, everything, time-continuous gradient flow processes, deterministic GD optimization methods, as well as stochastic GD optimization methods). For example, we refer the reader to [1, 1, 2] for results concerning gradient flows, to [1, 3] for results concerning deterministic gradient methods, to [14, 15, 2, 16] for results concerning stochastic gradient methods and to [16] for results concerning gradient based diffusion processes. Many of these results exploit _Kurdyka-Lojasiewicz_ gradient type inequalities to establishing convergence to a point (often a critical point) of the considered GD type optimization methods and we refer the reader
to [13, 14, 15] for classical results by Lojasiewicz concerning gradient inequalities for analytic target functions and direct consequences for the convergence of gradient flows.

Each of the above mentioned partial convergence results assumes that the considered GD type optimization process is (almost surely) bounded, loosely speaking, in the sense that

\[\sup_{t\in[0,\infty)}\|\Theta_{t}\|<\infty \tag{1}\]

where \(\Theta\colon[0,\infty)\to\mathbb{R}^{\mathfrak{d}}\) corresponds to the employed GD type optimization process (which could be a gradient flow optimization process or a time-continuous version of a discrete GD type optimization process), where \(\mathfrak{d}\in\mathbb{N}\) corresponds to the number of trainable parameters, and where \(\|\cdot\|\) refers to the standard norm on \(\mathbb{R}^{\mathfrak{d}}\).

In general, it remains an open problem to verify (1) (and, thus, whether the above mentioned achievements are actually applicable in context of the training of ANNs). The question whether (1) is satisfied seems to be closely related to _the existence of minimizers in the optimization landscape_; cf. [1, 10]. In particular, in [1] counterexamples to (1) are given and _divergence_ of GD type optimization processes is proved in the sense that

\[\liminf_{t\to\infty}\|\Theta_{t}\|=\infty \tag{2}\]

in certain cases where there do not exist minimizers in the optimization landscape. A divergence phenomena of the form (2) may _slow down_ (or even completely rule out) the convergence of the risk/error function, which is the highly relevant quantity in practical applications, and, in this aspect, it seems to be strongly advisable to _design the ANN architecture_ in a way so that there exist minimizers in the optimization landscape and so that the divergence in (2) _fails to happen_.

The question whether there exist minimizers in the optimization landscape, in turn, seems to be closely related to the choice of the _activation function_ and the specific architecture of the ANN. Indeed, in the case of _fully-connected feedforward ANNs with just one hidden layer and one dimensional input and output layer_, on the one hand for several common (smooth) activations such as

* the _standard logistic activation_,
* the _softplus activation_,
* the _inverse tangent (arctan) activation_,
* the _hyperbolic tangent activation_, and
* the _softsign activation_

it has been shown (cf. [1, Theorems 1.3 and 1.4] and [1, Section 1.2]) that, in general, there _do not exist minimizers in the optimization landscape_ even if the class of considered target functions is restricted to smooth functions or even polynomials but, on the other hand for the _ReLU activation_, it has been proved in [1, Theorem 1.1] that for every Lipschitz continuous target function there _do exist minimizer in the optimization landscape_. These existence/non-existence phenomena for minimizers in the optimization landscape thus reveals a _fundamental advantage of the ReLU activation_ compared to the above mentioned smooth activations and this issue may be one of the reasons why the ReLU activation seems in many simulations to outperform other smooth activations and also why the ReLU activation is probably the most popular employed activation function in simulations, even though it fails to be differentiable in contrast to the above mentioned continuously differentiable activation functions.

Theorem 1.1 in [1] is, however, restricted to Lipschitz continuous target functions, to the _standard mean square loss_, and to ANNs with one neuron on the input layer and excludes ANNs with multi-dimensional input layers. It is precisely the subject of this work to extend
Theorem 1.1 in [14] in several ways, in particular, to the situation of _residual ANNs with multi-dimensional input, general continuous target functions, and general loss functions_.

In particular, in this work we establish in one of our main results in Theorem 1.1 below, under suitable assumptions on the probability distribution of the input data, the existence of minimizers in the optimization landscape in the situation of shallow residual ANNs with multi-dimensional input layers and multi-dimensional hidden layers with the _ReLU activation_\(\mathbb{R}\ni x\mapsto\max\{x,0\}=x^{+}\in\mathbb{R}.\) To simplify the presentation we restrict ourselves in Theorem 1.1 (as in [14, Theorem 1.1]) to the situation of the standard mean square loss but in our more general result in Theorem 1.3 - which includes Theorem 1.1 as a special case - also covers a general class of loss functions.

**Theorem 1.1** (Existence of minimizer - residual ANNs).: _Let \(d_{\mathrm{in}},d\in\mathbb{N}\), \(\mathfrak{d}=d_{\mathrm{in}}(d+1)+2d+1\), let \(f\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) and \(h\colon\mathbb{R}^{d_{\mathrm{in}}}\to[0,\infty)\) be continuous, assume that \(h^{-1}((0,\infty))\) is a bounded and convex set, for every \(\theta=(\theta_{1},\ldots,\theta_{\mathfrak{d}})\in\mathbb{R}^{\mathfrak{d}}\) let \(\mathfrak{N}_{\theta}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) satisfy for all \(x=(x_{1},\ldots,x_{d_{\mathrm{in}}})\in\mathbb{R}^{d_{\mathrm{in}}}\) that_

\[\mathfrak{N}_{\theta}(x)=\theta_{\mathfrak{d}}+\left(\sum_{j=1}^{d}\theta_{d _{\mathrm{in}}(d+1)+d+j}\max\{\theta_{d_{\mathrm{in}}(d+1)+j}+\sum_{i=1}^{d_{ \mathrm{in}}}\theta_{jd_{\mathrm{in}}+i}x_{i},0\}\right)+\sum_{i=1}^{d_{ \mathrm{in}}}\theta_{i}x_{i}, \tag{3}\]

_and let \(\mathrm{err}\colon\mathbb{R}^{\mathfrak{d}}\to\mathbb{R}\) satisfy for all \(\theta\in\mathbb{R}^{\mathfrak{d}}\) that \(\mathrm{err}(\theta)=\int_{\mathbb{R}^{d_{\mathrm{in}}}}(f(x)-\mathfrak{N}_{ \theta}(x))^{2}h(x)\,\mathrm{d}x.\) Then there exists \(\theta\in\mathbb{R}^{\mathfrak{d}}\) such that \(\mathrm{err}(\theta)=\inf_{\vartheta\in\mathbb{R}^{\mathfrak{d}}}\mathrm{err }(\vartheta)\)._

Loosely speaking, Theorem 1.1 reveals in the situation of _shallow residual ReLU ANNs_ with a \(d_{\mathrm{in}}\)-dimensional input layer (with \(d_{\mathrm{in}}\) neurons on the input layer), with a \(d\)-dimensional hidden layer (with \(d\) neurons on the hidden layer), and with a skip-connection from the input layer to the output layer that there exist minimizers in the standard mean square loss optimization landscape.

Theorem 1.1 can also be reformulated in the setup of shallow fully-connected feedforward ANNs with multi-dimensional input layers and multi-dimensional hidden layers with the _ReLU activation_\(\mathbb{R}\ni x\mapsto\max\{x,0\}=x^{+}\in\mathbb{R}\) for all except of one neuron on the hidden layer and the _identity activation_\(\mathbb{R}\ni x\mapsto x\in\mathbb{R}\) for the remaining neuron on the hidden layer. This reformulation of Theorem 1.1 is precisely the subject of the next result in Theorem 1.2 below. We also refer to Figure 1 for a graphical visualization illustrating the in Theorem 1.2 employed ANN architectures with \(d_{\mathrm{in}}\) neurons on the input layer and \(d+1\) neurons on the hidden layer.

**Theorem 1.2** (Existence of minimizer - fully-connected feedforward ANNs).: _Let \(d_{\mathrm{in}},d\in\mathbb{N}\), \(\mathfrak{d}=(d_{\mathrm{in}}+2)(d+1)+1\), let \(f\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) and \(h\colon\mathbb{R}^{d_{\mathrm{in}}}\to[0,\infty)\) be continuous, assume that \(h^{-1}((0,\infty))\) is a bounded and convex set, for every \(\theta=(\theta_{1},\ldots,\theta_{\mathfrak{d}})\in\mathbb{R}^{\mathfrak{d}}\) let \(\mathfrak{N}_{\theta}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) satisfy for all \(x=(x_{1},\ldots,x_{d_{\mathrm{in}}})\in\mathbb{R}^{d_{\mathrm{in}}}\) that_

\[\begin{split}\mathfrak{N}_{\theta}(x)&=\theta_{ \mathfrak{d}}+\theta_{(d_{\mathrm{in}}+1)(d+1)+1}\big{(}\theta_{d_{\mathrm{in }}(d+1)+1}+\sum_{i=1}^{d_{\mathrm{in}}}\theta_{i}x_{i}\big{)}\\ &\qquad\qquad+\sum_{j=1}^{d}\theta_{(d_{\mathrm{in}}+1)(d+1)+j+1} \max\{\theta_{d_{\mathrm{in}}(d+1)+j+1}+\sum_{i=1}^{d_{\mathrm{in}}}\theta_{jd _{\mathrm{in}}+i}x_{i},0\},\end{split} \tag{4}\]

_and let \(\mathrm{err}\colon\mathbb{R}^{\mathfrak{d}}\to\mathbb{R}\) satisfy for all \(\theta\in\mathbb{R}^{\mathfrak{d}}\) that \(\mathrm{err}(\theta)=\int_{\mathbb{R}^{d_{\mathrm{in}}}}(f(x)-\mathfrak{N}_{ \theta}(x))^{2}h(x)\,\mathrm{d}x.\) Then there exists \(\theta\in\mathbb{R}^{\mathfrak{d}}\) such that \(\mathrm{err}(\theta)=\inf_{\vartheta\in\mathbb{R}^{\mathfrak{d}}}\mathrm{err }(\vartheta)\)._

Theorem 1.1 and Theorem 1.2 are both direct consequences of the more general result in Theorem 1.3 below (which covers more general supports of the probability distribution of the input data of the consider learning problem and also covers a general class of loss functions instead of only the standard mean square loss). In this regard, we note that the set of the realization functions of shallow fully-connected feedforward ANNs with \(d_{\mathrm{in}}\) neurons on the input layer, with \(d+1\) neurons on the hidden layer, with the ReLU activation for \(d\) neurons on the
hidden layer, and with the identity activation for the remaining neuron on the hidden layer (see Figure 1) coincides with the set of the realization functions of shallow residual ReLU ANNs with \(d_{\text{in}}\) neurons on the input layer, with \(d\) neurons on the hidden layer, and with a skip-connection from the input layer to the output layer.

Theorem 1.1 and Theorem 1.2 somehow employ a vectorized description of ANNs in the sense that every real vector \(\theta\in\mathbb{R}^{\mathfrak{d}}\) represents an ANN whose realization function is \(\mathfrak{N}_{\theta}\colon\mathbb{R}^{d_{\text{in}}}\to\mathbb{R}\). In our more general Theorem 1.3 and in our later arguments, we represent ANNs in a more structurized way (see (5) below). More specifically, in the following we describe every ANN by a quadrupel consisting

* of a matrix \(W^{1}=(w^{1}_{j,i})_{(j,i)\in\{0,1,\ldots,d\}\times\{1,2,\ldots,d_{\text{in}}\} }\in\mathbb{R}^{(d+1)\times d_{\text{in}}}\) (_weight matrix_ to describe the linear part in the affine transformation from the \(d_{\text{in}}\)-dimensional input layer to the \((d+1)\)-dimensional hidden layer),

Figure 1. Graphical illustration of a _shallow fully-connected feedforward ANN_ with \(d_{\text{in}}\) neurons on the input layer (with a \(d_{\text{in}}\)-dimensional input layer), with \(d+1\) neurons on the hidden layer (with a \((d+1)\)-dimensional hidden layer), and with 1 neuron on the output layer (with a \(1\)-dimensional output layer) as considered in Theorem 1.3. The blue colored circle around _neuron 0_ on the hidden layer represents the identity function \(\mathbb{R}\ni x\mapsto x\in\mathbb{R}\) as the activation function for this neuron and the purple colored circles around _neuron 1_, _neuron 2_,..., and _neuron \(d\)_ on the hidden layer represent the ReLU function \(\mathbb{R}\ni x\mapsto\max\{x,0\}\in\mathbb{R}\) as the activation function for these neurons.
* of a vector \(b^{1}=(b^{1}_{i})_{i=0,1,\ldots,d}\in\mathbb{R}^{d+1}\) (_bias vector_ to the additive part in the affine transformation from the \(d_{\mathrm{in}}\)-dimensional input layer to the \((d+1)\)-dimensional hidden layer),
* of a matrix \(W^{2}=(w^{2}_{i})_{i=0,1,\ldots,d}\in\mathbb{R}^{1\times(d+1)}\) (_weight matrix_ to describe the linear part in the affine transformation from the \((d+1)\)-dimensional hidden layer to the \(1\)-dimensional output layer), and
* of a scalar \(b^{2}\in\mathbb{R}\) (_bias_ to describe the additive part in the affine transformation from the \((d+1)\)-dimensional hidden layer to the \(1\)-dimensional output layer).

Moreover, for every \(j\in\{0,1,\ldots,d\}\) we write \(w^{1}_{j}=(w^{1}_{j,1},\ldots,w^{1}_{j,d_{\mathrm{in}}})\in\mathbb{R}^{d_{ \mathrm{in}}}\). We let

\[\mathbb{W}=(W^{1},b^{1},W^{2},b^{2})\in\mathbb{R}^{(d+1)\times d_{\mathrm{in} }}\times\mathbb{R}^{d+1}\times\mathbb{R}^{1\times(d+1)}\times\mathbb{R}=: \mathcal{W}_{d} \tag{5}\]

and call \(\mathbb{W}\) a _network configuration_ and \(\mathcal{W}_{d}\) the _parametrization class_. We often refer to a configuration of a neural network as the (_neural_) _network_\(\mathbb{W}\). A configuration \(\mathbb{W}\in\mathcal{W}_{d}\) describes a function \(\mathfrak{N}^{\mathbb{W}}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) via

\[\mathfrak{N}^{\mathbb{W}}(x)=w^{2}_{0}(w^{1}_{0}\cdot x+b^{1}_{0})+\sum_{j=1}^ {d}w^{2}_{j}\big{(}w^{1}_{j}\cdot x+b^{1}_{j}\big{)}^{\!\!+}+b^{2}, \tag{6}\]

where \(\cdot\) denotes the scalar product. We call \(\mathfrak{N}^{\mathbb{W}}\)_realization function_ or _response_ of the network \(\mathbb{W}\).

Note that in general the response of a network is a continuous and piecewise affine function from \(\mathbb{R}^{d_{\mathrm{in}}}\) to \(\mathbb{R}\). We conceive \(\mathbb{W}\mapsto\mathfrak{N}^{\mathbb{W}}\) as a parametrization of a class of potential response functions \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) in a minimization problem. More explicitly, let \(\mu\colon\mathcal{B}(\mathbb{R}^{d_{\mathrm{in}}})\to[0,\infty)\) be a finite measure on the Borel sets of \(\mathbb{R}^{d_{\mathrm{in}}}\), let \(\mathbb{D}=\mathrm{supp}(\mu)\) be the support of \(\mu\), and let \(\mathcal{L}\colon\mathbb{D}\times\mathbb{R}\to[0,\infty)\) be a product measurable function, the _loss function_. We aim to minimize the error

\[\mathrm{err}^{\mathcal{L}}(\mathbb{W})=\int_{\mathbb{D}}\mathcal{L}(x, \mathfrak{N}^{\mathbb{W}}(x))\,\mathrm{d}\mu(x) \tag{7}\]

over all \(\mathbb{W}\in\mathcal{W}_{d}\) for a given \(d\in\mathbb{N}_{0}\) and we let

\[\mathrm{err}^{\mathcal{L}}_{d}=\inf_{\mathbb{W}\in\mathcal{W}_{d}}\mathrm{err }^{\mathcal{L}}(\mathbb{W}) \tag{8}\]

the minimal error with \(d+1\) neurons in the hidden layer. We stress that if there does not exist a neural network \(\mathbb{W}\in\mathcal{W}_{d}\) satisfying \(\mathrm{err}^{\mathcal{L}}(\mathbb{W})=\mathrm{err}^{\mathcal{L}}_{d}\) then every sequence \((\mathbb{W}_{n})_{n\in\mathbb{N}}\subseteq\mathcal{W}_{d}\) of networks satisfying \(\lim_{n\to\infty}\mathrm{err}^{\mathcal{L}}(\mathbb{W}_{n})=\mathrm{err}^{ \mathcal{L}}_{d}\) diverges to infinity.

The class of responses \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) that is considered in this article clearly contains all responses of shallow ANNs having only \(d\) ReLU neurons in the hidden layer and no linear neuron. Conversely since an affine function \(\mathfrak{a}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) can be represented as the response of two ReLU neurons, \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) is also contained in the class of responses of shallow ANNs using \(d+2\) neurons in the hidden layer that all apply the ReLU activation function. If \(\mu\) is compactly supported, then the \(\mathbb{D}\)-restricted responses in \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) even can be expressed by shallow ReLU networks with \(d+1\) neurons.

_Overparametrized networks_ in the setting of empirical risk minimization (more ReLU neurons than data points to fit) are often able to perfectly interpolate the data such that there exists a network configuration achieving zero error and, thus, a global minimum in the search space \(\mathbb{W}\mapsto\mathrm{err}^{\mathcal{L}}(\mathbb{W})\). However, for general measures \(\mu\) not necessarily consisting of a finite number of Dirac measures or consisting of a finite number of Dirac measures but in the practically more realistic underparametrized regime, the literature on the existence of global minima is very limited and we refer the reader to [17, Theorem 1.1] for a result in the setting of _shallow ReLU ANNs_
with one-dimensional input. See also [13] for a similar result concerning the existence of global minima in the regression task of approximating functions in the space \(L_{p}([0,1]^{d},\|\cdot\|_{p})\) with shallow ANNs using _heavyside activation_. We also refer to [14] for a general introduction into best approximators in normed spaces. We also point, e.g., to [1, 10, 11, 12, 13] and the references therein for sophisticated convergence analysis for GD optimization methods in such overparametrized regimes.

A good literature review regarding the loss landscape in neural network training can be found in [15]. For statements about the existence of and convergence to _non-optimal local minima_ in the training of (shallow) networks we refer the reader, e.g., to [1, 13, 14, 15, 16, 17, 18].

In this article, we show under mild assumptions on the measure \(\mu\) and the loss function \(\mathcal{L}\) that there exists a global minimum in the loss landscape. We suggest a kind of closure of the search space and, in a second step, show that the additional artificial responses are not optimal in the minimization problem. We state the main result of this article.

**Theorem 1.3** (Existence of minimizers - general loss functions and fully-connected feedforward ANNs).: _Assume that \(\mathbb{D}=\operatorname{supp}(\mu)\) is compact, assume that \(\mu\) has a continuous Lebesgue density \(h\colon\mathbb{R}^{d_{\mathrm{in}}}\to[0,\infty)\), assume that for every hyperplane \(H\subseteq\mathbb{R}^{d_{\mathrm{in}}}\) intersecting the interior of the convex hull of \(\mathbb{D}\) there is an element \(x\in H\) with \(h(x)>0\), and assume that the loss function \(\mathcal{L}\colon\mathbb{D}\times\mathbb{R}\to[0,\infty)\) satisfies the following assumptions:_

1. _(Continuity in the first argument) For every_ \(y\in\mathbb{R}\) _it holds that_ \(\mathbb{D}\ni x\mapsto\mathcal{L}(x,y)\in\mathbb{R}\) _is continuous._
2. _(Strict convexity in the second argument) For all_ \(x\in\mathbb{D}\) _it holds that_ \(\mathbb{R}\ni y\mapsto\mathcal{L}(x,y)\in\mathbb{R}\) _is strictly convex and attains its minimum._

_Then it holds for every \(d\in\mathbb{N}_{0}\) that there exists an optimal network \(\mathbb{W}\in\mathcal{W}_{d}\) with \(\operatorname{err}^{\mathcal{L}}(\mathbb{W})=\operatorname{err}^{\mathcal{L}}_ {d}\)._

Theorem 1.3 is an immediate consequence of Proposition 3.3 below. We stress that the statement of Proposition 3.3 is stronger in the sense that it even shows that in many situations the newly added functions to the extended target space perform strictly worse than the representable responses.

**Example 1.4** (Regression problem).: _Let \(\mu\) be as in Theorem 1.3, let \(f\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) be continuous, and let \(L\colon\mathbb{R}\to[0,\infty)\) be a strictly convex function that attains its minimum. Then the function \(\mathcal{L}\colon\mathbb{R}^{d_{\mathrm{in}}}\times\mathbb{R}\to[0,\infty)\) given by_

\[\mathcal{L}(x,y)=L(y-f(x)) \tag{9}\]

_for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\), \(y\in\mathbb{R}\) satisfies the assumptions of Theorem 1.3 and Theorem 1.3 allows us to conclude that the infimum_

\[\inf_{\mathbb{W}\in\mathcal{W}_{d}}\int L(\mathfrak{R}^{\mathbb{W}}(x)-f(x)) \operatorname{d}\!\mu(x) \tag{10}\]

_is attained for a network \(\mathbb{W}\in\mathcal{W}_{d}\)._

## 2. Generalized response of neural networks

We will work with more intuitive geometric descriptions of realization functions of networks \(\mathbb{W}\in\mathcal{W}_{d}\). We slightly modify the ideas given in [1] and later introduce the notion of a _generalized response_. Then, we show in Proposition 2.4 that, in a general approximation problem, there always exists a generalized response that solves the minimization task at least as good as the class \(\{\mathfrak{W}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\).
We call a network \(\mathbb{W}\in\mathcal{W}_{d}\)_non-degenerate_ iff for all \(j\in\{1,2,\ldots,d\}\) we have \(w_{j}^{1}\neq 0\). For a non-degenerate network \(\mathbb{W}\), we say that the neuron \(j\in\{1,2,\ldots,d\}\) has

* _normal_\(\mathfrak{n}_{j}=|w_{j}^{1}|^{-1}w_{j}^{1}\in\mathcal{O}:=\{x\in\mathbb{R}^{d_{ \mathrm{in}}}\colon|x|=1\}\),
* _offset_\(o_{j}=-|w_{j}^{1}|^{-1}b_{j}^{1}\in\mathbb{R}\), and
* _kink_\(\Delta_{j}=|w_{j}^{1}|w_{j}^{2}\in\mathbb{R}\).

Moreover, we call the affine mapping \(\mathfrak{a}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) given by

\[\mathfrak{a}(x)=w_{0}^{2}(w_{0}^{1}\cdot x+b_{0}^{1})+b^{2} \tag{11}\]

_affine background._ We call \((\mathfrak{n},o,\Delta,\mathfrak{a})\) with \(\mathfrak{n}=(\mathfrak{n}_{1},\ldots,\mathfrak{n}_{d})\in\mathcal{O}^{d}\), \(o=(o_{1},\ldots,o_{d})\in\mathbb{R}^{d}\), \(\Delta=(\Delta_{1},\ldots,\Delta_{d})\in\mathbb{R}^{d}\) and the affine function \(\mathfrak{a}\) the _effective tuple of \(\mathbb{W}\)_ and write \(\mathcal{E}_{d}\) for the set of all effective tuples using \(d\) ReLU neurons.

First, we note that the response of a non-degenerate ANN \(\mathbb{W}\) can be represented in terms of its effective tuple:

\[\begin{split}\mathfrak{N}^{\mathbb{W}}(x)&=\mathfrak{ a}(x)+\sum_{j=1}^{d}w_{j}^{2}(w_{j}^{1}\cdot x+b_{j}^{1})^{+}=\mathfrak{a}(x)+ \sum_{j=1}^{d}\Delta_{j}\Big{(}\frac{1}{|w_{j}^{1}|}w_{j}^{1}\cdot x+\frac{1}{ |w_{j}^{1}|}b_{j}^{1}\Big{)}^{+}\\ &=\mathfrak{a}(x)+\sum_{j=1}^{d}\Delta_{j}\big{(}\mathfrak{n}_{j} \cdot x-o_{j}\big{)}^{+}.\end{split} \tag{12}\]

With slight misuse of notation we also write

\[\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}\colon\mathbb{R}^{d_{\mathrm{ in}}}\to\mathbb{R},\qquad x\mapsto\mathfrak{a}(x)+\sum_{j=1}^{d}\Delta_{j} \big{(}\mathfrak{n}_{j}\cdot x-o_{j}\big{)}^{+} \tag{13}\]

and \(\mathrm{err}^{\mathcal{L}}(\mathfrak{n},o,\Delta,\mathfrak{a})=\int\mathcal{ L}(x,\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}(x))\,\mathrm{d}\mu(x)\). Although the tuple \((\mathfrak{n},o,\Delta,\mathfrak{a})\) does not uniquely describe a neural network, it describes a response function uniquely and thus we will speak of the neural network with effective tuple \((\mathfrak{n},o,\Delta,\mathfrak{a})\).

We stress that also the response of a degenerate network \(\mathbb{W}\) can be described as response associated to an effective tuple. Indeed, for every \(j\in\{1,\ldots,d\}\) with \(w_{j}^{1}=0\) the respective neuron has a constant contribution \(w_{j}^{2}(b_{j}^{1})^{+}\). Now, one can choose an arbitrary normal \(\mathfrak{n}_{j}\) and offset \(o_{j}\), set the kink equal to zero (\(\Delta_{j}=0\)) and add the constant \(w_{j}^{2}(b_{j}^{1})^{+}\) to the affine background \(\mathfrak{a}\). Repeating this procedure for every such neuron we get an effective tuple \((\mathfrak{n},o,\Delta,\mathfrak{a})\in\mathcal{E}_{d}\) that satisfies \(\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}=\mathfrak{N}^{\mathbb{W}}\). Conversely, for every effective tuple \((\mathfrak{n},o,\Delta,\mathfrak{a})\in\mathcal{E}_{d}\), the mapping \(\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}\) is the response of an appropriate network \(\mathbb{W}\in\mathcal{W}_{d}\). In fact, for \(j=1,\ldots,d\), one can choose \(w_{j}^{1}=\mathfrak{n}_{j}\), \(b_{j}^{1}=-o_{j}\) and \(w_{j}^{2}=\Delta_{j}\) and gets that for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\)

\[w_{j}^{2}(w_{j}^{1}\cdot x+b_{j}^{1})^{+}=\Delta_{j}(\mathfrak{n}_{j}\cdot x-o _{j})^{+}. \tag{14}\]

Analogously, one can choose \(w_{0}^{1}=\mathfrak{a}^{\prime}\), \(b_{0}^{1}=\mathfrak{a}(0)\), \(w_{0}^{2}=1\) and \(b^{2}=0\) such that for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\)

\[\mathfrak{a}(x)=w_{0}^{2}(w_{0}^{1}\cdot x+b_{0}^{1})+b^{2}. \tag{15}\]

This entails that

\[\mathrm{err}_{d}^{\mathcal{L}}=\inf_{(\mathfrak{n},o,\Delta,\mathfrak{b})\in \mathcal{E}_{d}}\int\mathcal{L}(x,\mathfrak{N}^{\mathfrak{n},o,\Delta, \mathfrak{b}}(x))\,\mathrm{d}\mu(x) \tag{16}\]

and the infimum is attained iff there is a network \(\mathbb{W}\in\mathcal{W}_{d}\) for which the infimum in (8) is attained.
For an effective tuple \((\mathfrak{n},o,\Delta,\mathfrak{a})\in\mathcal{E}_{d}\), we say that the \(j\)th ReLU neuron has the _breakline_

\[H_{j}=\big{\{}x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}_{j}\cdot x=o_{j }\big{\}} \tag{17}\]

and we call

\[A_{j}=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}_{j}\cdot x>o_{j}\} \tag{18}\]

the _domain of activity_ of the \(j\)th ReLU neuron. By construction, we have

\[\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}(x)=\mathfrak{a}(x)+\sum_{j= 1}^{d}\bigl{(}\Delta_{j}(\mathfrak{n}_{j}\cdot x-o_{j})\bigr{)}\mathds{1}_{A_{ j}}(x). \tag{19}\]

Outside the breaklines the function \(\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}\) is differentiable with

\[D\mathfrak{N}^{\mathfrak{n},o,\Delta,\mathfrak{a}}(x)=\mathfrak{a}^{\prime}(x )+\sum_{j=1}^{d}\Delta_{j}\mathfrak{n}_{j}\mathds{1}_{A_{j}}(x). \tag{20}\]

Note that for each summand \(j=1,\ldots,d\) along the breakline the difference of the differential on \(A_{j}\) and \(\overline{A_{j}^{c}}\) equals \(\Delta_{j}\mathfrak{n}_{j}\) (which is also true for the response function \(\mathfrak{N}^{\mathbb{W}}\) provided that it is differentiable in the reference points and there does not exist a second neuron having the same breakline \(H_{j}\)).

For a better understanding of the optimization problem discussed in this article it makes sense to view the set of responses \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) as a subset of the locally convex vector space \(\mathcal{L}^{1}_{\mathrm{loc}}\) of locally integrable functions. Then the set of response functions is not closed and the main task of this article is to show that functions that can be approached by functions from \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) but are itself no response functions will provide larger errors than the response functions.

We now extend the family of response functions.

**Definition 2.1**.: _We call a function \(\mathcal{R}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) a generalized response (generalized realization function) if it admits the following representation: there are \(K\in\mathbb{N}_{0}\), a tuple of open half-spaces \(\mathbf{A}=(A_{1},\ldots,A_{K})\) of \(\mathbb{R}^{d_{\mathrm{in}}}\) with pairwise distinct boundaries \(\partial A_{1},\ldots,\partial A_{K}\), a vector \(\mathbf{m}=(m_{1},\ldots,m_{K})\in\{1,2\}^{K}\), an affine mapping \(\mathfrak{a}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\), vectors \(\delta_{1},\ldots,\delta_{K}\in\mathbb{R}^{d_{\mathrm{in}}}\), and reals \(\mathfrak{b}_{1},\ldots,\mathfrak{b}_{K}\in\mathbb{R}\) such that_

1. _it holds for all_ \(x\in\mathbb{R}^{d_{\mathrm{in}}}\) _that_ (21) \[\mathcal{R}(x)=\mathfrak{a}(x)+\sum_{k=1}^{K}\bigl{(}\delta_{k}\cdot x+ \mathfrak{b}_{k}\bigr{)}\mathds{1}_{A_{k}}(x)\] _and_
2. _it holds for all_ \(k\in\{1,\ldots,K\}\) _with_ \(m_{k}=1\) _that_ (22) \[\partial A_{k}\subseteq\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\delta_{k} \cdot x+\mathfrak{b}_{k}=0\}.\]

_We will represent generalized responses as in (21), we call \(A_{1},\ldots,A_{K}\) the active half-spaces of the response, and we call \(m_{1},\ldots,m_{K}\) the multiplicities of the half-spaces \(A_{1},\ldots,A_{K}\). The minimal number \(m_{1}+\ldots+m_{K}\) that can be achieved in such a representation is called the dimension of \(\mathcal{R}\). For every \(d\in\mathbb{N}_{0}\) we denote by \(\mathfrak{R}_{d}\) the family of all generalized responses of dimension \(d\) or smaller. We call a generalized response \(\mathrm{simple}\) if it is continuous which means that all mutliplicities can be chosen equal to one. A response \(\mathcal{R}\in\mathfrak{R}_{d}\) is called _strict at dimension \(d\) if the response has dimension \(d-1\) or is discontinuous. We denote by \(\mathfrak{R}_{d}^{\mathrm{strict}}\)_the responses in \(\mathfrak{R}_{d}\) that are strict at dimension \(d\)._

**Remark 2.2**.: _Note that the sets \(\{\mathfrak{N}^{\mathbb{W}}\colon\mathbb{W}\in\mathcal{W}_{d}\}\) and \(\{\mathcal{R}\colon\mathcal{R}\in\mathfrak{R}_{d}\) is simple\(\}\) agree._
In this section, we work with a general measure \(\mu\) which may have unbounded support. The assumptions on \(\mu\) are stated in the next definition.

**Definition 2.3**.:
1. _An element_ \(x\) _of a hyperplane_ \(H\subseteq\mathbb{R}^{d_{\mathrm{in}}}\) _is called_ \(H\)-regular _if_ \(x\in\operatorname{supp}\mu|_{A}\) _and_ \(x\in\operatorname{supp}\mu|_{\overline{A}^{c}}\)_, where_ \(A\) _is an open half-space with_ \(\partial A=H\)_._
2. _A measure_ \(\mu\) _is called_ nice _if all hyperplanes have_ \(\mu\)_-measure zero and if for every open half-space_ \(A\) _with_ \(\mu(A),\mu(\overline{A}^{c})>0\) _the set of_ \(\partial A\)_-regular points cannot be covered by finitely many hyperplanes different from_ \(\partial A\)_._

Next, we will show that under quite weak assumptions there exist generalized responses of dimension \(d\) or smaller that achieve an error of at most \(\operatorname{err}_{d}^{\mathcal{L}}\).

**Proposition 2.4**.: _Assume that \(\mu\) is a nice measure on a closed subset \(\mathbb{D}\subseteq\mathbb{R}^{d_{\mathrm{in}}}\) of \(\mathbb{R}^{d_{\mathrm{in}}}\) and assume that the loss function \(\mathcal{L}\colon\mathbb{D}\times\mathbb{R}\to[0,\infty)\) is measurable and satisfies the following assumptions:_

1. _(Lower-semincontinuity in the second argument) For all_ \(x\in\mathbb{D}\)_,_ \(y\in\mathbb{R}\) _we have_ (23) \[\liminf_{y^{\prime}\to y}\mathcal{L}(x,y^{\prime})\geq\mathcal{L}(x,y).\]
2. _(Unbounded in the second argument) For all_ \(x\in\mathbb{D}\) _we have_ (24) \[\lim_{|y|\to\infty}\mathcal{L}(x,y)=\infty.\]

_Let \(d\in\mathbb{N}_{0}\) with \(\operatorname{err}_{d}^{\mathcal{L}}<\infty\). Then there exists a generalized response \(\mathcal{R}\in\mathfrak{R}_{d}\) which satisfies_

\[\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x)=\overline{\operatorname{ err}_{d}^{\mathcal{L}}}:=\inf_{\tilde{\mathcal{R}}\in\mathfrak{R}_{d}}\int \mathcal{L}(x,\tilde{\mathcal{R}}(x))\,\mathrm{d}\mu(x). \tag{25}\]

_Furthermore, if \(d\geq 1\), then the infimum_

\[\inf_{\tilde{\mathcal{R}}\in\mathfrak{R}_{d}^{\mathrm{strict}}}\int\mathcal{ L}(x,\tilde{\mathcal{R}}(x))\,\mathrm{d}\mu(x) \tag{26}\]

_is attained on \(\mathfrak{R}_{d}^{\mathrm{strict}}\)._

Proof.: Let \((\mathcal{R}^{(n)})_{n\in\mathbb{N}}\) be a sequence of generalized responses in \(\mathfrak{R}_{d}\) that satisfies

\[\lim_{n\to\infty}\int\mathcal{L}(x,\mathcal{R}^{(n)}(x))\,\mathrm{d}\mu(x)= \overline{\operatorname{err}_{d}^{\mathcal{L}}}. \tag{27}\]

We use the representations as in (21) and write

\[\mathcal{R}^{(n)}(x)=\mathfrak{a}^{(n)}(x)+\sum_{k=1}^{K_{n}}\bigl{(}\delta_{k }^{(n)}\cdot x+\mathfrak{b}_{k}^{(n)}\bigr{)}\mathds{1}_{A_{k}^{(n)}}(x). \tag{28}\]

Moreover, denote by \(\mathfrak{n}_{k}^{(n)}\in\mathcal{O}\) and \(o_{k}^{(n)}\in\mathbb{R}\) the quantities with \(A_{k}^{(n)}=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}_{k}^{(n)} \cdot x>o_{k}^{(n)}\}\) and by \(\mathbf{m}^{(n)}=(m_{1}^{(n)},\dots,m_{K^{(n)}}^{(n)})\) the respective multiplicities.

1. Step: Choosing an appropriate subsequence.

Since \(K^{(n)}\leq d\) for all \(n\in\mathbb{N}\) we can choose a subsequence \((\ell_{n})_{n\in\mathbb{N}}\) such that there exists \(K\in\mathbb{N}_{0}\) with \(K^{(\ell_{n})}=K\) for all \(n\in\mathbb{N}\). For ease of notation we will assume that this is the case for the full sequence. With the same argument we can assume without loss of generality that there exists \(\mathbf{m}=(m_{1},\dots,m_{K})\in\{1,2\}^{K}\) such that for all \(n\in\mathbb{N}\) we have \(\mathbf{m}^{(n)}=\mathbf{m}\).
Moreover, after possibly thinning the sequence again we can assure that for all \(k\in\{1,2,\ldots,K\}\) we have convergence \(\mathfrak{n}_{k}^{(n)}\to\mathfrak{n}_{k}\) in the compact space \(\mathcal{O}\) and \(o_{k}^{(n)}\to o_{k}\) in the two point compactification \(\mathbb{R}\cup\{\pm\infty\}\). We assign each \(k\in\{1,2,\ldots,K\}\) an _asymptotic active area_\(A_{k}\) given by

\[A_{k}=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}_{k}\cdot x>o_{k}\} \tag{29}\]

which is degenerate in the case where \(o_{k}\in\{\pm\infty\}\).

We denote by \(H_{k}\) the respective breakline \(\partial A_{k}\). Even if, for every \(n\in\mathbb{N}\), the original breaklines \(\partial A_{1}^{(n)},\ldots,\partial A_{K}^{(n)}\) are pairwise distinct, this might not be true for the limiting ones. In particular, there may be several \(k\)'s for which the asymptotic active areas may be on opposite sides of the same breakline. In that case we choose one side and replace for each \(k\) with asymptotic active area on the opposite side its contribution in the representation (21) from \((\delta_{k}^{(n)}\cdot x+\mathfrak{b}_{k}^{(n)})\mathds{1}_{A_{k}^{(n)}}(x)\) to

\[\big{(}-\delta_{k}^{(n)}\cdot x-\mathfrak{b}_{k}^{(n)}\big{)}\mathds{1}_{( \overline{A}_{k}^{(n)})^{c}}(x)+\delta_{k}^{(n)}\cdot x+\mathfrak{b}_{k}^{(n)} \tag{30}\]

which agrees with the former term outside the breakline (which is a zero set). This means we replace \(\delta_{k}^{(n)}\), \(\mathfrak{b}_{k}^{(n)}\), and \(A_{k}^{(n)}\) by \(-\delta_{k}^{(n)}\), \(-\mathfrak{b}_{k}^{(n)}\), and \(\big{(}\overline{A}_{k}^{(n)}\big{)}^{c}\), respectively, and adjust the respective affine background accordingly. Thus we can assume without loss of generality that all asymptotic active areas sharing the same breakline are on the same side.

We use the asymptotic active areas to partition the space: let \(\mathbb{J}\) denote the collection of all subsets \(J\subseteq\{1,2,\ldots,K\}\) for which the set

\[A_{J}=\big{(}\bigcap_{j\in J}A_{j}\big{)}\cap\big{(}\bigcap_{j\in J^{c}} \overline{A}_{j}^{c}\big{)} \tag{31}\]

satisfies \(\mu(A_{J})>0\). We note that the sets \(A_{J}\), \(J\in\mathbb{J}\), are non-empty, open, and pairwise disjoint and their union has full \(\mu\)-measure since

\[\mu\Big{(}\mathbb{R}^{d_{\mathrm{in}}}\backslash\bigcup_{J\subseteq\{1,2, \ldots,K\}}A_{J}\Big{)}\leq\sum_{j=1}^{K}\mu\big{(}H_{j}\big{)}=0. \tag{32}\]

Moreover, for every \(J\in\mathbb{J}\) and every compact set \(B\) with \(B\subseteq A_{J}\) one has from a \(B\)-dependent \(n\) onwards that the generalized response \(\mathcal{R}^{(n)}\) satisfies for all \(x\in B\) that

\[\mathcal{R}^{(n)}(x)=\mathcal{D}_{J}^{(n)}\cdot x+\beta_{J}^{(n)}, \tag{33}\]

where

\[\mathcal{D}_{J}^{(n)}:=\mathsf{a^{\prime}}^{(n)}+\sum_{j\in J}\delta_{j}^{(n) }\qquad\text{and}\qquad\beta_{J}^{(n)}:=\mathsf{a}^{(n)}(0)+\sum_{j\in J} \mathfrak{b}_{j}^{(n)}. \tag{34}\]

Let \(J\in\mathbb{J}\). Next, we show that along an appropriate subsequence, we have convergence of \((\mathcal{D}_{J}^{(n)})_{n\in\mathbb{N}}\) in \(\mathbb{R}^{d_{\mathrm{in}}}\). First assume that along a subsequence one has that \((|\mathcal{D}_{J}^{(n)}|)_{n\in\mathbb{N}}\) converges to \(\infty\). For ease of notation we assume without loss of generality that \(|\mathcal{D}_{J}^{(n)}|\to\infty\). We let

\[\mathcal{H}_{J}^{(n)}=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathcal{D}_{J}^ {(n)}\cdot x+\beta_{J}=0\}. \tag{35}\]

For every \(n\) with \(\mathcal{D}_{J}^{(n)}\neq 0\), \(\mathcal{H}_{J}^{(n)}\) is a hyperplane which can be parametrized by taking a normal and the respective offset. As above we can argue that along an appropriate subsequence (which is again assumed to be the whole sequence) one has convergence of the normals in \(\mathcal{O}\) and of the offsets in \(\mathbb{R}\cup\{\pm\infty\}\). We denote by \(\mathcal{H}_{J}\) the hyperplane being associated to the limiting normal and offset (which is assumed to be the empty set in the case where the offsets do not converge
in \(\mathbb{R}\)). Since the norm of the gradient \(\mathcal{D}^{(n)}_{J}\) tends to infinity we get that for every \(x\in A_{J}\backslash\mathcal{H}_{J}\) one has \(|\mathcal{R}^{(n)}(x)|\to\infty\) and, hence, \(\mathcal{L}(x,\mathcal{R}^{(n)}(x))\to\infty\). Consequently, Fatou implies that

\[\liminf_{n\to\infty}\int_{A_{J}\backslash\mathcal{H}_{J}}\mathcal{L}(x, \mathcal{R}^{(n)}(x))\,\mathrm{d}\mu(x)\geq\int_{A_{J}\backslash\mathcal{H}_{ J}}\liminf_{n\to\infty}\mathcal{L}(x,\mathcal{R}^{(n)}(x))\,\mathrm{d}\mu(x)=\infty \tag{36}\]

contradicting the asymptotic optimality of \((\mathcal{R}^{(n)})_{n\in\mathbb{N}}\). We showed that the sequence \((\mathcal{D}^{(n)}_{J})_{n\in\mathbb{N}}\) is precompact and by switching to an appropriate subsequence we can guarantee that the limit \(\mathcal{D}_{J}=\lim_{n\to\infty}\mathcal{D}^{(n)}_{J}\) exists.

Similarly we show that along an appropriate subsequence \((\beta^{(n)}_{J})_{n\in\mathbb{N}}\) converges to a value \(\beta_{J}\in\mathbb{R}\). Suppose this were not the case, then there were a subsequence along which \(|\beta^{(n)}_{J}|\to\infty\) (again we assume for ease of notation that this is the case along the full sequence). Then for every \(x\in A_{J}\), \(|\mathcal{R}^{(n)}(x)|\to\infty\) and we argue as above that this contradicts the optimality of \((\mathcal{R}^{(n)})_{n\in\mathbb{N}}\). Consequently, we have for an appropriately thinned sequence on a compact set \(B\subseteq A_{J}\) uniform convergence

\[\lim_{n\to\infty}\mathcal{R}^{(n)}(x)=\mathcal{D}_{J}\cdot x+\beta_{J}. \tag{37}\]

Since \(\bigcup_{J\in\mathbb{J}}A_{J}\) has full \(\mu\)-measure we get with the lower semicontinuity of \(\mathcal{L}\) in the second argument and Fatou's lemma that for every measurable function \(\mathcal{R}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) satisfying for each \(J\in\mathbb{J}\), \(x\in A_{J}\) that

\[\mathcal{R}(x)=\mathcal{D}_{J}\cdot x+\beta_{J} \tag{38}\]

we have

\[\begin{split}\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x) &=\int\liminf_{n\to\infty}\mathcal{L}(x,\mathcal{R}^{(n)}(x))\, \mathrm{d}\mu(x)\\ &\leq\liminf_{n\to\infty}\int\mathcal{L}(x,\mathcal{R}^{(n)}(x)) \,\mathrm{d}\mu(x)=\overline{\mathrm{err}}_{d}^{\mathcal{L}}.\end{split} \tag{39}\]

Step 2: \(\mathcal{R}\) may be chosen as a generalized response of dimension \(d\) or smaller.

We call a summand \(k\in\{1,2,\ldots,K\}\) degenerate if \(A_{k}\) or \(\overline{A}_{k}^{c}\) has \(\mu\)-measure zero. We omit every degenerate summand \(k\) in the sense that we set \(\delta^{(n)}_{k}=0\) and \(\mathfrak{b}^{(n)}_{k}=0\) for all \(n\in\mathbb{N}\) and note that by adjusting the affine background appropriately we still have validity of (37) with the same limit on all relevant cells and, in particular, \(\mu\)-almost everywhere.

Let now \(k\) be a non-degenerate summand. Since \(\mu\) is nice there exists a \(\partial A_{k}\)-regular point \(x\) that is not in \(\bigcup_{A\in\mathbb{A}\colon\,A\neq A_{k}}\partial A\), where \(\mathbb{A}:=\{A_{j}\colon j\text{ is non-degenerate}\}\). We let

\[J^{x}_{-}=\{j\colon x\in A_{j}\}\qquad\text{and}\qquad J^{x}_{+}=J^{x}_{-} \cup\{j\colon A_{j}=A_{k}\}. \tag{40}\]

Since \(x\in\operatorname{supp}(\mu|_{\overline{A}_{j}^{c}})\) we get that the cell \(A_{J^{x}_{-}}\) has strictly positive \(\mu\)-measure so that \(J^{x}_{-}\in\mathbb{J}\). Analogously, \(x\in\operatorname{supp}(\mu|_{A_{j}})\) entails that \(J^{x}_{+}\in\mathbb{J}\). (Note that \(J^{x}_{+}\) and \(J^{x}_{-}\) are just the cells that lie on the opposite sides of the hyperplane \(\partial A_{k}\) at \(x\).) We thus get that

\[\delta^{(n)}_{A_{k}}:=\sum_{j\colon\,A_{j}=A_{k}}\delta^{(n)}_{j}=\mathcal{D}^ {(n)}_{J^{x}_{+}}-\mathcal{D}^{(n)}_{J^{x}_{-}}\to\mathcal{D}_{J^{x}_{+}}- \mathcal{D}_{J^{x}_{-}}=:\delta_{A_{k}}, \tag{41}\]

where the definitions of \(\delta^{(n)}_{A_{k}}\) and \(\delta_{A_{k}}\) do not depend on the choice of \(x\). Analogously,

\[\mathfrak{b}^{(n)}_{A_{k}}:=\sum_{j\colon\,A_{j}=A_{k}}\mathfrak{b}^{(n)}_{j}= \beta^{(n)}_{J^{x}_{+}}-\beta^{(n)}_{J^{x}_{-}}\to\beta_{J^{x}_{+}}-\beta_{J^{x} _{-}}=:\mathfrak{b}_{A_{k}}. \tag{42}\]
Now for general \(J\in\mathbb{J}\), we have

\[\mathcal{D}_{J}\leftarrow\mathcal{D}_{J}^{(n)}=\mathfrak{a}^{\prime(n)}+\sum_{A \in\{A_{j}:\,j\in J\}}\delta_{A}^{(n)}\qquad\text{and}\qquad\beta_{J}\gets \beta_{J}^{(n)}=\mathfrak{a}^{(n)}(0)+\sum_{A\in\{A_{j}:\,j\in J\}}\mathfrak{b}_{ A}^{(n)}. \tag{43}\]

Since \(\sum_{A\in\{A_{j}:\,j\in J\}}\delta_{A}^{(n)}\) and \(\sum_{A\in\{A_{j}:\,j\in J\}}\mathfrak{b}_{A}^{(n)}\) converge to \(\sum_{A\in\{A_{j}:\,j\in J\}}\delta_{A}\) and \(\sum_{A\in\{A_{j}:\,j\in J\}}\mathfrak{b}_{A}\), respectively, we have that \((\mathfrak{a}^{\prime(n)})_{n\in\mathbb{N}}\) and \((\mathfrak{a}^{(n)}(0))_{n\in\mathbb{N}}\) converge and there is an appropriate affine function \(\mathfrak{a}\) such that for all \(J\in\mathbb{J}\) and \(x\in A_{J}\)

\[\mathcal{R}(x)=\mathcal{D}_{J}\cdot x+\beta_{J}=\mathfrak{a}(x)+\sum_{A\in \mathbb{A}}\bigl{(}\delta_{A}\cdot x+\mathfrak{b}_{A}\bigr{)}\mathds{1}_{A}( x). \tag{44}\]

So far, we have only used the definition of \(\mathcal{R}\) on \(\bigcup_{J\in\mathbb{J}}A_{J}\) and we now assume that \(\mathcal{R}\) is chosen in such a way that the latter identity holds for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\) (by possibly changing the definition on a \(\mu\)-nullset). We still need to show that \(\mathcal{R}\) is a generalized response of dimension \(d\) or smaller.

Every active area \(A\in\mathbb{A}\) that is the asymptotic active area of a single non-degenerate summand \(k\) with \(m_{k}=1\) is assigned the multiplicity one. All other non-degenerate active areas get multiplicity two. Then the overall multiplicity (the sum of the individual multiplicities) is smaller or equal to the dimension \(d\). To see this recall that every active area \(A\in\mathbb{A}\) that is the asymptotic area of more than one summand or one summand of multiplicity two also contributed at least two to the multiplicity of the approximating responses. All degenerate summands do not contribute at all although they contribute to the approximating responses.

It remains to show that \(\mathcal{R}\) is indeed a generalized response with the active areas \(\mathbb{A}\) having the above multiplicities. For this it remains to show continuity of \(\mathds{1}_{A}(x)(\delta_{A}\cdot x+\mathfrak{b}_{A})\) for all \(A\in\mathbb{A}\) with assigned multiplicity one. Suppose that the \(k\)th summand is the unique summand that contributes to such an \(A\). Then \(\delta_{k}^{(n)}=\delta_{A}^{(n)}\to\delta_{A}\) and \(\mathfrak{b}_{k}^{(n)}=\mathfrak{b}_{A}^{(n)}\to\mathfrak{b}_{A}\). Moreover, one has

\[\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}_{k}^{(n)}\cdot x-o_{k}^{ (n)}=0\}\subseteq\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\delta_{k}^{(n)} \cdot x+\mathfrak{b}_{k}^{(n)}=0\} \tag{45}\]

which entails that, in particular, \(\delta_{k}^{(n)}\) is a multiple of \(\mathfrak{n}_{k}^{(n)}\). Both latter vectors converge which also entails that the limit \(\delta_{A}\) is a multiple of \(\mathfrak{n}_{k}\). To show that

\[\partial A\subseteq\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\delta_{A}\cdot x +\mathfrak{b}_{A}=0\} \tag{46}\]

is satisfied it thus suffices to verify that one point of the set on the left-hand side lies also in the set on the right-hand side. This is indeed the case since \(o_{k}\mathfrak{n}_{k}\) is in the set on the left-hand side and

\[\delta_{A}\cdot(o_{k}\mathfrak{n}_{k})+\mathfrak{b}_{A}=\lim_{n\to\infty} \underbrace{\delta_{k}^{(n)}\cdot(o_{k}^{(n)}\mathfrak{n}_{k}^{(n)})+ \mathfrak{b}_{k}^{(n)}}_{=0}, \tag{47}\]

where we used that \(x=o_{k}^{(n)}\mathfrak{n}_{k}^{(n)}\) satisfies by assumption \(\delta_{k}^{(n)}\cdot x+\mathfrak{b}_{k}^{(n)}=0\).

Step 3: The infimum over all strict responses is attained.

We suppose that \((\mathcal{R}^{(n)})_{n\in\mathbb{N}}\) is a sequence of strict generalized responses satisfying

\[\lim_{n\to\infty}\int\mathcal{L}(x,\mathcal{R}^{(n)}(x))\,\mathrm{d}\mu(x)= \inf_{\tilde{\mathcal{R}}\in\mathfrak{R}_{d}^{\mathrm{strict}}}\int\mathcal{L }(x,\tilde{\mathcal{R}}(x))\,\mathrm{d}\mu(x). \tag{48}\]

Then we can find a subsequence of responses in \(\mathfrak{R}_{d-1}\) or a subsequence of responses where at least one active area has multiplicity two. In the former case the response constructed above is a generalized response of dimension \(d-1\) or lower which is strict at dimension \(d\). Conversely, in the latter case the construction from above will lead to a generalized response that has at
least one active area with multiplicity two which, in turn, implies that \(\mathcal{R}\) is either of dimension strictly smaller than \(d\) or is discontinuous. 

**Remark 2.5** (Asymptotic ANN representations for generalized responses).: _In this remark, we show that every generalized response \(\mathcal{R}\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) of dimension \(d\) is, on \(\mathbb{R}^{d_{\mathrm{in}}}\backslash(\bigcup_{k=1,\ldots,K}\partial A_{k})\), the limit of ANN responses of ANNs in \(\mathcal{W}_{d}\)._

_Let \(\mathfrak{n}\in\mathcal{O}\), \(\delta\in\mathbb{R}^{d_{\mathrm{in}}}\) and \(o,\mathfrak{b}\in\mathbb{R}\) and set_

\[A=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathfrak{n}\cdot x>o\}\qquad\text{ and}\qquad\forall\,x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\mathcal{R}(x)= \mathds{1}_{A}(\delta\cdot x+\mathfrak{b}). \tag{49}\]

_We will show that the generalized response \(\mathcal{R}\) is on \(\mathbb{R}^{d_{\mathrm{in}}}\backslash(\partial A)\) the limit of the response of two ReLU neurons. For every \(n\in\mathbb{N}\), the following function_

\[\mathcal{R}_{n}(x)=\frac{1}{2}\big{(}(\delta+n\mathfrak{n})\cdot x+\mathfrak{ b}-no\big{)}^{+}-\frac{1}{2}\big{(}(-\delta+n\mathfrak{n})\cdot x-\mathfrak{b}- no\big{)}^{+} \tag{50}\]

_is the response of a shallow ReLU ANN with two hidden ReLU neurons. Now note that for all \(x\in A\) one has \(\mathfrak{n}\cdot x-o>0\) so that_

\[(\delta+n\mathfrak{n})\cdot x+\mathfrak{b}-no=n(\mathfrak{n}\cdot x-o)+\delta \cdot x+\mathfrak{b}\to\infty. \tag{51}\]

_Analogously, \((-\delta+n\mathfrak{n})\cdot x-\mathfrak{b}-no\to\infty\). Consequently, for these \(x\) one has for large \(n\) that_

\[\mathcal{R}_{n}(x)=\delta\cdot x+\mathfrak{b}=\mathcal{R}(x). \tag{52}\]

_Conversely, for all \(x\in\overline{A}^{c}\) one has \(\mathfrak{n}\cdot x-o<0\) so that analogously to above \((\delta+n\mathfrak{n})\cdot x+\mathfrak{b}-no\to-\infty\) and \((-\delta+n\mathfrak{n})\cdot x-\mathfrak{b}-no\to-\infty\). Consequently, for these \(x\) one has for large \(n\) that_

\[\mathcal{R}_{n}(x)=0=\mathcal{R}(x). \tag{53}\]

_We thus represented \(\mathcal{R}\) as asymptotic response of a ReLU ANN with two hidden neurons._

_For a generalized response one replaces every term \(\mathds{1}_{A_{k}}(\delta_{k}\cdot x+\mathfrak{b})\) of multiplicity two (see (21)) by two ReLU neurons exactly as above. Moreover, the terms with multiplicity one are responses of ANNs with one ReLU neuron._

## 3. Discontinuous responses are not optimal

In this section, we show that generalized responses that contain discontinuities are not optimal in the minimization task for a loss function that is continuous in the first argument. This proves Theorem 1.3 since all continuous generalized responses can be represented by shallow networks.

In the proofs we will make use of the following properties of the loss functions \(\mathcal{L}\) under consideration.

**Lemma 3.1**.: _Let \(\mathcal{L}\colon\mathbb{D}\times\mathbb{R}\to[0,\infty)\) be a function satisfying the following assumptions:_

1. _(Continuity in the first argument) For every_ \(y\in\mathbb{R}\) _it holds that_ \(\mathbb{D}\ni x\mapsto\mathcal{L}(x,y)\in\mathbb{R}\) _is continuous._
2. _(Strict convexity in the second argument) For all_ \(x\in\mathbb{D}\) _it holds that_ \(\mathbb{R}\ni y\mapsto\mathcal{L}(x,y)\in\mathbb{R}\) _is strictly convex and attains its minimum._

_Then_

1. _it holds that_ \(\mathcal{L}\colon\mathbb{D}\times\mathbb{R}\to[0,\infty)\) _is continuous,_
2. _it holds for every compact_ \(K\subseteq\mathbb{R}\) _that the function_ \(\mathbb{D}\ni x\mapsto\mathcal{L}(x,\cdot)|_{K}\in C(K,\mathbb{R})\) _is continuous with to the supremum norm,_
3. _it holds that there exists a unique_ \(\mathfrak{m}\colon\mathbb{D}\to\mathbb{R}\) _which satisfies for every_ \(x\in\mathbb{D}\) _that_ (54) \[\mathcal{L}(x,\mathfrak{m}(x))=\min_{y\in\mathbb{R}}\mathcal{L}(x,y),\] _and_
[MISSING_PAGE_POST]

We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\).

We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\).

We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\).

We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\).

We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\). We prove that \(\mathfrak{M}\) is not representable by a network with \(\mathcal{A}\).
where \((\cdot)^{\dagger}\) denotes the transpose of a vector or a matrix. Clearly, continuity of the summands is preserved and, hence, one can choose the same multiplicities. Therefore, \(\mathcal{R}\circ\varphi\) is again in \(\mathfrak{R}_{d}\) and it is even strict at dimension \(d\) or simple if this is the case for \(\mathcal{R}\). Applying the inverse affine transform \(\varphi^{-1}\) we also obtain equivalence of the properties. 

We are now in the position to prove the main statement of this article, Theorem 1.3. It is an immediate consequence of the following stronger result.

**Proposition 3.3**.: _Suppose that the assumptions of Theorem 1.3 are satisfied. Let \(d\in\mathbb{N}_{0}\). Then there exists an optimal network \(\mathbb{W}\in\mathcal{W}_{d}\) with_

\[\operatorname{err}^{\mathcal{L}}(\mathbb{W})=\operatorname{err}^{\mathcal{L} }_{d}=\overline{\operatorname{err}}^{\mathcal{L}}_{d}. \tag{61}\]

_If additionally \(d>1\) and \(\operatorname{err}^{\mathcal{L}}_{d}<\operatorname{err}^{\mathcal{L}}_{d-1}\), then one has that_

\[\inf_{\mathcal{R}\in\mathfrak{P}^{\operatorname{strict}}_{d}}\int\mathcal{L}( x,\mathcal{R}(x))\,\mathrm{d}\mu(x)>\operatorname{err}^{\mathcal{L}}_{d}. \tag{62}\]

Proof.: We can assume without loss of generality that \(\mu\neq 0\). First we verify the assumptions of Proposition 2.4 in order to conclude that there are generalized responses \(\mathcal{R}\) of dimension \(d\) for which

\[\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x)=\overline{ \operatorname{err}}^{\mathcal{L}}_{d}. \tag{63}\]

We verify that \(\mu\) is a nice measure: In fact, since \(\mu\) has Lebesgue-density \(h\), we have \(\mu(H)=0\) for all hyperplanes \(H\subseteq\mathbb{R}^{d_{\mathrm{in}}}\). Moreover, for every half-space \(A\) with \(\mu(A),\mu(\overline{A}^{c})>0\) we have that \(\partial A\) intersects the interior of the convex hull of \(\mathbb{D}\) so that there exists a point \(x\in\partial A\) with \(h(x)>0\). Since \(\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon h(x)>0\}\) is an open set, \(\{x\in\partial A:h(x)>0\}\) cannot be covered by finitely many hyperplanes different from \(\partial A\). Moreover, since for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\) the function \(y\mapsto\mathcal{L}(x,y)\) is strictly convex and attains its minimum we clearly have for fixed \(x\in\mathbb{R}^{d_{\mathrm{in}}}\) continuity of \(y\mapsto\mathcal{L}(x,y)\) and

\[\lim_{|y|\to\infty}\mathcal{L}(x,y)=\infty.\]

We prove the remaining statements via induction over the dimension \(d\). If \(d\leq 1\), all generalized responses of dimension \(d\) are representable by a neural network and we are done. Now let \(d\geq 2\) and suppose that \(\mathcal{R}\) is the best _strict_ generalized response at dimension \(d\). It suffices to show that one of the following two cases enters: one has

\[\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x)\geq\overline{ \operatorname{err}}^{\mathcal{L}}_{d-1} \tag{64}\]

or

\[\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x)>\overline{ \operatorname{err}}^{\mathcal{L}}_{d}. \tag{65}\]

Indeed, then in the case that (65) does not hold we have as consequence of (64)

\[\overline{\operatorname{err}}^{\mathcal{L}}_{d-1}\leq\int\mathcal{L}(x, \mathcal{R}(x))\,\mathrm{d}\mu(x)=\overline{\operatorname{err}}^{\mathcal{L}}_ {d} \tag{66}\]

and the induction hypothesis entails that \(\operatorname{err}^{\mathcal{L}}_{d-1}=\overline{\operatorname{err}}^{ \mathcal{L}}_{d-1}\leq\overline{\operatorname{err}}^{\mathcal{L}}_{d}\leq \operatorname{err}^{\mathcal{L}}_{d}\leq\operatorname{err}^{\mathcal{L}}_{d-1}\) so that \(\operatorname{err}^{\mathcal{L}}_{d}=\overline{\operatorname{err}}^{ \mathcal{L}}_{d}\) and \(\operatorname{err}^{\mathcal{L}}_{d}=\operatorname{err}^{\mathcal{L}}_{d-1}\). Thus, an optimal simple response \(\mathcal{R}\) of dimension \(d-1\) (which exists by induction hypothesis) is also optimal when taking the minimum over all generalized responses of dimension \(d\) or smaller. Conversely, if (65) holds, an optimal generalized response (which exists by Proposition 2.4) is simple so that, in particular, \(\operatorname{err}^{\mathcal{L}}_{d}=\overline{\operatorname{err}}^{\mathcal{L}}_ {d}\). This shows that
there always exists an optimal simple response. Moreover, it also follows that in the case where \(\operatorname{err}_{d}^{\mathcal{L}}<\operatorname{err}_{d-1}^{\mathcal{L}}\), either of the properties (64) and (65) entail property (62).

Suppose that \(\mathcal{R}\) is given by

\[\mathcal{R}(x)=\mathfrak{a}(x)+\sum_{k=1}^{K}\bigl{(}\delta_{k}\cdot x+ \mathfrak{b}_{k}\bigr{)}\mathds{1}_{A_{k}}(x), \tag{67}\]

with \(A_{1},\ldots,A_{K}\) being the pairwise different activation areas and \(m_{1},\ldots,m_{K}\) being the respective multiplicities. Note that \(\mathcal{R}\) has to be discontinuous, because otherwise \(\mathcal{R}\) is of dimension strictly smaller than \(\operatorname{d}\) and (64) holds. Therefore, we can assume without loss of generality that \(m_{K}=2\) and

\[\partial A_{K}\not\subseteq\{x\in\mathbb{R}^{d_{\operatorname{in}}}\colon \delta_{K}\cdot x+\mathfrak{b}_{K}=0\} \tag{68}\]

(otherwise we reorder the terms appropriately).

If \(\partial A_{K}\) does not intersect the interior of \(\mathbb{D}\), then one can replace the term \(\mathds{1}_{A_{K}}(x)\bigl{(}\delta_{K}\cdot x+\mathfrak{b}_{K}\bigr{)}\) by \(\delta_{K}\cdot x+\mathfrak{b}_{K}\) or \(0\) without changing the error on \(\mathbb{D}\). By doing so the new response has dimension \(d-2\) or smaller. Thus, we get that

\[\int_{\mathbb{D}}\mathcal{L}(x,\mathcal{R}(x))\operatorname{d}\!\mu(x)\geq \overline{\operatorname{err}}_{d-2}^{\mathcal{L}}. \tag{69}\]

Now suppose that \(\partial A_{K}\) intersects the interior of \(\mathbb{D}\). We prove that \(\mathcal{R}\) is not an optimal response in \(\mathfrak{R}_{d}\) by constructing a better response. To see this we apply an appropriate affine transformation on the coordinate mapping. For an invertible matrix \(B\in\mathbb{R}^{d_{\operatorname{in}}\times d_{\operatorname{in}}}\) and a vector \(c\in\mathbb{R}^{d_{\operatorname{in}}}\) we consider the invertible affine mapping

\[\varphi(x)=B(x+c). \tag{70}\]

By Lemma 3.2 the (strict) generalized responses are invariant under right applications of bijective affine transformations so that \(\hat{\mathcal{R}}=\mathcal{R}\circ\varphi\) is an optimal strict generalized response for the loss function \(\hat{\mathcal{L}}\) given by \(\hat{\mathcal{L}}(x,y)=\mathcal{L}(x,\varphi^{-1}(y))\).

Now we distinguish two cases. In line with the notation from before we denote by \(\mathfrak{n}_{K}\in\mathcal{O}\) and \(o_{K}\in\mathbb{R}\) the unique values for which \(A_{K}=\{x\in\mathbb{R}^{d_{\operatorname{in}}}\colon\mathfrak{n}_{K}\cdot x >o_{K}\}\). First suppose that \(\delta_{K}\) and \(\mathfrak{n}_{K}\) are linearly independent. We choose a basis \(b_{1},\ldots,b_{d_{\operatorname{in}}}\) of \(\mathbb{R}^{d_{\operatorname{in}}}\) such that \(b_{1}\cdot\mathfrak{n}_{K}=1\), \(b_{1}\perp\delta_{K}\), and for \(\forall\,l\in(\mathbb{N}\cap[1,d_{\operatorname{in}}])\backslash\{1\}\colon b _{l}\perp\mathfrak{n}_{K}\). This can be achieved by first choosing an arbitrary basis \(b_{2},\ldots,b_{d_{\operatorname{in}}}\) of the space of vectors being orthogonal to \(\mathfrak{n}_{K}\), secondly choosing a vector \(b_{1}^{\prime}\) that is orthogonal to \(\delta_{K}\) but not to \(\mathfrak{n}_{K}\) which is possible since \(\delta_{K}\) and \(\mathfrak{n}_{K}\) are linearly independent and finally letting \(b_{1}=b_{1}^{\prime}/(b_{1}^{\prime}\cdot\mathfrak{n}_{K})\).

We denote by \(B\) the matrix \((b_{1},\ldots,b_{d_{\operatorname{in}}})\) consisting of the basis vectors and choose \(c\in\mathbb{R}^{d_{\operatorname{in}}}\) so that

\[(B^{\dagger}\mathfrak{n}_{K})\cdot c=o_{K}\ \ \text{and}\ \ (B^{\dagger} \delta_{K})\cdot c=-\mathfrak{b}_{K}. \tag{71}\]

The latter is feasible since the expression on the left hand side only depends on the choice of \(c_{1}\) and the expression on the right hand side only on the coordinates \(c_{2},\ldots,c_{d_{\operatorname{in}}}\).

As is straight-forward to verify the respective response \(\hat{\mathcal{R}}\) has as \(K\)th active area \(\hat{A}_{K}=\{x\in\mathbb{R}^{d_{\operatorname{in}}}\colon x_{1}>0\}\) and on \(\hat{A}_{K}\) the \(K\)th summand in the respective representation of \(\hat{\mathcal{R}}\) is

\[\delta_{K}\cdot\varphi(x)+\mathfrak{b}_{K}=(\underbrace{B^{\dagger}\delta_{K}} _{=\colon\hat{A}_{K}})\cdot x. \tag{72}\]
Altogether the previous computations show that we can assume without loss of generality that the considered strict generalized response has as active area \(A_{K}=\hat{A}_{K}\) with \(\delta_{K}\) being perpendicular to the first unit vector and \(\mathfrak{b}_{K}\) being zero.

We compare the performance of the response \(\mathcal{R}\) with the \(\kappa\)-indexed family of generalized responses (\(\mathcal{R}^{\kappa}\colon\kappa\geq 1\)) of dimension \(d\) or smaller given by

\[\mathcal{R}^{\kappa}(x)=\mathfrak{a}(x)+\sum_{k=1}^{K-1}\mathds{1}_{A_{k}}(x) \big{(}\delta_{k}\cdot x+\mathfrak{b}_{k}\big{)}+\tilde{\mathcal{R}}^{\kappa}( x), \tag{73}\]

where

\[\tilde{\mathcal{R}}^{\kappa}(x)=\tfrac{1}{2}(\delta_{K}\cdot x+\kappa x_{1})^ {+}-\tfrac{1}{2}(-\delta_{K}\cdot x+\kappa x_{1})^{+}. \tag{74}\]

Let

\[\tilde{\mathcal{L}}(x,y)=\mathcal{L}\big{(}x,\mathfrak{a}(x)+\sum_{k=1}^{K-1 }(\delta_{k}\cdot x+\mathfrak{b}_{k})\mathds{1}_{A_{k}}(x)+y\big{)},\qquad \tilde{\mathcal{R}}(x)=\mathds{1}_{A_{K}}(x)(\delta_{K}\cdot x), \tag{75}\]

\(\delta_{K}^{\prime}=(\delta_{K,2},\ldots,\delta_{K,d_{\mathrm{in}}})^{\dagger}\) and similarly \(x^{\prime}=(x_{2},\ldots,x_{d_{\mathrm{in}}})^{\dagger}\) and note that

\[\tilde{\mathcal{R}}^{\kappa}(x)=\begin{cases}\tilde{\mathcal{R}}(x),&\text{ if }\kappa|x_{1}|\geq|\delta_{K}^{\prime}\cdot x^{\prime}|,\\ \text{lin. interpol. of }\tilde{\mathcal{R}}\text{ between }\tilde{x}^{(\kappa)} \text{ and }\hat{x}^{(\kappa)},&\text{ otherwise},\end{cases} \tag{76}\]

where

\[\tilde{x}^{(\kappa)}=\begin{pmatrix}-\tfrac{1}{\kappa}|\delta_{K}^{\prime} \cdot x^{\prime}|\\ x^{\prime}\end{pmatrix}\ \ \text{and }\ \ \hat{x}^{(\kappa)}=\begin{pmatrix}\tfrac{1}{ \kappa}|\delta_{K}^{\prime}\cdot x^{\prime}|\\ x^{\prime}\end{pmatrix}. \tag{77}\]

For \(\varepsilon>0\) denote by \(\mathcal{D}_{\varepsilon}\) the set

\[\mathcal{D}_{\varepsilon}=\left\{x^{\prime}\in\mathbb{R}^{d_{ \mathrm{in}}-1}\colon\text{ the segment }\Big{[}\begin{pmatrix}-\varepsilon\\ x^{\prime}\end{pmatrix},\begin{pmatrix}\varepsilon\\ x^{\prime}\end{pmatrix}\Big{]}\text{ has distance greater or equal to }\varepsilon\text{ to every }\partial A_{k}\ (k\neq K),\right.\] \[\text{ and there exists }x_{0}\in\mathbb{R}\text{ so that }(x_{0},x^{\prime})\in\mathbb{D}\right\}\]

It is straight-forward to verify that \(\mathcal{D}_{\varepsilon}\) is closed and due to the compactness of \(\mathbb{D}\) also compact. Moreover,

\[\bigcup_{\varepsilon>0}\mathcal{D}_{\varepsilon}=\underbrace{\left\{x^{\prime} \in\mathbb{R}^{d_{\mathrm{in}}-1}\colon\big{[}\exists x_{0}\in\mathbb{R}:(x_{0 },x^{\prime})\in\mathbb{D}\big{]}\text{ and }\big{[}\forall k\neq K:(0,x^{\prime})\not\in \partial A_{k}\big{]}\right\}}_{=\colon\mathcal{D}}. \tag{78}\]

We have

\[\begin{split}&\int_{\mathbb{D}}\mathcal{L}(x,\mathcal{R}^{\kappa} (x))\,h(x)\,dx-\int_{\mathbb{D}}\mathcal{L}(x,\mathcal{R}(x))\,h(x)\,dx\\ &\qquad=\int_{\mathbb{D}}\tilde{\mathcal{L}}(x,\tilde{\mathcal{R} }^{\kappa}(x))\,h(x)\,dx-\int_{\mathbb{D}}\tilde{\mathcal{L}}(x,\tilde{ \mathcal{R}}(x))\,h(x)\,dx\\ &\qquad=\int_{D_{\kappa}}\bigl{(}\tilde{\mathcal{L}}(x,\tilde{ \mathcal{R}}^{\kappa}(x))-\tilde{\mathcal{L}}(x,\tilde{\mathcal{R}}(x))\bigr{)} \,h(x)\,dx,\end{split} \tag{79}\]

where \(D_{\kappa}=\left\{\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\in\mathbb{R}^{d_{\mathrm{in}}}\colon x^{\prime}\in \mathcal{D},|x_{1}|\leq\tfrac{1}{\kappa}|\delta_{K}^{\prime}\cdot x^{\prime}|\right\}\). In dependence on \(\varepsilon>0\) we partition \(D_{\kappa}\) in two sets

\[D_{\kappa,\varepsilon}^{\prime}=\Bigl{\{}\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\in D_{\kappa}:x^{\prime}\in\mathcal{D}_{ \varepsilon}\Bigr{\}}\qquad\text{and}\qquad D_{\kappa,\varepsilon}^{\prime \prime}=\Bigl{\{}\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\in D_{\kappa}:x^{\prime}\in\mathcal{D}\backslash \mathcal{D}_{\varepsilon}\Bigr{\}}. \tag{80}\]

We note that \(\tilde{\mathcal{L}}\) is continuous on \(([-\varepsilon,\varepsilon]\times\mathcal{D}_{\varepsilon})\times\mathbb{R}^{d_{ \mathrm{in}}}\) as consequence of the continuity of \(\mathcal{L}\) (see Lemma 3.1) and the particular choice of \(\mathcal{D}_{\varepsilon}\). Moreover, for sufficiently large \(\kappa\) (depending on
\(\varepsilon\)), \(D^{\prime}_{\kappa,\varepsilon}\subseteq[-\varepsilon,\varepsilon]\times\mathcal{D }_{\varepsilon}\). Using that continuous functions are uniformly continuous on compacts, that \(h\) is uniformly bounded and the Lebesgue measure of \(D_{\kappa}\) is of order \(\mathcal{O}(1/\kappa)\) we conclude that in terms of \(\Upsilon_{x^{\prime}}:=|\delta^{\prime}_{K}\cdot x^{\prime}|\)

\[\int_{D^{\prime}_{\kappa,\varepsilon}}\bigl{(}\tilde{\mathcal{L}} (x,\tilde{\mathcal{R}}^{\kappa}(x))-\tilde{\mathcal{L}}(x,\tilde{\mathcal{R}} (x))\bigr{)}\,h(x)\,dx\] \[=\int_{D_{\kappa,\varepsilon}}\bigl{(}\tilde{\mathcal{L}} \Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{\kappa}(x)\Bigr{)}-\tilde{ \mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}(x)\Bigr{)}\Bigr{)}\,h\Bigl{(} \begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx+o\bigl{(}\tfrac{1}{\kappa}\bigr{)}\] \[=\int_{\mathcal{D}_{\varepsilon}}\Bigl{(}\int_{-\frac{1}{\kappa} \Upsilon_{x^{\prime}}}^{\frac{1}{\kappa}\Upsilon_{x^{\prime}}}\Bigl{(}\tilde{ \mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{\kappa}\Bigl{(}\begin{pmatrix}x_{ 1}\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)}-\tilde{\mathcal{L}}\Bigl{(}\begin{pmatrix} 0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}\Bigl{(}\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)}\Bigr{)}\,dx_{1}\Bigr{)}\,h\Bigl{(} \begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}+o\bigl{(}\tfrac{1}{\kappa}\bigr{)} \tag{81}\]

as \(\kappa\to\infty\). Moreover, for every \(x^{\prime}\in\mathcal{D}_{\varepsilon}\) one has

\[\int_{-\frac{1}{\kappa}\Upsilon_{x^{\prime}}}^{\frac{1}{\kappa} \Upsilon_{x^{\prime}}}\Bigl{(}\tilde{\mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{\kappa}\Bigl{(}\begin{pmatrix}x_{ 1}\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)}-\tilde{\mathcal{L}}\Bigl{(}\begin{pmatrix} 0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}\Bigl{(}\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)}\Bigr{)}\,dx_{1}=:\frac{1}{\kappa}\,Q(x^{ \prime}). \tag{82}\]

We represent \(Q(x^{\prime})\) in terms of the measure \(\nu_{x^{\prime}}=\operatorname{Leb}|_{[-\Upsilon_{x^{\prime}},\Upsilon_{x^{ \prime}}]}\), the strictly convex function

\[\xi_{x^{\prime}}\colon[-\Upsilon_{x^{\prime}},\Upsilon_{x^{\prime}}]\to[0, \infty),\ x_{1}\mapsto\tilde{\mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{1}\Bigl{(}\begin{pmatrix}x_{1}\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)} \tag{83}\]

and its secant \(\bar{\xi}_{x^{\prime}}\colon[-\Upsilon_{x^{\prime}},\Upsilon_{x^{\prime}}]\to[0,\infty)\) that equals \(\xi_{x^{\prime}}\) in the boundary points and is linear in between. One has

\[Q(x^{\prime})=\int\bigl{(}\xi_{x^{\prime}}(z)-\bar{\xi}_{x^{\prime}}(z)\bigr{)} \,d\nu_{x^{\prime}}\leq 0 \tag{84}\]

with strict inequality in the case where \(\Upsilon_{x^{\prime}}>0\) (due to strict convexity). Consequently, we get with (81) that

\[\lim_{\kappa\to\infty}\kappa\int_{D^{\prime}_{\kappa,\varepsilon}}\bigl{(} \tilde{\mathcal{L}}(x,\tilde{\mathcal{R}}^{\kappa}(x))-\tilde{\mathcal{L}}(x, \tilde{\mathcal{R}}(x))\bigr{)}\,h(x)\,dx=\int_{\mathcal{D}_{\varepsilon}}Q(x^ {\prime})\,h\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}. \tag{85}\]

To analyze the contribution of the integrals on \(D^{\prime\prime}_{\kappa,\varepsilon}\) we note that by uniform boundedness of \(\tilde{\mathcal{L}}(x,\tilde{\mathcal{R}}^{\kappa}(x))-\tilde{\mathcal{L}}(x, \tilde{\mathcal{R}}(x))\) over all \(x\in\mathbb{D}\) and \(\kappa\geq 1\) one has existence of a constant \(C\) not depending on \(\kappa\) and \(\varepsilon\) such that

\[\Bigl{|}\int_{D^{\prime\prime}_{\kappa,\varepsilon}}\bigl{(}\tilde{\mathcal{L}} (x,\tilde{\mathcal{R}}^{\kappa}(x))-\tilde{\mathcal{L}}(x,\tilde{\mathcal{R}}(x ))\bigr{)}\,h(x)\,dx\Bigr{|}\leq C\,|\mathcal{D}\backslash\mathcal{D}_{ \varepsilon}|\,\tfrac{1}{\kappa}, \tag{86}\]

where \(|\mathcal{D}\backslash\mathcal{D}_{\varepsilon}|\) is the \((d_{\mathrm{in}}-1)\)-dimensional Hausdorff measure of the set \(\mathcal{D}\backslash\mathcal{D}_{\varepsilon}\). By choosing \(\varepsilon>0\) arbitrarily small one can make \(|\mathcal{D}\backslash\mathcal{D}_{\varepsilon}|\) arbitrarily small and with a diagonalization argument we obtain with (79) and (85) that

\[\lim_{\kappa\to\infty}\kappa\int_{\mathbb{D}}\bigl{(}\mathcal{L}(x,\mathcal{R}^ {\kappa}(x))-\mathcal{L}(x,\mathcal{R}(x))\bigr{)}\,h(x)\,dx=\int_{\mathcal{D}}Q (x^{\prime})\,h\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}. \tag{87}\]

Now there exists \(x^{\prime}\in\mathcal{D}\) with \(h(x^{\prime})>0\) and by continuity of \(h\) we can choose \(x^{\prime}\) such that, additionally, \(\Upsilon_{x^{\prime}}=|\delta^{\prime}_{K}\cdot x^{\prime}|>0\). By continuity we thus get that

\[\int_{\mathcal{D}}Q(x^{\prime})\,h\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}<0 \tag{88}\]
and, consequently, there exists \(\kappa>0\) such that the generalized response \(\mathcal{R}^{\kappa}\) of dimension \(d\) or smaller has a strictly smaller error than \(\mathcal{R}\). Hence, it has to be simple.

It remains to treat the case where \(\delta_{K}\) and \(\mathfrak{n}_{K}\) are linearly dependent. In that case we choose \(\alpha\in\mathbb{R}\) with \(\delta_{K}=\alpha\mathfrak{n}_{K}\), we extend \(b_{1}=\mathfrak{n}_{K}\) to an orthonormal basis \((b_{1},\ldots,b_{d_{\mathrm{in}}})\) of \(\mathbb{R}^{d_{\mathrm{in}}}\) and denote by \(B\) the matrix formed by the vectors \(b_{1},\ldots,b_{d_{\mathrm{in}}}\). Moreover, choose \(c=(o_{K},0,\ldots,0)^{\dagger}\) and set \(\varphi(x)=B(x+c)\). Then the response \(\hat{\mathcal{R}}=\mathcal{R}\circ\varphi\) has as \(K\)th activation area \(\hat{A}_{K}=\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon x_{1}>0\}\) and on \(\hat{A}_{K}\) the \(K\)th summand in the respective representation of \(\hat{\mathcal{R}}\) is

\[\delta_{K}\cdot\varphi(x)+\mathfrak{b}_{K}=\alpha(x_{1}+o_{K})+\mathfrak{b}_{ K}=\alpha x_{1}+\hat{\mathfrak{b}}_{K}, \tag{89}\]

where \(\hat{\mathfrak{b}}_{K}=\alpha o_{K}+\mathfrak{b}_{K}\neq 0\) since otherwise we would have that

\[\partial A_{K}\subseteq\{x\in\mathbb{R}^{d_{\mathrm{in}}}\colon\delta_{K} \cdot x+\mathfrak{b}_{K}=0\}. \tag{90}\]

We showed that in the remaining case we can assume without loss of generality that \(A_{K}=\hat{A}_{K}\), \(\delta_{K}=(\alpha,0,\ldots,0)^{\dagger}\) for an \(\alpha\in\mathbb{R}\) and \(\mathfrak{b}_{K}\neq 0\).

In analogy to above we compare the response \(\mathcal{R}\) with the \(\kappa\)-indexed family of responses (\(\mathcal{R}^{\kappa}\colon\kappa\geq 1\)) given by

\[\mathcal{R}^{\kappa}(x)=\mathfrak{a}(x)+\sum_{k=1}^{K-1}\bigl{(}\delta_{k}\cdot x +\mathfrak{b}_{k}\bigr{)}\mathds{1}_{A_{k}}(x)+\tilde{\mathcal{R}}^{\kappa}( x), \tag{91}\]

where

\[\tilde{\mathcal{R}}^{\kappa}(x)=\tfrac{1}{2}(\alpha+\mathfrak{b}_{K}\kappa) \bigl{(}x_{1}+\tfrac{1}{\kappa}\bigr{)}^{+}+\tfrac{1}{2}(\alpha-\mathfrak{b}_ {K}\kappa)\bigl{(}x_{1}-\tfrac{1}{\kappa}\bigr{)}^{+}. \tag{92}\]

We use \(\tilde{\mathcal{L}}\) and \(\tilde{\mathcal{R}}\) as before, see (75), and note that \(\tilde{\mathcal{R}}^{\kappa}\) agrees with \(\tilde{\mathcal{R}}\) for all \(x\in\mathbb{R}^{d_{\mathrm{in}}}\) with \(|x_{1}|\geq\tfrac{1}{\kappa}\). In analogy to above, we conclude that

\[\begin{split}&\int_{\mathbb{D}}\bigl{(}\mathcal{L}(x,\mathcal{R}^{ \kappa}(x))-\mathcal{L}(x,\mathcal{R}(x))\bigr{)}\,h(x)\,dx\\ &\qquad=\int_{D_{\kappa}}\Bigl{(}\tilde{\mathcal{L}}\Bigl{(} \begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{\kappa}(x)\Bigr{)}-\tilde{ \mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}(x)\Bigr{)}\,h\Bigl{(}\begin{pmatrix} 0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx+o\bigl{(}\tfrac{1}{\kappa}\bigr{)},\end{split} \tag{93}\]

where \(D_{\kappa}=[-\tfrac{1}{\kappa},\tfrac{1}{\kappa}]\times\mathcal{D}\). As above we split the domain of integration into the two sets \(D^{\prime}_{\kappa,\varepsilon}\) and \(D^{\prime\prime}_{\kappa,\varepsilon}\). Now in terms of

\[\xi_{x^{\prime}}\colon[-1,1]\to[0,\infty),\qquad x_{1}\mapsto\tilde{\mathcal{ L}}\biggl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tfrac{1}{2}(x_{1}+1)\mathfrak{b}_{K}\biggr{)} \tag{94}\]

we get by using the uniform continuity of \(\tilde{\mathcal{L}}\) on \(D^{\prime}_{\kappa,\varepsilon}\) and the fact that \(|D_{\kappa,\varepsilon}|=\mathcal{O}(\tfrac{1}{\kappa})\) as \(\kappa\to\infty\) that

\[\begin{split}&\int_{D^{\prime}_{\kappa,\varepsilon}}\Bigl{(} \tilde{\mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}^{\kappa}(x)\Bigr{)}-\tilde{ \mathcal{L}}\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix},\tilde{\mathcal{R}}(x)\Bigr{)}\Bigr{)}\,h\Bigl{(} \begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\Bigr{)}\,dx\\ &\qquad=\frac{1}{\kappa}\int_{\mathcal{D}_{\varepsilon}}\int_{-1}^{1} \xi_{x^{\prime}}(x_{1})\,dx_{1}-\bigl{(}\xi_{x^{\prime}}(-1)+\xi_{x^{\prime} }(1)\bigr{)}\,h\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}+o\bigl{(}\tfrac{1}{\kappa}\bigr{)}. \end{split} \tag{95}\]

By strict convexity of \(\xi_{x^{\prime}}\), we get that \(Q(x^{\prime}):=\int_{-1}^{1}\xi_{x^{\prime}}(x_{1})\,dx_{1}-\bigl{(}\xi_{x^{ \prime}}(-1)+\xi_{x^{\prime}}(1)\bigr{)}<0\). With the same arguments as in the first case one obtains that

\[\lim_{\kappa\to\infty}\kappa\int_{\mathbb{D}}\bigl{(}\mathcal{L}(x,\mathcal{R}^ {\kappa}(x))-\mathcal{L}(x,\mathcal{R}(x))\bigr{)}\,h(x)\,dx=\int_{\mathcal{D}}Q (x^{\prime})\,h\Bigl{(}\begin{pmatrix}0\\ x^{\prime}\end{pmatrix}\Bigr{)}\,dx^{\prime}<0 \tag{96}\]

so that there exists a response \(\mathcal{R}^{\kappa}\) with strictly smaller error than \(\mathcal{R}\) and the proof is finished.
**Example 3.4**.: _If there exists a hyperplane \(H\) with \(h(x)=0\) for all \(x\in H\) such that \(H\) intersects the convex hull of \(\operatorname{supp}(\mu)\) the conclusion of Theorem 1.3 is in general not true. Consider a continuous function \(f\colon\mathbb{R}^{2}\to\mathbb{R}\) that satisfies \(f(x)=1\) for all \(x\in B((0,1),1)\) and \(f(x)=0\) for all \(x\in B((1,-1),1)\cup B((-1,-1),1)\). Now, let \(\mathcal{L}(x,y)=(f(x)-y)^{2}\) and \(\mu\) be the measure on \(\mathbb{R}^{2}\) with continuous Lebesgue density_

\[h(x)=\operatorname{\mathds{1}\hskip-2.845276pt\rule{0.0pt}{1.5pt}}_{B((0,1), 1)}(x)\,|x-(0,1)|+\operatorname{\mathds{1}\hskip-2.845276pt\rule{0.0pt}{1.5pt}}_ {B((1,-1),1)}(x)\,|x-(1,-1)|+\operatorname{\mathds{1}\hskip-2.845276pt\rule{0.0 pt}{1.5pt}}_{B((-1,-1),1)}(x)\,|x+(1,1)|. \tag{97}\]

_Then, we have \(h(x)\equiv 0\) on \(\mathbb{R}\times\{0\}\). Note that_

\[\mathcal{R}(x)=\begin{cases}0,&\text{ if }x_{2}\leq 0\\ 1,&\text{ if }x_{2}>0\end{cases} \tag{98}\]

_is a strict generalized response of dimension \(2\) with \(\int\mathcal{L}(x,\mathcal{R}(x))\,\mathrm{d}\mu(x)=0\). Conversely, there does not exist a network \(\mathbb{W}\in\mathcal{W}_{d}\) (for arbitrary \(d\in\mathbb{N}\)) with \(\int\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))\,\mathrm{d}\mu(x)=0\). In particular, assume that there exist \(d\in\mathbb{N}\) and \(\mathbb{W}\in\mathcal{W}_{d}\) with \(\operatorname{err}^{\mathcal{L}}(\mathbb{W})=0\). Then, every breakline \(H\neq\mathbb{R}\times\{0\}\) that intersects the interior of \(D\) contains an uncountable set of points \(x\in H\) with \(h(x)>0\) and for all such points the function \(f\) is constant in a neighborhood of \(x\). Therefore, the collective response of the neurons with breakline \(H\) has to be constant and we can without loss of generality assume that all ReLU neurons have the breakline \(H=\mathbb{R}\times\{0\}\). Moreover, since \(\operatorname{err}^{\mathcal{L}}(\mathbb{W})=0\) it holds that \(\mathfrak{N}^{\mathbb{W}}(x)=0\) for all \(x=(x_{1},x_{2})\in\mathbb{R}^{2}\) with \(x_{2}<0\) and \(\mathfrak{N}^{\mathbb{W}}(x)=1\) for all \(x=(x_{1},x_{2})\in\mathbb{R}^{2}\) with \(x_{2}>0\). This contradicts the continuity of \(\mathfrak{N}^{\mathbb{W}}\)._

_Thus, there does not exist a global minimum in the loss landscape \(\mathcal{W}_{d}\ni\mathbb{W}\to\operatorname{err}^{\mathcal{L}}(\mathbb{W})\) (for arbitrary \(d\in\mathbb{N}\)) and, in order to solve the minimization task iteratively, the sequence of networks returned by a gradient based algorithm have unbounded parameters._

_For a thorough investigation of the non-existence of global minima in the approximation of discontinuous target functions \(f\), see [1]._

Next, we show that in many situations if the class of network responses is not able to produce the function \(x\mapsto\mathbf{m}(x)\) defined in Lemma 3.1 the minimal error strictly decreases after adding a ReLU neuron to the network structure.

**Proposition 3.5**.: _Let \(d\in\mathbb{N}_{0}\), assume that \(\mathbb{D}=\operatorname{supp}(\mu)\) is a compact set, assume there exists \(\mathbb{W}\in\mathcal{W}_{d}\) with \(\operatorname{err}^{\mathcal{L}}(\mathbb{W})=\operatorname{err}^{\mathcal{L}} _{d}<\infty\), assume for every \(x\in\mathbb{D}\) that the function \(\mathcal{L}(x,\cdot)\) is convex,_

1. _assume for every compact_ \(K\subseteq\mathbb{R}\) _there exists_ \(L\in\mathbb{R}\) _such that for all_ \(x\in\mathbb{D}\)_,_ \(y,y^{\prime}\in K\) _that_ (99) \[|\mathcal{L}(x,y)-\mathcal{L}(x,y^{\prime})|\leq L|y-y^{\prime}|,\]

Figure 2. Visualization of the minimization task in Example 3.4. There exists a generalized response of dimension \(2\) but no neural network \(\mathbb{W}\in\mathcal{W}_{d}\) (\(d\in\mathbb{N}\)) attaining zero error.
_
* _assume for every affine function_ \(\varphi\colon\mathbb{R}^{d_{\mathrm{in}}}\to\mathbb{R}\) _that the set_ (100) \[\big{\{}x\in\mathbb{D}\colon\mathcal{L}\text{ is not $y$-differentiable in }(x,\varphi(x))\big{\}}\] _is a Lebesgue nullset,_

_and assume that there exist no neural network \(\mathbb{W}\in\mathcal{W}_{d}\) satisfying for \(\mu\)-almost all \(x\in\mathbb{D}\) that_

\[\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))=\inf_{y\in\mathbb{R}}\mathcal{L}(x,y). \tag{101}\]

_Then_

\[\mathrm{err}_{d}^{\mathcal{L}}>\mathrm{err}_{d+1}^{\mathcal{L}}. \tag{102}\]

**Remark 3.6**.: _We compare the assumptions of Proposition 3.5 with those of Theorem 1.3. In Proposition 3.5 we explicitly assume the existence of an optimal network \(\mathbb{W}\in\mathcal{W}_{d}\) and relax the continuity assumption on \(\mathcal{L}\) in the first component and the strict convexity assumption on \(\mathcal{L}\) in the second component. On the other hand, we introduce assumptions on the smoothness of \(\mathcal{L}\) in the second component. Under the assumptions of Theorem 1.3, condition (i) of Proposition 3.5 is satisfied (cf. [10, Thm. 10.6]) and we can apply the latter proposition if additionally condition (ii) is satisfied. In that case, the statement of Proposition 3.5 can be rewritten as follows: if \(\mathrm{err}_{d}^{\mathcal{L}}\leq\mathrm{err}_{d+1}^{\mathcal{L}}\), then there exists a neural network \(\mathbb{W}\in\mathcal{W}_{d}\) with \(\mathfrak{N}^{\mathbb{W}}(x)=\mathfrak{m}(x)\) for all \(x\in\mathbb{D}\)._

Proof of Proposition 3.5.: Let \(\mathbb{W}\in\mathcal{W}_{d}\) be a network with \(\mathrm{err}^{\mathcal{L}}(\mathbb{W})=\mathrm{err}_{d}^{\mathcal{L}}\). For \(\Delta,o\in\mathbb{R}\), \(\mathfrak{n}\in\mathcal{O}\) consider the function

\[N(\Delta,\mathfrak{n},o)(x)=\mathfrak{N}^{\mathbb{W}}(x)+\Delta(\mathfrak{n} \cdot x-o)^{+}. \tag{103}\]

If \(\mathrm{err}_{d}^{\mathcal{L}}\leq\mathrm{err}_{d+1}^{\mathcal{L}}\) then we have for all \(\Delta,o\in\mathbb{R}\), \(\mathfrak{n}\in\mathcal{O}\) that

\[\int_{\mathbb{D}}\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))\,h(x)\,dx\leq \int_{\mathbb{D}}\mathcal{L}(x,N(\Delta,\mathfrak{n},b)(x))\,h(x)\,dx \tag{104}\]

and taking the derivative with respect to \(\Delta\) at \(\Delta=0\) yields

\[\int_{\mathbb{D}}\Bigl{(}\frac{\partial}{dy}\mathcal{L}(x,\mathfrak{N}^{ \mathbb{W}}(x))\Bigr{)}(\mathfrak{n}\cdot x-o)^{+}\,h(x)\,dx=0, \tag{105}\]

where \(x\mapsto\frac{\partial}{dy}\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))\) is uniformly bounded and well-defined outside a Lebesgue nullset. Indeed, since \(\mathfrak{N}^{\mathbb{W}}\) is piecewise affine there exists a finite number of affine functions \(\varphi_{1},\ldots,\varphi_{m}\) such that the set of points \(x\in\mathbb{D}\) for which \(\mathcal{L}\) is not \(y\)-differential in \((x,\mathfrak{N}^{\mathbb{W}}(x))\) is contained in

\[\bigcup_{i=1}^{m}\{x\in\mathbb{D}\colon\mathcal{L}\text{ is not $y$-differentiable in }(x,\varphi_{i}(x))\}, \tag{106}\]

which by (ii) is a nullset. Moreover, the boundedness of the derivative follows from the Lipschitz continuity of \(\mathcal{L}\) in the second argument, see (i). We let

\[\tilde{h}(x):=\Bigl{(}\frac{\partial}{\partial y}\mathcal{L}(x,\mathfrak{N}^{ \mathbb{W}}(x))\Bigr{)}h(x) \tag{107}\]

and note that the space \(\mathcal{H}\) of all continuous functions \(g\colon\mathbb{D}\to\mathbb{C}\) satisfying

\[\int_{\mathbb{D}}g(x)\,\tilde{h}(x)\,dx=0 \tag{108}\]

is linear and closed under convergence in \(C(\mathbb{D},\mathbb{C})\) (endowed with supremum norm). We showed that \(\mathcal{H}\) contains all functions of the form \(x\mapsto(\mathfrak{n}\cdot x-o)^{+}\) and it is standard to deduce that \(\mathcal{H}\) contains all polynomials and, using the Stone-Weierstrass theorem, thus all continuous functions. By the Riesz-Markov-Kakutani representation theorem, the measure \(\tilde{h}(x)\,dx\) is
the zero-measure and \(\tilde{h}\) is zero except for \(\mu\)-nullsets. Note that \(h>0\), \(\mu\)-almost everywhere, and hence \(\frac{\partial}{\partial y}\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))=0\), \(\mu\)-almost everywhere. Using the convexity of \(y\mapsto\mathcal{L}(x,y)\) we get \(\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))=\inf_{y\in\mathbb{R}}\mathcal{L}( x,y)\), \(\mu\)-almost everywhere. 

In the next example, we show that the conclusion of Proposition 3.5 is in general false if condition (ii) is not satisfied. We note that the loss function \(\mathcal{L}\) in the example is not strictly convex but the statements of Lemma 3.1 still hold in this case.

**Example 3.7**.: _Consider the following regression problem. Let \(d_{\rm in}=1\), \(\ell\geq 1\), \(\mu={\rm Leb}|_{[-\ell-1,1+\ell]}\) be the Lebesgue measure on the interval \([-\ell-1,1+\ell]\) and \(\mathcal{L}(x,y)=|y-f(x)|\) where_

\[f(x)=\begin{cases}1-|x|,&\text{ if }|x|\leq 1\\ 0,&\text{ if }|x|>1\end{cases}. \tag{109}\]

_Note that \(\lambda(\{x\in[-\ell-1,\ell+1]:\mathcal{L}\text{ is not }y\text{-differentiable in }(x,0)\})=2\ell>0\) and_

\[f(x)=(x+1)^{+}-2(x)^{+}+(x-1)^{+} \tag{110}\]

_so that \(f\) is the response of a network using three ReLU neurons and \({\rm err}_{3}^{\mathcal{L}}=0\). We show that for \(\ell\geq 13\) we have_

\[{\rm err}_{0}^{\mathcal{L}}={\rm err}_{1}^{\mathcal{L}}={\rm err}_{2}^{ \mathcal{L}}=\int_{-\ell-1}^{\ell+1}|f(x)|\,dx=1, \tag{111}\]

_i.e. the best regression function in the set of response functions for networks having \(2\) ReLU neurons is the zero function although \(\inf_{y\in\mathbb{R}}\mathcal{L}(x,y)=f(x)\) is not the zero function. This shows that the conclusion of Proposition 3.5 is in general false if Assumption (ii) is not satisfied._

_Denote by \(E\colon\mathcal{W}_{2}\times\mathbb{I}\to\mathbb{R}\) the function_

\[(\mathbb{W},[s,t])\mapsto E(\mathbb{W},[s,t])=\int_{s}^{t}|\mathfrak{N}^{ \mathbb{W}}(x)-f(x)|-|f(x)|\,dx, \tag{112}\]

_where \(\mathbb{I}\) is the set of all closed intervals that are subsets of \([-\ell-1,\ell+1]\). We show that for \(\ell\geq 13\) we have \(E(\mathbb{W},[-\ell-1,1+\ell])\geq 0\) for all networks \(\mathbb{W}\in\mathcal{W}_{2}\)._

_Let \(\mathbb{W}\in\mathcal{W}_{2}\). Then \(\mathfrak{N}^{\mathbb{W}}\) is continuous and satisfies_

\[\mathfrak{N}^{\mathbb{W}}(x)=\begin{cases}\delta_{1}x+\mathfrak{b}_{1},&\text{ if }x\leq o_{1}\\ \delta_{2}x+\mathfrak{b}_{2},&\text{ if }o_{1}<x<o_{2}\\ \delta_{3}x+\mathfrak{b}_{3},&\text{ if }x>o_{2}\end{cases}, \tag{113}\]

_for \(\delta_{1},\delta_{2},\delta_{3},\mathfrak{b}_{1},\mathfrak{b}_{2},\mathfrak{ b}_{3},o_{1},o_{2}\in\mathbb{R}\) with \(o_{1}\leq o_{2}\)._

Figure 3. Visualization of the minimization task in Example 3.7. There exists a network \(\mathbb{W}\in\mathcal{W}_{3}\) with \(\mathfrak{N}^{\mathbb{W}}=f\) (blue). However, the realization function attaining minimal error in the class \(\mathcal{W}_{2}\), \(\mathcal{W}_{1}\), and \(\mathcal{W}_{0}\) is \(\mathfrak{N}^{\mathbb{W}}=0\) (red).
_First, we assume that \(o_{1}\leq 0\leq o_{2}\) and \(0\leq\mathfrak{N}^{\mathbb{W}}(0)\). We fix \(\delta_{2},\mathfrak{b}_{2}\in\mathbb{R}\) and derive a lower bound for \(E(\mathbb{W},[0,\ell+1])\) over all feasible choices of \(o_{2},\delta_{3}\) and \(\mathfrak{b}_{3}\). We start with the case \(\delta_{2}<0\). Note that the choice \(o_{2}=\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\), \(\delta_{3}=\mathfrak{b}_{3}=0\) yields a better result than all networks with \(o_{2}\geq\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\). Therefore we can restrict the optimization task to networks satisfying \(o_{2}\leq\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\). For \(\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\leq 1\) we show that, indeed, the optimal choice for the approximation of \(f\) on the right-hand side of the \(y\)-axis is \(o_{2}=\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\), \(\delta_{3}=\mathfrak{b}_{3}=0\). If \(o_{2}<\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\) and \(\mathfrak{N}^{\mathbb{W}}(1)\leq 0\) then_

\[E(\mathbb{W},[o_{2},a])\geq-\frac{1}{2}\mathfrak{N}^{\mathbb{W}}(o_{2})(a-o_ {2})\qquad\text{and}\qquad E(\mathbb{W},[a,\ell+1])\geq\frac{1}{2}\mathfrak{ N}^{\mathbb{W}}(o_{2})(a-o_{2}), \tag{114}\]

_where \(a\in[0,1]\) with \(\mathfrak{N}^{\mathbb{W}}(a)=0\). Conversely, if \(\mathfrak{N}^{\mathbb{W}}(1)\geq 0\) then \(E(\mathbb{W},[o_{2},1])\geq-\mathfrak{N}^{\mathbb{W}}(o_{2})(1-o_{2})\geq- \mathfrak{N}^{\mathbb{W}}(o_{2})\). We consider two cases. If \(\mathfrak{N}^{\mathbb{W}}(\ell+1)\geq 0\) then \(\delta_{3}\geq-\mathfrak{N}^{\mathbb{W}}(o_{2})/\ell\). Thus, \(E(\mathbb{W},[1,\ell+1])\geq\frac{1}{2}\ell\mathfrak{N}^{\mathbb{W}}(1)\geq \frac{\ell-1}{2}\mathfrak{N}^{\mathbb{W}}(o_{2})\) so that for \(\ell\geq 3\) we get \(E(\mathbb{W},[o_{2},\ell+1])\geq 0\). If \(\mathfrak{N}^{\mathbb{W}}(\ell+1)\leq 0\) then \(\delta_{3}\leq-\mathfrak{N}^{\mathbb{W}}(o_{2})/(\ell+1)\). In that case, \(E(\mathbb{W},[1,\ell+1])\) corresponds to the area of two triangles with slope \(|\delta_{3}|\) and baselines that add to \(\ell\). This is minimized by two congruent triangles so that_

\[E(\mathbb{W},[1,\ell+1])\geq\Big{(}\frac{\ell}{2}\Big{)}^{2}|\delta_{3}|\geq \frac{\ell-1}{4}\mathfrak{N}^{\mathbb{W}}(o_{2}). \tag{115}\]

_Thus, for \(\ell\geq 5\) we get \(E(\mathbb{W},[o_{2},\ell+1])\geq 0\) and the optimal choice is \(o_{2}=\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\), \(\delta_{3}=\mathfrak{b}_{3}=0\)._

_It remains to consider the case \(\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\geq 1\). In this case the choice \(\delta_{3}>0\) is clearly suboptimal and for \(o_{2}\leq 1\) we get \(E(\mathbb{W},[0,o_{2}])\geq-(\mathfrak{N}^{\mathbb{W}}(0)-\frac{1}{2}|\delta _{2}|)\). The above calculations show that_

\[E(\mathbb{W},[o_{2},\ell+1])\geq\frac{\ell-5}{4}\mathfrak{N}^{\mathbb{W}}(o_{2 })\geq\frac{\ell-5}{4}(\mathfrak{N}^{\mathbb{W}}(0)-|\delta_{2}|) \tag{116}\]

_so that for \(\ell\geq 7\) we have \(E(\mathbb{W},[0,\ell+1])\geq-\frac{1}{2}\mathfrak{N}^{\mathbb{W}}(0)\). If \(\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\geq\ell+1\) the choice \(o_{2}>1\) is suboptimal and in the case \(\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\leq\ell+1\) where \(o_{2}\leq 1\) is not the optimal choice it is easy to see that actually \(o_{2}=\mathfrak{N}^{\mathbb{W}}(0)/|\delta_{2}|\), \(\delta_{3}=\mathfrak{b}_{3}=0\) is the best choice. In that case_

\[E(\mathbb{W},[0,1])\geq-(\mathfrak{N}^{\mathbb{W}}(0)-\frac{1}{2}|\delta_{2}| )\quad\text{ and }\quad E(\mathbb{W},[1,\ell+1])\geq\frac{1}{2}(\mathfrak{N}^{ \mathbb{W}}(0)-|\delta_{2}|). \tag{117}\]

_In conclusion, in all of the above cases we get for \(\ell\geq 7\) that \(E(\mathbb{W},[0,\ell+1])\geq-\frac{1}{2}\mathfrak{N}^{\mathbb{W}}(0)\)._

_Next, we consider the case \(\delta_{2}\geq 0\). Set \(a:=\inf\{x\geq 0\colon\delta_{2}x+\mathfrak{b}_{2}\geq 1-x\}\). Choosing \(o_{2}>a\) or \(\delta_{3}\geq 0\) is clearly suboptimal. For \(0\leq o_{2}\leq a\) and \(\delta_{3}<0\) note that \(E(\mathbb{W},[0,1])\geq-\mathfrak{N}^{\mathbb{W}}(o_{2})\). Analogously to the case \(\delta_{2}<0\) we get_

\[E(\mathbb{W},[1,\ell+1])\geq\frac{\ell-1}{4}\mathfrak{N}^{\mathbb{W}}(o_{2}) \tag{118}\]

_so that for \(\ell\geq 7\)_

\[E(\mathbb{W},[0,\ell+1])\geq\frac{1}{2}\mathfrak{N}^{\mathbb{W}}(o_{2})\geq \frac{1}{2}\mathfrak{N}^{\mathbb{W}}(0). \tag{119}\]

_Now, if \(\mathfrak{N}^{\mathbb{W}}(0)\leq 0\) and \(\ell\geq 5\) then the above calculations imply that \(E(\mathbb{W},[-\ell-1,\ell+1])\geq 0\). Using the symmetry of the problem we showed that for all networks \(\mathbb{W}\in\mathcal{W}_{2}\) satisfying \(o_{1}\leq 0\leq o_{2}\) we have that_

\[\int_{-\ell-1}^{\ell+1}\mathcal{L}(x,\mathfrak{N}^{\mathbb{W}}(x))\,\mathrm{d}x \geq\int_{-\ell-1}^{\ell+1}\mathcal{L}(x,0)\,\mathrm{d}x. \tag{120}\]
_Using again the symmetry, we are therefore left with considering the case \(0<o_{1}\leq o_{2}\). We can cleary focus on the case \(o_{1}\leq 1\) and \(\mathfrak{N}^{\mathbb{W}}(0)\geq 0\). Note that one can use the above arguments in order to show that_

\[E(\mathbb{W},[o_{1},\ell+1])\geq\min\bigl{(}0,-\frac{1}{2}\mathfrak{N}^{ \mathbb{W}}(o_{1})\bigr{)}. \tag{121}\]

_If \(\delta_{1}<0\) we thus get_

\[E(\mathbb{W},[0,\ell+1])\geq-\frac{3}{2}\mathfrak{N}^{\mathbb{W}}(0). \tag{122}\]

_On the other hand, \(E(\mathbb{W},[-\ell-1,0])\geq(\ell-1)\mathfrak{N}^{\mathbb{W}}(0)\) so that, for \(l\geq 5/2\), we have \(E(\mathbb{W},[-\ell-1,\ell+1])\geq 0\). Conversely, if \(\delta_{1}\geq 0\) we get_

\[E(\mathbb{W},[0,\ell+1])\geq-\frac{3}{2}\mathfrak{N}^{\mathbb{W}}(0)-\delta_{1} \tag{123}\]

_and for \(\delta_{1}\geq\mathfrak{N}^{\mathbb{W}}(0)\) and \(\ell\geq 9/2\) we clearly have \(E(\mathbb{W},[-\ell-1,\ell+1])\geq 0\). For \(\delta_{1}\leq\mathfrak{N}^{\mathbb{W}}(0)\) we get_

\[E(\mathbb{W},[-1,\ell+1])\geq-\frac{5}{2}\mathfrak{N}^{\mathbb{W}}(0)-\frac{1 }{2}\delta_{1}. \tag{124}\]

_Now for \(\delta_{1}\leq\mathfrak{N}^{\mathbb{W}}(0)/(\ell+1)\) we get_

\[E(\mathbb{W},[-\ell-1,-1])\geq\frac{1}{2}\ell\mathfrak{N}^{\mathbb{W}}(1)\geq \frac{1}{2}(\ell-1)\mathfrak{N}^{\mathbb{W}}(0) \tag{125}\]

_and for \(\ell\geq 7\) we get that \(E(\mathbb{W},[-\ell-1,\ell+1])\geq 0\). If \(\mathfrak{N}^{\mathbb{W}}(0)/(\ell+1)\leq\delta_{1}\leq\mathfrak{N}^{\mathbb{ W}}(0)\) then_

\[E(\mathbb{W},[-\ell-1,-1])\geq\Bigl{(}\frac{\ell}{2}\Bigr{)}^{2}|\delta_{1}| \geq\frac{\ell-1}{4}\mathfrak{N}^{\mathbb{W}}(0). \tag{126}\]

_Thus, if \(\ell\geq 13\) have \(E(\mathbb{W},[-\ell-1,\ell+1])\geq 0\) and the proof of the assertion is finished._

_Acknowledgements._ We gratefully acknowledge the Cluster of Excellence EXC 2044-390685587, Mathematics Munster: Dynamics-Geometry-Structure funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation). This work has been partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - SFB 1283/2 2021 - 317210226.
