Pointwise convergence of Fourier series and deep neural network for the
  indicator function of d-dimensional ball
In this paper, we clarify the crucial difference between a deep neural
network and the Fourier series. For the multiple Fourier series of
periodization of some radial functions on $\mathbb{R}^d$, Kuratsubo (2010)
investigated the behavior of the spherical partial sum and discovered the third
phenomenon other than the well-known Gibbs-Wilbraham and Pinsky phenomena. In
particular, the third one exhibits prevention of pointwise convergence. In
contrast to it, we give a specific deep neural network and prove pointwise
convergence.
Ryota Kawasumi, Tsuyoshi Yoneda
# Pointwise convergence theorem of generalized mini-batch gradient descent in deep neural network

###### Abstract.

The theoretical structure of deep neural network (DNN) has been clarified gradually. Imaizumi-Fukumizu (2019) and Suzuki (2019) clarified that the learning ability of DNN is superior to the previous theories when the target function is non-smooth functions. However, as far as the author is aware, none of the numerous works to date attempted to mathematically investigate what kind of DNN architectures really induce pointwise convergence of gradient descent (without any statistical argument), and this attempt seems to be closer to the practical DNNs. In this paper we restrict target functions to non-smooth indicator functions, and construct a deep neural network inducing pointwise convergence provided by mini-batch gradient descent process in ReLU-DNN.

Key words and phrases:deep neural network, ReLU function, gradient descent, pointwise convergence 2020 Mathematics Subject Classification: Primary 68T27; Secondary 68T07, Tertiary 41A29

## 1. Introduction

Recently, deep leaning has been the successful tool for various tasks of data analysis (see [10, 9, 18, 21] for example). Also, the theoretical structure of deep neural network (DNN) has been clarified gradually. In particular, Amari [1] gave a simple observation showing that any target function is in a sufficiently small neighborhood of any randomly connected DNN, with sufficiently large number of neurons in a layer (see also Kawaguchi-Huang-Kaelbling [14] and references therein). Keeping these celebrated results in mind, our next work would be clarifying the precise convergence structure of DNN even if initial data are already close to the target function. Imaizumi-Fukumizu [11] examined learning of non-smooth functions, which was not covered by the previous theory, and clarified that, compare the DNN with the previous theories (such as the kernel methods), the convergence rates are almost optimal for non-smooth functions, while some of the popular models do not attain this optimal rate. Suzuki [23] (also references therein) clarified that the learning ability of ReLU-DNN is superior to the linear method when the target function is in the supercritical Besov spaces \(B_{p,q}^{s}\) with \(p<2\) and \(s<d/2\) (\(d\) is the dimension, note that the case \(s=d/2\) is called "critical"), which indicates the spatial inhomogeneity of the shape of the target function including the non-smooth functions. Thus, with the aid of these results, we can conclude that ReLU-DNN is suitable for recognizing jump discontinuity of the non-smooth functions.

We now briefly explain the key idea of [23]. To show the approximation error theorems, he first applied the wavelet expansion to the target functions and then
approximated each wavelet bases (composed of spline functions) by ReLU-DNN (see [28]). More specifically, let \(g:[0,1]\to[0,1]\) be a tent function such that

\[g(x)=\begin{cases}2x\quad(x<1/2),\\ 2(1-x)\quad(x\geq 1/2)\end{cases}\]

and let \(g_{s}\) be a \(s\) times composite function, and \(f_{m}\) be a function approximating to the second order polynomial such that

\[g_{s}(x)=\underbrace{g\circ g\circ\cdots\circ g}_{s}(x)\quad\text{and}\quad f _{m}(x)=x-\sum_{s=1}^{m}\frac{g_{s}(x)}{2^{2s}}.\]

Note that \(f_{m}(x)\to x^{2}\) (\(m\to\infty\)) uniformly. For deriving the multi-dimensional polynomials, it suffices to apply the following formula:

\[xy=\frac{1}{2}((x+y)^{2}-x^{2}-y^{2}),\]

and then we can easily approximate multi-dimensional spline functions by ReLU-DNN. The other idea (for variance) is applying the statistical argument in [22] combined with a covering number evaluation.

However, as far as the author is aware, none of the numerous works to date attempted to mathematically investigate what kind of DNN architectures really induce pointwise convergence of gradient descent (without any statistical argument) even if the initial data are already close to the target function, and this attempt seems to be closer to the practical DNNs. In what follows, we investigate this problem.

Before going any further, we point out that employing supercritical function spaces may not be enough to capture the discontinuity structure of the target functions. This means that we may need to directly analyze each DNNs, if the target function is bounded and discontinuous (i.e. in a critical function space). The flavor of this insight seems similar to the recent mathematical studies on the incompressible inviscid flows. See [2, 3, 6, 7, 19] for example. More precisely, these have been directly looking into the behavior of inviscid fluids in the critical function spaces (to show ill-posedness), and the argument seems quite different from the previous studies focusing on well-posedness in subcritical type of function spaces. See [4, 12, 13, 20, 27] for example. To show the well-posedness, the structure of function spaces, more precisely, commutator estimates are crucially indispensable.

This paper is organized as follows: In the next section, we construct target functions and the corresponding estimators. In Section 3, we investigate the pointwise convergence of gradient descent in terms of ReLU-DNN if initial data are already close to the target function. In the last section, we give the key lemma and its proof.

## 2. Target functions and the corresponding estimators

In this section, we define a set of target functions and the corresponding estimators, which is one of the typical function set assuring pointwise convergence. For \((y_{j},\tau_{j})\in[-1,1)^{d}\times\mathbb{S}^{d-1}\) (\(j=1,2,\cdots\)), let us define half spaces \(H^{\circ}\) and \(H^{\circ}_{\epsilon}\) as follows:

\[H^{\circ}(y_{j},\tau_{j}) :=\{x\in[-1,1)^{d}:x\cdot\tau_{j}-y_{j}\cdot\tau_{j}<0\},\] \[H^{\circ}_{\epsilon}(y_{j},\tau_{j}) :=\{x\in[-1,1)^{d}:x\cdot\tau_{j}-y_{j}\cdot\tau_{j}<-\epsilon\}.\]
We employ a set of indicator functions \(\{\chi_{\Omega},\Omega\in\mathcal{M}\}\) as the set of target functions, where \(\mathcal{M}\) is a set of convex smooth manifolds (with internal filling) as follows:

\[\mathcal{M}:= \bigg{\{}\Omega\subset[-1,1)^{d}:\partial\Omega\text{ is smooth, and the following three conditions hold:}\] \[\text{There exists }\{(y_{j},\tau_{j})\}_{j=1}^{\infty}\subset \partial\Omega\times\mathbb{S}^{d-1}\text{ such that }\bigcap_{j=1}^{\infty}H^{\circ}(y_{j},\tau_{j})=\Omega.\] \[\text{For each }j\text{ and any }N\in\mathbb{N},\] \[\max_{j^{\prime}\neq j,\ 1\leq j^{\prime}\leq N}\text{dist }\left(y_{j}-N^{-\frac{2}{d-1}}\tau_{j},\partial H^{\circ}_{N^{-\frac{2}{d-1} }}(y_{j^{\prime}},\tau_{j^{\prime}})\right)\lesssim N^{-\frac{1}{d-1}}.\] \[\text{For each }j,\text{ there is a set of points}\] \[\{c_{ji}\}_{i=1}^{d}\subset(\Omega\cap\partial H^{\circ}_{N^{-\frac{2}{d- 1}}}(y_{j},\tau_{j}))\text{ which are linearly independent.}\bigg{\}}\]

The first condition is nothing more than expressing the convexity. The second one is needed for the estimate of the difference between the target function and the corresponding estimator (see (2)). Note that, we choose \(\bigcap_{j^{\prime}=1}^{N}\partial H^{\circ}_{e}(y_{j^{\prime}},\tau_{j^{ \prime}})\) as a regular polytope, and by the dimensional analysis, the power \(-\frac{1}{d-1}\) naturally appears. The points \(\{c_{ji}\}_{i=1}^{d}\) in the third one is needed for the construction of training samples for mini-batch (see the next section).

**Remark 1**.: An interesting question naturally arises: for \(\Omega\in\mathcal{M}\), whether or not \(\partial\Omega\) is a manifold isometric to the sphere. We leave it as an open question (c.f. Tsukamoto [25, 26]).

**Definition 1**.: (Definition of estimator.) For a target function \(f^{\circ}=\chi_{\Omega}\ (\Omega\in\mathcal{M})\), we define the corresponding estimator \(f^{\circ}_{N}\) as follows:

\[f^{\circ}_{N}:=\chi_{\Omega^{\circ}_{N}},\quad\Omega^{\circ}_{N}:=\bigcup_{j=1 }^{N}H^{\circ}_{N^{-\frac{2}{d-1}}}(y_{j},\tau_{j}).\]

**Lemma 1**.: _We have_

\[f^{\circ}_{N}(x)\to f^{\circ}(x)\quad(N\to\infty)\quad\text{for any}\quad x\in[- 1,1)^{d}. \tag{1}\]

_Moreover we have the following convergence rate:_

\[\|f^{\circ}_{N}-f^{\circ}\|_{L^{r}}^{r}\lesssim_{d}N^{-\frac{2}{d-1}}\quad \text{for}\quad 1\leq r<\infty. \tag{2}\]

Proof.: By applying a diagonal argument, we immediately have (1). To show (2), let us choose a set of \(\{\tau_{ji}^{\perp}\}_{i=1}^{d-1}\subset\mathbb{S}^{d-1}\) satisfying \(\tau_{ji}^{\perp}\cdot\tau_{j}=0\ (i=1,2,\cdots,d-1)\) and \(\tau_{ji}^{\perp}\cdot\tau_{ji^{\prime}}^{\perp}\ (i\neq i^{\prime})\). Then by using a standard local coordinate system, we have

\[\left\{y_{j}-N^{-\frac{2}{d-1}}\tau_{j}+\sum_{i=1}^{d-1}s_{i}\tau_{ji}^{\perp}: s_{i}\in\mathbb{R},\ |s|\lesssim N^{-\frac{1}{d-1}}\right\}\subset\partial H^{\circ}_{N^{-\frac{2}{d- 1}}}(y_{j},\tau_{j})\]

and

\[\left\{y_{j}+\sum_{i=1}^{d-1}s_{i}\tau_{ji}^{\perp}+g(s)\tau_{j}:s_{i}\in \mathbb{R},\ |s|\lesssim N^{-\frac{1}{d-1}}\right\}\subset\partial\Omega^{\circ},\]
where \(g(s)=c_{1}s_{1}^{2}+\cdots+c_{d-1}s_{d-1}^{2}+O(|s|^{3})\) for some positive constants \(c_{i}>0\) (independent of \(N\)). Thus we have

\[|\Omega_{N}^{\circ}\setminus\Omega^{\circ}|\lesssim(N^{-\frac{2}{d-1}}+c_{1}s_ {1}^{2}+\cdots c_{d-1}s_{d-1}^{2})|\partial\Omega^{\circ}|\lesssim_{d}N^{- \frac{2}{d-1}}.\]

Therefore

\[\|f_{N}^{\circ}-f^{\circ}\|_{L^{r}}^{r}\lesssim_{d}N^{-\frac{2}{d-1}}\quad \text{for}\quad 1\leq r<\infty.\]

## 3. Pointwise convergence of gradient descent

In what follows we mathematically investigate the pointwise convergence of gradient descent in terms of ReLU-DNN, which seems to be closer to the practical ReLU-DNN. In order to do that, first we formulate the mini-batch gradient descent in pure mathematics. Let \(f^{\circ}\) be a target function and \(\{f_{N}(W^{t})\}_{t=0}^{\infty}\) (\(t\in\mathbb{Z}_{\geq 0}\)) be a sequence of functions generated by the following gradient descent:

\[E(W^{t}):=\frac{1}{2}\int_{\mathcal{D}}|f_{N}(W^{t},x)-f^{\circ}(x)|^{2}dx,\]

\[W^{t+1}=W^{t}-\epsilon\frac{1}{|\mathcal{D}|}\nabla_{W^{t}}E(W^{t}),\]

where \(f_{N}\) is a prescribed neural network with \(N\)-nodes, \(\{W^{t}\}_{t}\) is a set of weight and bias, \(\epsilon\in\mathbb{R}_{>0}\) is a leaning coefficient and \(\mathcal{D}\subset[-1,1)^{d}\) is a set of training samples for mini-batch. Note that, since the gradient is normalized by \(1/|\mathcal{D}|\), \(\mathcal{D}\) can be replaced by a non-zero measure set or a set of lines. Let \(\{f_{N}^{\circ}\}_{N}\) be a sequence of estimators such that

\[f_{N}^{\circ}(x):=\lim_{t\to\infty}f_{N}(W^{t},x).\]

Our specific purpose is to find neural networks \(f_{N}\), suitable \(\mathcal{D}\) and \(\epsilon\) assuring pointwise convergence to the corresponding estimator \(f_{N}^{\circ}\), which is already given in the last section.

**Remark 2**.: This problem setting clarifies the crucial difference between the shallow and deep neural networks, as follows: Since \(\sin\) and \(\cos\) functions are continuous, we can recover them from linear combination of activation functions (see [5] for example). Thus mathematical analysis of shallow neural network can be replaced by a linear combination of \(\sin\) and \(\cos\) functions, which is nothing more than the Fourier series. For \(x\in[-1,1)^{d}\), we set the target function \(f^{\circ}\) as the indicator function of the \(d\) dimensional ball such that

\[f^{\circ}(x)=\begin{cases}1,&|x|\leq 1/2,\\ 0,&|x|>1/2,\end{cases}\]

and let \(f^{N}\) be a Fourier series with spherical partial sum:

\[f_{N}(W^{t},x):=\sum_{|k|<N}c_{k}^{t}e^{ik\cdot x}\in\mathbb{R},\quad W^{t}: =\{c_{k}^{t}\}_{k\in\mathbb{Z}^{d}}\subset\mathbb{C},\ c_{-k}^{t}=\bar{c}_{k}^ {t},\ k\in\mathbb{Z}^{d}.\]

Let \(\mathcal{D}=[-1,1)^{d}\) (\(t=0,1,2\cdots\)), and by the Parseval's identity, we immediately have the following estimator (of course, different from the one given in the last section):

\[f_{N}^{\circ}(x)=\sum_{|k|<N}\tilde{c}_{k}e^{ik\cdot x}\quad\text{for}\quad \tilde{c}_{k}=\int_{[-1,1)^{d}}f^{\circ}(x)e^{-ik\cdot x}dx.\]
Then we obtain the following counterexample, which clarifies the crucial difference between the shallow and deep neural networks.

Counterexample.Let \(d\geq 5\) and \(\mathcal{D}=[-1,1)^{d}\). Then, for any \(x\in\mathbb{Q}^{d}\cap[-1,1)^{d}\),

\[f_{N}^{\circ}(x)-f^{\circ}(x)\quad\text{diverges as}\quad N\to\infty.\]

The proof is just direct consequence of Kuratsubo [15] (see also [16, 17]). Thus we omit its detail.

In contrast with the Fourier series case (shallow neural network), we will show pointwise convergence to \(f_{N}^{\circ}\) which is already given in the last section. Let \(N=2^{n}\) (\(n\in\mathbb{N}\)) and let us now construct a deep neural network \(f_{N}\). For the initial layer, we define

\[z^{1}:=h(w^{1}x+b^{1}):=\begin{pmatrix}h(w_{1}^{1}\cdot x+b_{1}^{1})\\ \vdots\\ h(w_{2^{n}}^{1}\cdot x+b_{2^{n}}^{1})\end{pmatrix}\]

for \(x\in[-1,1)^{d}\), \(w^{1}:=\{w_{j}^{1}\}_{j=1}^{2^{n}}:=\{w_{ji}^{1}\}_{ji}\in\mathbb{R}^{2^{n} \times d}\), \(b^{1},z^{1}\in\mathbb{R}^{2^{n}}\). Recall that \(w\) is the weight and \(b\) is the bias. For the \(2k\)-th layer, we set

\[z^{2k}:=h(w^{2k}z^{2k-1}+b^{2k})\]

for \(w^{2k}\in\mathbb{R}^{3\cdot 2^{n-k}\times 2^{n-k+1}}\), \(b^{2k},z^{2k}\in\mathbb{R}^{3\cdot 2^{n-k}}\). Moreover, we impose the following sparsity condition: for \(J=1,2,\cdots,2^{n-k}\) and \(1\leq k\leq n\),

\[\begin{split} z_{3J-2}^{2k}&=h(w_{3J-2,2J-1}^{2k}z_{ 2J-1}^{2k-1}+w_{3J-2,2J}^{2k}z_{2J}^{2k-1}),\\ z_{3J-1}^{2k}&=h(w_{3J-1,2J-1}^{2k}z_{2J-1}^{2k-1}+w_{3J-1,2J}^ {2k}z_{2J}^{2k-1}),\\ z_{3J}^{2k}&=h(w_{3J,2J-1}^{2k}z_{2J-1}^{2k-1}+w_{3J,2J}^ {2k}z_{2J}^{2k-1}),\end{split} \tag{3}\]

where \(z^{2k}=\{z_{j}^{2k}\}_{j}\), \(b^{2k}=\{b_{j}^{2k}\}_{j}\), and \(w^{2k}=\{w_{ji}^{2k}\}_{ji}\), and also impose the following restriction:

\[\begin{split} w_{3J-2,2J-1}^{2k}&=w_{3J-1,2J-1}^{2k} =-w_{3J,2J-1}^{2k},\\ w_{3J-2,2J}^{2k}&=-w_{3J-1,2J}^{2k}=w_{3J,2J}^{2k}. \end{split} \tag{4}\]

For the \(2k+1\) layer, we set

\[z^{2k+1}=w^{2k+1}z^{2k},\]

\(w^{2k+1}\in\mathbb{R}^{2^{n-k}\times 3\cdot 2^{n-k}}\), \(z^{2k+1}\in\mathbb{R}^{2^{n-k}}\). In this layer, we impose the following restriction: for \(J=1,2,\cdots 2^{n-k}\),

\[z_{J}^{2k+1}=z_{3J-2}^{2k}-z_{3J-1}^{2k}-z_{3J}^{2k}.\]

Then we see that, in the \(2n+1\) layer, \(z^{2n+1}\) becomes a real number. In the final layer, we apply the following clipping:

\[f_{N}:=z^{2n+2} =\max\{z^{2n+1},1\}\] \[=1-h(1-z^{2n+1}).\]

**Remark 3**.: In this paper we employed ReLU function as the activate function, for simplicity. Of course, employing the sigmoid function case is also attractive problem.

Then the main theorem is as follows:
**Theorem 2**.: _Assume that the initial function \(f_{N}(W^{t=0})\) is already close to the target function \(f^{\circ}\), namely, \(W^{t=0}\) satisfies the initial conditions (8) and (9). Let \(\epsilon=\gamma^{2}\) (\(\gamma\) is given in Proposition 9). Then, by choosing \(\mathcal{D}\) appropriately, and by suitable change of variables: \(W^{t}\mapsto(\alpha^{t},\beta^{t})\), then \(f_{N}(\alpha^{t},\beta^{t})\) converges to \(f^{\circ}_{N}\) pointwisely (as \(t\to\infty\)). The change of variables are explicitly written as_

\[\alpha_{j}:=m_{j}^{2}|w_{j}^{1}|^{2}\quad\text{and}\quad\beta_{j}=m_{j}(w_{j}^ {1}\cdot c_{ji}+b_{j}^{1}),\]

_where the definition of \(m_{j}\) is given in (5). Moreover we have the following convergence rate:_

\[\|f_{N}(\alpha^{t},\beta^{t})-f^{\circ}_{N}\|_{L^{r}}^{r}\lesssim_{d}t^{-1/3} \quad\text{for}\quad 1\leq r<\infty.\]

**Remark 4**.: It is an open question whether or not the original coefficient \(W^{t}\) case is also converging to the same estimator \(f^{\circ}_{N}\).

**Remark 5**.: The initial conditions (8) and (9) are just for the technical reason. We can relax these conditions further.

_Proof._ First we consider a pair of \(2k-1\), \(2k\) and \(2k+1\) layers. Let us rewrite (3) in the simpler description as follows:

\[\begin{cases}z_{3J-2}^{2k}=h(m^{1}z_{2J-1}^{2k-1}+m^{0}z_{2J}^{2k-1}),\\ z_{3J-1}^{2k}=h(m^{1}z_{2J-1}^{2k-1}-m^{0}z_{2J}^{2k-1}),\\ z_{3J}^{2k}=h(-m^{1}z_{2J-1}^{2k-1}+m^{0}z_{2J}^{2k-1}),\end{cases}\]

where

\[m_{k,J}^{1}=m^{1} :=w_{3J-2,2J-1}^{2k}=w_{3J-1,2J-1}^{2k}=-w_{3J,2J-1}^{2k},\] \[m_{k,J}^{0}=m^{0} :=w_{3J-2,2J}^{2k}=-w_{3J-1,2J}^{2k}=w_{3J,2J}^{2k}.\]

Recall that

\[z_{J}^{2k+1}=z_{3J-2}^{2k}-z_{3J-1}^{2k}-z_{3J}^{2k}.\]

Taking a derivative, we have

\[\partial_{z_{2J-1}^{2k-1}}z_{J}^{2k+1}= m^{1}\partial h(m^{1}z_{2J-1}^{2k-1}+m^{0}z_{2J}^{2k-1})\] \[-m^{1}\partial h(m^{1}z_{2J-1}^{2k-1}-m^{0}z_{2J}^{2k-1})\] \[+m^{1}\partial h(-m^{1}z_{2J-1}^{2k-1}+m^{0}z_{2J}^{2k-1}).\]

Due to the cancellation of Heaviside functions in the following domain,

\[D_{k,J}^{0}:=\left\{x:m^{0}z_{2J}^{2k-1}<m^{1}z_{2J-1}^{2k-1}\right\},\]

we have

\[\partial_{z_{2J-1}^{2k-1}}z_{J}^{2k+1}=0\quad\text{for}\quad x\in D_{k,J}^{0}.\]

Note that, rigorously saying, \(z^{2k-1}:=z^{2k-1}\circ z^{2k}\circ\cdots\circ z^{1}\). To the contrary, there is no cancellation of Heaviside functions in the following domain:

\[D_{k,J}^{1}:=\{x:m^{0}z_{2J}^{2k-1}>m^{1}z_{2J-1}^{2k-1}\}.\]

In other words,

\[\partial_{z_{2J-1}^{2k-1}}z_{J}^{2k+1}=2m^{1}\quad\text{for}\quad x\in D_{k,J} ^{1}.\]
The same argument goes through also in the case \(\partial_{z^{2k-1}_{2J}}z^{2k+1}_{J}\) (omit its detail). In this case, we have

\[\partial_{z^{2k-1}_{2J}}z^{2k+1}_{J} =2m^{0}\quad\text{for}\quad x\in D^{0}_{k,J},\] \[\partial_{z^{2k-1}_{2J}}z^{2k+1}_{J} =0\quad\text{for}\quad x\in D^{1}_{k,J}.\]

We apply this property inductively in the reverse direction (as the back propergation), and we divide the non-zero region \(\{x:f_{N}(W^{t},x)>0\}\) into several parts appropriately. To do that, we suitably rewrite the natural number \(j\in\{1,2,\cdots,2^{n}\}\) as follows:

\[j=\delta^{j}_{1}+2\delta^{j}_{2}+2^{2}\delta^{j}_{3}+\cdots+2^{n-1}\delta^{j} _{n},\]

where \(\delta^{j}_{k}\in\{0,1\}\). Let

\[D_{j}:=\bigcap_{k=1}^{n}D^{\delta^{j}_{k}}_{k,J^{j}_{k}}\quad\text{for}\quad J ^{j}_{k}:=\sum_{\ell=k}^{n}2^{\ell-k}\delta^{j}_{\ell}.\]

By using this \(D_{j}\), the derivative formula becomes much simpler:

\[\partial_{x}z^{2n+1}(x)=m_{j}w^{1}_{j}\quad\text{for}\quad x\in D_{j},\quad \text{where}\quad m_{j}:=\prod_{k=1}^{n}\left(2m^{\delta^{j}_{k}}_{k,J^{j}_{k} }\right). \tag{5}\]

By the construction of \(D_{j}\), we observe that

\[\{x:f_{N}=0\}\cap\partial D_{j}\subset\{x:w^{1}_{j}\cdot x+b^{1}_{j}=0\},\]

then, by the fundamental theorem of calculus, we have

\[z^{2n+1}(x)=\sum_{j=1}^{2^{n}}\left(h(m_{j}w^{1}_{j}\cdot x+m_{j}b^{1}_{j}) \chi_{D_{j}}(x)\right).\]

Therefore we obtain the following explicit formula:

\[f_{N}(x)=\max\left\{\sum_{j=1}^{2^{n}}\left(h(\tilde{w}^{1}_{j}\cdot x+\tilde{ b}^{1}_{j})\chi_{D_{j}}(x)\right),1\right\}, \tag{6}\]

where \(\tilde{w}^{1}_{j}:=m_{j}w^{1}_{j}\) and \(\tilde{b}^{1}_{j}:=m_{j}b^{1}_{j}\). Then we can apply Lemma 4 in the next section, and complete the proof.

## 4. key lemma for pointwise convergence

In this section we give several assumptions and a geometric a-priori region, just for providing much simpler argument. First let us assume

\[\tilde{w}^{t=0}_{j}\perp\partial H^{\circ}_{N^{-\frac{2}{d-1}}}(y_{j},\tau_{j }). \tag{7}\]

To give the geometric a-priori region, we use parametrized hyper planes. For \(r=\{r_{ji}\}_{ji}\in(-1,1)\) (\(i=1,\cdots,d\), \(j=1,\cdots,2^{n}\)), let \(h_{j}(r)\) be a unique hyper plane composed of points which are also linearly independent (due to the assumption (7)):

\[\{\tilde{c}_{ji}(r)\}_{i=1}^{d}:=\left\{2\frac{\tilde{w}_{j}}{|\tilde{w}_{j}| ^{2}}r_{ji}+c_{ji}\right\}_{i=1}^{d}.\]

To be more precise. By the Gram-Schmidt process, there is \(\tilde{\tau}_{j}\in\mathbb{S}^{d-1}\) such that

\[(\tilde{c}_{ji}(r)-\tilde{c}_{ji^{\prime}}(r))\cdot\tilde{\tau}_{j}=0\quad(i \neq i^{\prime}),\]
and then we define the hyper plane \(h_{j}(s)\) as follows:

\[h_{j}(r):=\{x:(x-\tilde{c}_{j1}(r))\cdot\tilde{\tau}_{j}=0\}.\]

By using this \(h_{j}(s)\), we now define the a-priori region \(\mathcal{L}_{j}\) as follows:

\[\mathcal{L}_{j}:=\bigcup_{r_{j1}\in(-1,1)}\bigcup_{r_{j2}\in(-1,1)}\cdots \bigcup_{r_{jd}\in(-1,1)}h_{j}(r).\]

Before we state the key lemma, we need the following proposition.

**Proposition 3**.: _Assume_

\[\{c_{ji}\}_{i=1}^{d}\subset D_{j}\quad\text{and}\quad\tilde{w}_{j}\perp \partial H^{\circ}_{N^{-\frac{2}{d-1}}}(y_{j},\tau_{j}). \tag{8}\]

_Then there exists \(\gamma>0\) such that if_

\[|\tilde{w}_{j}|>\gamma, \tag{9}\]

_then_

\[\ell_{ji}(s):=\tilde{w}_{j}^{1}s+c_{ji}\in D_{j}\setminus(\cup_{j\neq j^{ \prime}}\mathcal{L}_{j^{\prime}})\quad\text{for}\quad s\in[-2|\tilde{w}_{j}|^ {-2},2|\tilde{w}_{j}|^{-2}]. \tag{10}\]

We need this (10) for providing the simple induction argument (see the proof of Lemma 4). This means that, by a careful computation, we may be able to relax this (10) further.

Proof.: The case \(|\tilde{w}_{j}|=\infty\) automatically satisfies (10) and then we just apply the continuity argument. 

**Lemma 4**.: _Assume that the initial weight and bias \(W^{t=0}\) satisfy (8) and (9). Let \(\epsilon=\gamma^{2}\). Then, by choosing \(\mathcal{D}\subset[-1,1)^{d}\) appropriately, and by suitable change of variables: \(W^{t}\mapsto(\alpha^{t},\beta^{t})\), \(f_{N}(\alpha^{t},\beta^{t})\) converges to \(f_{N}^{\circ}\) pointwisely (as \(t\to\infty\)). The change of variables are explicitly written as_

\[\alpha_{j}:=m_{j}^{2}|w_{j}^{1}|^{2}\quad\text{and}\quad\beta_{j}=m_{j}(w_{j} ^{1}\cdot c_{ji}+b_{j}^{1}),\]

_where the definition of \(m_{j}\) is given in (5). Also we have the following convergence rate:_

\[\|f_{N}(\alpha^{t},\beta^{t})-f_{N}^{\circ}\|_{L^{r}}^{r}\lesssim t^{-1/3}.\]

**Remark 6**.: Formally, the coefficients of \(f_{N}^{\circ}\) include infinity. But this is rather reasonable, since we need to express discontinuity by using finite times composite function of the ReLU function.

Proof of Lemma.For \(t\) times gradient descent, we choose \(2^{n}\cdot d\) elements of straight lines passing through \(\{c_{ji}\}_{ji}\), and we denote them \(\ell_{ji}^{t}\) (\(i=1,2,\cdots,d,j=1,\cdots,2^{n}\)):

\[\ell_{ji}^{t}(s)=\tilde{w}_{j}^{1,t}s+c_{ji}^{t}.\]

Rigorously this \(\tilde{w}_{j}^{1,t}\) is frozen. More precisely, if we take derivative in \(w_{j}^{1}\) or \(b_{j}^{1}\), we regard this \(\tilde{w}_{j}^{1,t}\) as a constant, not variable. Let \(\mathcal{D}\) be such that

\[\mathcal{D}:=\bigcup_{k=1}^{d}\bigcup_{j=1}^{2^{n}}\bigcup_{s=-2\gamma^{-2}}^ {2\gamma^{-2}}\ell_{ji}(s).\]
First we show that this \(\mathcal{D}\) is independent of \(t\). We plug this lines into (6), and introduce the new variables \(\alpha_{j}\), \(\beta_{ji}\):

\[(\tilde{w}_{j}^{1}\ell_{ji}(s)+\tilde{b}_{j}^{1}) =|\tilde{w}_{j}^{1}|^{2}s+(\tilde{w}_{j}^{1}\cdot c_{ji}+\tilde{b} _{j}^{1})\] \[=:\alpha_{j}s+\beta_{j}. \tag{11}\]

Note that \(\tilde{w}_{j}^{1}\cdot c_{ji}\) is independent of \(i\). This means that,

\[\text{if}\quad\tilde{w}_{j}^{1,t}\perp\partial H^{\circ}_{N^{-\frac{2}{d-1}}} (y_{j},\tau_{j}),\quad\text{then}\quad\tilde{w}_{j}^{1,t+1}\perp\partial H^{ \circ}_{N^{-\frac{2}{d-1}}}(y_{j},\tau_{j}).\]

Now we rewrite the error function \(E\) as follows:

\[E(\alpha,\beta) :=\frac{d}{2|\mathcal{D}|}\sum_{j=1}^{2^{n}}E_{j}(\alpha_{j}, \beta_{j})\] \[:=\frac{d}{2|\mathcal{D}|}\sum_{j=1}^{2^{n}}\left(\int_{-2\gamma }^{0}(\alpha_{j}s+\beta_{j})^{2}ds+\int_{0}^{2\gamma}(1-(\alpha_{j}s+\beta_{j} ))^{2}ds\right)\] \[=\frac{d}{2|\mathcal{D}|}\sum_{j=1}^{2^{n}}\left(\int_{-\frac{ \beta_{j}}{\alpha_{j}}}^{0}(\alpha_{j}s+\beta_{j})^{2}ds+\int_{0}^{\frac{1- \beta_{j}}{\alpha_{j}}}(1-(\alpha_{j}s+\beta_{j}))^{2}ds\right).\]

Direct calculations yield \(|\mathcal{D}|=4d2^{n}/\gamma^{2}\) and

\[E_{j}(\alpha_{j},\beta_{j})=\frac{1-3\beta_{j}+3\beta_{j}^{2}}{3\alpha_{j}}.\]

Then we have

\[\partial_{\alpha_{i}}E_{j}=-\frac{1}{3\alpha_{j}^{2}}\left(1-3\beta_{j}+3 \beta_{j}^{2}\right)\quad\text{and}\quad\partial_{\beta_{j}}E_{j}=-\frac{1}{ \alpha_{j}}(1-2\beta_{j}).\]

Since \(1-3\beta_{j}+3\beta_{j}^{2}\geq 1/4>0\) for any \(\beta_{j}\in\mathbb{R}\), we always have \(\partial_{\alpha_{j}}E<0\) and

\[\alpha_{j}^{t+1}=\alpha_{j}^{t}-\epsilon\partial_{\alpha_{j}}E\geq\alpha_{j}^ {t}+\frac{\gamma^{2}}{24|\mathcal{D}|(\alpha_{j}^{t})^{2}}.\]

Thus \(\alpha^{t+1}>\alpha^{t}\) and then we have \(|\tilde{w}_{j}^{1,t+1}|>\gamma\) inductively. By directly solving the ODE: \(\frac{d}{dt}g(t)=1/g(t)^{2}\), applying the mean-value theorem and the comparison principle, we have \(\alpha_{j}^{t}\gtrsim t^{1/3}\). Next we consider \(\beta_{j}\). First we show \(0<\beta_{j}<1\). By

\[\partial_{\beta_{j}}E=-\frac{1}{\alpha_{j}^{t}}(1-2\beta_{j}^{t})\in(0,1/ \alpha_{j}^{t})\quad\text{for}\quad 1/2<\beta_{j}<1,\]

and \(\epsilon=\gamma^{2}<\alpha_{j}^{t}\), we have \(\beta_{j}^{t}>\beta_{j}^{t+1}>0\). To the contrary, if \(\beta_{j}^{t}\in(0,1/2)\), then \(\beta_{j}^{t}<\beta_{j}^{t+1}<1\). Thus \(\beta_{j}^{t}\in(0,1)\). This means that \(\{c_{ji}\}_{i=1}^{d}\subset D_{j}^{t+1}\) inductively. Thus the next step \((\alpha^{t+1},\beta^{t+1})\) also satisfies (10) inductively. Moreover, since \(|\beta_{j}^{t+1}-\beta_{j}^{t}|\to 0\), \(\beta_{j}^{t}\) converges. Since \(\beta_{j}^{t+1}-\beta_{j}^{t}=0\) if and only if \(\beta_{j}^{t}=1/2\), thus \(\beta_{j}^{t}\) converges to \(1/2\). Therefore \(f_{N}(\alpha^{t},\beta^{t})\) converges to \(f_{N}^{\circ}\) pointwisely. Moreover we immediately have the following estimate:

\[\|f_{N}(\alpha^{t},\beta^{t})-f_{N}^{\circ}\|_{L^{r}}^{r}\lesssim t^{-\frac{1} {3}}|\partial\Omega_{N}^{\circ}|\lesssim t^{-1/3}.\]

This is the desired estimate.
## 5. Conclusion

In the previous approximation error analyses, ReLU deep neural networks had been crucially applied for constructing one-dimensional polynomials (spline functions), which is needed for wavelet expansions. In contrast with these, in this paper, we found a ReLU DNN architecture which is suitable for capturing convex shape of discontinuity on indicator functions (target functions), accompanied by pointwise convergence. Our next question would be what kind of ReLU-DNN architectures really attain pointwise convergence (or not) for mixed concave and convex discontinuities, and this is our future work.

**Acknowledgments.** I am grateful to Professors Masaharu Nagayama, Eiichi Nakai, Kengo Nakai, Yoshitaka Saiki and Yuzuru Sato for valuable comments. Research of TY was partly supported by the JSPS Grants-in-Aid for Scientific Research 20H01819 and 21K03304. This paper was a part of the lecture note on the class: Mathematical Analysis I (spring semester 2023) for undergrad/graduate courses in Hitotsubashi University.
