Non-asymptotic approximations of Gaussian neural networks via
  second-order Poincaré inequalities
There is a growing interest on large-width asymptotic properties of Gaussian
neural networks (NNs), namely NNs whose weights are initialized according to
Gaussian distributions. A well-established result is that, as the width goes to
infinity, a Gaussian NN converges in distribution to a Gaussian stochastic
process, which provides an asymptotic or qualitative Gaussian approximation of
the NN. In this paper, we introduce some non-asymptotic or quantitative
Gaussian approximations of Gaussian NNs, quantifying the approximation error
with respect to some popular distances for (probability) distributions, e.g.
the $1$-Wasserstein distance, the total variation distance and the
Kolmogorov-Smirnov distance. Our results rely on the use of second-order
Gaussian Poincar\'e inequalities, which provide tight estimates of the
approximation error, with optimal rates. This is a novel application of
second-order Gaussian Poincar\'e inequalities, which are well-known in the
probabilistic literature for being a powerful tool to obtain Gaussian
approximations of general functionals of Gaussian stochastic processes. A
generalization of our results to deep Gaussian NNs is discussed.
Alberto Bordino, Stefano Favaro, Sandra Fortini
# Non-asymptotic approximations of Gaussian neural networks via second-order Poincare inequalities

###### Abstract

There is a growing interest on large-width asymptotic properties of Gaussian neural networks (NNs), namely NNs whose weights are initialized according to Gaussian distributions. A well-established result is that, as the width goes to infinity, a Gaussian NN converges in distribution to a Gaussian stochastic process, which provides an asymptotic or qualitative Gaussian approximation of the NN. In this paper, we introduce some non-asymptotic or quantitative Gaussian approximations of Gaussian NNs, quantifying the approximation error with respect to some popular distances for (probability) distributions, e.g. the 1-Wasserstein distance, the total variation distance and the Kolmogorov-Smirnov distance. Our results rely on the use of second-order Gaussian Poincare inequalities, which provide tight estimates of the approximation error, with optimal rates. This is a novel application of second-order Gaussian Poincare inequalities, which are well-known in the probabilistic literature for being a powerful tool to obtain Gaussian approximations of general functionals of Gaussian stochastic processes. A generalization of our results to deep Gaussian NNs is discussed.

## 1 Introduction

There is a growing interest on large-width asymptotic properties of Gaussian neural networks (NNs), namely NNs whose weights or parameters are initialized according to Gaussian distributions (Neal, 1996; Williams, 1997; Der and Lee, 2005; Garriga-Alonso et al., 2018; Lee et al., 2018; Matthews et al., 2018; Novak et al., 2018; Antognini, 2019; Hanin, 2019; Yang, 2019; Aitken and Gur-Ari, 2020; Andreassen and Dyer, 2020; Bracale et al., 2021; Eldan et al., 2021; Basteri and Trevisan, 2022). Let \(\mathcal{N}(\mu,\sigma^{2})\) be a Gaussian distribution with mean \(\mu\) and variance \(\sigma^{2}\), and consider: i) an input \(\mathbf{x}\in\mathbb{R}^{d}\), with \(d\geq 1\); ii) a collection of (random) weights \(\theta=\{w_{i}^{(0)},w,b_{i}^{(0)},b\}_{i\geq 1}\) such that \(w_{i,j}^{(0)}\stackrel{{ d}}{{=}}w_{j}\), with the \(w_{i,j}^{(0)}\)'s being independent and identically distributed as \(\mathcal{N}(0,\sigma_{w}^{2})\), and \(b_{i}^{(0)}\stackrel{{ d}}{{=}}b\), with the \(b_{i}^{(0)}\)'s being independent and identically distributed as \(\mathcal{N}(0,\sigma_{b}^{2})\) for \(\sigma_{w}^{2},\sigma_{b}^{2}>0\); iii) an activation function \(\tau:\mathbb{R}\rightarrow\mathbb{R}\). Then, a (fully connected
feed-forward) Gaussian NN is defined as follows:

\[f_{\mathbf{x}}(n)[\tau,n^{-1/2}]=b+\frac{1}{n^{1/2}}\sum_{j=1}^{n}w_{j}\tau(\langle w _{j}^{(0)},\mathbf{x}\rangle_{\mathbb{R}^{d}}+b_{j}^{(0)}), \tag{1}\]

with \(n^{-1/2}\) being a scaling factor. Neal (1996) characterized the infinitely wide limit of the NN (1), showing that, as \(n\to+\infty\), for any \(\mathbf{x}\in\mathbb{R}^{d}\) the NN \(f_{\mathbf{x}}(n)[\tau,n^{-1/2}]\) converges in distribution to a Gaussian random variable (RV). That is, as a function of \(\mathbf{x}\), the infinitely wide limit of the NN is a Gaussian stochastic process. The proof is an application of the classical Central Limit Theorem (CLT), thus relying on minimal assumptions on \(\tau\) to ensure that \(\mathbb{E}[(g_{j}(\mathbf{x}))^{2}]\) is finite, where \(g_{j}(\mathbf{x})=w_{j}\tau(\langle w_{j}^{(0)},\mathbf{x}\rangle_{\mathbb{R}^{d}}+b_ {j}^{(0)})\). The result of Neal (1996) has been extended to more general matrix input, i.e. \(p>1\) inputs of dimension \(d\), and to deep Gaussian NNs, assuming a "sequential growth" (Der and Lee, 2005) and a "joint growth" (Matthews et al., 2018) of the width over the NN's layers. These results provide asymptotic or qualitative Gaussian approximations of Gaussian NNs, as they do not provide the rate at which the NN converges to the infinitely wide limit.

### Our contribution

In this paper, we consider non-asymptotic or quantitative Gaussian approximations of the NN (1), quantifying the approximation error with respect to some popular distances for (probability) distributions. To introduce our results, let \(d_{W_{1}}\) be the 1-Wasserstein distance and consider a Gaussian NN with a 1-dimensional unitary input, i.e. \(d=1\) and \(x=1\), unit variance's weight, i.e. \(\sigma_{w}^{2}=1\), and no biases, i.e. \(b_{i}^{(0)}=b=0\) for any \(i\geq 1\). Under this setting, our result states as follows: if \(\tau\in C^{2}(\mathbb{R})\) such \(\tau\) and its first and second derivatives are bounded above by the linear envelope \(a+b|x|^{\gamma}\), for \(a,b,\gamma>0\), and \(N\sim\mathcal{N}(0,\sigma^{2})\) with \(\sigma^{2}\) being the variance of the NN, then for any \(n\geq 1\)

\[d_{W_{1}}(f_{1}(n)[\tau,n^{-1/2}],N)\leq\frac{K_{\sigma^{2}}}{n^{1/2}}, \tag{2}\]

with \(K_{\sigma^{2}}\) being a constant that can be computed explicitly. The polynomial envelope assumption is not new in the study of large-width properties of Gaussian NNs (Matthews et al., 2018; Yang, 2019), and it is critical to achieve the optimal rate \(n^{-1/2}\) in the estimate (2) of the approximation error. In general, we show that an approximation analogous to (2) holds true for the Gaussian NN (1), with the approximation being with respect to the 1-Wasserstein distance, the total variation distance and the Kolmogorov-Smirnov distance. Our results rely on the use of second-order Gaussian Poincare inequalities, or simply second-order Poincare inequalities, first introduced in Chatterjee (2009) and Nourdin et al. (2009) as a powerful tool to obtain Gaussian approximation of general functionals of Gaussian stochastic processes. Here, we make use of some refinements of second-order Poincare inequalities developed in Vidotto (2020), which have the advantage of providing tight estimates of the approximation error, with (presumably) optimal rates. An extension of (2) is presented for Gaussian NNs with \(p>1\) inputs, whereas a generalization of our results to deep Gaussian NNs is discussed with respect to the "sequential growth" and the "joint growth" of the width over the NN's layers.

### Related work

While there exists a vast literature on infinitely wide limits of Gaussian NNs, as well as their corresponding asymptotic approximations, only a few recent works have investigated non-asymptotic approximations of Gaussian NNs. To the best of our knowledge, the work of Eldan et al. (2021) is the first to consider the problem of non-asymptotic approximations of Gaussian NNs, focusing
on NNs with Gaussian distributed weights \(w_{i,j}\)'s and Rademacher distributed weights \(w_{i}\)'s. For such a class of NNs, they established a quantitative CLT in an infinite-dimensional functional space, metrized with the Wasserstein distance, providing rates of convergence to a Gaussian stochastic process. For deep Gaussian NNs (Der and Lee, 2005; Matthews et al., 2018), the work of Basteri and Trevisan (2022) first established a quantitative CLT in the 2-Wasserstein distance, providing the rate at which a deep Gaussian NN converges to its infinitely wide limit. Such a result relies on basic properties of the Wasserstein distance, which allow for a quantitatively tracking the hidden layers and yield a proof by induction, with the triangular inequality being applied to obtain independence from the previous layers. See Favaro et al. (2022) for an analogous result in the sup-norm distance. Our work is close to that of Basteri and Trevisan (2022), in the sense that we deal with NNs for which all the weights are initialized according to Gaussian distributions, and we consider their approximations through Gaussian RVs. The novelty of our work lies on the use of second-order Poincare inequalities, which allow reducing the problem to a direct computation of the gradient and Hessian of the NN, and provide estimates of the approximation error with optimal rate, and tight constants, with respect to other distances than sole Wasserstein distance. This is the first to make use of second-order Poincare inequalities as a tool to obtain non-asymptotic Gaussian approximations of Gaussian NNs.

### Organization of the paper

The paper is structured as follows. In Section 2 we present an overview on second-order Poincare inequalities, recalling some of the main results of Vidotto (2020) that are critical to prove our non-asymptotic Gaussian approximations of Gaussian NNs. Section 3 contains the non-asymptotic Gaussian approximation of the NN (1), as well as its extension for the NN (1) with \(p>1\) inputs, where Section 4 contains some numerical illustrations of our approximations. In Section 5 we discuss the extension of our results to deep Gaussian NNs, and we present some directions for future research.

## 2 Preliminaries on second-order Poincare inequalities

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a generic probability space on which all the RVs are assumed to be defined. We denote by \(\perp\!\!\!\perp\) the independence between RVs, and we make use of the acronym "iid" to refer to RVs that are independent and identically distributed and by \(\|X\|_{L^{q}}:=(\mathbb{E}[X^{q}])^{1/q}\) the \(L^{q}\) norm of the RV \(X\). In this work, we consider some popular distances between (probability) distributions of real-valued RVs. Let \(X\) and \(Y\) be two RVs in \(\mathbb{R}^{d}\), for some \(d\geq 1\). We denote by \(d_{W_{1}}\) the 1-Wasserstein distance, that is,

\[d_{W_{1}}(X,Y)=\sup_{h\in\mathscr{H}}|\mathbb{E}[h(X)]-\mathbb{E}[h(Y)]|,\]

where \(\mathscr{H}\) is the class of all functions \(h:\mathbb{R}^{d}\to\mathbb{R}\) such that it holds true that \(\|h\|_{\mathrm{Lip}}\;\leq 1\), with \(\|h\|_{\mathrm{Lip}}\;=\sup_{x,y\in\mathbb{R}^{d},x\neq y}|h(x)-h(y)|/\|x-y\|_ {\mathbb{R}^{d}}\). We denote by \(d_{TV}\) the total variation distance, that is,

\[d_{TV}(X,Y)=\sup_{B\in\mathscr{B}(\mathbb{R}^{m})}|\mathbb{P}(X\in B)-\mathbb{ P}(Y\in B)|,\]

where \(\mathscr{B}\left(\mathbb{R}^{d}\right)\) is the Borel \(\sigma\)-field of \(\mathbb{R}^{d}\). Finally, we denote by \(d_{KS}\) the Kolmogorov-Smirnov distance, i.e.

\[d_{KS}(X,Y)=\sup_{z_{1},\ldots,z_{d}\in\mathbb{R}}|\mathbb{P}\left(X\in\times_ {i=1}^{d}\left(-\infty,z_{i}\right]\right)-\mathbb{P}\left(Y\in\times_{i=1}^{ d}\left(-\infty,z_{i}\right]\right)|.\]
We recall the following interplays between some of the above distances: i) \(d_{KS}(\cdot,\cdot)\leq d_{TV}(\cdot,\cdot)\); ii) if \(X\) is a real-valued RV and \(N\sim\mathcal{N}(0,1)\) is the standard Gaussian RV then \(d_{KS}(X,N)\leq 2\sqrt{d_{W_{1}}(X,N)}\).

Second-order Poincare inequalities provide a useful tool for Gaussian approximation of general functionals of Gaussian fields (Chatterjee, 2009; Nourdin et al., 2009). See also Nourdin and Peccati (2012) and references therein for a detailed account. For our work, it is useful to recall some results developed in Vidotto (2020), which provide improved versions of the second-order Poincare inequality first introduced in Chatterjee (2009) for random variables and then extended in Nourdin et al. (2009) to general infinite-dimensional Gaussian fields. Let \(N\sim\mathcal{N}(0,1)\). Second-order Poincare inequalities can be seen as an iteration of the so-called Gaussian Poincare inequality, which states that

\[\mathrm{Var}[f(N)]\leq\mathbb{E}[f^{\prime}(N)^{2}] \tag{3}\]

for every differentiable function \(f:\mathbb{R}\to\mathbb{R}\), a result that was first discovered in a work by Nash (1956) and then reproved by Chernoff (1981). The inequality (3) implies that if the \(L^{2}\) norm of the RV \(f^{\prime}(N)\) is small, then so are the fluctuations of the RV \(f(N)\). The first version of a second-order Poincare inequality was obtained in Chatterjee (2009), where it is proved that one can iterate (3) in order to assess the total variation distance between the distribution of \(f(N)\) and the distribution of a Gaussian RV with matching mean and variance. The precise result is stated in the following theorem.

**Theorem 2.1** (Chatterjee (2009) - second-order Poincare inequality).: _Let \(X\sim\mathcal{N}\left(0,I_{d\times d}\right)\). Take any \(f\in C^{2}(\mathbb{R}^{d})\), and \(\nabla f\) and \(\nabla^{2}f\) denote the gradient of \(f\) and Hessian of \(f\), respectively. Suppose that \(f(X)\) has a finite fourth moment, and let \(\mu=\mathbb{E}[f(X)]\) and \(\sigma^{2}=\mathrm{Var}[f(X)]\). Let \(N\sim\mathcal{N}(\mu,\sigma^{2})\) then_

\[d_{TV}(f(X),N)\leq\frac{2\sqrt{5}}{\sigma^{2}}\left\{\mathbb{E}\left[\|\nabla f (X)\|_{\mathbb{R}^{d}}^{4}\right]\right\}^{1/4}\left\{\mathbb{E}\left[\|\nabla ^{2}f(X)\|_{op}^{4}\right]\right\}^{1/4}, \tag{4}\]

_where \(\|\cdot\|_{op}\) stands for the operator norm of the Hessian \(\nabla^{2}f(X)\) regarded as a random \(d\times d\) matrix._

Nourdin et al. (2009) pointed out that the Stein-type inequalities that lead to (4) are special instances of a more general class of inequalities, which can be obtained by combining Stein's method and Malliavin calculus on an infinite-dimensional Gaussian space. In particular, Nourdin et al. (2009) obtained a general version of (4), involving functionals of arbitrary infinite-dimensional Gaussian fields. Both (4) and its generalization in Nourdin et al. (2009) are known to give suboptimal rates of convergence. This is because, in general, it is not possible to obtain an explicit computation of the expectation of the operator norm involved in the estimate of total variation distance, which leads to move further away from the distance in distribution and use bounds on the operator norm instead of computing it directly. To overcome this drawback, Vidotto (2020) adapted to the Gaussian setting an approach recently developed in Last et al. (2016) to obtain second-order Poincare inequalities for Gaussian approximation of Poisson functionals, yielding estimates of the approximation error that are (presumably) optimal. The next theorem states Vidotto (2020, Theorem 2.1) for the special case of a function \(f(X)\), with \(f\in C^{2}\left(\mathbb{R}^{d}\right)\) such that its partial derivatives have sub-exponential growth, and \(X\sim\mathcal{N}\left(0,I_{d\times d}\right)\). See Appendix A for an overview of Vidotto (2020, Theorem 2.1).

**Theorem 2.2** (Vidotto (2020) - 1-dimensional second-order Poincare inequality).: _Let \(F=f(X)\), for some \(f\in C^{2}\left(\mathbb{R}^{d}\right)\), and \(X\sim\mathcal{N}\left(0,I_{d\times d}\right)\) such that \(E[F]=0\) and \(E\left[F^{2}\right]=\sigma^{2}\). Let
\(N\sim\mathcal{N}\left(0,\sigma^{2}\right)\), then_

\[d_{M}(F,N)\leq c_{M}\sqrt{\sum_{l,m=1}^{d}\left\{\mathbb{E}\left[\left(\langle \nabla_{l,\cdot}^{2}F,\nabla_{m,\cdot}^{2}F\rangle\right)^{2}\right]\right\}^{1/ 2}\left\{\mathbb{E}\left[\left(\nabla_{l}F\nabla_{m}F\right)^{2}\right]\right\} ^{1/2}}, \tag{5}\]

_where \(\langle\cdot,\cdot\rangle\) is the scalar product, \(M\in\{TV,KS,W_{1}\},c_{TV}=\frac{4}{\sigma^{2}},c_{KS}=\frac{2}{\sigma^{2}},c _{W_{1}}=\sqrt{\frac{8}{\sigma^{2}\pi}}\) and \(\nabla_{i,\cdot}^{2}F\) is the \(i\)-th row of the Hessian matrix of \(F=f(X)\) while \(\nabla_{i}F\) is the \(i\)-th element of the gradient of \(F\)._

The next theorem generalizes Theorem 2.2 to multidimensional functionals. In particular, for any \(p>1\), the next theorem states Vidotto (2020, Theorem 2.3) for the special case of a function \((f_{1}(X),\ldots,f_{p}(X))\), with \(f_{1},\ldots,f_{p}\in C^{2}\left(\mathbb{R}^{d}\right)\) such that its partial derivatives have sub-exponential growth, and \(X\sim\mathcal{N}\left(0,I_{d\times d}\right)\). See Appendix A for a brief overview of Vidotto (2020, Theorem 2.3).

**Theorem 2.3** (Vidotto (2020) - \(p\)-dimensional second-order Poincare inequality).: _For any \(p>1\) let \((F_{1},\ldots,F_{p})=(f_{1}(X),\ldots,f_{p}(X))\), for some \(f_{1},\ldots,f_{p}\in C^{2}(\mathbb{R}^{d})\), and \(X\sim\mathcal{N}\left(0,I_{d\times d}\right)\) such that \(E\left[F_{i}\right]=0\) for \(i=1,\ldots,p\) and \(E\left[F_{i}F_{j}\right]=c_{ij}\) for \(i,j=1,\ldots,p\), with \(C=\{c_{ij}\}_{i,j=1,\ldots,p}\) being a symmetric and positive definite matrix, i.e. a variance-covariance matrix. Let \(N\sim\mathcal{N}(0,C)\), then_

\[d_{W_{1}}(F,N) \tag{6}\] \[\quad\leq 2\sqrt{p}\left\|C^{-1}\right\|_{2}\|C\|_{2}\sqrt{\sum_{i, k=1}^{p}\sum_{l,m=1}^{d}\left\{\mathbb{E}\left[\left(\langle\nabla_{l,\cdot}^{2}F_{i },\nabla_{m,\cdot}^{2}F_{i}\rangle\right)^{2}\right]\right\}^{1/2}\left\{ \mathbb{E}\left[\left(\nabla_{l}F_{k}\nabla_{m}F_{k}\right)^{2}\right]\right\} ^{1/2}}\]

_where \(\left\|\cdot\right\|_{2}\) is the spectral norm of a matrix._

## 3 Main results

In this section, we present the main result of the paper, namely a non-asymptotic Gaussian approximation of the NN (1), quantifying the approximation error with respect to the 1-Wasserstein distance, the total variation distance and the Kolmogorov-Smirnov distance. It is useful to start with the simple setting of a Gaussian NN with a 1-dimensional unitary input, i.e. \(d=1\) and \(x=1\), unit variance's weight, i.e. \(\sigma_{w}^{2}=1\), and no biases, i.e. \(b_{i}^{(0)}=b=0\) for any \(i\geq 1\). That is, we consider the NN

\[F:=f_{1}(n)[\tau,n^{-1/2}]=\frac{1}{n^{1/2}}\sum_{j=1}^{n}w_{j}\tau(w_{j}^{(0 )}). \tag{7}\]

By means of a straightforward calculation, one has \(\mathbb{E}[F]=0\) and \(\mathrm{Var}[F]=\mathbb{E}_{Z\sim\mathcal{N}(0,1)}[\tau^{2}(Z)]\). As \(F\) in (7) is a function of independent standard Gaussian RVs, Theorem 2.2 can be applied to approximate \(F\) with a Gaussian RV with the same mean and variance as \(F\), quantifying the approximation error.

**Theorem 3.1**.: _Let \(F\) be the NN (7) with \(\tau\in C^{2}(\mathbb{R})\) such that \(|\tau(x)|\leq a+b|x|^{\gamma}\) and \(\left|\frac{d}{dx^{2}}\tau(x)\right|\leq a+b|x|^{\gamma}\) for \(l=1,2\) and some \(a,b,\gamma\geq 0\). If \(N\sim\mathcal{N}(0,\sigma^{2})\) with \(\sigma^{2}=\mathbb{E}_{Z\sim\mathcal{N}(0,1)}[\tau^{2}(Z)]\), then for any \(n\geq 1\)_

\[d_{M}\left(F,N\right)\leq\frac{c_{M}}{\sqrt{n}}\sqrt{3(1+\sqrt{2})}\cdot\|a+b |Z|^{\gamma}\|_{L_{4}}^{2}, \tag{8}\]
_where \(Z\sim\mathcal{N}(0,1)\), \(M\in\{TV,KS,W_{1}\}\), with corresponding constants \(c_{TV}=4/\sigma^{2}\), \(c_{KS}=2/\sigma^{2}\), and \(c_{W_{1}}=\sqrt{8/\sigma^{2}\pi}\)._

Proof.: To apply Theorem 2.2, we start by computing some first and second order partial derivatives. That is,

\[\begin{cases}\frac{\partial F}{\partial w_{j}}=n^{-1/2}\tau(w_{j}^{(0)})\\ \\ \frac{\partial F}{\partial w_{j}^{(0)}}=n^{-1/2}w_{j}\tau^{\prime}(w_{j}^{(0)}) \\ \\ \nabla_{w_{j},w_{i}}^{2}F=0\\ \\ \nabla_{w_{j}^{(0)},w_{i}^{(0)}}^{2}F=n^{-1/2}\tau^{\prime}(w_{j}^{(0)}) \delta_{ij}\\ \\ \nabla_{w_{j}^{(0)},w_{i}^{(0)}}^{2}F=n^{-1/2}w_{j}\tau^{\prime\prime}(w_{j}^{(0 )})\delta_{ij}\end{cases}\]

with \(i,j=1\ldots n\). Then, by a direct application of Theorem 2.2, we obtain the following preliminary estimate

\[d_{M}\left(F,N\right) \leq c_{M}\Bigg{\{}\sum_{j=1}^{n}2\left\{\mathbb{E}\left[\left( \left\langle\nabla_{w_{j}}^{2}.F,\nabla_{w_{j}^{(0)},\cdot}^{2}F\right\rangle \right)^{2}\right]\mathbb{E}\left[\left(\frac{\partial F}{\partial w_{j}} \frac{\partial F}{\partial w_{j}^{(0)}}\right)^{2}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\langle\nabla_{w_{j}}^{ 2}.F,\nabla_{w_{j},\cdot}^{2}F\right\rangle\right)^{2}\right]\mathbb{E}\left[ \left(\frac{\partial F}{\partial w_{j}}\frac{\partial F}{\partial w_{j}} \right)^{2}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\langle\nabla_{w_{j}^{ (0)},\cdot}^{2}.F,\nabla_{w_{j}^{(0)},\cdot}^{2}F\right\rangle\right)^{2} \right]\mathbb{E}\left[\left(\frac{\partial F}{\partial w_{j}^{(0)}}\frac{ \partial F}{\partial w_{j}^{(0)}}\right)^{2}\right]\right\}^{1/2}\Bigg{\}}^{1/2},\]

which can be further developed. In particular, we can write the right-hand side of the previous estimate as

\[c_{M}\bigg{\{}\sum_{j=1}^{n}2\left\{\mathbb{E}\left[\left(\frac{ 1}{n}w_{j}\tau^{\prime}\left(w_{j}^{(0)}\right)\tau^{\prime\prime}\left(w_{j }^{(0)}\right)\right)^{2}\right]\mathbb{E}\left[\left(\frac{1}{n}w_{j}\tau \left(w_{j}^{(0)}\right)\tau^{\prime}\left(w_{j}^{(0)}\right)\right)^{2} \right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\frac{1}{\sqrt{n}}\tau^{ \prime}\left(w_{j}^{(0)}\right)\right)^{4}\right]\mathbb{E}\left[\left(\frac{ 1}{\sqrt{n}}\tau\left(w_{j}^{(0)}\right)\right)^{4}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\frac{1}{n}\left\{\tau^{ \prime}\left(w_{j}^{(0)}\right)\right\}^{2}+\frac{1}{n}w_{j}^{2}\left\{\tau^{ \prime\prime}\left(w_{j}^{(0)}\right)\right\}^{2}\right)^{2}\right]\mathbb{E }\left[\left(\frac{1}{\sqrt{n}}w_{j}\tau^{\prime}\left(w_{j}^{(0)}\right) \right)^{4}\right]\right\}^{1/2}\bigg{\}}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\tau^{\prime}\left(w_{j}^{(0) }\right)\right)^{4}\right]\mathbb{E}\left[\left(\tau\left(w_{j}^{(0)}\right) \right)^{4}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\{\tau^{\prime}\left(w _{j}^{(0)}\right)\right\}^{2}+w_{j}^{2}\left\{\tau^{\prime\prime}\left(w_{j}^{(0 )}\right)\right\}^{2}\right)^{2}\right]\mathbb{E}\left[\left(w_{j}\tau^{ \prime}\left(w_{j}^{(0)}\right)\right)^{4}\right]\right\}^{1/2}\bigg{\}}^{1/2}\]
\[\stackrel{{(idid)}}{{=}}\frac{c_{M}}{\sqrt{n}}\bigg{\{}2 \left\{\mathbb{E}\left[\left(\tau^{\prime}\left(Z\right)\tau^{\prime\prime}\left(Z \right)\right)^{2}\right]\mathbb{E}\left[\left(\tau\left(Z\right)\tau^{\prime} \left(Z\right)\right)^{2}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\tau^{\prime}\left(Z\right) \right)^{4}\right]\mathbb{E}\left[\left(\tau\left(Z\right)\right)^{4}\right] \right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\{\tau^{\prime}\left(Z \right)\right\}^{2}+w_{j}^{2}\left\{\tau^{\prime\prime}\left(Z\right)\right\}^ {2}\right)^{2}\right]\mathbb{E}\left[\left(w_{j}\tau^{\prime}\left(Z\right) \right)^{4}\right]\right\}^{1/2}\bigg{\}}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\{\tau^{\prime}\left(Z \right)\right\}^{4}\right]\mathbb{E}\left[\left(\tau\left(Z\right)\right)^{4} \right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(\left\{\tau^{\prime}\left(Z \right)\right\}^{4}\right]+2\mathbb{E}\left[\left\{\tau^{\prime}\left(Z \right)\right\}^{2}\left\{\tau^{\prime\prime}\left(Z\right)\right\}^{2}\right] +3\mathbb{E}\left[\left\{\tau^{\prime\prime}\left(Z\right)\right\}^{4}\right] \right)3\mathbb{E}\left[\left\{\tau^{\prime}\left(Z\right)\right\}^{4} \right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left|\tau^{\prime}\left(Z\right) \right|^{4}\right]\mathbb{E}\left[\left|\tau\left(Z\right)\right|^{4}\right] \right\}^{1/2}\] \[\quad+\left\{\left(\mathbb{E}\left[\left|\tau^{\prime}\left(Z \right)\right|^{4}\right]+2\mathbb{E}\left[\left|\tau^{\prime}\left(Z\right) \right|^{2}\right]+3\mathbb{E}\left[\left|\tau^{\prime\prime}\left(Z\right) \right|^{4}\right]\right)3\mathbb{E}\left[\left|\tau^{\prime}\left(Z\right) \right|^{4}\right]\right\}^{1/2}\bigg{\}}^{1/2},\]

where \(Z\sim\mathcal{N}(0,1)\). Now, since \(\tau\) is polynomially bounded and the square root is an increasing function,

\[d_{M}\left(F,N\right)\leq\frac{c_{M}}{\sqrt{n}}\bigg{\{}2 \left\{\mathbb{E}\left[\left(a+b|Z|^{\gamma}\right)^{4}\right]\mathbb{E} \left[\left(a+b|Z|^{\gamma}\right)^{4}\right]\right\}^{1/2}\] \[\quad+\left\{\mathbb{E}\left[\left(a+b|Z|^{\gamma}\right)^{4} \right]\mathbb{E}\left[\left(a+b|Z|^{\gamma}\right)^{4}\right]\right\}^{1/2}\] \[\quad+\left\{18\mathbb{E}\left[\left(a+b|Z|^{\gamma}\right)^{4} \right]\mathbb{E}\left[\left(a+b|Z|^{\gamma}\right)^{4}\right]\right\}^{1/2} \bigg{\}}^{1/2}\] \[=\frac{c_{M}}{\sqrt{n}}\sqrt{3\sqrt{2+3}}\left\{\mathbb{E}\left[ \left(a+b|Z|^{\gamma}\right)^{4}\right]\right\}^{1/2}\] \[=\frac{c_{M}}{\sqrt{n}}\sqrt{3(1+\sqrt{2})}\|a+b|Z|^{\gamma}\|_{L _{4}}^{2},\]

where \(Z\sim\mathcal{N}(0,1)\). 

The proof of Theorem 3.1 shows how a non-asymptotic approximation of \(F\) can be obtained by a direct application of Theorem 2.2. In particular, the estimate (8) of the approximation error \(d_{M}\left(F,N\right)\) has the optimal rate \(n^{-1/2}\) with respect to the \(1\)-Wasserstein distance, the total variation distance and the Kolmogorov-Smirnov distance. As for the constant, it depends on the variance \(\mathbb{E}_{Z\sim\mathcal{N}(0,1)}[\tau^{2}(Z)]\) of \(F\). Once the activation function \(\tau\) is specified, \(\mathbb{E}_{Z\sim\mathcal{N}(0,1)}[\tau^{2}(Z)]\) can be evaluated by means of an exact or approximate calculation, or a suitable lower bound for it can be provided.
Now, we extend Theorem 3.1 to the more general case of the Gaussian NN (1), showing that the problem still reduces to an application of Theorem 2.2. In particular, it is convenient to write (1) as follows:

\[F:=\frac{1}{n^{1/2}}\sigma_{w}\sum_{j=1}^{n}w_{j}\tau(\sigma_{w} \langle w_{j}^{(0)},\mathbf{x}\rangle+\sigma_{b}b_{j}^{(0)})+\sigma_{b}b, \tag{9}\]

with \(w_{j}^{(0)}=[w_{j,1}^{(0)},\ldots,w_{j,d}^{(0)}]^{T}\) and \(w_{j}\overset{d}{=}w_{j,i}^{(0)}\overset{\text{iid}}{\sim}\ \mathcal{N}(0,1)\). We set \(\Gamma^{2}=\sigma_{w}^{2}\|\mathbf{x}\|^{2}+\sigma_{b}^{2}\), and for \(n\geq 1\) we consider a collection \((Y_{1},\ldots,Y_{n})\) of independent standard Gaussian RVs. Then, from (9) we can write

\[F\overset{d}{=}\frac{1}{n^{1/2}}\sigma_{w}\sum_{j=1}^{n}w_{j}\tau \left(\Gamma Y_{j}\right)+\sigma_{b}b.\]

As before, straightforward calculations leads to \(\mathbb{E}[F]=0\) and \(\operatorname{Var}[F]=\sigma_{w}^{2}\mathbb{E}_{Z\sim\mathcal{N}(0,1)}\left[ \tau^{2}\left(\Gamma Z\right)\right]+\sigma_{b}^{2}\). As \(F\) in (9) is a function of independent standard Gaussian RVs, Theorem 2.2 can be applied to approximate \(F\) with a Gaussian RV with the same mean and variance as \(F\), quantifying the approximation error. This approximation is stated in the next theorem, whose proof is in Appendix B.

**Theorem 3.2**.: _Let \(F\) be the NN (9) with \(\tau\in C^{2}(\mathbb{R})\) such that \(|\tau(x)|\leq a+b|x|^{\gamma}\) and \(\left|\frac{d}{dx^{2}}\tau(x)\right|\leq a+b|x|^{\gamma}\) for \(l=1,2\) and some \(a,b,\gamma\geq 0\). If \(N\sim\mathcal{N}(0,\sigma^{2})\) with \(\sigma^{2}=\sigma_{w}^{2}\mathbb{E}_{Z\sim\mathcal{N}(0,1)}\left[\tau^{2} \left(\Gamma Z\right)\right]+\sigma_{b}^{2}\) and \(\Gamma=(\sigma_{w}^{2}\|\mathbf{x}\|^{2}+\sigma_{b}^{2})^{1/2}\), then for any \(n\geq 1\)_

\[d_{M}\left(F,N\right)\leq\frac{c_{M}\sqrt{\Gamma^{2}+\Gamma^{4}(2 +\sqrt{3(1+2\Gamma^{2}+3\Gamma^{4})})}\|a+b|\Gamma Z|^{\gamma}\|_{L^{4}}^{2}} {\sqrt{n}}, \tag{10}\]

_where \(Z\sim\mathcal{N}(0,1)\), \(M\in\{TV,KS,W_{1}\}\), with corresponding constants \(c_{TV}=4/\sigma^{2},c_{KS}=2/\sigma^{2},c_{W_{1}}=\sqrt{8/\sigma^{2}\pi}\)._

We observe that Theorem 3.1 can be recovered from Theorem 3.2. In particular, the estimate (8) of the approximation \(d_{M}\left(F,N\right)\) can be recovered from the estimate (10) by setting \(\sigma_{b}=0\), \(\sigma_{w}=1\) and \(\mathbf{x}=1\). As for Theorem 3.1, the constant depends on the variance \(\sigma_{w}^{2}\mathbb{E}_{Z\sim\mathcal{N}(0,1)}\left[\tau^{2}\left(\Gamma Z \right)\right]+\sigma_{b}^{2}\) of \(F\). Therefore, to apply Theorem 3.2 one needs to evaluate the variance of \(F\), by means of an exact or approximate calculation, or to provide a suitable lower bound for it, as we have discussed previously.

We conclude by presenting an extension of Theorem 3.2 to a Gaussian NN with \(p>1\) inputs \([\mathbf{x_{1}},\ldots,\mathbf{x_{p}}]^{T}\), where \(\mathbf{x_{i}}\in\mathbb{R}^{d}\) for \(i=1,\ldots,p\). In particular, we consider the NN \(F:=[F_{1},\ldots,F_{p}]^{T}\) where

\[F_{i}:=\frac{1}{n^{1/2}}\sigma_{w}\sum_{j=1}^{n}w_{j}\tau(\sigma_ {w}\langle w_{j}^{(0)},\mathbf{x_{i}}\rangle+\sigma_{b}b_{j}^{(0)})+\sigma_{b}b, \tag{11}\]

with \(w_{j}^{(0)}=[w_{j,1}^{(0)},\ldots,w_{j,d}^{(0)}]^{T}\) and \(w_{j}\overset{d}{=}w_{j,i}^{(0)}\overset{d}{=}b_{j}^{(0)}\overset{d}{=}b \overset{\text{iid}}{\sim}\ \mathcal{N}(0,1)\). Since the parameter are jointly distributed according to multivariate standard Gaussian distribution, Theorem 2.3 can be applied to approximate \(F\) with a Gaussian random vector whose mean and covariance are the same as \(F\). The resulting estimate of the approximation error depends on the maximum and the minimum eigenvalues, i.e. \(\lambda_{1}(C)\) and \(\lambda_{p}(C)\) respectively, of the covariance matrix \(C\), whose \((i,k)\)-th entry is given by

\[\mathbb{E}[F_{i}F_{k}]=\sigma_{w}^{2}\mathbb{E}[\tau(Y_{i})\tau(Y_{k})]+\sigma _{b}^{2}, \tag{12}\]
where \(Y\sim\mathcal{N}(0,\sigma_{w}^{2}X^{T}X+\sigma_{b}^{2}\mathbf{1}\mathbf{1}^{T})\), with \(\mathbf{1}\) being the all-one vector of dimension \(p\) and \(X\) being the \(n\times p\) matrix of the inputs \(\{\mathbf{x_{i}}\}_{i\in[p]}\). This approximation is stated in the next theorem, whose proof is in Appendix C.

**Theorem 3.3**.: _Let \(F=[F_{1}\ldots,F_{p}]^{T}\) with \(F_{i}\) being the NN (11), for \(i=1,\ldots,p\), with \(\tau\in C^{2}(\mathbb{R})\) such that \(|\tau(x)|\leq a+b|x|^{\gamma}\) and \(\left|\frac{d}{dx^{t}}\tau(x)\right|\leq a+b|x|^{\gamma}\) for \(l=1,2\) and some \(a,b,\gamma\geq 0\). Furthermore, let \(C\) be the covariance matrix of \(F\), whose entries are given in (12), and define \(\Gamma_{i}^{2}=\sigma_{w}^{2}||\mathbf{x_{i}}||^{2}+\sigma_{b}^{2}\) and \(\Gamma_{ik}=\sigma_{w}^{2}\sum_{j=1}^{d}|x_{ij}x_{kj}|+\sigma_{b}^{2}\). If \(N=[N_{1},\cdots N_{p}]^{T}\sim\mathcal{N}(0,C)\), then for any \(n\geq 1\)_

\[d_{W_{1}}\left(F,N\right)\leq 2\sigma_{w}^{2}\tilde{K}\frac{\lambda_{1}(C)}{ \lambda_{p}(C)}\sqrt{\frac{p}{n}}, \tag{13}\]

_where \(\lambda_{1}(C)\) and \(\lambda_{p}(C)\) are the maximum and the minimum eigenvalues of \(C\), respectively, and where_

\[\tilde{K}=\bigg{\{}\sum_{i,k=1}^{p}(\Gamma_{i}^{2}+\sqrt{3(1+2\Gamma_{i}^{2}+3 \Gamma_{i}^{4})}\Gamma_{ik}^{2}+2\Gamma_{i}^{2}\Gamma_{ik})\|a+b|\Gamma_{i}Z| ^{\gamma}\|_{L^{4}}^{2}\|a+b|\Gamma_{k}Z|^{\gamma}\|_{L^{4}}^{2}\bigg{\}}^{1/2},\]

_with \(Z\sim\mathcal{N}(0,1)\)._

The estimate (13) of the approximation error \(d_{W_{1}}\left(F,N\right)\) depends on the spectral norms of the covariance matrix \(C\) and the precision matrix \(C^{-1}\). Such spectral norms must be computed explicitly for the specific activation \(\tau\) in use, or at least bounded from above, in order to apply Theorem 3.3. This boils down to finding the greatest eigenvalue \(\lambda_{1}\) and the smallest eigenvalue \(\lambda_{p}\) of the matrix \(C\), which can be done for a broad class of activations with classical optimization techniques, or at least bounding \(\lambda_{1}\) from above and \(\lambda_{p}\) from below (Diaconis and Stroock, 1991; Guattery et al., 1999).

## 4 Numerical illustrations

In this section, we present a brief simulation study for two specific choices of the activation function: i) \(\tau(x)=\tanh x\), which is polynomially bounded with parameters \(a=1\) and \(b=0\); ii) \(\tau(x)=x^{3}\), which is polynomially bounded with parameters \(a=6\), \(b=1\) and \(\gamma=3\). Each of the plots below is obtained as follows: for a fixed width of \(n=k^{3}\), with \(k\in\{1,\cdots,16\}\), we simulate 5000 points from a SLNN as in Theorem 3.1 to produce an estimate of the distance between the NN and a Gaussian RV with mean \(0\) and variance \(\sigma^{2}\), which is estimated by means of a Monte-Carlo approach. Estimates of the KS and TV distance are produced by means of the functions _KolmogorovDist_ and _TotVarDist_ from the package **distEx** by Ruckdeschel et al. (2006) while those of the 1-Wasserstein distance using the function _wasserstein1d_ from the package **transport** by Schuhmacher et al. (2022). We repeat this procedure 500 times for every fixed \(n=k^{3}\) with \(k\in\{1,\cdots,16\}\), compute the sample mean (black dots) and the 2.5-th and the 97.5-th sample percentiles (red dashed lines), and compare these estimates with the theoretical bound given by Theorem 3.1 (blue line).

The plots confirm that the distance between a shallow NN and an arbitrary Gaussian RV, with the same mean and variance, is asymptotically bounded from above by \(n^{-1/2}\) and that the approximation gets better and better as \(n\to\infty\). This is evident in the case \(\tau(x)=x^{3}\), where there is a clear decay between \(n=1\) and \(n=1000\). This behaviour does not show up for \(\tau(x)=\tanh x\), since \(\tanh x\sim x\), for \(x\to 0\), and Gaussian RVs are more likely to attain values in a neighbourhood of zero.
Figure 1: Estimates of the Kolmogorov-Smirnov distance for a Shallow NN of varying width \(n=k^{3}\), \(k\in\{1,\cdots,16\}\), with \(\tau(x)=\tanh x\) (left) and \(\tau(x)=x^{3}\) (right). The blue line is the theoretical bound of Theorem 3.1, the black dots are sample means of the Monte-Carlo sample, while the red-dashed lines represent a \(95\%\) sample confidence interval.

Figure 3: Estimates of the 1-Wasserstein distance for a Shallow NN of varying width \(n=k^{3}\), \(k\in\{1,\cdots,16\}\), with \(\tau(x)=\tanh x\) (left) and \(\tau(x)=x^{3}\) (right).

Figure 2: Estimates of the Total Variation distance for a Shallow NN of varying width \(n=k^{3}\), \(k\in\{1,\cdots,16\}\), with \(\tau(x)=\tanh x\) (left) and \(\tau(x)=x^{3}\) (right).
Discussion

We introduced some non-asymptotic Gaussian approximations of Gaussian NNs, quantifying the approximation error with respect to the 1-Wasserstein distance, the total variation distance and the Kolmogorov-Smirnov distance. As a novelty, our work relies on the use of second-order Poincare inequalities, which lead to estimates of the approximation error with optimal rate and tight constants. This is the first work to make use of second-order Poincare inequalities for non-asymptotic Gaussian approximations of Gaussian NNs. For a Gaussian NN with a single input, the estimate in Theorem 3.2 requires to evaluate or estimate, whereas for a Gaussian NN with inputs, the estimate in Theorem 3.3 requires to evaluate or estimate. Our approach based on second-order Poincare inequalities remains valid in the more general setting of deep Gaussian NNs. Both Theorem 3.2 and Theorem 3.3 can be extended to deep Gaussian NNs, at the cost of more involved algebraic calculations, as well as more involved estimates of the approximation errors. For instance, for an input one may consider a deep Gaussian NN with layers, i.e.

(14)

with

(15)

and apply Theorem 2.2 to as defined in (14) and (15). Such an application implies to deals with complicated expressions of the gradient and the Hessian that, however, is a purely algebraic problem.

Related to the choice of the activation, one can also try to relax the hypothesis of polynomially boundedness and use a whatever. There is nothing wrong in doing it, as Corollary 2.2 and 2.3 still apply, with the only difference that the bound would be less explicit than the one we found here. Furthermore, one could also think about relaxing the hypothesis to include or just continuous activations, like the famous ReLU function (i.e. ) which is excluded from our analysis. Some results in this direction can be found in Eldan et al. (2021), though using Rademacher weights for the hidden layer. In this regard, we try to derive a specific bound for the ReLU function applying Corollary 2.2 to a sequence of smooth approximating functions and then passing to the limit. In particular, we approximated the ReLU function with for and applied Theorem 2.2 to a generic using the 1-Wasserstein distance and obtained a bound dependent on. Then, the idea would have been to take the limit of this bound for and hopefully obtain a non-trivial
bound, but that was not the case as the limit exploded. The same outcome was found using the SAU approximating sequence, i.e.

\[H(m,x):=\frac{1}{m\sqrt{2\pi}}\exp\biggl{\{}-\frac{1}{2}m^{2}x^{2}\biggr{\}}+ \frac{x}{2}+\frac{x}{2}\operatorname{erf}\left\{\frac{mx}{\sqrt{2}}\right\},\]

where \(\operatorname{erf}\left(\cdot\right)\) denotes the error function. This fact probably indicates the impossibility to apply the results of Vidotto (2020) in the context of continuous activation functions as the ReLU function, and the necessity to come up with new results on second-order Poincare inequalities to fill this gap. These results would not be trivial aft all, since Theorem A.2 needs each \(F_{1},\ldots,F_{d}\) to be in \(\mathbb{D}^{2,4}\), and so two degrees of smoothness are required. This is not "the fault" of Vidotto (2020), but it is due to the intrinsic character of the equation \(f^{\prime\prime}(x)-xf^{\prime}(x)=h(x)-Eh(Z)\) with \(Z\sim\mathcal{N}(0,1)\) in dimension \(p\geq 2\).

## Acknowledgements

Stefano Favaro is grateful to Professor Larry Goldstein for having introduced him to second-order Poincare inequalities, and to Professor Dario Trevisan for the many stimulating conversations on Gaussian NNs. Stefano Favaro received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No 817257. Stefano Favaro is also affiliated to IMATI-CNR "Enrico Magenes" (Milan, Italy).
