Deterministic equivalent of the Conjugate Kernel matrix associated to
  Artificial Neural Networks
We study the Conjugate Kernel associated to a multi-layer linear-width
feed-forward neural network with random weights, biases and data. We show that
the empirical spectral distribution of the Conjugate Kernel converges to a
deterministic limit. More precisely we obtain a deterministic equivalent for
its Stieltjes transform and its resolvent, with quantitative bounds involving
both the dimension and the spectral parameter. The limiting equivalent objects
are described by iterating free convolution of measures and classical matrix
operations involving the parameters of the model.
ClÃ©ment Chouard
# Deterministic equivalent of the conjugate kernel matrix associated to artificial neural networks

###### Abstract.

We study the Conjugate Kernel associated to a multi-layer linear-width feed-forward neural network with random weights, biases and data. We show that the empirical spectral distribution of the Conjugate Kernel converges to a deterministic limit. More precisely we obtain a deterministic equivalent for its Stieltjes transform and its resolvent, with quantitative bounds involving both the dimension and the spectral parameter. The limiting equivalent objects are described by iterating free convolution of measures and classical matrix operations involving the parameters of the model.

**Acknowledgments:** This work received support from the University Research School EUR-MINT (reference number ANR-18-EURE-0023).
###### Contents

* 1 Introduction
	* 1.1 Overview of the article
	* 1.2 General notations and definitions
		* 1.2.1 Concentration framework
		* 1.2.2 Polynomial bounds in \(z\) and notation \(O_{z}(\epsilon_{n})\)
		* 1.2.3 Covariance matrices of functions of Gaussian vectors
		* 1.2.4 \(O_{z}(\epsilon_{n})\)
		* 1.2.5 \(O_{z}(\epsilon_{n})\)
		* 1.2.6 \(O_{z}(\epsilon_{n})\)
		* 1.2.7 \(O_{z}(\epsilon_{n})\)
		* 1.2.8 \(O_{z}(\epsilon_{n})\)
		* 1.2.9 \(O_{z}(\epsilon_{n})\)
	* 1.3 \(O_{z}(\epsilon_{n})\)
	* 1.4 \(O_{z}(\epsilon_{n})\)
	* 1.5 \(O_{z}(\epsilon_{n})\)
	* 1.6 \(O_{z}(\epsilon_{n})\)
	* 1.7 \(O_{z}(\epsilon_{n})\)
	* 1.8 \(O_{z}(\epsilon_{n})\)
	* 1.9 \(O_{z}(\epsilon_{n})\)
* 1.10 \(O_{z}(\epsilon_{n})\)
* 1.11 \(O_{z}(\epsilon_{n})\)
* 1.12 \(O_{z}(\epsilon_{n})\)
* 1.13 \(O_{z}(\epsilon_{n})\)
* 1.14 \(O_{z}(\epsilon_{n})\)
* 1.15 \(O_{z}(\epsilon_{n})\)
* 1.16 \(O_{z}(\epsilon_{n})\)
* 1.17 \(O_{z}(\epsilon_{n})\)
* 1.18 \(O_{z}(\epsilon_{n})\)
* 1.19 \(O_{z}(\epsilon_{n})\)
* 1.20 \(O_{z}(\epsilon_{n})\)
* 1.21 \(O_{z}(\epsilon_{n})\)
* 1.22 \(O_{z}(\epsilon_{n})\)
* 1.23 \(O_{z}(\epsilon_{n})\)
* 1.24 \(O_{z}(\epsilon_{n})\)
* 1.25 \(O_{z}(\epsilon_{n})\)
* 1.26 \(O_{z}(\epsilon_{n})\)
* 1.27 \(O_{z}(\epsilon_{n})\)
* 1.28 \(O_{z}(\epsilon_{n})\)
* 1.29 \(O_{z}(\epsilon_{n})\)
* 1.30 \(O_{z}(\epsilon_{n})\)
* 1.31 \(O_{z}(\epsilon_{n})\)
* 1.32 \(O_{z}(\epsilon_{n})\)
* 1.33 \(O_{z}(\epsilon_{n})\)
* 1.34 \(O_{z}(\epsilon_{n})\)
* 1.35 \(O_{z}(\epsilon_{n})\)
* 1.36 \(O_{z}(\epsilon_{n})\)
* 1.37 \(O_{z}(\epsilon_{n})\)
* 1.38 \(O_{z}(\epsilon_{n})\)
* 1.39 \(O_{z}(\epsilon_{n})\)
* 1.40 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.40 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.40 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.40 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.40 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.41 \(O_{z}(\epsilon_{n})\)
* 1.42 \(O_{z}(\epsilon_{n})\)
* 1.43 \(O_{z}(\epsilon_{n})\)
* 1.44 \(O_{z}(\epsilon_{n})\)
* 1.45 \(O_{z}(\epsilon_{n})\)
* 1.46 \(O_{z}(\epsilon_{n})\)
* 1.47 \(O_{z}(\epsilon_{n})\)
* 1.48 \(O_{z}(\epsilon_{n})\)
* 1.49 \(O_{z}(\epsilon_{n})\)
## 1. Introduction

Artificial Neural Networks are computing systems inspired by a simplified model of interconnected neurons, storing and exchanging information inside a biological brain. Theorized in the late fifties [11], this class of algorithms only established oneself as a cutting-edge technology in the two thousands, thanks to the ever-increasing computing power, and perhaps even more thanks to the accessibility of huge databases in the modern-age internet. Compared to the delicate craftsmanship of artificial intelligence specialists, the mathematical understanding of these networks remained at a very basic level until recently. The classical tools of random matrix theory in particular were helpless regarding the intrinsic non-linear connections between the artificial neurons.

This article is concerned about the Conjugate Kernel model, first introduced from the point of view of random matrices in [14]. This model is a random matrix ensemble that corresponds to a multi-layer feed-forward neural network with weights initialized at random, in the asymptotic regime where the network width grows linearly with the sample size. In this setting, the Conjugate Kernel matrix is simply the sample covariance matrix of the output vectors produced by the final layer of the network. It can be shown that the Conjugate Kernel governs the training and generalization properties of the underlying network ([1], [15]). Its spectral properties are thus of great theoretical and practical interest.

The limiting spectral distribution of the Conjugate Kernel associated to a single-layer network was first characterized in [14] under the form of a quartic polynomial equation satisfied by the Stieltjes transform of the limiting measure. This was proven rigorously in [1] under more general assumptions, using advanced combinatorics. Some results on the extremal eigenvalues of the model were even obtained in [1]. [10] first noticed a connection between these equations and the measures obtained from a free multiplicative convolution with some Marcenko-Pastur distributions. Let us also mention the work of [11], where a variant of model including a rank-one additive perturbation is examined, also by means of combinatorial techniques.

A first step towards a deeper understanding of the model using analytical techniques was done in [13] in the case of a single-layer network with deterministic data. [14] later generalized this result to random data and multi-layer networks, and was the first to convincingly explain the appearance of free probability convolutions in the limiting spectral distribution of the Conjugate Kernel matrix. A similar analysis was done in [15] in the regime where the network width is much larger than the sample size.

The aforementioned results may be classified as global laws, in the sense that they establish the existence of a limiting spectral distribution, a problem that is directly linked to the convergence of the Stieltjes transform via the classical inversion formulas. The next natural question that arises in random matrix theory is to look for a deterministic equivalent of the resolvent of the Conjugate Kernel matrix in the sense of [12]. This consists in finding a deterministic matrix that is asymptotically close to the expected resolvent of the matrix, and thus close to the random resolvent itself provided
enough concentration in the random matrices. Establishing a deterministic equivalent result allows for instance to approximate linear statistics of the eigenvectors [10], to examine the convergence of the eigenvectors empirical spectral distributions, and possibly to study the outliers of spiked models [11]. Regarding classical sample covariance random ensembles, this task was done a decade ago in the series of articles [13], [1], [14]. Such results were recently partially extended to models with a general dependence structure in [12], [15].

The most important contribution of this article is Theorem 7.2, that provides a deterministic equivalent of the Conjugate Kernel of a multi-layer neural network model. This theorem extends previously known results in various directions. First we include models with non-differentiable activation functions, as well as potential biases inside or outside the activation function. Secondly we give a quantitative estimate for the convergence of the Stieltjes transforms, which translates into a quantitative convergence of the measures in Kolmogorov distance. Finally and most importantly, we obtain local results, taking the form of a deterministic equivalent for the resolvent matrix of the Conjugate Kernel, quantitatively on both the dimension and the spectral parameter.

In the rest of the paper, we also establish intermediary results that may be interesting by themselves. In particular we show that the free convolution of measures with a Marcenko-Pastur distribution is regular with respect to the Stieltjes transform of these distributions (Theorem 4.7). We obtain a similar property for the deterministic equivalent matrices that appear in our results (Proposition 4.11). We also show how our framework applies to other models involving entry-wise operations on random matrices in Section 5.5.

This paper relies mostly on analytical methods that study spectral functions of random matrices. We make great use of the theory of Stieltjes transforms and resolvent matrices, particularly its recent developments towards the deterministic equivalent of general sample covariance matrices ([12], [15]). We introduce a new notion of asymptotic equivalence for objects that depend both on a dimension and on a complex spectral parameter. Our definition is convenient to work with, whilst keeping enough precision to imply some quantitative results like the convergence of measures in Kolmogorov distance.

This analytical toolbox works in conjunction with concentration of measure principles. We follow the framework of [12], in particular we use the notion of Lipschitz concentration that is remarkably compatible with entry-wise operations on random matrices. We also use a linearization argument to study the covariance matrices of functions of weakly correlated Gaussian vectors. Using the theory of Hermite polynomials, we provide estimates for this approximation, not only on an entry-wise basis like it was done in [10], but also in spectral norm which is new to our knowledge.

After we released the first version of this manuscript, we came across the preprint [16], which we were not aware of. A future comparison of the two different perspectives would be certainly interesting, as [16] studies similar objects and seems to describe related phenomena to those considered here.
### Overview of the article

In Section 2, we remind the notations and basic properties of concentrated random vectors and matrices, following the framework of [10]. We also introduce the new notion of asymptotic equivalence that will be key in the rest of the article. In Section 3, we address the problem of the approximation the covariance matrices of functions of weakly correlated Gaussian vectors. In Section 4, we remind the general deterministic equivalent results on which this article is based, and we study thoroughly the properties of the matrices appearing in these equivalents. In Section 5, we study a first model of artificial neural network with a single layer and deterministic data. We also explain how our framework applies to other models involving entry-wise operations on random matrices (5.5). In Section 6, we analyze a second model of artificial neural network, still consisting of a single layer but with random data instead. In Section 7, we explain how to study multi-layer networks by induction on the model with one layer. In the Appendix 8, we prove an independent result about the convergence the convergence in Kolmogorov distance of some empirical spectral measures.

### General notations and definitions

The set of matrices with \(d\) lines, \(n\) columns, and entries belonging to a set \(\mathbb{K}\) is denoted as \(\mathbb{K}^{d\times n}\). We use the following norms for vectors and matrices: \(\left\|\cdot\right\|\) the Euclidean norm, \(\left\|\cdot\right\|_{F}\) the Frobenius norm, \(\left\|\cdot\right\|\) the spectral norm, and \(\left\|\cdot\right\|_{\max}\) the entry-wise maximum norm.

Given \(M\in\mathbb{C}^{n\times n}\), we denote by \(M^{\top}\) its transpose, and by \(\mathrm{Tr}(M)=\sum_{i=1}^{n}M_{ii}\) its trace. If \(M\) is real and diagonalizable we denote by \(\mathrm{Sp}M\) its spectrum and by \(\mu_{M}=\frac{1}{n}\sum_{\lambda\in\mathrm{Sp}M}\delta_{\lambda}\) its empirical spectral distribution. If \(M\) is symmetric positive semi-definite, then \(\mathrm{Sp}M\subset\mathbb{R}^{+}\), and given a spectral parameter \(z\in\mathbb{C}^{+}=\{\omega\in\mathbb{C}\text{ such that }\Im(\omega)>0\}\), we define its resolvent \(\mathcal{G}_{M}(z)=(M-zI_{n})^{-1}\) and its Stieltjes transform \(g_{M}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{M}(z)\).

If \(\mu\) is a real probability distribution, its Stieltjes transform is \(g_{\mu}(z)=\int_{\mathbb{R}}\frac{\mu(dt)}{t-z}\), well defined for \(z\in\mathbb{C}^{+}\). It is easy to see that the Stieltjes transform of a matrix is the same as the Stieltjes transform of its empirical spectral distribution, that is \(g_{\mu_{M}}(z)=g_{M}(z)\). We denote by \(\mathcal{F}_{\mu}\) be the cumulative distribution function of the measure \(\mu\), and by \(D(\mu,\nu)=\sup_{t\in\mathbb{R}}|\mathcal{F}_{\mu}(t)-\mathcal{F}_{\nu}(t)|\) the Kolmogorov distance between \(\mu\) and \(\nu\).

Let us also introduce two operations on measures: if \(\gamma\in\mathbb{R}\), we denote by \(\gamma\cdot\mu+(1-\gamma)\cdot\nu\) the new measure such that \((\gamma\cdot\mu+(1-\gamma)\cdot\nu)(B)=\gamma\mu(B)+(1-\gamma)\nu(B)\) for any measurable set \(B\) (it may be a signed measure). If \(a,b\in\mathbb{R}\), we denote by \(a+b\mu\) the distribution of the random variable \(a+bX\), where \(X\) is a random variable distributed according to \(\mu\).

We denote by \(\mathcal{N}\) a standard (centered and reduced) Gaussian random variable, or the law of this random variable depending on the context. We denote by \(\delta_{a}\) the Dirac delta distribution centered at \(a\in\mathbb{R}\), and by \(\mathrm{MP}(\gamma)\) the Marcenko-Pastur distribution with shape parameter \(\gamma>0\). The operator
\(\boxtimes\) denotes the multiplicative free convolution between measures ([4]). The distribution \(\operatorname{MP}(\gamma)\boxtimes\mu\) may be also defined by its Stieltjes transform \(g(z)\), which is the unique solution of the self-consistent equation [10]:

\[g(z)=\int_{\mathbb{R}}\frac{1}{(1-\gamma-\gamma zg(z))t-z}\mu(dt).\]

For a better readability, we will sometimes omit to mention indices \(n\) and spectral parameters \(z\) in our notations, especially in the course of technical proofs, even if we are dealing implicitly with sequences and complex functions.
## 2. Technical tools

### Concentration framework

**Definition 2.1**.: Let \(X_{n}\) be a sequence of random vectors in finite dimensional normed vector spaces \((E_{n},\left\|\cdot\right\|)\), and let \(\sigma_{n}>0\) be a sequence.

1. We say that \(X\) is Lipschitz concentrated if there is a constant \(C>0\) such that for any sequence of \(1\)-Lipschitz maps \(f_{n}:E_{n}\to\mathbb{C}\), for any \(n\in\mathbb{N}\) and \(t\geq 0\): \[\mathbb{P}(|f_{n}(X_{n})-\mathbb{E}[f_{n}(X_{n})]|\geq t)\leq Ce^{-\frac{1}{C }\left(\frac{t}{\sigma_{n}}\right)^{2}}.\] We note this Lipschitz concentration property \(X\propto_{\left\|\cdot\right\|}\mathcal{E}(\sigma_{n})\), or simply \(X\propto\mathcal{E}(\sigma_{n})\) when there is no ambiguity on the chosen norm.
2. If \(Z_{n}\in E_{n}\) is a sequence of deterministic vectors, we say that \(X\) is linearly concentrated around \(Z\) if there is a constant \(C>0\) such that for any sequence of \(1\)-Lipschitz linear maps \(f_{n}:E_{n}\to\mathbb{C}\), for any \(n\in\mathbb{N}\) and \(t\geq 0\): \[\mathbb{P}(|f_{n}(X_{n})-f_{n}(Z_{n})|\geq t)\leq Ce^{-\frac{1}{C}\left(\frac{ t}{\sigma_{n}}\right)^{2}}.\] We note this linear concentration property \(X\in_{\left\|\cdot\right\|}Z\pm\mathcal{E}(\sigma_{n})\), or simply \(X\in Z\pm\mathcal{E}(\sigma_{n})\).

We refer to [10] for a comprehensive study of these notions of concentration. Let us enumerate the main properties that will be used throughout this article:

**Proposition 2.2**.:
1. _A random vector (respectively a matrix) with i.i.d._ \(\mathcal{N}\) _entries is_ \(\propto\mathcal{E}(1)\) _concentrated with respect to the Euclidean norm (respectively the Frobenius norm). Other examples are listed in_ _[_10_, Theorem 1]__._
2. _A Lipschitz transformation of a Lipschitz concentrated vector still verifies a Lipschitz concentration property_ _[_10_, Proposition 1]__._
3. _If_ \(X\propto\mathcal{E}(\sigma_{n})\)_,_ \(Y\propto\mathcal{E}(\sigma_{n}^{\prime})\)_, and_ \(X\) _and_ \(Y\) _are independent, then_ \((X,Y)\propto\mathcal{E}(\sigma_{n}+\sigma_{n}^{\prime})\) _(_[_10_, Proposition 7]__)._
4. _Lipschitz concentration implies linear concentration: if_ \(X\propto\mathcal{E}(\sigma_{n})\)_, then_ \(X\in\mathbb{E}[X]\pm\mathcal{E}(\sigma_{n})\)_(_[_10_, Proposition 4]__). Both definitions are equivalent in one dimension._
5. _If_ \(X\in Z\pm\mathcal{E}(\sigma_{n})\) _and_ \(\tilde{Z}\) _is another deterministic vector, we have the equivalence:_ \(X\in\tilde{Z}\pm\mathcal{E}(\sigma_{n})\iff\left\|Z-\tilde{Z}\right\|\leq O( \sigma_{n})\) _(_[_10_, Lemma 3]__)._
6. _If_ \(X\in Z\pm\mathcal{E}(\sigma_{n})\) _and_ \(X^{\prime}\) _is another random vector such that_ \(\left\|X^{\prime}\right\|\leq O(\sigma_{n})\) _a.s., then_ \(X+X^{\prime}\in Z\pm\mathcal{E}(\sigma_{n})\)_._
7. _Since_ \(\left\|\cdot\right\|\leq\left\|\cdot\right\|_{F}\) _on matrices, the concentration with respect to the Frobenius norm is a stronger property than with the spectral norm._
8. _The map_ \(Y\mapsto\left(Y^{\top}Y/n-zI_{p}\right)^{-1}\) _is_ \(\frac{2^{3/2}|z|^{1/2}}{\Im(z)^{2}\sqrt{n}}\)_-Lipschitz with respect to the Frobenius norm (__[_11_, Proposition 4.3]__). In particular, if_ \(Y\propto_{\left\|\cdot\right\|_{F}}\mathcal{E}(1)\)_, then_ \(\mathcal{G}_{Y^{\top}Y/n}(z)\propto_{\left\|\cdot\right\|_{F}}\mathcal{E} \Bigg{(}\frac{|z|^{1/2}}{\Im(z)^{2}\sqrt{n}}\Bigg{)}\)_._
_._
9. _The map_ \(M\mapsto\frac{1}{n}\mathrm{Tr}(M)\) _is_ \(1/\sqrt{n}\)_-Lipschitz in Frobenius norm. In particular, if_ \(\mathcal{G}_{K}(z)\propto_{\left|\kern-1.075pt\left|\kern-1.075pt\left|\cdot\right| \kern-1.075pt\right|_{F}}\mathcal{E}(\sigma_{n})\) _for some random matrix_ \(K\in\mathbb{R}^{n\times n}\)_, then_ \(g_{K}(z)\propto\mathcal{E}(\sigma_{n}/\sqrt{n})\)_._
10. _If a random variable_ \(X\) _is_ \(\propto\mathcal{E}(\sigma_{n})\) _concentrated, then_ \(|X-\mathbb{E}[X]|\leq O(\sqrt{\log n}\,\sigma_{n})\) _a.s. (_[_10_, Proposition 3.3]__)._

We will also need a result for a product of matrices, and introduce for this reason a notion of conditional concentration. If \(\mathcal{B}\) is a measurable subset of the universe \(\Omega\) with \(\mathbb{P}(\mathcal{B})>0\), the random matrix \(X\) conditioned with the event \(\mathcal{B}\), denoted as \((X|\mathcal{B})\), designates the measurable mapping \((\mathcal{B},\mathcal{F}_{\mathcal{B}},\mathbb{P}/\mathbb{P}(\mathcal{B})) \mapsto E_{n}\) satisfying: \(\forall\omega\in\mathcal{B}\), \((X|\mathcal{B})(\omega)=X(\omega)\).

**Proposition 2.3** (Proposition 9 in [10]).: _If \(X\) and \(Y\) are sequences of random matrices with dimensions bounded by \(O(n)\), such that \((X,Y)\propto_{\left|\kern-1.075pt\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt \right|_{F}}\mathcal{E}(1)\), \(\mathbb{E}[\left|\kern-1.075pt\left|\kern-1.075pt\left|\chi\right|\right| \kern-1.075pt\right|]\leq O(\sqrt{n})\) and \(\mathbb{E}[\left|\kern-1.075pt\left|\kern-1.075pt\left|\kern-1.075pt\left|Y \right|\kern-1.075pt\right|]\right|\leq O(\sqrt{n})\), then there exists a constant \(c>0\) such that:_

1. \(\mathbb{E}[\left|\kern-1.075pt\left|\kern-1.075pt\left|\kern-1.075pt\left|X \right|\kern-1.075pt\right|]\right|\leq c\sqrt{n}\) _and_ \(\mathbb{E}[\left|\kern-1.075pt\left|\kern-1.075pt\left|Y\right|\kern-1.075pt \right|]\right|\leq c\sqrt{n}\)_._
2. _With_ \(\mathcal{B}=\{\left|\kern-1.075pt\left|\kern-1.075pt\left|X\right|\kern-1.075pt \right|\kern-1.075pt\right|\kern-1.075pt\right|\kern-1.075pt\}\) _and_ \(\left|\kern-1.075pt\left|\kern-1.075pt\left|Y\right|\kern-1.075pt\right| \kern-1.075pt\right|\kern-1.075pt\}\)__\(\leq 2c\sqrt{n}\)_),_ \(\mathbb{P}(\mathcal{B}^{c})\leq ce^{-n/c}\)_._
3. \((XY|\mathcal{B})\propto_{\left|\kern-1.075pt\left|\kern-1.075pt\left|\cdot \right|\kern-1.075pt\right|_{F}}\mathcal{E}(\sqrt{n})\)_._

### Polynomial bounds in \(z\) and notation \(O_{z}(\epsilon_{n})\)

Throughout this article we will deal with quantitative estimates involving both a dimension parameter \(n\in\mathbb{N}\) and a spectral parameter \(z\in\mathbb{C}^{+}\). As it turns out, for many applications it is not useful to track the exact dependence in \(z\), which motivates the following definition:

**Definition 2.4**.: Let \(\zeta:\mathbb{N}\times\mathbb{C}^{+}\to\mathbb{R}^{+}\) be a function and \(\epsilon_{n}>0\) a sequence. We say that \(\zeta\) is bounded by \(\epsilon_{n}\) in \(n\) and polynomially in \(z\), and we note \(\zeta(n,z)\leq O_{z}(\epsilon_{n})\), if there exists \(\alpha\geq 0\) such that uniformly in \(z\in\mathbb{C}^{+}\) with bounded \(\Im(z)\):

\[\zeta(n,z)\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha}}{\Im(z)^{2\alpha}} \bigg{)}.\]

We say that a family of functions is uniformly \(O_{z}(\epsilon_{n})\) bounded if the above bound holds uniformly for any function of the family.

_Remark 2.5_.: If \(\Im(z)\) is bounded, then \(\frac{|z|}{\Im(z)}\geq 1\) and \(\frac{1}{\Im(z)}\) is bounded away from \(0\). As a consequence, if \(\alpha\geq\alpha^{\prime}\geq 0\), then \(\frac{|z|^{\alpha}}{\Im(z)^{2\alpha}}\leq O\bigg{(}\frac{|z|^{\alpha^{\prime}} }{\Im(z)^{2\alpha^{\prime}}}\bigg{)}\). The classical rules of calculus thus apply to our notation: \(O_{z}(\epsilon_{n})+O_{z}(\epsilon_{n}^{\prime})=O_{z}(\epsilon_{n}+\epsilon_{n} ^{\prime})\) and \(O_{z}(\epsilon_{n})O_{z}(\epsilon_{n}^{\prime})=O_{z}(\epsilon_{n}\epsilon_{n} ^{\prime})\).

The next technical lemma will be key to translate results like the deterministic equivalent Theorem 4.3 into simplified versions using our \(O_{z}(\epsilon_{n})\) notations.

**Lemma 2.6**.: _If for some constants \(\alpha_{0},\alpha_{1},\ldots,\alpha_{6}>0\), the function \(\zeta\) satisfies an a priori inequality \(|\zeta(n,z)|\leq O\bigg{(}\frac{|z|^{\alpha_{1}}}{\epsilon_{n}^{\alpha_{0}} \Im(z)^{\alpha_{1}+\alpha_{2}}}\bigg{)}\), and if the bound:_

\[\zeta(n,z)\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha_{3}}}{\Im(z)^{\alpha_{3} +\alpha_{4}}}\bigg{)}\]
_holds uniformly for \(z\in\mathbb{C}^{+}\) with bounded \(\Im(z)\) and satisfying \(\epsilon_{n}\frac{|z|^{\alpha_{5}}}{\Im(z)^{\alpha_{5}+\alpha_{6}}}\leq c\) for some constant \(c>0\), then for some exponent \(\alpha>0\), the bound:_

\[\zeta(n,z)\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha}}{\Im(z)^{2\alpha}} \bigg{)}\]

_holds uniformly in \(z\in\mathbb{C}^{+}\) with bounded \(\Im(z)\), that is \(\zeta\leq O_{z}(\epsilon_{n})\)._

Proof.: Let us choose \(\alpha=\max\left(\alpha_{5}(\alpha_{0}+1)+\alpha_{1},\alpha_{6}(\alpha_{0}+1)+ \alpha_{2},\alpha_{3},\alpha_{4}\right)\). If \(\epsilon_{n}\frac{|z|^{\alpha_{5}}}{\Im(z)^{\alpha_{5}+\alpha_{6}}}\leq c\), we use the main inequality on \(\zeta\):

\[\zeta(n,z)\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha_{3}}}{\Im(z)^{\alpha_{ 3}+\alpha_{4}}}\bigg{)}\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha}}{\Im(z)^{2 \alpha}}\bigg{)}.\]

In the case where \(\epsilon_{n}\frac{|z|^{\alpha_{5}}}{\Im(z)^{\alpha_{5}+\alpha_{6}}}\geq c\), we use the _a priori_ inequality on \(\zeta\):

\[|\zeta(n,z)| \leq O\bigg{(}\frac{|z|^{\alpha_{1}}}{\epsilon_{n}^{\alpha_{0}} \Im(z)^{\alpha_{1}+\alpha_{2}}}\bigg{)}\] \[\leq O\bigg{(}\epsilon_{n}\bigg{(}\frac{|z|^{\alpha_{5}}}{\Im(z) ^{\alpha_{5}+\alpha_{6}}}\bigg{)}^{\alpha_{0}+1}\frac{|z|^{\alpha_{1}}}{\Im(z )^{\alpha_{1}+\alpha_{2}}}\bigg{)}\] \[\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha}}{\Im(z)^{2\alpha}} \bigg{)}.\]

_Remark 2.7_.: As we will see later in this article, the lemma is particularly suited to simplify some quantitative statements that require the spectral parameter \(z\) to be away from the real axis. Indeed the Stieltjes tranforms and resolvent matrices satisfy the classical bounds \(|g(z)|\leq 1/\Im(z)\) and \(\left\|\mathcal{G}(z)\right\|_{F}\leq\sqrt{n}\|\mathcal{G}(z)\|\|\leq\sqrt{n} /\Im(z)\), which translate into _a priori_ inequalities for any difference of such objects.

Let us wrap this subsection by giving a general setting on which \(O_{z}(\epsilon_{n})\) bounds between Stieltjes transforms imply polynomial bounds in Kolmogorov distance. We remind our reader that for measures \(\nu\) and \(\mu\) with cumulative distribution functions \(\mathcal{F}_{\nu}\) and \(\mathcal{F}_{\mu}\), the Kolmogorov distance is defined as \(D(\mu,\nu)=\sup\limits_{t\in\mathbb{R}}|F_{\nu}(t)-\mathcal{F}_{\mu}(t)|\). The following result is an immediate consequence of [10, Theorem 3.6].

**Proposition 2.8**.: _Let \(\mu_{n}\) and \(\nu_{n}\) be sequences of probability measures on \(\mathbb{R}^{+}\) such that:_

1. \(\int_{\mathbb{R}}|\mathcal{F}_{\mu_{n}}(t)-\mathcal{F}_{\nu_{n}}(t)|dt<\infty\)_._
2. \(\mathcal{F}_{\nu_{n}}\) _are uniformly Holder continuous with exponent_ \(\beta\in(0,1]\)_._
3. \(\mu_{n}\) _and_ \(\nu_{n}\) _have uniformly bounded second moments._
4. \(|g_{\mu_{n}}(z)-g_{\nu_{n}}(z)|\leq O_{z}(\epsilon_{n})\) _for some sequence_ \(\epsilon_{n}\to 0\)_._

_Then there exists \(\theta>0\) such that \(D(\mu_{n},\nu_{n})\leq O(\epsilon_{n}^{\theta})\)._

_Remark 2.9_.: In some cases, the uniform Holder continuity hypothesis may be adapted using a simple change of measures. For instance, given shape
parameters \(\gamma_{n}>1\) and measures \(\tau_{n}\) supported on the same compact of \((0,\infty)\), the cumulative distribution function of \(\nu_{n}=\operatorname{MP}(\gamma_{n})\boxtimes\tau_{n}\) are not even continuous at \(0\). However we can consider the new measures \(\check{\nu}_{n}=(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot\nu_{n}\), and use the fact that \(\mathcal{F}_{\check{\nu}_{n}}\) are uniformly \(1/2\)-Holder continuous to deduce the same result (see [10, Section 8.3]).

A variant of this result for empirical spectral distributions may also be adapted, see Proposition 8.1 in the Appendix.
## 3. Covariance matrices of functions of Gaussian vectors

In this section we consider the random vector \(y=f(u)\in\mathbb{R}^{n}\), obtained by applying a real function on each of the coordinates of a centered Gaussian vector \(u\in\mathbb{R}^{n}\). We are particularly interested in the case where the entries of \(u\) are weakly correlated. In this instance, we will give approximations of the covariance matrix \(\Sigma=\mathbb{E}\Big{[}yy^{\top}\Big{]}\), with a control of the error either entry-wise, or in spectral norm, or for their respective Stieltjes transforms. We first present the tools making these approximations possible.

### Hermite polynomials

Let us remind the notation \(\mathcal{N}\) for a standard Gaussian random variable. We denote by \(\mathcal{H}\) the Hilbert space of real Borel functions such that \(\mathbb{E}\big{[}f(\mathcal{N})^{2}\big{]}<\infty\), endowed with the Gaussian inner product:

\[<f,g>=\mathbb{E}[f(\mathcal{N})g(\mathcal{N})]=\frac{1}{\sqrt{2\pi}}\int_{ \mathbb{R}}f(t)g(t)e^{-t^{2}/2}dt.\]

Remark that all Lipschitz continuous functions automatically belong to \(\mathcal{H}\). The \(r\)-th non-normalized Hermite polynomial is by definition:

\[h_{r}(t)=(-1)^{r}e^{t^{2}/2}\frac{\mathrm{d}^{r}}{\mathrm{d}t^{r}}\Big{(}e^{-t ^{2}/2}\Big{)}.\]

The first Hermite polynomials are: \(h_{0}=1\), \(h_{0}=\mathbf{X}\), \(h_{2}=\mathbf{X}^{2}-1\), and \(h_{3}=\mathbf{X}^{3}-3\mathbf{X}\). The \(r\)-th normalized Hermite polynomial is \(\hat{h}_{r}=\frac{h_{r}}{\sqrt{r!}}\), and the \(r\)-th Hermite coefficient of a function \(f\in\mathcal{H}\) is \(\zeta_{r}(f)=<f,\hat{h}_{r}>\). Hereafter we remind some classical properties of Hermite polynomials without proofs. For more details, we invite our reader to consult [10, Chapter 4].

**Proposition 3.1**.:
1. _[label=()]_
2. \(h_{r}\) _are monic polynomials of degree_ \(r\)_._ \(\hat{h}_{r}\) _form a complete orthonormal basis of_ \(\mathcal{H}\)_._
3. _Every function of_ \(\mathcal{H}\) _can be expanded as the converging sum in_ \(\mathcal{H}\)_:_ \(f=\sum_{r\geq 0}\zeta_{r}(f)\hat{h}_{r}\)_. In particular_ \(\|f\|_{\mathcal{H}}^{2}=\mathbb{E}\big{[}f(\mathcal{N})^{2}\big{]}=\sum_{r \geq 0}\zeta_{r}(f)^{2}\)_._
4. _The Hermite polynomials satisfy the relations:_ \[h_{k}^{\prime}(t) =kh_{k-1}(t),\] \[h_{r+1}(t) =th_{r}(t)-h_{r}^{\prime}(t).\]

We move on to more specific properties that shall be used in the course of this section.

**Lemma 3.2** (Lemma D.2 in [11]).: _If \(z_{1}\) and \(z_{2}\) are standard Gaussian random variables, such that \((z_{1},z_{2})\) forms a Gaussian vector, then:_

\[\mathbb{E}\Big{[}\hat{h}_{r}(z_{1})\hat{h}_{s}(z_{2})\Big{]}=\mathbf{1}_{r=s} \operatorname{Cov}[z_{1},z_{2}]^{r}.\]

**Proposition 3.3**.: _Let us fix \(f\in\mathcal{H}\), and for \(r\in\mathbb{N}\) let \(\Psi_{r}(\sigma)=\sigma^{-r}\mathbb{E}[f(\sigma\mathcal{N})h_{r}(\mathcal{N})]\). Then \(\Psi_{r}\) is \(\mathcal{C}^{\infty}\) on \((0,\infty)\) and \(\Psi_{r}^{\prime}(\sigma)=\sigma\Psi_{r+2}(\sigma)\)._
Proof.: From the identity \(\mathbb{E}[f(\sigma\mathcal{N})h_{r}(\mathcal{N})]=\frac{1}{\sigma\sqrt{2\pi}}\int_ {\mathbb{R}}f(t)h_{r}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^{2}}}dt\) and the Leibniz integral rule, we deduce that \(\Psi_{r}\) is \(\mathcal{C}^{\infty}\) and that:

\[\Psi_{r}^{\prime}(\sigma) =\frac{d}{d\sigma}\bigg{(}\frac{1}{\sigma^{r+1}\sqrt{2\pi}}\int_ {\mathbb{R}}f(t)h_{r}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^{2}}}dt\bigg{)}\] \[=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}f(t)\frac{d}{d\sigma} \bigg{(}\frac{1}{\sigma^{r+1}}h_{r}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^{2}}} \bigg{)}dt.\]

The derivative inside the integral can be computed in a few steps. Firstly from the definition \(\frac{d}{dt}\Big{(}h_{r}(t)e^{-t^{2}/2}\Big{)}=(-1)^{r}\frac{d^{r+1}}{dt^{r+1} }\Big{(}e^{-t^{2}/2}\Big{)}=-h_{r+1}(t)e^{-t^{2}/2}\). Then applying the identity \(th_{r+1}(t)=h_{r+2}(t)+(r+1)h_{r}(t)\), which can be deduced from Proposition 3.1, leads to:

\[\frac{d}{d\sigma}\bigg{(}h_{r}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^ {2}}}\bigg{)} =\frac{t}{\sigma^{2}}h_{r+1}(t/\sigma)e^{-t^{2}/2\sigma^{2}}\] \[=\frac{1}{\sigma}(h_{r+2}(t/\sigma)+(r+1)h_{r}(t/\sigma))e^{- \frac{t^{2}}{2\sigma^{2}}},\] \[\frac{d}{d\sigma}\bigg{(}\frac{1}{\sigma^{r+1}}h_{r}(t/\sigma)e^ {-\frac{t^{2}}{2\sigma^{2}}}\bigg{)} =\frac{1}{\sigma^{r+2}}(h_{r+2}(t/\sigma)+(r+1)h_{r}(t/\sigma))e^{ -\frac{t^{2}}{2\sigma^{2}}}-\frac{r+1}{\sigma^{r+2}}h_{r}(t/\sigma)e^{-\frac{t ^{2}}{2\sigma^{2}}}\] \[=\frac{1}{\sigma^{r+2}}h_{r+2}(t/\sigma)e^{-\frac{t^{2}}{2\sigma ^{2}}}.\]

We obtain finally:

\[\Psi_{r}^{\prime}(\sigma) =\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}f(t)\frac{1}{\sigma^{r+2}} h_{r+2}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^{2}}}dt\] \[=\sigma\cdot\bigg{(}\frac{1}{\sigma^{r+3}\sqrt{2\pi}}\int_{ \mathbb{R}}f(t)h_{r+2}(t/\sigma)e^{-\frac{t^{2}}{2\sigma^{2}}}dt\bigg{)}\] \[=\sigma\cdot\Psi_{r+2}(\sigma).\]

**Corollary 3.4**.: _Let \(f\in\mathcal{H}\), and for \(\sigma>0\) let \(f_{\sigma}:t\mapsto f(\sigma t)\). Then for any integer \(r\geq 0\):_

\[\zeta_{r}(f_{\sigma}) =\sigma^{r}(\zeta_{r}(f)+O(\sigma-1))\] \[=\sigma^{r}\Big{(}\zeta_{r}(f)+\sqrt{(r+1)(r+2)}(\sigma-1)\zeta_{ r+2}(f)+O((\sigma-1)^{2})\Big{)}.\]

Proof.: These expressions are straightforward consequences of first and second order Taylor expansions of \(\Psi_{r}\). For instance for the second formula:

\[\zeta_{r}(f_{\sigma}) =\frac{\sigma^{r}}{\sqrt{r!}}\Psi_{r}(\sigma)\] \[=\frac{\sigma^{r}}{\sqrt{r!}}\big{(}\Psi_{r}(1)+(\sigma-1)\Psi_{ r+2}(1)+O((\sigma-1)^{2})\big{)}\] \[=\sigma^{r}\Big{(}\zeta_{r}(f)+\sqrt{(r+1)(r+2)}(\sigma-1)\zeta_{ r+2}(f)+O((\sigma-1)^{2})\Big{)}.\]
### Iterated Hadamard products

In what follows, \(M^{\circ r}\) denotes the Hadamard product of \(r\) copies of a matrix \(M\in\mathbb{R}^{n\times n}\). We denote by \(\operatorname{diag}(M)\) and \(\operatorname{off}(M)\) respectively the diagonal and off-diagonal sub-matrices of \(M\). \(\operatorname{diag}(M)\in\mathbb{R}^{n}\) denotes the diagonal elements of \(M\) reshaped as a vector in \(\mathbb{R}^{n}\).

**Lemma 3.5**.: _If \(\Delta\in\mathbb{R}^{n\times n}\) is a symmetric matrix such that \(I_{n}+\Delta\) is positive semi-definite, then for any integer \(r\geq 1\):_

\[\left|\!\left|\!\left|\Delta^{\circ r+1}\right|\!\right|\!\right| \leq(1+\left|\!\left|\Delta\right|\!\right|_{\max})\!\left|\! \left|\Delta^{\circ r}\right|\!\right|+\left|\!\left|\Delta\right|\!\right|_{ \max}^{r},\] \[\left|\!\left|\!\left|\Delta^{\circ 2r}\right|\!\right|\!\right| \leq\left|\!\left|\Delta\right|\!\right|_{\max}^{2r-2}\!\left|\! \left|\!\left|\Delta\circ\Delta\right|\!\right|\!\right|.\]

Proof.: For any matrices \(A\) and \(B\in\mathbb{R}^{n\times n}\) with \(A\) positive semi-definite, the inequality holds true ([10, Proposition 3.7.9.]). As a consequence:

\[\left|\!\left|\!\left|\Delta^{\circ r+1}\right|\!\right|\!\right| \leq\left|\!\left|\!\left|\left(I_{n}+\Delta\right)\circ\Delta^{ \circ r}\right|\!\right|\!\right|+\left|\!\left|\!\left|I_{n}\circ\Delta^{ \circ r}\right|\!\right|\] \[\leq\left|\!\left|I_{n}+\Delta\right|\!\right|\!\left|\!\left| \Delta^{\circ r}\right|\!\right|\!\right|+\left|\!\left|\!\left|\operatorname {diag}(\Delta)^{r}\right|\!\right|\] \[\leq(1+\left|\!\left|\Delta\right|\!\right|\!\left|\Delta^{ \circ r}\right|\!\right|+\left|\!\left|\Delta\right|\!\right|_{\max}^{r}.\]

For the second inequality, since \(\Delta^{\circ 2r}\) has non negative entries, there is \(\alpha\in\left(\mathbb{R}^{+}\right)^{n}\) such that \(\left|\!\left|\alpha\right|\!\right|=1\) and. We deduce that:

\[\left|\!\left|\!\left|\Delta^{\circ 2r}\right|\!\right|\!\right| =\left(\sum_{i=1}^{n}\left(\sum_{j=1}^{n}\Delta_{ij}^{2r}\alpha_ {j}\right)^{2}\right)^{1/2}\] \[\leq\left|\!\left|\Delta\right|\!\right|_{\max}^{2r-2}\!\left(\sum _{i=1}^{n}\left(\sum_{j=1}^{n}\Delta_{ij}^{2}\alpha_{j}\right)^{2}\right)^{1/2}\] \[\leq\left|\!\left|\Delta\right|\!\right|_{\max}^{2r-2}\!\left|\! \left|\!\left|\Delta\circ\Delta\right|\!\right|\!\right|\!\right|\!.\]

**Proposition 3.6**.: _If \(\Delta\in\mathbb{R}^{n\times n}\) is a sequence of symmetric matrices such that \(I_{n}+\Delta\) is positive semi-definite, \(\left|\!\left|\Delta\right|\!\right|\!\right|\) converges to \(0\) and \(\left|\!\left|\!\left|\!\left|\Delta\right|\!\right|\!\right|\) is bounded, then for any \(r\geq 1\) the quantities,, and are all bounded by \(O\!\left(\left|\Delta\right|\!\right|\!\right|\!\right|\)._

Proof.: According to the preceding lemma,. This implies that: and, and. The main results follow after remarking that \(\left|\!\left|\Delta^{\circ 2r+1}\right|\!\right|\leq(1+\left|\!\left|\Delta\right|\! \right|\!\left|\Delta\right|\!\right|\!\left|\!\right|+\left|\!\left|\Delta \right|\!\right|\!\left|\!\left|\Delta^{\circ 2r}\right|\!\right|\!\left|\!\right|+\left|\! \left|\Delta\right|\!\right|_{\max}^{2r-2}\!\left|\!\left|\!\left|\Delta\circ \Delta\right|\!\right|\!\right|\leq O\!\left(\left|\!\left|\Delta\right|\! \right|\!\right|\!\right|_{\max}^{2r-2}\!\left|\!\left|\!\left|\!\left|\! \left|\Delta\right|\!\right|\!\right|_{\max}^{2r-2}\!\right)\). The remaining results follow after remarking that \(\left|\!\left|\operatorname{diag}(\Delta^{\circ r})\right|\!\right|\!\leq O( \left|\!\left|\Delta\right|\!\right|\!\right|_{\max}^{r})\). 

### General expansion of the covariance matrix \(\Sigma\)

In the following paragraphs we are considering a function \(f\in\mathbf{H}\), and \(u\in\mathbb{R}^{n}\) a centered Gaussian vector with covariance matrix \(S\in\mathbb{R}^{n\times n}\). The random vector \(y=f(u)\) is obtained by applying the function \(f\) entry-wise on \(u\), and we let \(\Sigma=\mathbb{E}\!\left[yy^{\top}\right]\). For \(1\leq i\leq n\), we denote by \(f_{i}\) the function \(f_{i}:t\mapsto f\!\left(S_{ii}^{1/2}\,t\right)\), and by \(D_{r}\in\mathbb{R}^{n\times n}\) the diagonal matrix with entries \({S_{ii}}^{-r/2}\zeta_{r}(f_{i})\).
**Proposition 3.7**.: _The convergence of the following sum holds entry-wise:_

\[\Sigma=\sum_{r\geq 0}D_{r}\,S^{\sigma r}D_{r}.\]

Proof.: The random variables \(\tilde{u}_{i}=S_{ii}^{-1/2}u_{i}\) are standard Gaussian random variables, and they form a Gaussian vector. Using Lemma 3.2, we have:

\[\Sigma_{ij} =\mathbb{E}[f_{i}(\tilde{u}_{i})f_{j}(\tilde{u}_{j})]\] \[=\sum_{r,s\geq 0}\zeta_{r}(f_{i})\zeta_{s}(f_{j})\mathbb{E}\Big{[} \hat{h}_{r}(\tilde{u}_{i})\hat{h}_{s}(\tilde{u}_{j})\Big{]}\] \[=\sum_{r\geq 0}\zeta_{r}(f_{i})\zeta_{r}(f_{j})\operatorname{Cov} [\tilde{u}_{i},\tilde{u}_{j}]^{r}\] \[=\sum_{r\geq 0}\zeta_{r}(f_{i})\zeta_{r}(f_{j}){S_{ii}}^{-r/2}{S_ {jj}}^{-r/2}{S_{ij}}^{r}\] \[=\sum_{r\geq 0}\left(D_{r}\right)_{i}(D_{r})_{j}(S^{\sigma r})_ {ij},\]

which proves the Proposition. 

### Approximation of \(\Sigma\) for weakly correlated Gaussian vectors

We will now explain how the expansion of \(\Sigma\) given by Proposition 3.7 can be simplified when \(u\) has weakly correlated entries, in the sense that its covariance matrix \(S\) is close to \(I_{n}\). To this end we let \(\Delta=S-I_{n}\), and:

\[\Sigma_{\text{approx}}=\left\|f\right\|_{\mathcal{H}}^{2}I_{n}+\frac{\zeta_{2 }(f)^{2}}{2}\vec{\operatorname{diag}}(\Delta)\vec{\operatorname{diag}}( \Delta)^{\top}+\sum_{r=1,2,3}\zeta_{r}(f)^{2}\Delta^{\sigma r}\]

**Assumptions 3.8**.:
1. \(f\) is Lipschitz and Gaussian centered, i.e. \(\zeta_{0}(f)=\mathbb{E}[f(\mathcal{N})]=0\).
2. \(\left|\!\left|\Delta\right|\!\right|\) and \(\left|\!\left|\vec{\operatorname{diag}}(\Delta)\right|\!\right|\) are bounded.
3. \(\left|\!\left|\Delta\right|\!\right|_{\text{max}}\) converges to \(0\).

**Theorem 3.9**.: _Under Assumptions 3.8, \(\left|\!\left|\!\left|\Sigma-\Sigma_{\text{approx}}\right|\!\right|\!\right| \leq O(\left|\!\left|\Delta\right|\!\right|_{\text{max}})\)._

Proof.: Let \(\epsilon_{n}=\left|\Delta\right|\!\right|_{\text{max}}\). Since \(f\) is Lipschitz, \(\left\|f-f_{i}\right\|_{L^{2}}=O\Big{(}\left|1-S_{ii}^{1/2}\right|\Big{)}=O( \epsilon_{n})\). We deduce that \(\left|\!\left|\!\left|\operatorname{diag}(\Sigma)-\left\|f\right|I_{n}\right| \!\right|=\max\limits_{1\leq i\leq n}\left|\left|f_{i}\right|-\left\|f\right| _{\mathcal{H}}\right|\leq O(\epsilon_{n})\). For the off-diagonal terms, let \(\nu\) be the vector in \(\mathbb{R}^{n}\) defined by its coordinates \(\nu_{i}=\zeta_{0}(f_{i})\). Let also \(\mu=\frac{\zeta_{2}(f)^{2}}{2}\vec{\operatorname{diag}}(\Delta)\). The decomposition:

\[\operatorname{off}(\Sigma)=\operatorname{off}\Big{(}\nu\nu^{\top}\Big{)}+\sum _{r\geq 1}D_{r}\operatorname{off}(\Delta^{\sigma r})\,D_{r}\]

follows easily from the expansion of \(\Sigma\) given by Proposition 3.7. Since \(\zeta_{0}(f)=0\) and \(S_{ii}^{1/2}=1+\frac{\Delta_{ii}}{2}+O(\Delta_{ii}^{2})\), using the second order expansion given by
Corollary 3.4 applied to the function \(f\), uniformly in \(i\in[\![1,n]\!]\) we have:

\[\nu_{i}=\zeta_{0}(f_{i}) =\sqrt{2}\Big{(}S_{ii}^{1/2}-1\Big{)}\zeta_{2}(f)+O\bigg{(}\Big{(}S _{ii}^{1/2}-1\Big{)}^{2}\bigg{)}\] \[=\frac{\zeta_{2}(f)}{\sqrt{2}}\Delta_{ii}+O(\Delta_{ii}^{2})\] \[=\mu_{i}+O(\Delta_{ii}^{2}).\]

Therefore \(\left\|\nu-\mu\right\|^{2}=O\Bigg{(}\sum_{i=1}^{n}\Delta_{ii}^{4}\Bigg{)}\leq O \bigg{(}\max_{1\leq i\leq n}\Delta_{ii}^{2}\bigg{)}\left\|\vec{\operatorname{ diag}}(\Delta)\right\|^{2}\leq O(\epsilon_{n}^{2})\), and:

For the remaining sum \(\sum_{r\geq 1}D_{r}\operatorname{off}(\Delta^{\circ r})\,D_{r}\), we use again that \(|\zeta_{r}(f_{i})-\zeta_{r}(f)|\leq\left\|f_{i}-f\right\|_{\mathcal{H}}\leq O (\epsilon_{n})\), hence \(\left\|D_{r}-\zeta_{r}(f)I_{n}\right\|\leq O(\epsilon_{n})\), and in particular \(\left\|D_{r}\right\|\leq O(1)\). Using Proposition 3.6, for \(r=1,2\) or \(3\):

\[\left\|D_{r}\operatorname{off}(\Delta^{\circ r})\,D_{r}-\zeta_{r} (f)^{2}\Delta^{\circ r}\right\|\] \[\quad+\left\|D_{r}-\zeta_{r}(f)I_{n}\right\|\left\|\Delta^{\circ r }\right\|\left\|(\left\|D_{r}\right\|+|\zeta_{r}(f)|\right)\] \[\leq O(\epsilon_{n}).\]

Finally for the remaining terms, using Proposition 3.6 for \(r\geq 4\):

\[\left\|\left\|\sum_{r\geq 4}D_{r}\operatorname{off}(\Delta^{\circ r})\,D_{r} \right\|\right\|\leq\sum_{r\geq 2}O\big{(}\epsilon_{n}^{2r-2}\big{)}\leq O \big{(}\epsilon_{n}^{2}\big{)}.\]

### Linearization of \(\Sigma\) in specific settings

Let:

\[\Sigma_{\operatorname{lin}}=\left\|f\right\|_{\mathcal{H}}^{2}I_{n}+\zeta_{1} (f)^{2}\Delta\]

The matrix \(\Sigma_{\operatorname{lin}}\) is obtained by means of usual matrix operations, easy to handle with classical random matrix theory tools, in contrast to \(\Sigma_{\operatorname{approx}}\) which involves Hadamard products. may not converge to \(0\) under the mere Assumptions 3.8. We can nonetheless identify conditions involving \(\Delta\) and \(f\), under which \(\Sigma_{\operatorname{lin}}\) is a good approximation of \(\Sigma\) either entry-wise, or in spectral norm, or for their respective Stieltjes transforms.

**Proposition 3.10**.: _Under Assumptions 3.8, \(\left\|\Sigma-\Sigma_{\operatorname{lin}}\right\|\) is bounded, and with \(\epsilon_{n}=\zeta_{2}(f)^{2}\|\Delta\|_{\max}^{2}+\zeta_{3}(f)^{2}\|\Delta\|_ {\max}^{3}\):_

\[\left\|\Sigma-\Sigma_{\operatorname{lin}}\right\|_{\max} \leq O(\left\|\Delta\right\|_{\max}),\] \[\left\|\Sigma-\Sigma_{\operatorname{lin}}\right\| \leq O(\left\|\Delta\right\|_{\max}+n\epsilon_{n}),\] \[\left|g_{\Sigma}(z)-g_{\Sigma_{\operatorname{lin}}}(z)\right| \leq\frac{1}{\Im(z)^{2}}O\big{(}\left\|\Delta\right\|_{\max}+ \sqrt{n}\epsilon_{n}\big{)}.\]

Proof.: We start from the expression:

\[\Sigma_{\operatorname{approx}}-\Sigma_{\operatorname{lin}}=\frac{\zeta_{2}(f )^{2}}{2}\vec{\operatorname{diag}}(\Delta)\vec{\operatorname{diag}}(\Delta) ^{\top}+\zeta_{2}(f)^{2}\Delta^{\circ 2}+\zeta_{3}(\tilde{f})^{2}\Delta^{\circ 3}\]
The matrices \(\vec{\mathrm{diag}}(\Delta)\vec{\mathrm{diag}}(\Delta)^{\top}\), \(\Delta^{\circ 2}\) and \(\Delta^{\circ 3}\) are all bounded in spectral norm, thus \(\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin}}\right|\kern-1.075pt \right\|\) is bounded and from Theorem 3.9\(\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin}}\right|\kern-1.075pt\right\|\) is always bounded. The bound \(\left\|\Sigma_{\mathrm{approx}}-\Sigma_{\mathrm{lin}}\right\|_{\max}\leq O( \epsilon_{n})\) is immediate given the above expression for the difference of these matrices, and from Theorem 3.9:

\[\left\|\Sigma-\Sigma_{\mathrm{lin}}\right\|_{\max} \leq\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{approx}} \right|\kern-1.075pt\right\|+\left\|\Sigma_{\mathrm{approx}}-\Sigma_{\mathrm{ lin}}\right\|_{\max}\] \[\leq O(\left\|\Delta\right\|_{\max}+\epsilon_{n})\leq O(\left\| \Delta\right\|_{\max}).\]

The other bounds follow from classical matrix inequalities. Indeed:

\[\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin}}\right| \kern-1.075pt\right\| \leq\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{approx}} \right|\kern-1.075pt\right\|+\left\|\kern-1.075pt\left|\Sigma_{\mathrm{approx }}-\Sigma_{\mathrm{lin}}\right|\kern-1.075pt\right\|\] \[\leq O(\left\|\Delta\right\|_{\max})+n\left\|\Sigma_{\mathrm{ approx}}-\Sigma_{\mathrm{lin}}\right\|_{\max}\] \[\leq O(\left\|\Delta\right\|_{\max}+n\epsilon_{n}).\]

Finally for the inequality on Stieltjes transforms, for any \(A,B\in\mathbb{R}^{n\times n}\) and \(z\in\mathbb{C}^{+}\):

\[\left|g_{A}(z)-g_{B}(z)\right| =\left|\frac{1}{n}\mathrm{Tr}(\mathcal{G}_{A}(z)(B-A)\mathcal{G} _{B}(z))\right|\] \[\leq\frac{1}{n}\left\|\kern-1.075pt\left|\mathcal{G}_{A}(z) \right|\kern-1.075pt\right\|\left|I_{n}\right\|_{F}\kern-1.075pt\right\|B-A \kern-1.075pt\left|\kern-1.075pt\left|\mathcal{G}_{B}(z)\right|\kern-1.075pt\right\|\] \[\leq\frac{\left\|B-A\kern-1.075pt\right\|_{F}}{\sqrt{n}\Im(z)^ {2}}.\]

We deduce that:

\[\left|g_{\Sigma}(z)-g_{\Sigma_{\mathrm{lin}}}(z)\right| \leq\left|g_{\Sigma}(z)-g_{\Sigma_{\mathrm{approx}}}(z)\right| +\left|g_{\Sigma_{\mathrm{approx}}}(z)-g_{\Sigma_{\mathrm{lin}}}(z)\right|\] \[\leq\frac{\left\|\Sigma-\Sigma_{\mathrm{lin}}\right\|_{F}}{ \sqrt{n}\Im(z)^{2}}+\frac{\left\|\Sigma_{\mathrm{approx}}-\Sigma_{\mathrm{ lin}}\right\|_{F}}{\sqrt{n}\Im(z)^{2}}\] \[\leq\frac{\left\|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin} }\right|\kern-1.075pt\right\|}{\Im(z)^{2}}+\sqrt{n}\frac{\left\|\Sigma_{ \mathrm{approx}}-\Sigma_{\mathrm{lin}}\right\|_{\max}}{\Im(z)^{2}}\] \[\leq\frac{1}{\Im(z)^{2}}O\big{(}\left\|\Delta\right\|_{\max}+ \sqrt{n}\epsilon_{n}\big{)}.\]

_Remark 3.11_.: In practice, \(\left\|\kern-1.075pt\Sigma-\Sigma_{\mathrm{lin}}\right\|\kern-1.075pt\right|\) converges to \(0\) provided one of the following additional hypothesis holds true:

* \(\left\|\kern-1.075pt\Delta\right\|_{\max}=o\Big{(}n^{-1/2}\Big{)}\),
* \(\zeta_{2}(f)=0\) and \(\left\|\kern-1.075pt\Delta\right\|_{\max}=o\Big{(}n^{-1/3}\Big{)}\),
* \(\zeta_{2}(f)=\zeta_{3}(f)=0\).

For the Stieltjes transforms, \(g_{\Sigma}(z)-g_{\Sigma_{\mathrm{lin}}}(z)\) converges to \(0\) pointwise, and thus \(\Sigma\) and \(\Sigma_{\mathrm{lin}}\) have the same limiting spectral distribution if it exists, provided one of the following additional hypothesis holds true:

* \(\left\|\kern-1.075pt\Delta\right\|_{\max}=o\Big{(}n^{-1/4}\Big{)}\),
* \(\zeta_{2}(f)=0\) and \(\left\|\kern-1.075pt\Delta\right\|_{\max}=o\Big{(}n^{-1/6}\Big{)}\),
* \(\zeta_{2}(f)=\zeta_{3}(f)=0\).
## 4. Deterministic equivalent of sample covariance matrices

In this section we recall the latest results about the deterministic equivalent of the resolvents of sample covariance matrices on which this article is based. These estimates were first established in [10] with a convergence speed in \(n\), and complemented with quantitative estimates in the spectral parameter \(z\) in [11]. In a second step, we will thoroughly study the properties of the deterministic equivalent matrices appearing in these results.

Given \(\Sigma\in\mathbb{R}^{n\times n}\) a positive semi-definite matrix and a sequence of shape parameters \(\gamma_{n}>0\), we build the matrix function \(\mathbf{G}_{\boxtimes}^{\Sigma}:\mathbb{C}^{+}\to\mathbb{C}^{n\times n}\) from the following objects:

\[\nu^{\Sigma} =\mathrm{MP}(\gamma_{n})\boxtimes\mu_{\Sigma},\] \[\bar{\nu}^{\Sigma} =(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot\nu^{\Sigma},\] \[l_{\bar{\nu}^{\Sigma}}(z) =-1/g_{\bar{\nu}^{\Sigma}}(z),\] \[\mathbf{G}_{\boxtimes}^{\Sigma}(z) =(-zg_{\bar{\nu}^{\Sigma}}(z)\Sigma-zI_{n})^{-1},\] \[=z^{-1}l_{\bar{\nu}^{\Sigma}}(z)\mathcal{G}_{L}(l_{\bar{\nu}^{ \Sigma}}(z)).\]

Let us state without proofs some useful properties of these objects. \(\bar{\nu}^{\Sigma}\) is always a true probability measure ([11, Lemma 6.1]). The usual resolvent inequality \(\left\|\mathbf{G}_{\boxtimes}^{\Sigma}(z)\right\|\leq 1/\Im(z)\) holds ([11, Lemma 5.1]), and \(\mathrm{Tr}\mathbf{G}_{\boxtimes}^{\Sigma}(z)=ng_{\nu^{\Sigma}}(z)\) ([11, Proposition 6.2]).

_Remark 4.1_.: The notation \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\) is inspired from the free probability theory, which appears as the profound canvas hidden within the above definitions. Indeed it is not difficult to see that the pair \(\left(g_{\bar{\nu}^{\Sigma}}(z),\mathbf{G}_{\boxtimes}^{\Sigma}(z)\right)\) satisfies the system of self-consistent equations:

\[g_{\bar{\nu}^{\Sigma}}(z) =\left(-z-z\frac{\gamma_{n}}{n}\mathrm{Tr}\big{(}\mathbf{G}_{ \boxtimes}^{\Sigma}(z)\Sigma\big{)}\right)^{-1},\] \[\mathbf{G}_{\boxtimes}^{\Sigma}(z) =(-zI_{n}-zg_{\bar{\nu}^{\Sigma}}(z)\Sigma)^{-1}.\]

This may be rewritten as the operator-valued self-consistent equation:

\[z\mathcal{H}(z)=I_{n+1}+z\eta(\mathcal{H}(z))\mathcal{H}(z),\]

where \(\mathcal{H}(z)=\begin{pmatrix}g_{\bar{\nu}^{\Sigma}}(z)&0\\ 0&\mathbf{G}_{\boxtimes}^{\Sigma}(z)\end{pmatrix}\), and:

\[\eta:\mathbb{C}\oplus\mathbb{C}^{n\times n} \to\mathbb{C}\oplus\mathbb{C}^{n\times n},\] \[\begin{pmatrix}g&0\\ 0&G\end{pmatrix} \mapsto\begin{pmatrix}\frac{\gamma_{n}}{n}\mathrm{Tr}(G\Sigma)&0\\ 0&g\Sigma\end{pmatrix}.\]

\(z\mathcal{H}(z^{2})\) thus corresponds to the resolvent of an operator-valued free semi-circular variable with covariance \(\eta\), see [12, Section 3.3]. In this sense the definition of \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\) extends the notion of free convolution, and it should come as no surprise that such objects appear as deterministic equivalent of sample covariance matrices.
### General results

Let \(Y\in\mathbb{R}^{d\times n}\) be a sequence of random matrices. The associated sample covariance matrix is \(K=Y^{\top}Y/d\), and for \(z\in\mathbb{C}^{+}\) we define its resolvent \(\mathcal{G}_{K}(z)=\left(K-zI_{n}\right)^{-1}\) and Stieltjes transform \(g_{K}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{K}(z)\). From the expected covariance matrix \(\Sigma=\mathbb{E}[K]\), we follow the above procedure to build the matrix function \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\).

**Assumptions 4.2**.:
1. \(Y\propto_{\left\|\mspace{-1.0mu }\right\|_{F}}\mathcal{E}(1)\), the rows of \(Y\) are i.i.d. sampled from the distribution of a random vector \(y\), with \(\left\|\mathbb{E}[y]\right\|\) and \(\left\|\left\|\mathbb{E}[yy^{\top}]\right\|\right\|=\left\|\left\|\Sigma \right\|\right\|\) bounded.
2. The ratio \(\gamma_{n}=\dfrac{n}{d}\) is bounded from above and away from \(0\).

**Theorem 4.3**.: _[_2_, Theorem 2.3]_ _Uniformly under Assumptions 4.2, there exists \(C>0\) such that uniformly in \(z\in\mathbb{C}^{+}\) with \(\Im(z)\) bounded and \(\dfrac{|z|^{7}}{\Im(z)^{16}}\leq n/C\), the following inequality holds:_

\[\left\|\mathbb{E}[\mathcal{G}_{K}(z)]-\mathbf{G}_{\boxtimes}^{\Sigma}(z) \right\|_{F}\leq O\Bigg{(}\dfrac{|z|^{5/2}}{\Im(z)^{9}n^{1/2}}\Bigg{)}.\]

_Remark 4.4_.: By "uniformly under Assumptions 4.2", we mean that the implicit constants in the result only depend on the constants chosen in the assumptions. Said otherwise, if a family of random matrices satisfies Assumptions 4.2 uniformly, then the results holds uniformly for any matrix of this family.

Keeping track of the exponents appearing in this theorem will become more and more difficult as we go further into this article. We choose for this reason to work with a weaker notion of approximation using our concept of \(O_{z}(\epsilon_{n})\) polynomial bounds. As a side effect, we will see that we do not need anymore to specify conditions on \(z\) on which the approximation holds true.

**Theorem 4.5** (Simplified version of the Deterministic Equivalent).: _Uniformly under Assumptions 4.2, the following concentration properties hold true:_

1. \(g_{K}(z)\propto\mathcal{E}(O_{z}(1/n))\) _and_ \(\mathcal{G}_{K}(z)\propto_{\left\|\mspace{-1.0mu }\right\|_{F}}\mathcal{E} \big{(}O_{z}(1/\sqrt{n})\big{)}\)_._
2. \(g_{K}(z)\in g_{\nu}(z)\pm\mathcal{E}(O_{z}(1/n))\) _and_ \(\mathcal{G}_{K}(z)\in_{\left\|\mspace{-1.0mu }\right\|_{F}}\mathbf{G}_{ \boxtimes}^{\Sigma}(z)\pm\mathcal{E}\big{(}O_{z}(1/\sqrt{n})\big{)}\)_._

Proof.: From Theorem 4.3, Lemma 2.6 and the _a priori_ bound \(\left\|\mathbb{E}[\mathcal{G}_{K}(z)]-\mathbf{G}_{\boxtimes}^{\Sigma}(z) \right\|_{F}\leq 2\sqrt{n}/\Im(z)\), we have \(\left\|\mathbb{E}[\mathcal{G}_{K}(z)]-\mathbf{G}_{\boxtimes}^{\Sigma}(z) \right\|_{F}\leq O_{z}(1/\sqrt{n})\). The general properties of concentration recalled in Proposition 2.2 prove the Theorem. 

Similarly to [2, Corollaries 2.5 and 2.8], we may deduce the following spectral properties of \(K\). We will not prove this result immediately, but rather prompt our reader to consult the proof of Corollary 5.9 which is extremely similar.

**Corollary 4.6**.: _Uniformly under Assumptions 4.2 :_

1. \(|g_{K}(z)-g_{\nu^{\Sigma}}(z)|\leq O_{z}\Big{(}\sqrt{\log n}/n\Big{)}\) _and_ \(\left\|\mathcal{G}_{K}(z)-\mathbf{G}_{\boxtimes}^{\Sigma}(z)\right\|_{\max} \leq O_{z}\Big{(}\sqrt{\log n/n}\Big{)}\)_._
2. _If the eigenvalues of_ \(\Sigma\) _are bounded from below, there exists_ \(\theta>0\) _such that_ \(D(\mu_{K},\nu^{\Sigma})\leq O(n^{-\theta})\) _a.s._
_._
3. _If additionally_ \(\mu_{\Sigma}\to\mu_{\infty}\) _weakly and_ \(\gamma_{n}\to\gamma_{\infty}\)_, then_ \(\mu_{K}\to\nu_{\infty}=\operatorname{MP}(\gamma_{\infty})\boxtimes\mu_{\infty}\) _weakly a.s., and more precisely:_ \[D(\mu_{K},\nu_{\infty})\leq D(\mu_{\Sigma},\mu_{\infty})+O(|\gamma_{n}-\gamma _{\infty}|)+O(n^{-\theta})\quad\text{a.s.}\]

### Regularity of the Stieltjes transform with respect to the free convolution

Let \(\Sigma\in\mathbb{R}^{n\times n}\) be a sequence of positive semi-definite matrices and \(\tau\) probability measures such that \(g_{\Sigma}(z)-g_{\tau}(z)\) converges to \(0\) pointwise in \(\mathbb{C}^{+}\). Then \(\mu_{\Sigma}\) and \(\tau\) converge weakly to the same limit if it exists. The free multiplicative convolutions \(\nu=\operatorname{MP}(\gamma_{n})\boxtimes\mu_{\Sigma}\) and \(\chi=\operatorname{MP}(\gamma_{n})\boxtimes\tau\) also converge weakly to the same limit, which is equivalent to their Stieltjes transforms \(g_{\nu}(z)\) and \(g_{\chi}(z)\) having the same limit pointwise. In this section we would like to refine this result by quantifying the convergence of \(g_{\nu}(z)-g_{\chi}(z)\) to \(0\).

In the upcoming paragraphs, we always consider shape parameters \(\gamma_{n}\) that are bounded from above and away from \(0\), like in Assumptions 4.2. We remind our reader that we are dealing with sequences of matrices, measures, and complex functions, even if we sometimes omit the indices \(n\) and \(z\) for a better readability.

**Theorem 4.7**.: _Let \(\Sigma\in\mathbb{R}^{n\times n}\) be deterministic positive semi-definite matrices, and \(\tau\) deterministic probability measures supported on \(\mathbb{R}^{+}\)._

_If \(|g_{\Sigma}(z)-g_{\tau}(z)|\leq O_{z}(\epsilon_{n})\), then \(|g_{\nu}(z)-g_{\chi}(z)|\leq O_{z}(\epsilon_{n})\)._

The proof of this result may be decomposed in several steps. First we translate the definition of the measures \(\nu\) and \(\chi\) into appropriate self-consistent equations on their reciprocal Cauchy transforms (Proposition 4.8). Then we see that \(\chi\) is an approximate fixed point of the equation corresponding to \(\nu\) (Proposition 4.10). Finally we use the stability of these self-consistent equations and the tools developed in [10] and [11] to conclude.

The key function in the upcoming paragraphs \(l_{\hat{\nu}\Sigma}(z)=-1/g_{\hat{\nu}}(z)\) is known as the reciprocal Cauchy transform of the measure \(\tilde{\nu}^{\Sigma}\). As such some classical properties of this function may be found in the seminal book [14, Section 3.4]. We will nonetheless provide short proofs for all the properties we need in this article.

**Proposition 4.8** (Self-consistent equation for reciprocal Cauchy transforms).: _If \(\mu\) is a measure supported on \(\mathbb{R}^{+}\), we let \(\nu=\operatorname{MP}(\gamma_{n})\boxtimes\mu\), \(\tilde{\nu}=(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot\nu\), and \(l_{\hat{\nu}}(z)=-1/g_{\hat{\nu}}(z)\)._

_Then \(l_{\hat{\nu}}(z)\) is the only solution on \(\mathbb{C}^{+}\) of the self consistent equation in \(l\):_

\[l=z+\gamma_{n}l+\gamma_{n}l^{2}g_{\mu}(l)=z+\gamma_{n}z\int_{\mathbb{R}}\frac{ t}{\frac{zt}{l}-z}\mu(dt).\]

Proof.: The two right hand side terms are always equal since:

\[\int_{\mathbb{R}}\frac{zt}{\frac{zt}{l}-z}\mu(dt)=l\int_{\mathbb{R}}\frac{t}{ t-l}\mu(dt)=l\int_{\mathbb{R}}\bigg{(}1+\frac{l}{t-l}\bigg{)}\mu(dt)=l+l^{2}g_{ \mu}(l).\]

Let us work with the first formulation. It is a classical property of Stieltjes transforms that \(g_{\hat{\nu}}(z)\in\mathbb{C}^{+}\) when \(z\in\mathbb{C}^{+}\), hence \(l_{\hat{\nu}}(z)\in\mathbb{C}^{+}\). Since \(g_{\hat{\nu}}=\dfrac{\gamma_{n}-1}{z}+\gamma_{n}g_{\nu}\), we have the identity \(1-\gamma_{n}-\gamma_{n}zg_{\nu}=\dfrac{z}{l_{\hat{\nu}}}\). By definition of
the free multiplicative convolution with a Marcenko-Pastur distribution, \(g_{\nu}\) is the only solution on \(\mathbb{C}^{+}\) of:

\[g_{\nu} =\int_{\mathbb{R}}\frac{1}{(1-\gamma_{n}-\gamma_{n}zg_{\nu})t-z}\mu (dt)\] \[=\int_{\mathbb{R}}\frac{1}{zt/l_{\hat{\nu}}-z}\mu(dt)\] \[=\frac{l_{\hat{\nu}}}{z}\int_{\mathbb{R}}\frac{1}{t-l_{\hat{\nu}} }\mu(dt)\] \[=\frac{l_{\hat{\nu}}}{z}g_{\mu}(l_{\hat{\nu}}).\]

Using again the identity \(1-\gamma_{n}-\gamma_{n}zg_{\nu}=\frac{z}{l_{\hat{\nu}}}\), the self-consistent equation characterizing \(g_{\nu}\) is equivalent for \(l_{\hat{\nu}}\) to satisfy:

\[\gamma_{n}l_{\hat{\nu}}+\gamma_{n}l_{\hat{\nu}}^{2}g_{\mu}(l_{\hat{\nu}})= \gamma_{n}l_{\hat{\nu}}(1+zg_{\nu})=l_{\hat{\nu}}-z.\]

**Lemma 4.9**.: _With the same notations and hypothesis as in Proposition 4.8:_

\[0 \leq\Im\big{(}z^{-1}l_{\hat{\nu}}(z)\big{)},\] \[\Im(z) \leq\Im(l_{\hat{\nu}}(z))\leq O(1+\Im(z)),\] \[|l_{\hat{\nu}}(z)| \leq O\bigg{(}\frac{|z|}{\Im(z)}\bigg{)}.\]

_In particular if a function \(\zeta:\mathbb{N}\times\mathbb{C}^{+}\to\mathbb{R}\) satisfies \(\zeta(n,z)\leq O_{z}(\epsilon_{n})\), then \(\zeta(n,l_{\hat{\nu}}(z))\leq O_{z}(\epsilon_{n})\)._

Proof.: \(\nu=\operatorname{MP}(\gamma_{n})\boxtimes\mu\) is supported on \(\mathbb{R}^{+}\), thus:

\[-\Im\bigg{(}\frac{z}{l_{\hat{\nu}}}\bigg{)} =-\Im(1-\gamma_{n}-\gamma_{n}zg_{\nu})\] \[=\gamma_{n}\int_{\mathbb{R}^{+}}\Im\bigg{(}\frac{z}{t-z}\bigg{)} \nu(dt)\] \[=\gamma_{n}\Im(z)\int_{\mathbb{R}^{+}}\frac{t}{|t-z|^{2}}\nu(dt) \geq 0,\]

which proves that \(\Im\big{(}z^{-1}l_{\hat{\nu}}\big{)}\geq 0\). Secondly:

\[\Im(l_{\hat{\nu}}-z) =\Im\Bigg{(}\gamma_{n}z\int_{\mathbb{R}}\frac{t}{\frac{zt}{l_{ \hat{\nu}}}-z}\mu(dt)\Bigg{)}\] \[=\gamma_{n}\int_{\mathbb{R}^{+}}\Im\bigg{(}\frac{1}{1/l_{\hat{\nu }}-1/t}\bigg{)}\mu(dt)\] \[=\gamma_{n}\int_{\mathbb{R}^{+}}\frac{\Im(l_{\hat{\nu}})t^{2}}{ \Re(l_{\hat{\nu}})^{2}+\Im(l_{\hat{\nu}})^{2}+t^{2}-2\Re(l_{\hat{\nu}})t}\mu( dt)\geq 0.\]
The right hand side integral is bounded from above by \(\dfrac{1}{\Im(l_{\tilde{\nu}})}\int_{\mathbb{R}^{+}}t^{2}\mu(dt)\), hence \(\Im(l_{\tilde{\nu}})^{2}\leq\Im(z)\Im(l_{\tilde{\nu}})+\gamma_{n}\int_{\mathbb{ R}^{+}}t^{2}\mu(dt)\). Solving this second order polynomial inequality gives \(\Im(l_{\tilde{\nu}})\in\left[\Im(z)/2\pm\left(\gamma_{n}\int_{\mathbb{R}^{+}}t^ {2}\mu(dt)+\Im(z)^{2}/4\right)^{1/2}\right]\) and \(\Im(l_{\tilde{\nu}})\leq O(1+\Im(z))\). We also have \(\Im\bigg{(}\dfrac{zt}{l_{\tilde{\nu}}}\bigg{)}\leq 0\) for any \(t\geq 0\), hence:

\[|l_{\tilde{\nu}}| \leq|z|+\gamma_{n}|z|\int_{\mathbb{R}^{+}}\dfrac{t}{\left|\Im \left(\frac{zt}{l_{\tilde{\nu}}}-z\right)\right|}\mu(dt)\] \[\leq|z|+\gamma_{n}\dfrac{|z|}{\Im(z)}\int_{\mathbb{R}^{+}}t\,\mu (dt)\leq O\bigg{(}\dfrac{|z|}{\Im(z)}\bigg{)}.\]

For the last statement, let \(\zeta\) be a function such that \(\zeta(n,z)\leq O_{z}(\epsilon_{n})\). If \(\Im(z)\) is bounded, so is \(\Im(l_{\tilde{\nu}}(z))\), hence there exists \(\alpha>0\) such that:

\[\zeta(n,l_{\tilde{\nu}}(z))\leq O\bigg{(}\epsilon_{n}\dfrac{|l_{\tilde{\nu}}( z)|^{\alpha}}{\Im(l_{\tilde{\nu}}(z))^{2\alpha}}\bigg{)}\leq O\bigg{(} \epsilon_{n}\dfrac{|z|^{\alpha}}{\Im(z)^{3\alpha}}\bigg{)}\leq O\bigg{(} \epsilon_{n}\dfrac{|z|^{2\alpha}}{\Im(z)^{4\alpha}}\bigg{)},\]

from which we deduce that \(\zeta(n,l_{\tilde{\nu}}(z))\leq O_{z}(\epsilon_{n})\). 

We may now move on to the second part of the proof of Theorem 4.7. Let us define the mapping:

\[\mathcal{F}:l\in\mathbb{C}^{+}\mapsto z+\gamma_{n}\dfrac{z}{n}\mathrm{Tr} \bigg{(}\Big{(}\dfrac{z}{l}\Sigma-zI_{n}\Big{)}^{-1}\Sigma\bigg{)}.\]

As shown in Proposition 4.8, the definition of \(\mathcal{F}\) is equivalent to:

\[\mathcal{F}(l)=z+\gamma_{n}z\int_{\mathbb{R}}\dfrac{t}{\frac{zt}{l}-z}\mu_{ \Sigma}(dt)=z+\gamma_{n}l+\gamma_{n}l^{2}g_{\Sigma}(l).\]

We set \(l_{\tilde{\nu}}(z)=-1/g_{\tilde{\nu}}(z)\) and \(l_{\tilde{\chi}}(z)=-1/g_{\tilde{\chi}}(z)\). In Proposition 4.8 we have proved that \(l_{\tilde{\nu}}(z)\) is a fixed point of \(\mathcal{F}\). We will see in the next Proposition that \(l_{\tilde{\chi}}(z)\) is almost a fixed point of \(\mathcal{F}\).

**Proposition 4.10**.: \(|\mathcal{F}(l_{\tilde{\chi}}(z))-l_{\tilde{\chi}}(z)|\leq O_{z}(\epsilon_{n})\)_._

Proof.: Using the first formulation of the self-consistent equation of Proposition 4.8, we have \(l_{\tilde{\chi}}=z+\gamma_{n}n_{l}\tilde{\chi}+\gamma_{n}n_{l}^{2}\tilde{ \chi}_{\tilde{\chi}}g_{\nu}(l_{\tilde{\chi}})\), thus:

\[\mathcal{F}(l_{\tilde{\chi}})-l_{\tilde{\chi}}=\gamma_{nn}l_{\tilde{\chi}}^{2} (g_{\mu_{\Sigma}}(l_{\tilde{\chi}})-g_{\tau}(l_{\tilde{\chi}})).\]

As seen in Lemma 4.9, since \(|g_{\mu_{\Sigma}}(z)-g_{\tau}(z)|\leq O_{z}(\epsilon_{n})\), we also have \(|g_{\mu_{\Sigma}}(l_{\tilde{\chi}}(z))-g_{\tau}(l_{\tilde{\chi}}(z))|\leq O_{z }(\epsilon_{n})\), and we obtain:

\[|\mathcal{F}(l_{\tilde{\chi}})-l_{\tilde{\chi}}| \leq|\gamma_{n_{n}}||l_{\tilde{\chi}}|^{2}|g_{\mu_{\Sigma}}(l_{ \tilde{\chi}})-g_{\tau}(l_{\tilde{\chi}})|\] \[\leq O(1)O\bigg{(}\dfrac{|z|}{\Im(z)}\bigg{)}^{2}O_{z}(\epsilon_{ n})\leq O_{z}(\epsilon_{n}).\]
The last step to prove Theorem 4.7 is to use the stability of the self-consistent equation \(l=\mathcal{F}(l)\). Let us recall the tools and results established in [14, Section 6]. For a fixed \(z\in\mathbb{C}^{+}\), we introduce the domain \(\mathbf{D}=\{\omega\in\mathbb{C}\) such that \(\Im(\omega)\geq\Im(z)\) and \(\Im(z^{-1}\omega)\geq 0\}\), and the semi-metric on \(\mathbb{C}^{+}\):

\[d(\omega_{1},\omega_{2})=\frac{|\omega_{1}-\omega_{2}|}{\Im(\omega_{1})^{1/2} \Im(\omega_{2})^{1/2}}.\]

\(\mathcal{F}\) is a contraction mapping on \(\mathbf{D}\) with respect to \(d\). More precisely, \(\mathcal{F}\) is \(k_{\mathcal{F}}\)-Lipschitz with \(k_{\mathcal{F}}=\frac{\frac{|z|}{\Im(z)^{2}}}{1+\frac{|z|}{\Im(z)^{2}}}\) ([14, Proposition 6.11]). Moreover, if \(c\in\mathbf{D}\) is a fixed point of \(\mathcal{F}\) and \(b\in\mathbf{D}\) any other point, provided \(k_{\mathcal{F}}(1+d(b,\mathcal{F}(b)))<1\), the following inequality holds true ([14, Lemma 6.14]):

\[|c-b|\leq\frac{|\mathcal{F}(b)-b|}{1-k_{\mathcal{F}}(1+d(b,\mathcal{F}(b)))}.\]

We have now collected all the arguments required to compare the Stieltjes transforms of \(\nu\) and \(\chi\).

Proof of Theorem 4.7.: If \(\Im(z)\) is bounded, using Proposition 4.10 there exists \(\alpha\) and \(C>0\) such that \(|\mathcal{F}(l_{\tilde{\chi}})-l_{\tilde{\chi}}|\leq C\epsilon_{n}\frac{|z|^{ \alpha}}{\Im(z)^{2\alpha}}\). For values of \(z\) such that

\[\epsilon_{n}\frac{|z|^{\alpha+1}}{\Im(z)^{2\alpha+3}}\leq 1/2C,\,\frac{|z|}{ \Im(z)^{2}}d(\mathcal{F}(l_{\tilde{\chi}}),l_{\tilde{\chi}})\leq 1/2,\,\text{ thus:}\]

\[k_{\mathcal{F}}(1+d(\mathcal{F}(l_{\tilde{\chi}}),l_{\tilde{\chi}})) =\frac{\frac{|z|}{\Im(z)^{2}}+\frac{|z|}{\Im(z)^{2}}d(\mathcal{F }(l_{\tilde{\chi}}),l_{\tilde{\chi}})}{1+\frac{|z|}{\Im(z)^{2}}}\] \[\leq 1-\frac{1}{2\Big{(}1+\frac{|z|}{\Im(z)^{2}}\Big{)}}.\]

In particular \(k_{\mathcal{F}}(1+d(\mathcal{F}(l_{\tilde{\chi}}),l_{\tilde{\chi}}))<1\), and from [14, Lemma 6.14]:

\[|l_{\tilde{\chi}}-l_{\tilde{\nu}}| \leq\frac{|\mathcal{F}(l_{\tilde{\chi}})-l_{\tilde{\chi}}|}{1-k_ {\mathcal{F}}(1+d(\mathcal{F}(l_{\tilde{\chi}}),l_{\tilde{\chi}}))}\] \[\leq 2\bigg{(}1+\frac{|z|}{\Im(z)^{2}}\bigg{)}C\epsilon_{n}\frac{|z| ^{\alpha}}{\Im(z)^{2\alpha}}\] \[\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha+1}}{\Im(z)^{2\alpha+ 2}}\bigg{)}.\]

To conclude:

\[|g_{\nu}-g_{\chi}|=\frac{|g_{\tilde{\nu}}-g_{\tilde{\chi}}|}{\gamma}=\frac{| g_{\tilde{\nu}}||g_{\tilde{\chi}}||l_{\tilde{\nu}}-l_{\tilde{\chi}}|}{ \gamma_{n}}\leq O\bigg{(}\epsilon_{n}\frac{|z|^{\alpha+1}}{\Im(z)^{2\alpha+2 }}\bigg{)},\]

for values of \(z\) such that \(\epsilon_{n}\frac{|z|^{\alpha+1}}{\Im(z)^{2\alpha+3}}\leq\frac{1}{2C}\). Given the _a priori_ bound \(|g_{\nu}(z)-g_{\chi}(z)|\leq\frac{2}{\Im(z)}\) and Lemma 2.6, the bound \(|g_{\nu}(z)-g_{\chi}(z)|\leq O_{z}(\epsilon_{n})\) holds true.
### Approximation of deterministic equivalent built from deterministic matrices

Given an approximation for the Stieltjes transform \(g_{\Sigma}(z)\approx g_{\tau}(z)\), and another approximation for the resolvent \(\mathcal{G}_{\Sigma}(z)\approx\mathbf{H}(z)\), can we find an approximation for the matrix \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\)? We build a matrix function \(\mathbf{K}\) using the same procedure we used earlier to build \(\mathbf{G}_{\boxtimes}^{\Sigma}\) from \(\mu_{\Sigma}\) and \(\mathcal{G}_{\Sigma}\):

\[\chi =\mathrm{MP}(\gamma_{n})\boxtimes\tau,\] \[\tilde{\chi} =(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot\chi,\] \[l_{\tilde{\chi}}(z) =-1/g_{\tilde{\chi}}(z),\] \[\mathbf{K}(z) =z^{-1}l_{\tilde{\chi}}(z)\mathbf{H}(l_{\tilde{\chi}}(z)).\]

**Proposition 4.11**.: _Let \(\Sigma\in\mathbb{R}^{n\times n}\) be deterministic positive semi-definite matrices, \(\tau\) deterministic probability measures supported on \(\mathbb{R}^{+}\), and \(\mathbf{H}:\mathbb{C}^{+}\to\mathbb{C}^{n\times n}\) deterministic complex functions._

_If \(|g_{\Sigma}(z)-g_{\tau}(z)|\leq O_{z}(\epsilon_{n})\) and \(\left\|\!\left|\mathcal{G}_{\Sigma}(z)-\mathbf{H}(z)\right|\!\right|\leq O_{z }(\epsilon_{n}^{\prime})\), then \(\left\|\!\left|\mathbf{G}_{\boxtimes}^{\Sigma}(z)-\mathbf{K}(z)\right|\! \right|\leq O_{z}(\epsilon_{n}+\epsilon_{n}^{\prime})\)._

Proof.: We use a triangular inequality in the following decomposition:

\[\mathbf{G}_{\boxtimes}^{\Sigma}(z)-\mathbf{K}(z) =z^{-1}(l_{\tilde{\nu}}-l_{\tilde{\chi}})\mathcal{G}_{\Sigma}(l_{ \tilde{\nu}})\] \[\qquad+z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{\tilde{\nu }})-\mathcal{G}_{\Sigma}(l_{\tilde{\chi}}))\] \[\qquad+z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{\tilde{ \chi}})-\mathbf{H}(l_{\tilde{\chi}})).\]

For the first term, \(|l_{\tilde{\nu}}-l_{\tilde{\chi}}|\leq O_{z}(\epsilon_{n})\) and \(\left\|\!\left|\mathcal{G}_{\Sigma}(l_{\tilde{\nu}})\right|\!\right|\leq\dfrac {1}{\Im(l_{\tilde{\nu}})}\leq\dfrac{1}{\Im(z)}\), thus

\(\left\|\!\left|z^{-1}(l_{\tilde{\nu}}-l_{\tilde{\chi}})\mathcal{G}_{\Sigma}(l _{\tilde{\nu}})\right|\!\right\|\leq O_{z}(\epsilon_{n})\). For the second term, using a resolvent identity:

\[\left\|\!\left|z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{ \tilde{\nu}})-\mathcal{G}_{\Sigma}(l_{\tilde{\chi}}))\right|\!\right\| \leq|z^{-1}l_{\tilde{\chi}}(z)|\!\left|\!\left|\mathcal{G}_{\Sigma}( l_{\tilde{\nu}})\right|\!\right|\!\left|l_{\tilde{\nu}}-l_{\tilde{\chi}}|\! \right|\!\left|\!\left|\mathcal{G}_{\Sigma}(l_{\tilde{\chi}})\right|\!\right|\] \[\leq\dfrac{1}{|z|}O\bigg{(}\dfrac{|z|}{\Im(z)}\bigg{)}\dfrac{O_{z }(\epsilon_{n})}{\Im(z)^{2}}\leq O_{z}(\epsilon_{n}).\]

Finally for the third term, \(\left\|\!\left|\mathcal{G}_{\Sigma}(l_{\tilde{\chi}})-\mathbf{H}(l_{\tilde{ \chi}})\right|\!\right|\leq O_{z}(\epsilon_{n}^{\prime})\) using Lemma 4.9, thus:

\(\left\|\!\left|z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{\tilde{\chi}})- \mathbf{H}(l_{\tilde{\chi}}))\right|\!\right|\leq\dfrac{1}{|z|}O\bigg{(}\dfrac {|z|}{\Im(z)}\bigg{)}O_{z}(\epsilon_{n}^{\prime})\leq O_{z}(\epsilon_{n}^{ \prime}).\)

**Corollary 4.12**.: _The map \(\Sigma\mapsto\mathbf{G}_{\boxtimes}^{\Sigma}(z)\) is \(O_{z}(1)\) Lipschitz with respect to the spectral norm._

Proof.: Using a resolvent identity we have:

\(\left|\!\left|\!\left|\mathcal{G}_{\Sigma}(z)-\mathcal{G}_{\Sigma^{\prime}}(z) \right|\!\right|\leq\left|\!\left|\mathcal{G}_{\Sigma}(z)\right|\!\right|\! \right|\!\left|\!\left|\mathcal{\Sigma}^{\prime}-\Sigma\right|\!\right|\! \left|\!\left|\!\left|\mathcal{G}_{\Sigma}(z)\right|\!\right|\!\right|\leq\dfrac {\left|\!\left|\!\left|\Sigma-\Sigma^{\prime}\right|\!\right|\!\right|}{\Im(z) ^{2}}\).

In particular \(\left|g_{\Sigma}(z)-g_{\Sigma^{\prime}(z)}\right|\leq\left|\!\left|\mathcal{G}_{ \Sigma}(z)-\mathcal{G}_{\Sigma^{\prime}}(z)\right|\!\right|\leq O_{z}\big{(} \!\left|\!\left|\!\left|\Sigma-\Sigma^{\prime}\right|\!\right|\!\right|\! \big{)}\). The result follows from Theorem 4.7 and Proposition 4.11.
### Concentration of deterministic equivalents built from random matrices

If \(\Sigma\) is random and satisfies a typical \(\mathcal{E}(1/\sqrt{n})\) Lipschitz concentration property, from the approximations \(\mathbb{E}[g_{\Sigma}(z)]\approx g_{\tau}(z)\) and \(\mathbb{E}[\mathcal{G}_{\Sigma}(z)]\approx\mathbf{H}(z)\), we may deduce that \(g_{\Sigma}(z)\approx g_{\tau}(z)\) a.s., but we cannot expect that \(\mathcal{G}_{\Sigma}(z)\approx\mathbf{H}(z)\) since \(\mathcal{G}_{\Sigma}(z)\) and \(\mathbb{E}[\mathcal{G}_{\Sigma}(z)]\) are not necessarily close in spectral norm. We can however prove that \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\) is linearly concentrated around \(\mathbf{K}(z)\).

**Proposition 4.13**.: _Let \(\Sigma\in\mathbb{R}^{n\times n}\) be random positive semi-definite matrices such that \(\Sigma\propto_{\left|\kern-1.0pt\left|\cdot\right|\kern-1.0pt\right|_{F}} \mathcal{E}(1/\sqrt{n})\). Then:_

1. \(g_{\nu}(z)\propto\mathcal{E}(O_{z}(1/n))\) _and_ \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\propto_{\left|\kern-1.0pt\left|\cdot\right| \kern-1.0pt\right|}\mathcal{E}(O_{z}(1/\sqrt{n}))\)_._
2. _If_ \(\tau\) _are deterministic probability measures supported on_ \(\mathbb{R}^{+}\) _such that_ \(\left|\mathbb{E}[g_{\Sigma}(z)]-g_{\tau}(z)\right|\leq O_{z}(\epsilon_{n})\)_, then_ \(\left|g_{\nu}(z)-g_{\chi}(z)\right|\leq O_{z}(\epsilon_{n}+\sqrt{\log n}/n)\) _a.s._
3. _If in addition_ \(\mathbf{H}:\mathbb{C}^{+}\to\mathbb{C}^{n\times n}\) _are deterministic complex functions such that_ \(\left|\kern-1.0pt\left|\mathbb{E}[\mathcal{G}_{\Sigma}(z)]-\mathbf{H}(z) \right|\kern-1.0pt\right|\kern-1.0pt\right|\leq O_{z}(\epsilon_{n}^{\prime})\)_, then_ \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\in_{\left|\kern-1.0pt\left|\cdot\right| \kern-1.0pt\right|}\mathbf{K}(z)\pm\mathcal{E}(O_{z}(\epsilon_{n}+\epsilon_{n} ^{\prime}+1/\sqrt{n}))\)_._

Proof.: We refer to Proposition 2.2 for the properties of Lipschitz and linear concentration used in this proof. The map \(\Sigma\mapsto\mathbf{G}_{\boxtimes}^{\Sigma}\) is \(O_{z}(1)\) Lipschitz with respect to the spectral norm, thus \(\mathbf{G}_{\boxtimes}^{\Sigma}\propto_{\left|\kern-1.0pt\left|\cdot\right| \kern-1.0pt\right|}\mathcal{E}(O_{z}(1/\sqrt{n}))\). Remembering the identity \(\mathrm{Tr}\mathbf{G}_{\boxtimes}^{\Sigma}=ng_{\nu}\), we also deduce that \(g_{\nu}\propto\mathcal{E}(O_{z}(1/\sqrt{n}))\).

The map \(\Sigma\mapsto\mathcal{G}_{\Sigma}\) is \(1/\Im(z)^{2}\) Lipschitz, thus \(\mathcal{G}_{\Sigma}\propto_{\left|\kern-1.0pt\left|\cdot\right|\kern-1.0pt \right|_{F}}\mathcal{E}(O_{z}(1/\sqrt{n}))\) and \(g_{\Sigma}\propto\mathcal{E}(O_{z}(1/n))\). We deduce that \(\left|\mathbb{E}[g_{\Sigma}]-g_{\Sigma}\right|\leq O_{z}(\sqrt{\log n}/n)\) a.s., hence \(\left|g_{\Sigma}-g_{\tau}\right|\leq O_{z}(\epsilon_{n}+\sqrt{\log n}/n)\) a.s. We can apply Theorem 4.7 uniformly in this set of full measure, and obtain that \(\left|g_{\nu}-g_{\chi}\right|\leq O_{z}(\epsilon_{n}+\sqrt{\log n}/n)\) a.s.

In the decomposition:

\[\mathbf{G}_{\boxtimes}^{\Sigma}(z)-\mathbf{K}(z) =z^{-1}(l_{\tilde{\nu}}-l_{\tilde{\chi}})\mathcal{G}_{\Sigma}(l_{ \tilde{\nu}})\] \[\qquad+z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{\tilde{\nu }})-\mathcal{G}_{\Sigma}(l_{\tilde{\chi}}))\] \[\qquad+z^{-1}l_{\tilde{\chi}}(\mathcal{G}_{\Sigma}(l_{\tilde{\chi }})-\mathbf{H}(l_{\tilde{\chi}})),\]

the first two terms are bounded by \(O_{z}(\epsilon_{n}+\sqrt{\log n}/n)\leq O_{z}(\epsilon_{n}+1/\sqrt{n})\) a.s. in spectral norm (see the proof of Proposition 4.11). For the third term, \(\mathcal{G}_{\Sigma}(z)\propto_{\left|\kern-1.0pt\left|\cdot\right|\kern-1.0pt \right|_{F}}\mathcal{E}(O_{z}(1/\sqrt{n}))\) and \(\left|\kern-1.0pt\left|\mathbb{E}[\mathcal{G}_{\Sigma}(z)]-\mathbf{H}(z) \right|\kern-1.0pt\right|\leq O_{z}(\epsilon_{n}^{\prime})\), thus \(\mathcal{G}_{\Sigma}(z)\in_{\left|\kern-1.0pt\left|\cdot\right|\kern-1.0pt \right|}\mathbf{H}(z)\pm\mathcal{E}(O_{z}(\epsilon_{n}^{\prime}+1/\sqrt{n}))\). From Lemma 4.9, we also have \(\mathcal{G}_{\Sigma}(l_{\tilde{\chi}})\in_{\left|\kern-1.0pt\left|\cdot\right| \kern-1.0pt\right|}\mathbf{H}(l_{\tilde{\chi}})\pm\mathcal{E}(O_{z}(\epsilon_{ n}^{\prime}+1/\sqrt{n}))\). Finally \(\left|z^{-1}l_{\tilde{\chi}}\right|\leq 1/\Im(z)\), and combining the above estimates and concentration properties leads to \(\mathbf{G}_{\boxtimes}^{\Sigma}(z)\in_{\left|\kern-1.0pt\left|\cdot\right| \kern-1.0pt\right|}\mathbf{K}(z)\pm\mathcal{E}(O_{z}(\epsilon_{n}+\epsilon_{n} ^{\prime}+1/\sqrt{n}))\).
## 5. Single-layer neural network with deterministic data

### Setting

In this section we consider the Conjugate Kernel matrix associated to a single-layer artificial neural network with deterministic input. The model is made of:

* a random weight matrix \(W\in\mathbb{R}^{d\times d_{0}}\), with variance parameter \(\sigma_{W}^{2}>0\),
* a deterministic data matrix \(X\in\mathbb{R}^{d_{0}\times n}\), \(\sigma_{X}^{2}>0\),
* two random biases matrices \(B\) and \(D\in\mathbb{R}^{d\times n}\), \(\sigma_{B}^{2},\sigma_{D}^{2}\geq 0\),
* and an activation function \(f:\mathbb{R}\to\mathbb{R}\).

As output of the neuron, we set \(Y=f(WX/\sqrt{d_{0}}+B)+D\in\mathbb{R}^{d\times n}\), where the function \(f\) is applied entry-wise. Our goal is to investigate the spectral properties of the Conjugate Kernel matrix \(K=Y^{\top}Y/d\). For \(z\in\mathbb{C}^{+}\) we define its resolvent \(\mathcal{G}_{K}(z)=\left(K-zI_{n}\right)^{-1}\) and Stieltjes transform \(g_{K}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{K}(z)\). We also define the following objects:

\[\tilde{\sigma}^{2} =\sigma_{W}^{2}\sigma_{X}^{2}+\sigma_{B}^{2},\] \[\tilde{f}(t) =f(\tilde{\sigma}t),\] \[\mathfrak{a} =\left\|\tilde{f}\right\|_{\mathcal{H}}^{2}-\frac{\sigma_{W}^{2 }\sigma_{X}^{2}}{\tilde{\sigma}^{2}}\zeta_{1}(\tilde{f})^{2}+\sigma_{D}^{2},\] \[\mathfrak{b} =\zeta_{1}(\tilde{f})^{2}\frac{\sigma_{W}^{2}}{\tilde{\sigma}^{ 2}},\] \[K_{X} =X^{\top}X/d_{0},\] \[\Delta_{X} =K_{X}-\sigma_{X}^{2}I_{n},\] \[\Sigma =\mathbb{E}[K],\] \[\Sigma_{\mathrm{lin}} =\mathfrak{a}I_{n}+\mathfrak{b}K_{X}.\]

_Remark 5.1_.: We always have \(\mathfrak{a}\geq 0\) since \(\zeta_{1}(f)^{2}\leq\left\|\tilde{f}\right\|_{\mathcal{H}}^{2}\) and \(\sigma_{W}^{2}\sigma_{X}^{2}\leq\tilde{\sigma}^{2}\). Moreover \(\mathfrak{a}=0\) if and only if \(f\) is a linear function and there is no bias in the model (that is \(\sigma_{B}^{2}=\sigma_{D}^{2}=0\)).

Like in the rest of this article, we sometimes omit the indices \(n\) and \(z\) for a better readability, even if we are implicitly dealing with sequences of matrices, measures, and complex functions.

**Assumptions 5.2**.:
1. \(W\), \(B\) and \(D\) are random, independent, with i.i.d. \(\mathcal{N}(\sigma_{W}^{2})\), \(\mathcal{N}(\sigma_{B}^{2})\) and \(\mathcal{N}(\sigma_{D}^{2})\) entries respectively.
2. \(\tilde{f}\) is Lipschitz continuous and Gaussian centered, that is \(\mathbb{E}\Big{[}\tilde{f}(\mathcal{N})\Big{]}=\mathbb{E}[f(\tilde{\sigma} \mathcal{N})]=0\).
3. \(X\) is deterministic and is bounded.
4. \(\left\|\mathrm{diag}(\Delta_{X})\right\|\) is bounded and converges to \(0\).
5. The ratio \(\gamma_{n}=\frac{n}{d}\) is bounded from above and away from \(0\).

We refer to the Remark 5.8 for a detailed discussion about the assumption (4). The main result of this section is Theorem 5.7 that gives a deterministic equivalent for \(\mathcal{G}_{K}(z)\) and \(g_{K}(z)\). To prove this result, we will combine the
general results on resolvent matrices recalled in Section 4, with the linearization techniques of Section 3. We will wrap up this section by applying our framework to a simple yet original model having weakly correlated entries.

### Technicalities and linearization of \(\Sigma\)

**Proposition 5.3**.: _Under Assumptions 5.2, \(Y\propto_{\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt\right|_{F}}\mathcal{ E}(1)\). The rows of \(Y\) are i.i.d. sampled from the distribution of a random vector \(y=f(X^{\top}w/\sqrt{d_{0}}+b)+\tilde{b}\), where \(w\in\mathbb{R}^{d_{0}}\) and \(b,\tilde{b}\in\mathbb{R}^{n}\) are independent Gaussian vectors with i.i.d. \(\mathcal{N}(\sigma_{W}^{2})\), \(\mathcal{N}(\sigma_{B}^{2})\) and \(\mathcal{N}(\sigma_{D}^{2})\) coordinates respectively. \(\left|\kern-1.075pt\left|\mathbb{E}[y]\right|\kern-1.075pt\right|\) is moreover bounded._

Proof.: The map \((W,B,D)\mapsto f(WX/\sqrt{d_{0}}+B)+D\) is Lipschitz with respect to the product Frobenius norm since \(f\) is Lipschitz and \(\left|\kern-1.075pt\left|\kern-1.075pt\left|X/\sqrt{d_{0}}\right|\kern-1.075pt \right|\kern-1.075pt\right|\) is bounded. Given the Gaussian concentration \((W,B,D)\propto_{\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt\right|_{F}} \mathcal{E}(1)\), we immediately obtain that \(Y\propto_{\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt\right|_{F}} \mathcal{E}(1)\). From the expression:

\[Y_{ij}=f\!\left(\sum_{k=1}^{d_{0}}W_{ik}X_{kj}/\sqrt{d_{0}}+B_{ij}\right)+D_{ ij},\]

we see that the rows of \(Y\) are independent and have the same distribution as \(y=f\!\left(X^{\top}w/\sqrt{d_{0}}+b\right)+\tilde{b}\). For the last statement, we may write \(y=\tilde{f}(u)+\tilde{b}\), where \(u\) is a centered Gaussian vector with covariance matrix \(S=\big{(}\sigma_{W}^{2}K_{X}+\sigma_{B}^{2}I_{n}\big{)}/\tilde{\sigma}^{2}\). We have \(S-I_{n}=(\sigma_{W}^{2}/\tilde{\sigma}^{2})\Delta_{X}\), hence the random variables \(u_{i}\) are centered Gaussian with covariance \(1+O((\Delta_{X})_{ii})\). Since \(\zeta_{0}(\tilde{f})=\mathbb{E}[f(\tilde{\sigma}\mathcal{N})]=0\), using the first order expansion given by Corollary 3.4 applied to the function \(\tilde{f}\), we have \(\mathbb{E}\!\left[\tilde{f}(u_{i})\right]=O((\Delta_{X})_{ii})\) uniformly on \(i\in\llbracket 1,n\rrbracket\). We deduce that \(\left|\kern-1.075pt\left|\mathbb{E}[y]\right|\kern-1.075pt\right|=\left| \kern-1.075pt\left|\mathbb{E}\!\left[\tilde{f}(u)\right]\right|\kern-1.075pt \right|=O\!\left(\left|\kern-1.075pt\left|\mathrm{diag}(\Delta_{X})\right| \kern-1.075pt\right|\kern-1.075pt\right|\kern-1.075pt\right)=O(1)\). 

**Corollary 5.4**.: _Under Assumptions 5.2, \(\left|\kern-1.075pt\left|\Sigma\right|\kern-1.075pt\right|\kern-1.075pt\right|\) is bounded, and moreover:_

\[\left|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin}}\right| \kern-1.075pt\right|\leq O(\left|\kern-1.075pt\left|\Delta_{X} \right|\kern-1.075pt\right|_{\max}+n\zeta_{2}(\tilde{f})^{2}\|\Delta_{X}\|_{ \max}^{2}+n\zeta_{3}(\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{3}),\] \[\left|\kern-1.075pt\left|g_{\Sigma}(z)-g_{\Sigma_{\mathrm{lin}}}( z)\right|\kern-1.075pt\right|\leq O(\left|\kern-1.075pt\left|\Delta_{X}\right|\kern-1.075pt\right|_{ \max}+\sqrt{n}\zeta_{2}(\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{2}+\sqrt{n}\zeta_{3 }(\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{3}).\]

Proof.: As seen in the last Proposition, the rows of \(Y\) are i.i.d. sampled from the distribution of a vector \(y=\tilde{f}(u)+\tilde{b}\) where \(u\) is a centered Gaussian vector with covariance matrix \(S=I_{n}+\Delta\), and \(\Delta=(\sigma_{W}^{2}/\tilde{\sigma}^{2})\Delta_{X}\). The Assumptions 3.8 are satisfied, and we deduce from Proposition 3.10 that \(\left|\kern-1.075pt\left|\kern-1.075pt\left|\Sigma-\Sigma_{\mathrm{lin}}\right| \kern-1.075pt\right|\kern-1.075pt\right|\) is bounded. Since \(\left|\kern-1.075pt\left|\kern-1.075pt\left|K_{X}\right|\kern-1.075pt\right| \kern-1.075pt\right|\kern-1.075pt\right|\kern-1.075pt\right|\kern-1.075pt\) is bounded and \(\Sigma_{\mathrm{lin}}=\mathfrak{a}I_{n}+\mathfrak{b}K_{X}\), \(\left|\kern-1.075pt\left|\Sigma\right|\kern-1.075pt\right|\kern-1.075pt\) is also bounded. From the same proposition, we have the following estimates
in spectral norm, with \(\epsilon_{n}=\left\|\Delta_{X}\right\|_{\max}+n\zeta_{2}(\tilde{f})^{2}\|\Delta_{X }\|_{\max}^{2}+n\zeta_{3}(\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{3}\):

\[\mathbb{E}\Big{[}\tilde{f}(u)\tilde{f}(u)^{\top}\Big{]} =\left\|\tilde{f}\right\|^{2}I_{n}+\zeta_{1}(\tilde{f})^{2}\Delta +O(\epsilon_{n})\] \[=\left\|\tilde{f}\right\|^{2}I_{n}+\zeta_{1}(\tilde{f})^{2}( \sigma_{W}^{2}/\tilde{\sigma}^{2})\big{(}K_{X}-\sigma_{X}^{2}I_{n}\big{)}+O( \epsilon_{n})\] \[\Sigma=\mathbb{E}\Big{[}yy^{\top}\Big{]} =\mathbb{E}\Big{[}\tilde{f}(u)\tilde{f}(u)^{\top}\Big{]}+\sigma_ {D}^{2}I_{n}\] \[=\left(\left\|\tilde{f}\right\|^{2}-\frac{\sigma_{W}^{2}\sigma_{X }^{2}}{\tilde{\sigma}^{2}}\zeta_{1}(\tilde{f})^{2}+\sigma_{D}^{2}\right)I_{n} +\zeta_{1}(\tilde{f})^{2}\frac{\sigma_{W}^{2}}{\tilde{\sigma}^{2}}K_{X}+O( \epsilon_{n})\] \[=\Sigma_{\mathrm{lin}}+O(\epsilon_{n}).\]

The proof for the Stieltjes transforms is similar. 

### Propagation of the approximate orthogonality

The contents of this paragraph will not be useful in this section, but rather in later stages of the article to study multi-layers networks by induction. Similar results may be found in [10, Section D] under the name of propagation of approximate orthogonality (see Remark 5.8 for more details on this concept), and proved by slightly different means.

**Lemma 5.5**.: _Let \(\sigma_{Y}^{2}=\left\|\tilde{f}\right\|_{\mathcal{H}}^{2}+\sigma_{D}^{2}\), and \(\Delta_{Y}=K-\sigma_{Y}^{2}I_{n}=Y^{\top}Y/d-\sigma_{Y}^{2}I_{n}\). Under Assumptions 5.2, there exists an event \(\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{c})\leq ce^{-n/c}\) for some constant \(c>0\), such that uniformly on \(\mathcal{B}\), \(\left\|\mathrm{\widetilde{diag}}(\Delta_{Y})\right\|\) and \(\left\|\!\left|K\right|\!\right|\) are bounded, and \(\left\|\!\left|\Delta_{Y}\right\|\!\right|_{\max}\leq O\Big{(}\left\|\Delta_{ X}\right\|\!\right|_{\max}+\sqrt{\log n/n}\Big{)}\)._

Proof.: We have \(\left|\!\left|\!\left|\mathbb{E}[Y]\right|\!\right|^{2}\leq\left\|\mathbb{E}[ Y]\right\|_{F}^{2}\leq d\left\|\mathbb{E}[y]\right\|^{2}\leq O(n)\). From Proposition 2.3, there exists a constant \(c>0\) and an event \(\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{c})\leq ce^{-n/c}\), such that \((K|\mathcal{B})\propto_{\left\|\!\left|\cdot\right|\!\right|_{F}}\mathcal{E}(1/ \sqrt{n})\), and \(\left\|\!\left|\!\left|Y\right|\!\right|\leq 2c\sqrt{n}\) on \(\mathcal{B}\), thus \(\left|\!\left|\!\left|K\right|\!\right|\!\right|\) is bounded on \(\mathcal{B}\). We will check the other statements in expectation first, and in a second stage use concentration to obtain bounds on the random objects.

For any \(i\in\llbracket 1,n\rrbracket\), \(Y_{ii}\) has the same distribution as \(\tilde{f}(u_{i})+\tilde{b}_{i}\), where \(u_{i}\) and \(\tilde{b}_{i}\) are centered Gaussian variables, independent, with variances \(1+O((\Delta_{X})_{ii})\) and \(\sigma_{D}^{2}\) respectively. Using the first order expansion given by Corollary 3.4 applied to the function \(\tilde{f}^{2}\), uniformly on \(i\in\llbracket 1,n\rrbracket\) we have:

\[\mathbb{E}\big{[}Y_{ii}^{2}\big{]} =\mathbb{E}\bigg{[}\Big{(}\tilde{f}(u_{i})+b_{i}\Big{)}^{2}\bigg{]}\] \[=\zeta_{0}(\tilde{f}^{2})+O((\Delta_{X})_{ii})+\sigma_{D}^{2}\] \[=\left\|\tilde{f}\right\|_{\mathcal{H}}^{2}+\sigma_{D}^{2}+O(( \Delta_{X})_{ii})\] \[=\sigma_{Y}^{2}+O((\Delta_{X})_{ii}).\]

We deduce that \(\mathbb{E}\bigg{[}\left\|\mathrm{\widetilde{diag}}(\Delta_{Y})\right\|^{2} \bigg{]}\leq\sum_{i=1}^{n}O\big{(}(\Delta_{X})_{ii}^{2}\big{)}\leq O\bigg{(} \left\|\mathrm{\widetilde{diag}}(\Delta_{X})\right\|^{2}\bigg{)}\leq O(1)\). Since \(\Big{(}\mathrm{\widetilde{diag}}(\Delta_{Y})|\mathcal{B}\Big{)}\propto_{ \left\|\!\left|\cdot\right|\!\right|}\mathcal{E}(1/\sqrt{n})\), \(\left\|\mathrm{\widetilde{diag}}(\Delta_{Y})-\mathbb{E}\Big{[}\mathrm{ \widetilde{diag}}(\Delta_{Y})\Big{]}\right\|\leq O(1)\) a.s. on \(\mathcal{B}\), and \(\left\|\mathrm{\widetilde{diag}}(\Delta_{Y})\right\|\leq O(1)\) a.s. on \(\mathcal{B}\).
Finally \(\left\|K-\Sigma\right\|_{\max}\leq O\Big{(}\sqrt{\log n/n}\Big{)}\) a.s. on \(\mathcal{B}\) using the general properties of concentration (see Proposition 2.2(10)), and from Proposition 3.10:

\[\left\|\Sigma-\sigma_{Y}^{2}I_{n}\right\|_{\max}\leq\left\|\Sigma-\Sigma_{\rm lin }\right\|_{\max}+O(\left\|\Delta_{X}\right\|_{\max})\leq O(\left\|\Delta_{X} \right\|_{\max}),\]

which proves the result after a final triangular inequality. 

### Deterministic equivalent and consequences

We let \(\nu^{\Sigma_{\rm lin}}=\mathrm{MP}(\gamma_{n})\boxtimes\mu_{\Sigma_{\rm lin}}\), and we refer to the beginning of Section 4 for the definition of the deterministic equivalent matrix \(\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z)\).

_Remark 5.6_.: If \(\mathfrak{b}=0\), then \(\Sigma_{\rm lin}\) does not depend on \(X\) and the objects \(\nu^{\Sigma_{\rm lin}}\) and \(\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z)\) are fully explicit. Indeed \(\Sigma_{\rm lin}=\mathfrak{a}I_{n}\), \(\mu_{\Sigma_{\rm lin}}=\delta_{\mathfrak{a}}\), and \(\nu^{\Sigma_{\rm lin}}=\mathrm{MP}(\gamma_{n})\boxtimes\delta_{\mathfrak{a}}= \mathfrak{a}\mathrm{MP}(\gamma_{n})\). Since \(\Sigma_{\rm lin}\) is a multiple of the identity matrix, so are \(\mathcal{G}_{\Sigma_{\rm lin}}(z)\) and \(\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}\), hence:

\[\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z)=\bigg{(}\frac{1}{n}\mathrm{Tr} \mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z)\bigg{)}I_{n}=g_{\mathfrak{a} \mathrm{MP}(\gamma_{n})}(z)I_{n}.\]

Note that if \(f\neq 0\), then \(\mathfrak{a}\neq 0\) and \(g_{\mathfrak{a}\mathrm{MP}(\gamma_{n})}(z)=\mathfrak{a}g_{\mathrm{MP}(\gamma_{ n})}(z/\mathfrak{a})\).

In the case where \(\mathfrak{b}\neq 0\), we can describe the deterministic equivalents as functions of \(X\). Indeed \(\mu_{\Sigma_{\rm lin}}=\mathfrak{a}+\mathfrak{b}\mu_{K_{X}}\), \(\nu^{\Sigma_{\rm lin}}=\mathrm{MP}(\gamma_{n})\boxtimes(\mathfrak{a}+ \mathfrak{b}\mu_{K_{X}})\), and since \(\mathcal{G}_{\Sigma_{\rm lin}}(z)=\frac{1}{\mathfrak{b}}\mathcal{G}_{K_{X}} \bigg{(}\frac{z-\mathfrak{a}}{\mathfrak{b}}\bigg{)}\):

\[\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z) =z^{-1}l_{\tilde{\nu}^{\Sigma_{\rm lin}}}(z)\mathcal{G}_{L}(l_{ \tilde{\nu}^{\Sigma_{\rm lin}}}(z))\] \[=\frac{l_{\tilde{\nu}^{\Sigma_{\rm lin}}}(z)}{\mathfrak{b}z} \mathcal{G}_{K_{X}}\bigg{(}\frac{l_{\tilde{\nu}^{L}}(z)-\mathfrak{a}}{ \mathfrak{b}}\bigg{)},\]

where \(\tilde{\nu}^{\Sigma_{\rm lin}}=(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot \nu^{\Sigma_{\rm lin}}\) and \(l_{\tilde{\nu}^{\Sigma_{\rm lin}}}(z)=-1/g_{\tilde{\nu}^{\Sigma_{\rm lin}}}(z)\).

For the next result let us denote:

\[\epsilon_{n} =\frac{1}{n}+\left\|\Delta_{X}\right\|_{\max}+\sqrt{n}\zeta_{2} (\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{2}+\sqrt{n}\zeta_{3}(\tilde{f})^{2}\| \Delta_{X}\|_{\max}^{3},\] \[\epsilon_{n}^{\prime} =\frac{1}{\sqrt{n}}+\left\|\Delta_{X}\right\|_{\max}+n\zeta_{2} (\tilde{f})^{2}\|\Delta_{X}\|_{\max}^{2}+n\zeta_{3}(\tilde{f})^{2}\|\Delta_{X} \|_{\max}^{3}.\]

**Theorem 5.7**.: _Uniformly under Assumptions 5.2, the following concentration properties hold true:_

1. \(g_{K}(z)\propto\mathcal{E}(O_{z}(1/n))\) _and_ \(\mathcal{G}_{K}(z)\propto_{\left|\kern-1.0pt\cdot\kern-1.0pt\right|_{F}} \mathcal{E}\big{(}O_{z}(1/\sqrt{n})\big{)}\)_._
2. \(g_{K}(z)\in g_{\nu^{L}}(z)\pm\mathcal{E}(O_{z}(\epsilon_{n}))\) _and_ \(\mathcal{G}_{K}(z)\in_{\left|\kern-1.0pt\cdot\kern-1.0pt\right|\kern-1.0pt \right|}\mathbf{G}_{\boxtimes}^{\Sigma_{\rm lin}}(z)\pm\mathcal{E}\big{(}O_{z }(\epsilon_{n}^{\prime})\big{)}\)_._

_Remark 5.8_.: Assumption 5.2(4) states roughly that the data matrix is close to being an orthogonal matrix (up to rescaling), a notion that was first introduced in [13] under the name of approximate orthogonality. If this is true, the Conjugate Kernel model may be compared to a classical model derived from matrices with i.i.d. entries. To clarify this point, the assumption on \(\left\|\Delta_{X}\right\|_{\max}\) measures how the entries of \(Y\) are dependent and how far they are from being standard Gaussian random variables individually. The additional bound on \(\left\|\vec{\mathrm{diag}}(\Delta_{X})\right\|\) ensures that the latter phenomenon occurs somewhat uniformly.
In order for the deterministic equivalents to be meaningful, we need that \(\epsilon_{n}\) converges to \(0\) for the Stieltjes transforms, and that \(\epsilon_{n}^{\prime}\) converges to \(0\) for the resolvent matrices. This is a stronger assumption that depends both on the data matrix \(X\) and on the activation function \(f\). More precisely, \(K_{X}\) should not be too far from \(I_{n}\) entry-wise, to an extent that also depends on how far \(f\) is from acting linearly on Gaussians.

In practice, for the Stieltjes transforms and a general activation function \(f\) we need \(\left\|\Delta_{X}\right\|_{\max}=o\Big{(}n^{-1/4}\Big{)}\). If \(\zeta_{2}(\tilde{f})=0\), which happens for instance if \(f\) is odd symmetric, this convergence can be relaxed to \(\left\|\Delta_{X}\right\|_{\max}=o\Big{(}n^{-1/6}\Big{)}\). If \(\zeta_{2}(\tilde{f})=\zeta_{3}(\tilde{f})=0\), no additional hypothesis is required as \(\epsilon_{n}=\left\|\Delta_{X}\right\|_{\max}\) converges already to \(0\). For the resolvent matrices, the equivalent statements boil down to \(\left\|\Delta_{X}\right\|_{\max}=o\Big{(}n^{-1/2}\Big{)}\) in general, \(\left\|\Delta_{X}\right\|_{\max}=o\Big{(}n^{-1/3}\Big{)}\) if \(\zeta_{2}(\tilde{f})=0\), and no additional hypothesis if \(\zeta_{2}(\tilde{f})=\zeta_{3}(\tilde{f})=0\).

Proof of Theorem 5.7.: \(Y\) satisfies the Assumptions 4.2 as seen in Proposition 5.3. We deduce from Theorem 4.5 the Lipschitz concentration properties (1) and the linear concentration properties \(\mathcal{G}_{K}(z)\in_{\llbracket\cdot\rrbracket_{F}}\mathbf{G}_{\boxtimes}^{ \Sigma}(z)\pm\mathcal{E}\big{(}O_{z}(1/\sqrt{n})\big{)}\) and \(g_{K}(z)\in g_{\nu}(z)\pm\mathcal{E}(O_{z}(1/n))\). Moreover from Theorem 4.7 and Corollaries 5.4 and 4.12:

\[\left|g_{\nu^{\Sigma}}(z)-g_{\nu}x_{\text{\tiny lin}}(z)\right| \leq O_{z}(\left|g_{\Sigma}(z)-g_{\Sigma_{\text{\tiny lin}}}(z) \right|)\] \[\leq O_{z}(\epsilon_{n}),\] \[\left\|\!\left|\mathbf{G}_{\boxtimes}^{\Sigma}(z)-\mathbf{G}_{ \boxtimes}^{\Sigma_{\text{\tiny lin}}}(z)\right|\!\right|\!\right| \leq O_{z}(\left\|\!\left\|\Sigma-\Sigma_{\text{\tiny lin}} \right\|\!\right|)\] \[\leq O_{z}(\epsilon_{n}^{\prime}),\]

which imply the linear concentration properties (2). 

Similarly to Corollary 4.6, we may deduce from the deterministic equivalents the following spectral properties:

**Corollary 5.9**.: _Uniformly under Assumptions 4.2:_

1. \(\left|g_{K}(z)-g_{\nu}x_{\text{\tiny lin}}(z)\right|\leq\sqrt{\log n}\,O_{z} (\epsilon_{n})\) _a.s. and_ \(\left\|\mathcal{G}_{K}(z)-\mathbf{G}_{\boxtimes}^{\Sigma_{\text{\tiny lin}}}(z )\right\|_{\max}\leq\sqrt{\log n}\,O_{z}(\epsilon_{n}^{\prime})\) _a.s._
2. _If_ \(f\) _is not linear, or if is_ \(f\) _linear and the eigenvalues of_ \(K_{X}\) _are bounded from below, there exists_ \(\theta>0\) _such that_ \(D(\mu_{K},\nu^{\Sigma_{\text{\tiny lin}}})\leq O(\epsilon_{n}^{\theta})\) _a.s._
3. _If moreover_ \(\mu_{K_{X}}\) _converges weakly to a measure_ \(\mu_{\infty}\) _and if_ \(\gamma_{n}\to\gamma_{\infty}\)_, then_ \(\mu_{K}\) _converges weakly to_ \(\nu_{\infty}=\operatorname{MP}(\gamma_{\infty})\boxtimes(\mathfrak{a}+ \mathfrak{b}\mu_{\infty})\)_, and more precisely:_ \[D(\mu_{K},\nu_{\infty})\leq O\Big{(}D(\mu_{K_{X}},\mu_{\infty})+\left|\gamma_ {n}-\gamma_{\infty}\right|+\epsilon_{n}^{\theta}\Big{)}\quad\text{a.s.}\]

Proof.: By definition of the \(O_{z}(\epsilon_{n}^{\prime})\) notation, we may find \(\alpha>0\) such that uniformly in \(z\in\mathbb{C}^{+}\) with bounded \(\Im(z)\), \(\mathcal{G}_{K}(z)\in_{\llbracket\cdot\rrbracket_{\rrbracket}}\mathbf{G}_{ \boxtimes}^{\Sigma_{\text{\tiny lin}}}(z)\pm\mathcal{E}\big{(}O(\epsilon_{n}^{ \prime}(z))\big{)}\) where \(\epsilon_{n}^{\prime}(z)=\epsilon_{n}^{\prime}\frac{|z|^{\alpha}}{\Im(z)^{2\alpha}}\). Let us now fix \(z\in\mathbb{C}^{+}\). The maps \(M\mapsto M_{ij}\) are linear and \(1\)-Lipschitz with respect to the spectral norm. By definition of
the linear concentration, there are constants \(C>0\) such that for any \(n,t,i\) and \(j\):

\[\mathbb{P}\bigg{(}\bigg{|}\Big{(}\mathcal{G}_{K}(z)-\mathbf{G}_{\mathbb{S}}^{ \Sigma_{\mathrm{lin}}}(z)\Big{)}_{ij}\bigg{|}\geq t\bigg{)}\leq Ce^{-\frac{t^{ 2}}{Ct_{n}^{\prime}(z)^{2}}}.\]

We choose \(t_{n}=\epsilon_{n}^{\prime}(z)\sqrt{4C\log n}\), and we use a union bound:

\[\mathbb{P}\Big{(}\bigg{\|}\mathcal{G}_{K}(z)-\mathbf{G}_{\mathbb{S}}^{\Sigma_ {\mathrm{lin}}}(z)\Big{\|}_{\max}\geq t_{n}\Big{)}\leq n^{2}Ce^{-\frac{t_{n}^{ 2}}{Ct_{n}^{\prime}(z)^{2}}}=Ce^{2\log n-4\log n}=C/n^{2}.\]

These probabilities are summable, so using Borel-Cantelli lemma:

\[\bigg{\|}\mathcal{G}_{K}(z)-\mathbf{G}_{\mathbb{S}}^{\Sigma_{\mathrm{lin}}}( z)\Big{\|}_{\max}\leq t_{n}\leq O\Big{(}\sqrt{\log n}\,\epsilon_{n}^{\prime}(z) \Big{)}\quad\text{a.s.}\]

We deduce that \(\bigg{\|}\mathcal{G}_{K}(z)-\mathbf{G}_{\mathbb{S}}^{\Sigma_{\mathrm{lin}}}( z)\Big{\|}_{\max}\leq\sqrt{\log n}\,O_{z}(\epsilon_{n}^{\prime})\) a.s. The proof for the Stieltjes transforms is similar.

If \(f\) is not linear, or if \(f\) is linear and the eigenvalues of \(K_{X}\) are bounded from below, then the eigenvalues of \(\Sigma_{\mathrm{lin}}\) are bounded from below. In this case, for values of \(\gamma_{n}\leq 1\) the cumulative distribution function \(\mathcal{F}_{\nu}\Sigma_{\mathrm{lin}}\) are uniformly Holder continuous with exponent \(\beta=1/2\), and for values of \(\gamma_{n}\geq 1\) the cumulative distribution function \(\mathcal{F}_{\bar{\nu}}\Sigma_{\mathrm{lin}}\) are uniformly Holder continuous with exponent \(\beta=1/2\) (see [1, Section 8.3] and Remark 2.9). In both cases, from Proposition 2.8 we deduce the second assertion.

Finally for the third assertion, using the properties of free multiplicative convolution [1, Proposition 4.13] we deduce that \(\nu^{\Sigma_{\mathrm{lin}}}\) converges weakly to \(\nu_{\infty}\), and that:

\[D(\nu^{\Sigma_{\mathrm{lin}}},\nu^{\infty}) \leq D(\mathrm{MP}(\gamma_{n}),\mathrm{MP}(\gamma_{\infty}))+D( \mu_{\Sigma_{\mathrm{lin}}},\mathfrak{a}+\mathfrak{b}\mu_{\infty})\] \[\leq O(|\gamma_{n}-\gamma_{\infty}|+D(\mu_{K_{X}},\mu_{\infty})).\]

### Application to another model involving entry-wise operations

In this paragraph we show how our framework applies to other models of random matrices, not strictly related to artificial neural networks.

We consider \(U\in\mathbb{R}^{n\times n}\) a Gaussian random matrix filled with \(\mathcal{N}\) random variables, i.i.d. within the columns and weakly correlated within the rows. More precisely, we consider \(u\in\mathbb{R}^{n}\) a Gaussian random vector, centered, with covariance matrix:

\[S=\begin{pmatrix}1&1/n&\dots&&1/n\\ 1/n&1&1/n&&\vdots\\ \vdots&1/n&\ddots&&\\ &&&\ddots&1/n\\ 1/n&\dots&&1/n&1\end{pmatrix}.\]

We let \(U\) be the random matrix with i.i.d. sampled columns from the distribution of \(u\). Let \(f=\tanh\) be the hyperbolic tangent function, \(B\) and \(D\in\mathbb{R}^{n\times n}\) independent random matrices filled with i.i.d. \(\mathcal{N}\) random variables, and \(Y=f(U+B)+D\). We want to apply the contents of this section to study the spectral properties of the sample covariance matrix
\(K=Y^{\top}Y/n\), its resolvent \(\mathcal{G}_{K}(z)=\left(K-zI_{n}\right)^{-1}\) and Stieltjes transform \(g_{K}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{K}(z)\).

Let us denote by \(J\in\mathbb{R}^{n\times n}\) the matrix whose entries are all equal to \(1\), so that \(S=I_{n}+(J-I_{n})/n\). In law \(U=XW\) where \(W\in\mathbb{R}^{n\times n}\) is a matrix filled with i.i.d. \(\mathcal{N}\) entries, independent from the other sources of randomness, and \(X=S^{1/2}\). Up to a transposition in the independence structure, the model \(Y\) is equal in law as \(f(XW+B)+D\), and satisfies the Assumptions 5.2, with \(\Delta_{X}=(J-I_{n})/n\) and \(\sigma_{W}^{2}=\sigma_{X}^{2}=\sigma_{B}^{2}=\sigma_{D}^{2}=1\). Indeed \(\tilde{\sigma}^{2}=2\), \(\tilde{f}(t)=\tanh(\sqrt{2}t)\), and by odd symmetry of \(\tanh\) it is clear that \(\zeta_{0}(\tilde{f})=\zeta_{2}(\tilde{f})=0\). We also have \(\left\|\!\left|\Delta_{X}\right|\!\right\|=1-1/n\), \(\widetilde{\mathrm{diag}}(\Delta_{X})=0\), \(\left\|\Delta_{X}\right\|_{\max}=1/n\) and \(\epsilon_{n}=O(1/n)\).

The constants \(\mathfrak{a}\) and \(\mathfrak{b}\) are linked to Gaussian moments of the hyperbolic tangent function and may be numerically approximated. The matrix \(\Sigma_{\mathrm{lin}}=\mathfrak{a}I_{p}+\mathfrak{b}S=(\mathfrak{a}+\mathfrak{ b})I_{p}+\mathfrak{b}(J-I_{n})/n\) is explicitly diagonalizable and \(\mu_{\Sigma_{\mathrm{lin}}}=\frac{n-1}{n}\cdot\delta_{\mathfrak{a}+ \mathfrak{b}-\mathfrak{b}/n}+\frac{1}{n}\cdot\delta_{\mathfrak{a}+2\mathfrak{ b}-\mathfrak{b}/n}\). The measure \(\nu^{\Sigma_{\mathrm{lin}}}=\tilde{\nu}^{\Sigma_{\mathrm{lin}}}=\mathrm{MP}(1) \boxtimes\mu_{L}\) is characterized by its Stieltjes transform \(g_{\nu}\Sigma_{\mathrm{lin}}(z)\), which is the only solution \(g\in\mathbb{C}^{+}\) of the self-consistent equation:

\[g =\int_{\mathbb{R}}\frac{1}{-zgt-z}\mu_{\Sigma_{\mathrm{lin}}}(dt)\] \[=\frac{1}{-zg(\mathfrak{a}+\mathfrak{b}-\mathfrak{b}/n)-z}+\frac {1}{n}\bigg{(}\frac{1}{-zg(\mathfrak{a}+2\mathfrak{b}-\mathfrak{b}/n)-z}- \frac{1}{-zg(\mathfrak{a}+\mathfrak{b}-\mathfrak{b}/n)-z}\bigg{)}\]

This equation may be rewritten as a cubic polynomial equation and solved explicitly. The deterministic equivalent matrix \(\mathbf{G}_{\boxtimes}^{\Sigma_{\mathrm{lin}}}(z)\) may also be explicitly computed. The interested reader can check that:

\[\mathbf{G}_{\boxtimes}^{\Sigma_{\mathrm{lin}}}(z) =-\frac{g_{\nu}\Sigma_{\mathrm{lin}}(z)\mathfrak{b}}{(g_{\nu} \Sigma_{\mathrm{lin}}(z)(\mathfrak{a}+\mathfrak{b}-\mathfrak{b}/n)+1)(g_{\nu }\Sigma_{\mathrm{lin}}(z)(\mathfrak{a}+2\mathfrak{b}-\mathfrak{b}/n)+1)}\frac{ J}{n}\] \[\quad-\frac{1}{(g_{\nu}\Sigma_{\mathrm{lin}}(z)(\mathfrak{a}+ \mathfrak{b}-\mathfrak{b}/n)+1)}\frac{I_{n}}{z}\]

Theorem 5.7 applied to this model provides the following deterministic equivalents: \(g_{K}(z)\in g_{\nu}\Sigma_{\mathrm{lin}}(z)\pm\mathcal{E}(O_{z}(1/n))\), and \(\mathcal{G}_{K}(z)\in_{\left|\!\left|\cdot\right|\!\right|}\mathbf{G}_{ \boxtimes}^{\Sigma_{\mathrm{lin}}}(z)\pm\mathcal{E}\big{(}O_{z}(1/\sqrt{n}) \big{)}\).

Since \(\mu_{\Sigma_{\mathrm{lin}}}\) converges weakly to \(\delta_{\mathfrak{a}+\mathfrak{b}}\), \(\nu^{L}\) and \(\mu_{K}\) converge weakly to \(\nu_{\infty}=\mathrm{MP}(1)\boxtimes\delta_{\mathfrak{a}+\mathfrak{b}}=( \mathfrak{a}+\mathfrak{b})\mathrm{MP}(1)\), and \(D(\mu_{K},\nu^{\Sigma_{\mathrm{lin}}})\leq O(n^{-\theta})\) for some \(\theta>0\). To go further, Corollary 5.9(3) is not helpful since the measures are discrete. As a matter of fact, \(D(\mu_{\Sigma_{\mathrm{lin}}},\delta_{\mathfrak{a}+\mathfrak{b}})=1-1/n\) does not vanish. However we can compare directly the Stieltjes transforms of \(\Sigma_{\mathrm{lin}}\) and \(\delta_{\mathfrak{a}+\mathfrak{b}}\):

\[\left|g_{\Sigma_{\mathrm{lin}}}(z)-g_{\delta_{\mathfrak{a}+ \mathfrak{b}}}(z)\right| =\left|\frac{1-1/n}{\mathfrak{a}+\mathfrak{b}-\mathfrak{b}/n-z}+ \frac{1/n}{\mathfrak{a}+2\mathfrak{b}-\mathfrak{b}/n-z}-\frac{1}{\mathfrak{a}+ \mathfrak{b}-z}\right|\] \[\leq O_{z}(1/n)\]

From Theorem 4.7, we have \(\left|g_{\nu}\Sigma_{\mathrm{lin}}(z)-g_{\nu^{\infty}}(z)\right|\leq O_{z}(1/n)\), hence \(D(\nu^{\Sigma_{\mathrm{lin}}},\nu^{\infty})\leq O(n^{-\theta})\) for some \(\theta>0\) from Proposition 2.8. We conclude that \(\mu_{K}\) converges to \(\nu_{\infty}\) as speed \(O(n^{-\theta})\) in Kolmogorov distance.
## 6. Single-layer neural network with random data

### Setting

In this section we study the Conjugate Kernel matrix associated to a single-layer artificial neural network with random input. The hypothesis are thus the same as in the last section, excepted for the random data matrix. We consider:

* a random weight matrix \(W\in\mathbb{R}^{d\times d_{0}}\), with variance parameter \(\sigma_{W}^{2}>0\),
* a random data matrix \(X\in\mathbb{R}^{d_{0}\times n}\), \(\sigma_{X}^{2}>0\),
* two random biases matrices \(B\) and \(D\in\mathbb{R}^{d\times n}\), \(\sigma_{B}^{2},\sigma_{D}^{2}\geq 0\),
* and an activation function \(f:\mathbb{R}\to\mathbb{R}\).

As output of the neuron, we set \(Y=f(WX/\sqrt{d_{0}}+B)+D\in\mathbb{R}^{d\times n}\), where the function \(f\) is applied entry-wise. Our goal is to investigate the spectral properties of the Conjugate Kernel matrix \(K=Y^{\top}Y/d\). For \(z\in\mathbb{C}^{+}\) we define its resolvent \(\mathcal{G}_{K}(z)=\left(K-zI_{n}\right)^{-1}\) and Stieltjes transform \(g_{K}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{K}(z)\). We also define the following objects:

\[\tilde{\sigma}^{2} =\sigma_{W}^{2}\sigma_{X}^{2}+\sigma_{B}^{2},\] \[\tilde{f}(t) =f(\hat{\sigma}t),\] \[\mathfrak{a} =\left\|\tilde{f}\right\|_{\mathcal{H}}^{2}-\frac{\sigma_{W}^{2 }\sigma_{X}^{2}}{\tilde{\sigma}^{2}}\zeta_{1}(\tilde{f})^{2}+\sigma_{D}^{2},\] \[\mathfrak{b} =\zeta_{1}(\tilde{f})^{2}\frac{\sigma_{W}^{2}}{\tilde{\sigma}^{ 2}},\] \[K_{X} =X^{\top}X/d_{0},\] \[\Delta_{X} =K_{X}-\sigma_{X}^{2}I_{n},\] \[\Sigma_{X} =\mathfrak{a}I_{n}+\mathfrak{b}K_{X}.\]

Like in the rest of this article, we sometimes omit the indices \(n\) and \(z\) for a better readability, even if we are implicitly dealing with sequences of matrices, measures, and complex functions. For reasons that we will understand later, we do not make hypothesis on the random matrix \(X\) itself, but rather on \(X\) conditioned with a high probability event (see the definition before Proposition 2.3).

**Assumptions 6.1**.:
1. \(W\), \(B\) and \(D\) are random, independent, with i.i.d. \(\mathcal{N}(\sigma_{W}^{2})\), \(\mathcal{N}(\sigma_{B}^{2})\) and \(\mathcal{N}(\sigma_{D}^{2})\) entries respectively.
2. \(\tilde{f}\) is Lipschitz continuous and Gaussian centered, that is \(\mathbb{E}\Big{[}\tilde{f}(\mathcal{N})\Big{]}=\mathbb{E}[f(\tilde{\sigma} \mathcal{N})]=0\).
3. \(X\) is random, independent from \(W\), \(B\) and \(D\). There is an event \(\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{c})\leq O(\sqrt{\log n}/n)\), such that \((X|\mathcal{B})\propto_{\left\|\cdot\right\|_{F}}\mathcal{E}(1)\).
4. There is a sequence \(\epsilon_{n}\) converging to \(0\), such that uniformly in \(\omega\in\mathcal{B}\), and \(\left\|\widetilde{\mathrm{diag}}(\Delta_{X})\right\|\) are bounded, and \(\left\|\Delta_{X}\right\|_{\max}\leq O(\epsilon_{n})\).
5. The ratio \(\gamma_{n}=\frac{n}{d}\) is bounded from above and away from \(0\).
6. There is a sequence \(\hat{\epsilon}_{n}\geq 0\) such that \(\left|\mathbb{E}[g_{K_{X}}(z)]-g_{\tau}(z)\right|\leq O_{z}(\hat{\epsilon}_{n})\) for some sequence of measures \(\tau\) supported on \(\mathbb{R}^{+}\).
7. There is a sequence \(\hat{\epsilon}^{\prime}_{n}\geq\hat{\epsilon}_{n}\) such that \(\left|\kern-1.075pt\left|\kern-1.075pt\left|\mathbb{E}[\mathcal{G}_{K_{X}}(z)]- \mathbf{H}(z)\right|\kern-1.075pt\right|\kern-1.075pt\right|\leq O_{z}(\hat{ \epsilon}^{\prime}_{n})\) for some sequence of matrix functions \(\mathbf{H}:\mathbb{C}^{+}\to\mathbb{C}^{n\times n}\) satisfying \(\left|\kern-1.075pt\left|\kern-1.075pt\left|\mathbf{H}\right|\kern-1.075pt \right|\kern-1.075pt\right|\leq 1/\Im(z)\).

_Remark 6.2_.: Compared to Assumptions 5.2, in the assertions (3) and (4) we ask the data matrix \(X\) to be independent from the other random matrices, well concentrated, and uniformly approximately orthogonal on an event of high probability.

Let us explain why \(\mathbb{P}(\mathcal{B}^{c})\leq O(\sqrt{\log n}/n)\) is a convenient choice to simplify our results. We claim that, up to a \(O_{z}(\sqrt{\log n}/n)\) error that will blend into similar terms, it will be equivalent for us to compute expectations on the whole probability set or on \(\mathcal{B}\). Indeed, if a random function \(\zeta:\mathbb{N}\times\mathbb{C}^{+}\to\mathbb{C}\) satisfies the _a priori_ bound \(\zeta(n,z)\leq 1/\Im(z)\), then we have:

\[\left|\mathbb{E}_{\mathcal{B}}[\zeta]-\mathbb{E}[\zeta]\right| =\left|\mathbb{E}[\mathbf{1}_{\mathcal{B}}\zeta]/\mathbb{P}( \mathcal{B})-\mathbb{E}[\zeta]\right|\] \[=\frac{\left|\mathbb{E}[\mathbf{1}_{\mathcal{B}^{c}}\zeta]+ \mathbb{E}[\zeta]\mathbb{P}(\mathcal{B}^{c})\right|}{\mathbb{P}(\mathcal{B})}\] \[\leq\frac{2\mathbb{P}(\mathcal{B}^{c})}{1-\mathbb{P}(\mathcal{B} ^{c})}\frac{1}{\Im(z)}\] \[\leq O_{z}(\mathbb{P}(\mathcal{B}^{c}))\leq O_{z}(\sqrt{\log n}/ n).\]

The assertions (6) and (7) correspond to deterministic equivalents for the Stieltjes transform and the resolvent of \(K_{X}\). If \(\zeta_{1}(\tilde{f})=0\) then \(\mathfrak{b}=0\), and as seen in Remark 5.6 the deterministic equivalents will not depend on \(X\). In this case \(\tau\) and \(\mathbf{H}(z)\) will not appear in the results and the assertions (6) and (7) are in essence empty.

### Deterministic equivalent and consequences

We let \(\nu^{\Sigma_{X}}=\operatorname{MP}(\gamma_{n})\boxtimes\mu_{\Sigma_{X}}\), and we refer to the beginning of Section 4 for the definition of the deterministic equivalent matrix \(\mathbf{G}_{\boxtimes}^{\Sigma_{X}}(z)\). Similarly to the process used to express \(\mathbf{G}_{\boxtimes}^{\Sigma_{X}}(z)\) as a function of \(\mu_{K_{X}}\) and \(\mathcal{G}_{K_{X}}\) (see Remark 5.6), we define the objects:

\[\chi =\operatorname{MP}(\gamma_{n})\boxtimes(\mathfrak{a}+\mathfrak{b }\tau),\] \[\check{\chi} =(1-\gamma_{n})\cdot\delta_{0}+\gamma_{n}\cdot\chi,\] \[l_{\check{\chi}}(z) =-1/g_{\check{\chi}}(z),\] \[\mathbf{K}(z) =\frac{z^{-1}l_{\check{\chi}}(z)}{\mathfrak{b}}\mathbf{H}\!\left( \frac{l_{\check{\chi}}(z)-\mathfrak{a}}{\mathfrak{b}}\right) \text{if }\mathfrak{b}\neq 0,\] \[\mathbf{K}(z) =g_{\mathfrak{a}\mathrm{MP}(\gamma_{n})}(z)I_{n} \text{if }\mathfrak{b}=0.\]

**Lemma 6.3**.: _Under Assumptions 6.1, \(\left|\mathbb{E}[g_{\nu^{\Sigma_{X}}}(z)]-g_{\chi}(z)\right|\leq\zeta_{1}( \tilde{f})O_{z}(\hat{\epsilon}_{n}+\sqrt{\log n}/n)\), and \(\left|\kern-1.075pt\left|\mathbb{E}\!\left[\mathbf{G}_{\boxtimes}^{\Sigma_{X} }(z)\right]-\mathbf{K}(z)\right|\kern-1.075pt\right|\kern-1.075pt\right|\leq \zeta_{1}(\tilde{f})O_{z}(\hat{\epsilon}^{\prime}_{n}+1/\sqrt{n})\)._

Proof.: If \(\zeta_{1}(\tilde{f})=0\), then \(\Sigma_{X}\) does not depend on \(X\), \(\mathbf{G}_{\boxtimes}^{\Sigma_{X}}(z)=\mathbf{K}(z)\) and \(\nu^{\Sigma_{X}}=\chi\). If \(\zeta_{1}(\tilde{f})\neq 0\), from Proposition 2.3 applied to \((X|\mathcal{B})\) there is a constant \(c>0\) and an event \(\mathcal{B}^{\prime}\subset\mathcal{B}\) with \(\mathbb{P}\!\left(\mathcal{B}^{\prime c}\right)\leq\mathbb{P}(\mathcal{B}^{c} )+ce^{-n/c}\leq O(\sqrt{\log n}/n)\), on which \((\Sigma_{X}|\mathcal{B}^{\prime})\propto_{\left|\kern-1.075pt\left|\kern-1.075pt \left|\cdot\right|\kern-1.075pt\right|_{F}}\mathcal{E}\!\left(1/\sqrt{n}\right)\). As explained in the Remark 6.2, given the _a priori_ bounds on Stieltjes transforms and on the spectral norm of resolvent matrices, we may pass from expectations on \(\mathcal{B}^{\prime}\) to
expectations on the full probability set at the cost of a \(O_{z}(\sqrt{\log n}/n)\) error term. We thus have:

\[\left|\mathbb{E}_{\mathcal{B}^{\prime}}[g_{\Sigma_{X}}(z)]-g_{ \mathfrak{a}+\mathfrak{b}\tau}(z)\right| =\left|\mathbb{E}[\mathbf{1}_{\mathcal{B}^{\prime}}(g_{\Sigma_{X} }(z)-g_{\mathfrak{a}+\mathfrak{b}\tau}(z))]\right|/\mathbb{P}(\mathcal{B}^{ \prime})\] \[\leq O_{z}(\hat{\epsilon}_{n}+\sqrt{\log n}/n).\]

Proposition 4.13 applied to \((\Sigma_{X}|\mathcal{B}^{\prime})\) implies that:

\[\left|\mathbb{E}_{\mathcal{B}^{\prime}}[g_{\nu}\Sigma_{X}(z)]-g_{\chi}(z) \right|\leq O_{z}(\hat{\epsilon}_{n}+\sqrt{\log n}/n),\]

hence \(\left|\mathbb{E}[g_{\nu}\Sigma_{X}(z)]-g_{\chi}(z)\right|\leq O_{z}(\hat{ \epsilon}_{n}+\sqrt{\log n}/n)\). Similarly for the resolvents:

\[\left|\left|\left|\mathbb{E}_{\mathcal{B}^{\prime}}[\mathcal{G} _{\Sigma_{X}}(z)]-\frac{1}{\mathfrak{b}}\mathbf{H}\bigg{(}\frac{z-\mathfrak{a }}{\mathfrak{b}}\bigg{)}\right|\right| =\left|\left|\left|\mathbb{E}\bigg{[}\frac{1}{\mathfrak{b}} \mathcal{G}_{K_{X}}\bigg{(}\frac{z-\mathfrak{a}}{\mathfrak{b}}\bigg{)}\right| \right|-\frac{1}{\mathfrak{b}}\mathbf{H}\bigg{(}\frac{z-\mathfrak{a}}{ \mathfrak{b}}\bigg{)}\right|\right|\] \[\quad+O_{z}(\sqrt{\log n}/n)\] \[\leq O_{z}(\hat{\epsilon}^{\prime}_{n}+\sqrt{\log n}/n),\]

which implies by Proposition 4.13 that \(\left|\left|\left|\mathbb{E}_{\mathcal{B}^{\prime}}\Big{[}\mathbf{G}_{\mathbb{ S}}^{\Sigma_{X}}(z)\Big{]}-\mathbf{K}(z)\right|\right|\right|\leq O_{z}(\hat{ \epsilon}_{n}+\tilde{\epsilon}^{\prime}_{n}+1/\sqrt{n})\leq O_{z}(\hat{ \epsilon}^{\prime}_{n}+1/\sqrt{n})\), and \(\left|\left|\left|\mathbb{E}\Big{[}\mathbf{G}_{\mathbb{S}}^{\Sigma_{X}}(z) \Big{]}-\mathbf{K}(z)\right|\right|\right|\leq O_{z}(\hat{\epsilon}^{\prime}_ {n}+1/\sqrt{n})\). 

We remind our reader that the sequences \(\epsilon\) appearing in the Assumptions 6.1 measure the lack of orthogonality of \(X\) for \(\epsilon_{n}\), and the convergence speeds in the deterministic equivalents \(g_{K_{X}}(z)\approx g_{\tau}(z)\) for \(\hat{\epsilon}_{n}\) and \(\mathcal{G}_{K_{X}}(z)\approx\mathbf{H}(z)\) for \(\hat{\epsilon}^{\prime}_{n}\) respectively. For the next result let us denote:

\[\tilde{\epsilon}_{n} =\sqrt{\log n}/n+\zeta_{1}(\tilde{f})\hat{\epsilon}_{n}+\epsilon _{n}+\sqrt{n}\zeta_{2}(\tilde{f})^{2}\epsilon_{n}^{2}+\sqrt{n}\zeta_{3}(\tilde {f})^{2}\epsilon_{n}^{3},\] \[\tilde{\epsilon}^{\prime}_{n} =1/\sqrt{n}+\zeta_{1}(\tilde{f})\hat{\epsilon}^{\prime}_{n}+ \epsilon_{n}+n\zeta_{2}(\tilde{f})^{2}\epsilon_{n}^{2}+n\zeta_{3}(\tilde{f})^{ 2}\epsilon_{n}^{3}.\]

\(\tilde{\epsilon}_{n}\) and \(\tilde{\epsilon}^{\prime}_{n}\) will correspond to the new convergence speeds in the deterministic equivalents \(g_{K}(z)\approx g_{\chi}(z)\) and \(\mathcal{G}_{K}(z)\approx\mathbf{K}(z)\).

**Theorem 6.4**.: _Uniformly under Assumptions 6.1, there is an event \(\mathcal{B}^{\prime}\subset\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{\prime c})\leq O(\sqrt{\log n}/n)\), such that the following conditional expectation properties hold true:_

1. \((Y|\mathcal{B}^{\prime})\left.\propto_{\left|\cdot\right|\right|_{F}}\mathcal{ E}(1)\)_,_ \((g_{K}(z)|\mathcal{B}^{\prime})\propto\mathcal{E}(O_{z}(1/n))\)_, and_ \((\mathcal{G}_{K}(z)|\mathcal{B}^{\prime})\left.\propto_{\left|\cdot\right|_{F}} \mathcal{E}(O_{z}(1/\sqrt{n}))\)_._
2. \((g_{K}(z)|\mathcal{B}^{\prime})\in g_{\chi}(z)\pm\mathcal{E}(O_{z}(\tilde{ \epsilon}_{n}))\) _and_ \((\mathcal{G}_{K}(z)|\mathcal{B}^{\prime})\in_{\left|\cdot\right|\right|} \mathbf{K}(z)\pm\mathcal{E}\big{(}O_{z}(\tilde{\epsilon}^{\prime}_{n})\big{)}\)_._

Proof.: For a better readability in the upcoming arguments, we choose to omit the spectral parameters \(z\) in most of our notations. From Proposition 2.3 applied to \((W,(X|\mathcal{B}))\left.\propto_{\left|\cdot\right|\right|_{F}}\mathcal{E}(1)\) there is a constant \(c>0\) and an event \(\mathcal{B}^{\prime}\subset\mathcal{B}\), with \(\mathbb{P}\big{(}\mathcal{B}^{\prime c}\big{)}\leq\mathbb{P}(\mathcal{B}^{c}) +ce^{-n/c}\leq O(\sqrt{\log n}/n)\), such that \((WX|\mathcal{B}^{\prime})\left.\propto_{\left|\cdot\right|_{F}}\mathcal{E} \big{(}\sqrt{n}\big{)}\). The map \((U,B,D)\mapsto f(U+B)+D\) is Lipschitz with respect to the Frobenius norm, and by independence \(\Big{(}(WX/\sqrt{d_{0}}\,|\mathcal{B}^{\prime}),B,D\Big{)}\propto_{\left| \cdot\right|_{F}}\mathcal{E}(1)\), thus \((Y|\mathcal{B}^{\prime})=f\Big{(}(WX/\sqrt{d_{0}}\,|\mathcal{B}^{\prime})+B \Big{)}+D\propto_{\left|\cdot\right|\right|_{F}}\mathcal{E}(1)\). The general
concentration properties recalled in Proposition 2.2 imply that \((g_{K}|\mathcal{B}^{\prime})\propto\mathcal{E}(O_{z}(1/n))\) and \((\mathcal{G}_{K}|\mathcal{B}^{\prime})\propto_{\llbracket\cdot\rrbracket_{F}} \mathcal{E}(O_{z}(1/\sqrt{n}))\).

For the second assertion, given the linear concentration properties \((\mathcal{G}_{K}|\mathcal{B}^{\prime})\in_{\llbracket\cdot\rrbracket} \mathbb{E}\big{[}(\mathcal{G}_{K}|\mathcal{B}^{\prime})\big{]}\pm\mathcal{E} (O_{z}(1/\sqrt{n}))\) and \(\big{(}g_{K}|\mathcal{B}^{\prime})\in\mathbb{E}\big{[}(g_{K}|\mathcal{B}^{ \prime})\big{]}\pm\mathcal{E}\big{(}O_{z}(1/n))\), we only need to prove that \(\big{|}\mathbb{E}\big{[}\big{(}g_{K}|\mathcal{B}^{\prime}\big{)}\big{]}-g_{ \chi}\big{|}\leq O_{z}(\tilde{\epsilon}_{n})\), and that \(\big{|}\big{|}\mathbb{E}\big{[}\big{(}\mathcal{G}_{K}|\mathcal{B}^{\prime} \big{)}\big{]}-\mathbf{K}\big{|}\big{|}\leq O_{z}(\tilde{\epsilon}_{n}^{ \prime})\). To compare these expectations, as explained in the Remark 6.2, we may assume without loss of generality that \(\mathcal{B}^{\prime}=\Omega\), because the additional \(O_{z}(\sqrt{\log n}/n)\) error terms will not change the final estimates.

We apply Theorem 5.7 to the models with deterministic data matrices \(X(\omega)\), uniformly in the outcomes \(\omega\in\Omega_{X}\). Since \(1/\sqrt{n}\leq\tilde{\epsilon}_{n}^{\prime}\) and \(\epsilon_{n}+n\zeta_{2}(\tilde{f})^{2}\epsilon_{n}^{2}+n\zeta_{3}(\tilde{f})^{ 2}\epsilon_{n}^{3}\leq\tilde{\epsilon}_{n}^{\prime}\), we obtain that \(\mathcal{G}_{K(\omega)}\in_{\llbracket\cdot\rrbracket\cdot\rrbracket} \mathbf{G}_{\boxtimes}^{\Sigma_{X}(\omega)}\pm\mathcal{E}\big{(}O_{z}(\tilde{ \epsilon}_{n}^{\prime})\). As a consequence, we get uniformly in \(\omega\in\Omega_{X}\):

\[\big{|}\big{|}\mathbb{E}_{W,B,D}[\mathcal{G}_{K}(\omega)]-\mathbf{G}_{ \boxtimes}^{\Sigma_{X}(\omega)}\big{|}\big{|}\leq O_{z}(\tilde{\epsilon}_{n}^ {\prime}).\]

Since \(X\) is independent from the other sources of randomness, for any measurable function \(\Phi\) we have \(\mathbb{E}[\Phi(W,X,B,D)|X]=\mathbb{E}_{W,B,D}[\Phi(W,X,B,D)]\). We can thus integrate on \(X\) the above inequality and using the tower property of conditional expectation:

\[\big{|}\big{|}\mathbb{E}[\mathcal{G}_{K}]-\mathbb{E}\big{[} \mathbf{G}_{\boxtimes}^{\Sigma_{X}}\big{]}\big{|}\big{|}\] \[\leq\mathbb{E}_{X}\Big{[}\big{|}\big{|}\mathbb{E}_{W,B,D}[\mathcal{ G}_{K}]-\mathbf{G}_{\boxtimes}^{\Sigma_{X}}\big{|}\big{|}\Big{]}\leq O_{z}( \tilde{\epsilon}_{n}^{\prime}).\]

From Lemma 6.3 we also have \(\big{|}\big{|}\mathbb{E}\big{[}\mathbf{G}_{\boxtimes}^{\Sigma_{X}}\big{]}- \mathbf{K}\big{|}\big{|}\leq\zeta_{1}(\tilde{f})O_{z}(\tilde{\epsilon}_{n}^{ \prime}+1/\sqrt{n})\leq O_{z}(\tilde{\epsilon}_{n}^{\prime})\), hence \(\big{|}\!\big{|}\mathbb{E}[\mathcal{G}_{K}]-\mathbf{K}\big{|}\!\big{|}\leq O_{ z}(\tilde{\epsilon}_{n}^{\prime})\). The proof for the Stieltjes transforms is similar. 

**Corollary 6.5**.: _Uniformly under Assumptions 6.1:_

1. \(|g_{K}(z)-g_{\chi}(z)|\leq\sqrt{\log n}\,O_{z}(\tilde{\epsilon}_{n})\) _a.s., and_ \(\left\|\mathcal{G}_{K}(z)-\mathbf{K}(z)\right\|_{\max}\leq\sqrt{\log n}\,O_{z}( \tilde{\epsilon}_{n}^{\prime})\) _a.s._
2. _If_ \(f\) _is not linear, or if is_ \(f\) _linear and the measures_ \(\tau\) _are supported on the same compact of_ \((0,\infty)\)_, there exists_ \(\theta>0\) _such that_ \(D(\mu_{K},\chi)\leq O(\tilde{\epsilon}_{n}^{\theta})\) _a.s._
3. _If moreover_ \(\tau\) _converges weakly to a measure_ \(\tau_{\infty}\)_, and if_ \(\gamma_{n}\to\gamma_{\infty}\)_, then_ \(\mu_{K}\) _converges weakly to_ \(\chi_{\infty}=\mathrm{MP}(\gamma_{\infty})\boxtimes(\mathfrak{a}+\mathfrak{b} \tau_{\infty})\)_, and more precisely:_ \[D(\mu_{K},\chi_{\infty})\leq O\Big{(}D(\tau,\tau_{\infty})+|\gamma_{n}-\gamma_{ \infty}|+\tilde{\epsilon}_{n}^{\theta}\Big{)}\quad\text{a.s.}\]

As we did for Corollary 4.6, we will not prove this result here, but rather prompt our reader to consult the proof of Corollary 5.9 which is extremely similar.

### Application to data matrices with i.i.d. columns

In this paragraph we focus on a fairly general setting where the data matrix \(X\) is made of independent samples, and we explore the consequences given by our deterministic equivalents. Let us first mention a general framework on which the Assumption 6.1(4) holds true.
**Proposition 6.6** ([20], Proposition 3.3).: _Let \(X\in\mathbb{R}^{d_{0}\times n}\) be a random matrix whose columns are i.i.d. sampled from the distribution of a random vector \(x\in\mathbb{R}^{d_{0}}\), such that \(x\propto_{\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt\right|}\mathcal{E}(1)\), \(\mathbb{E}[x]=0\), and \(\mathbb{E}\Big{[}\big{\|}x\big{\|}^{2}\Big{]}=\sigma_{x}^{2}d_{0}\). We also assume the ratio \(\dfrac{n}{d_{0}}\) to be bounded from above and away from \(0\)._

_Then there is an event \(\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{c})\leq O(1/n)\), such that uniformly in \(\omega\in\mathcal{B}\), and are bounded, and \(\|\Delta_{X}\|_{\max}\leq O(\epsilon_{n})\) with \(\epsilon_{n}=\sqrt{\log n/n}\)._

_Remark 6.7_.: In [20] the result is stated for a weaker notion called called convex concentration. We will not digress on this here but rather refer our reader to [10, Section 1.7] which presents in details this variant of concentration. Also note that in the above result we only need concentration for the columns of \(X\), while in our deterministic equivalent result we need concentration for the whole matrix \(X\).

To obtain deterministic equivalents for the input data matrix, it is of course possible to use again Theorem 4.5:

**Proposition 6.8**.: _If \(X\in\mathbb{R}^{d_{0}\times n}\) is a random matrix, \(\propto_{\left|\kern-1.075pt\left|\cdot\right|\kern-1.075pt\right|_{F}} \mathcal{E}(1)\) concentrated, whose columns are i.i.d. sampled from the distribution of a random vector \(x\in\mathbb{R}^{d_{0}}\), with \(\left|\mathbb{E}[x]\right|\) and \(\left|\kern-1.075pt\left|\mathbb{E}[K_{X}]\right|\kern-1.075pt\right|\kern-1.075pt\right|\) bounded, and if the ratio \(n/d_{0}\) is bounded from above and away from \(0\), then with \(\tau=\operatorname{MP}(n/d_{0})\boxtimes\mu_{\mathbb{E}[K_{X}]}\):_

\[\left|\mathbb{E}[g_{K_{X}}(z)]-g_{\tau}(z)\right| \leq O_{z}(1/n)\] \[\Big{|}\Big{|}\mathbb{E}[\mathcal{G}_{K_{X}}(z)]-\mathbf{G}_{ \boxtimes}^{\mathbb{E}[K_{X}]}(z)\Big{|}\Big{|} \leq O_{z}(1/\sqrt{n}).\]

If we combine the previous propositions, we obtain deterministic equivalents for the Conjugate Kernel model in a fairly general setting where the data matrix is made of i.i.d. training samples. This encompasses in particular the case where \(X\) is a matrix with i.i.d. \(\mathcal{N}\) entries, which was the original model studied in [20].

_Remark 6.9_.: Note that the typical order of magnitude \(\epsilon_{n}=\sqrt{\log n/n}\) given by Proposition 6.6 is good enough for a meaningful equivalent of the Stieltjes transform, with a \(O_{z}(\log n/\sqrt{n})\) convergence speed. However it is not small enough for the resolvent, where we would have an \(O_{z}(\zeta_{2}(\tilde{f})^{2}\log n)\) error term, unless of course \(\zeta_{2}(\tilde{f})=0\) (see Remark 5.8). If \(\zeta_{2}(\tilde{f})=0\), we obtain a \(O_{z}\Big{(}(\log n)^{3/2}/\sqrt{n}\Big{)}\) error term for the deterministic equivalent of the resolvent.

This condition \(\zeta_{2}(\tilde{f})=0\), and more generally the Hermite coefficients of the activation function \(f\), appear in other articles that study the Conjugate Kernel model. Let us consider indeed \(\tilde{Y}=\sqrt{\mathfrak{a}}Z+\sqrt{\mathfrak{b}}WX/\sqrt{d_{0}}\), and \(\tilde{K}=\tilde{Y}^{\top}\tilde{Y}/d\), where \(Z\) is a third random matrix, independent from the others, and filled with i.i.d. \(\mathcal{N}\) entries. Using Theorem 5.7 conditionally on \(X\) and similar arguments to those of this paper, we see that \(\mathcal{G}_{\tilde{K}}(z)\) also admits \(\mathbf{G}_{\boxtimes}^{\mathbb{E}[K_{X}]}(z)\) as deterministic equivalent, with a \(O_{z}\Big{(}(\log n)^{3/2}/\sqrt{n}\Big{)}\) error term in the case where \(\zeta_{2}(\tilde{f})=0\). In [1, Theorem 2.3], using combinatorics
it is shown that the biggest eigenvalues of both models behave similarly if \(\zeta_{2}(\tilde{f})=0\). Although we could not manage to retrieve this property with our deterministic equivalent solely, there is without a doubt a connection between these statements. [1] also provides other equivalent models in the case where \(\zeta_{2}(\tilde{f})\neq 0\), which we could not relate to our results.
## 7. Multi-layer neural network model

In this section we consider the Conjugate Kernel matrix associated to an artificial neural network with \(L\) hidden layers and a random input matrix:

\[X_{0}\to X_{1} =f_{1}\Big{(}W_{1}X_{0}/\sqrt{d_{0}}+B_{1}\Big{)}+D_{1}\] \[X_{1}\to X_{2} =f_{2}\Big{(}W_{2}X_{1}/\sqrt{d_{1}}+B_{2}\Big{)}+D_{2}\] \[\quad\vdots\] \[X_{l}\to X_{l+1}=f_{l+1}\Big{(}W_{l+1}X_{l}/\sqrt{d_{l}}+B_{l+1} \Big{)}+D_{l+1}\] \[\quad\vdots\] \[X_{L-1}\to X_{L}=f_{L}\Big{(}W_{L}X_{L-1}/\sqrt{d_{L-1}}+B_{L} \Big{)}+D_{L}\]

The initial data \(X_{0}\in\mathbb{R}^{d_{0}\times n}\) is a random matrix with variance parameter \(\sigma^{2}_{X_{0}}>0\). Each layer \(l\in\llbracket 1,L\rrbracket\) is made of:

* a random weight matrix \(W_{l}\in\mathbb{R}^{d_{l}\times d_{l-1}}\), with variance parameter \(\sigma^{2}_{W_{l}}>0\),
* two random biases matrices \(B_{l}\) and \(D_{l}\in\mathbb{R}^{d_{l}\times n}\), \(\sigma^{2}_{B_{l}},\sigma^{2}_{D_{l}}\geq 0\),
* and an activation function \(f_{l}:\mathbb{R}\to\mathbb{R}\).

At each layer, we define the conjugate kernel matrix \(K_{l}=X_{l}^{\top}X_{l}/d_{l}\), and for \(z\in\mathbb{C}^{+}\) its resolvent \(\mathcal{G}_{l}(z)=\left(K_{l}-zI_{n}\right)^{-1}\), and Stieltjes transform \(g_{l}(z)=(1/n)\mathrm{Tr}\mathcal{G}_{l}(z)\). We define by induction the following objects:

\[\tilde{\sigma}_{l}^{2} =\sigma^{2}_{W_{l}}\sigma^{2}_{X_{l-1}}+\sigma^{2}_{B_{l}},\] \[\tilde{f}_{l}(t) =f_{l}(\tilde{\sigma}_{l}t),\] \[\sigma_{X_{l}} =\left\|\tilde{f}_{l}\right\|^{2}_{\mathcal{H}}+\sigma^{2}_{D_{l}},\] \[\mathfrak{a}_{l} =\left\|\tilde{f}_{l}\right\|^{2}_{\mathcal{H}}-\frac{\sigma^{2 }_{W_{l}}\sigma^{2}_{X_{l}}}{\tilde{\sigma}_{l}^{2}}\zeta_{1}(\tilde{f}_{l})^{ 2}+\sigma^{2}_{D_{l}},\] \[\mathfrak{b}_{l} =\zeta_{1}(\tilde{f}_{l})^{2}\frac{\sigma^{2}_{W_{l}}}{\tilde{ \sigma}_{l}^{2}},\] \[\Delta_{X_{l}} =K_{l}-\sigma^{2}_{X_{l}}I_{n},\] \[\Sigma_{X_{l}} =\mathfrak{a}_{l}I_{n}+\mathfrak{b}_{l}K_{X}.\]

**Assumptions 7.1**.:
1. \(W_{l}\), \(B_{l}\) and \(D_{l}\) are random, independent as a family for \(l\in\llbracket 1,L\rrbracket\), with i.i.d. \(\mathcal{N}(\sigma^{2}_{W_{l}})\), \(\mathcal{N}(\sigma^{2}_{B_{l}})\) and \(\mathcal{N}(\sigma^{2}_{D_{l}})\) entries respectively.
2. \(\tilde{f}_{l}\) are Lipschitz continuous and Gaussian centered, that is \(\mathbb{E}\Big{[}\tilde{f}_{l}(\mathcal{N})\Big{]}=\mathbb{E}[f(\tilde{\sigma }_{l}\mathcal{N})]=0\).
3. \(X_{0}\) is random, independent from all the other matrices. There is an event \(\mathcal{B}\) with \(\mathbb{P}(\mathcal{B}^{c})\leq O(\sqrt{\log n}/n)\), such that \((X_{0}|\mathcal{B})\propto_{\llbracket 1\rrbracket_{F}}\mathcal{E}(1)\).
4. There is a sequence \(\epsilon_{n}\) converging to \(0\), with \(\sqrt{\log n/n}\leq O(\epsilon_{n})\), such that uniformly in \(\omega\in\mathcal{B}\), \(\left|\!\left|\!\left|K_{0}\right|\!\right|\!\right|\) and \(\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left| \!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\left|\! \left|\!\left|\!\left|\!\left|\!\left|\!\left|\!\!\left|\!\left|\!\left|\! \left|\!\left|\!\left|\!\left|\!\!\left|\!\left|\!\left|\!\!\left|\!\left|\! \left|\!\left|\!\left|\!\!\left|\!\left|\!\!\left|\!\left|\!\!\left|\!\! \left|\!\left|\!\!\left|\!\left|\!\!\left|\!\left|\!\!\left|\!\!\left|\! \left|\!\!\left|\!\left|\!\!\left|\!\left|\!\!\left|\!\left|\!\!\left|\! \left|\!\!\left|\!\!\left|\!\!\left|\!\left|\!\!\left|\!\!\left|\!\! \left|\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left| \!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\! \left|\!\!\left|\!\!\left|\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\! \!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\!\left|\!\! \left|\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\!\left|\!\!\left|\!\! \left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\left|\!\!\!\left|\!\!\left|\!\
using Theorem 6.4 we get the deterministic equivalents for all layers. The error terms \(\hat{\epsilon}_{n}^{(l)}\) and \(\hat{\epsilon}_{n}^{\prime(l)}\) are given by the formulas above Theorem 6.4. 

_Remark 7.3_.: The above deterministic equivalents are only meaningful if the error terms \(\hat{\epsilon}_{n}^{\prime(l)}\) and \(\hat{\epsilon}_{n}^{(l)}\) vanish when \(n\to\infty\). Similarly to Remark 5.8, let us mention a few cases where their expressions may be greatly simplified:

* If \(\zeta_{1}(\tilde{f}_{l})=0\) for some layer \(l\), then for all subsequent layers \(k\geq l\) the error terms \(\hat{\epsilon}_{n}^{\prime(k)}\) and \(\hat{\epsilon}_{n}^{(k)}\) do not depend on \(\hat{\epsilon}_{n}^{(0)}\) and \(\hat{\epsilon}_{n}^{\prime(0)}\) anymore.
* If \(\epsilon_{n}=o(n^{-1/4})\), then \(\hat{\epsilon}_{n}^{(l)}=O\Big{(}\hat{\epsilon}_{n}^{(0)}+\sqrt{n}\epsilon_{n }^{2}\Big{)}\).
* If \(\zeta_{2}(\tilde{f}_{l})=0\) and \(\epsilon_{n}=o(n^{-1/6})\), then \(\hat{\epsilon}_{n}^{(l)}=O\Big{(}\hat{\epsilon}_{n}^{(0)}+\sqrt{n}\epsilon_{n }^{3}\Big{)}\).
* If \(\zeta_{2}(\tilde{f}_{l})=\zeta_{3}(\tilde{f}_{l})=0\), then \(\hat{\epsilon}_{n}^{(l)}=O\Big{(}\hat{\epsilon}_{n}^{(0)}+\sqrt{\log n}/n\Big{)}\).
* If \(\epsilon_{n}=o(n^{-1/2})\), then \(\hat{\epsilon}_{n}^{\prime(l)}=O\Big{(}\hat{\epsilon}_{n}^{(0)}+\hat{\epsilon }_{n}^{\prime(0)}+n\epsilon_{n}^{2}\Big{)}\).
* If \(\zeta_{2}(\tilde{f}_{l})=0\) and \(\epsilon_{n}=o(n^{-1/3})\), then \(\hat{\epsilon}_{n}^{\prime(l)}=O\Big{(}\hat{\epsilon}_{n}^{\prime(0)}+n \epsilon_{n}^{3}\Big{)}\).
* If \(\zeta_{2}(\tilde{f}_{l})=\zeta_{3}(\tilde{f}_{l})=0\), then \(\hat{\epsilon}_{n}^{\prime(l)}=O\Big{(}\hat{\epsilon}_{n}^{\prime(0)}+1/\sqrt{ n}\Big{)}\).

In the case \(\epsilon_{n}=O(\sqrt{\log n/n})\) corresponding to a data matrix \(X_{0}\) with i.i.d. columns (see Proposition 6.6), and starting from typical \(\hat{\epsilon}_{n}^{(0)}=O_{z}(1/n)\) and \(\hat{\epsilon}_{n}^{\prime(0)}O_{z}(1/\sqrt{n})\) equivalents for the Stieltjes transform and the resolvent of \(K_{0}\) respectively, Theorem 7.2 gives an \(O_{z}(\log n/\sqrt{n})\) equivalent for the Stieltjes transform. The error for the resolvents does not vanish in general because of the \(O_{z}(\zeta_{2}(\tilde{f})^{2}\log n)\) error term. If \(\zeta_{2}(\tilde{f})=0\) however, we obtain a \(O_{z}((\log n)^{3/2}/\sqrt{n})\) approximation.

**Corollary 7.4**.: _Uniformly under Assumptions 7.1:_

1. \(\Big{|}g_{K_{l}}(z)-g_{\chi_{n}^{(l)}}(z)\Big{|}\leq\sqrt{\log n}\,O_{z}(\hat {\epsilon}_{n}^{(l)})\) _a.s., and_ \(\left\|\mathcal{G}_{K_{l}}(z)-\mathbf{G}_{l}(z)\right\|_{\max}\leq\sqrt{\log n }\,O_{z}(\hat{\epsilon}_{n}^{\prime})\) _a.s._
2. _If_ \(\hat{f}_{l}\) _is not linear, or if the measures_ \(\chi_{n}^{(0)}\) _are supported on the same compact of_ \((0,\infty)\)_, there exists_ \(\theta>0\) _such that_ \(D(\mu_{K_{l}},\chi_{n}^{(l)})\leq O(\hat{\epsilon}_{n}^{(l)\theta})\) _a.s._
3. _If moreover_ \(\chi_{n}^{(0)}\) _converges weakly to a measure_ \(\chi_{\infty}^{(0)}\)_, and if all ratios_ \(\gamma_{n}^{(k)}\to\gamma_{\infty}^{(k)}>0\) _, then_ \(\mu_{K_{l}}\) _converges a.s. to the measure_ \(\chi_{\infty}^{(l)}\) _defined by induction as:_ \[\chi_{\infty}^{(l)}=\mathrm{MP}\Big{(}\gamma_{\infty}^{(l)}\Big{)}\boxtimes \Big{(}\mathfrak{a}_{l}+\mathfrak{b}_{l}\chi_{\infty}^{(l-1)}\Big{)}.\] _More precisely:_ \[D(\mu_{K_{l}},\chi_{\infty}^{(l)})\leq O\bigg{(}D(\chi_{n}^{(0)},\chi_{\infty}^ {(0)})+\max_{0\leq k\leq l}|\gamma_{n}^{(k)}-\gamma_{\infty}^{(k)}|+\hat{ \epsilon}_{n}^{(l)\theta}\bigg{)}\quad\text{a.s.}\]

Again we will not prove this result here, but refer to the proof of Corollary 5.9 which is similar.

_Remark 7.5_.: Our result generalizes previously known global laws on the Conjugate Kernel model. Adapted to our notations, [10, Theorem 3.4] states that, without bias in the model, if \(f\) is twice differentiable, \(\epsilon_{n}=o(n^{-1/4})\), and \(\chi_{n}^{(0)}\) converges weakly to \(\chi_{\infty}^{(0)}\), then \(\mu_{K_{l}}\) converges weakly to
\(\chi_{\infty}^{(l)}\) a.s. Taking into account the Remark 7.3, in this setting we have \(\hat{\epsilon}_{n}^{(l)}=O(\hat{\epsilon}_{n}^{(0)}+\sqrt{n}\epsilon_{n}^{2})=O( \hat{\epsilon}_{n}^{(0)}+o(1))\), and we retrieve the a.s. convergence of \(\mu_{K_{l}}\) towards \(\chi_{\infty}^{(l)}\) weakly, supplemented with quantitative estimates for the Stieltjes transforms and the Kolmogorov distances.
## 8. Appendix: Bounds on Kolmogorov distances between empirical spectral measures

Let us remind the notations \(\mathcal{F}_{\nu}\) for the cumulative distribution function of a measure \(\nu\), and \(D(\nu,\mu)=\sup\limits_{t\in\mathbb{R}}|\mathcal{F}_{\nu}(t)-\mathcal{F}_{\mu}(t)|\) for the Kolmogorov distance between two measures \(\nu\) and \(\mu\).

It is a well-known fact that the convergence in Kolmogorov distance implies the weak convergence for probability measures, and there is even an equivalence if the limiting measure admits a Holder continuous cumulative distribution function ([1]). In [1] and [11], the authors propose a general method to derive a convergence speed in Kolmogorov distance from estimates on the Stieltjes transforms. This method implies for instance our Proposition 2.8. However the techniques employed are not well suited to work with two discrete measures like empirical spectral distributions.

Two matrices close in spectral norm admit the same limiting spectral distribution if it exists. This does not imply any bound on the Kolmogorov distances however in general because the measures are discrete. In this section, we show how a quantitative result may still be obtained, provided the limiting empirical measure is regular enough. We strongly incite the reader to first examine [11, Section 8] where the technical tools are explained in full details.

**Proposition 8.1**.: _Let \(\Sigma\) and \(\tilde{\Sigma}\in\mathbb{R}^{p\times p}\) be symmetric matrices such that:_

1. \(\left\|\Sigma\right\|\) _and_ \(\left\|\tilde{\Sigma}\right\|\) _are bounded, and_ \(\left\|\Sigma-\tilde{\Sigma}\right\|\) _converges to_ \(0\)_._
2. \(\mu_{\tilde{\Sigma}}\) _converges weakly to some probability measure_ \(\nu^{\infty}\)_, and_ \(\mathcal{F}_{\nu^{\infty}}\) _is Holder continuous for some parameter_ \(\beta>0\)_._

_Then \(\mu_{\Sigma}\) converges weakly to \(\nu^{\infty}\), and more precisely in Kolmogorov distance:_

**Lemma 8.2**.: _For any \(y\in(0,1)\) and \(A>0\):_

Proof.: We closely follow the [11, Section 3.1] with the only key difference that we plug in Bai's Inequality the following bound:

\[\left|\mathcal{F}_{\mu_{\tilde{\Sigma}}}(x+t)-\mathcal{F}_{\mu_{\tilde{\Sigma }}}(x)\right|\leq|\mathcal{F}_{\nu^{\infty}}(x+t)-\mathcal{F}_{\nu^{\infty}}( x)|+2\,D\big{(}\mu_{\tilde{\Sigma}},\nu^{\infty}\big{)}.\]

We thus obtain:

\[D(\mu_{\tilde{\Sigma}},\mu_{\Sigma}) \leq\frac{2}{\pi}\Bigg{(}\int_{\mathbb{R}}|g_{\mu_{\tilde{\Sigma }}}-g_{\mu_{\Sigma}}\big{|}(t+iy)dt+\frac{1}{y}\sup\limits_{x\in\mathbb{R}} \int_{\left[\pm 2y\tan\left(\frac{3\pi}{8}\right)\right]}|\mathcal{F}_{\nu^{\infty }}(x+t)-\mathcal{F}_{\nu^{\infty}}(x)|dt\Bigg{)}\] \[\quad+O\big{(}D\big{(}\mu_{\tilde{\Sigma}},\nu^{\infty}\big{)} \big{)}.\]

For \(z\in\mathbb{C}^{+}\) a classical application of the resolvent identity gives \(\big{|}g_{\Sigma}(z)-g_{\tilde{\Sigma}}(z)\big{|}\leq\left\|\Sigma-\tilde{ \Sigma}\right\|\). The rest of the proof is exactly the same as in [11, Section 3.1].
Proof of Theorem 8.1.: We optimize \(y\) and \(A\) in the above lemma by choosing \(y_{n}\,=\,\left\|\left|\Sigma-\tilde{\Sigma}\right|\right\|^{\frac{1}{4+2\beta}}\) and \(A_{n}\,=\,\left\|\left|\Sigma-\tilde{\Sigma}\right|\right\|^{-1/2}\), which leads to the bound \(D(\mu_{\tilde{\Sigma}},\mu_{\Sigma})\leq O\bigg{(}\left\|\left|\Sigma-\tilde{ \Sigma}\right|\right\|^{\frac{\beta}{4+2\beta}}+D\big{(}\mu_{\tilde{\Sigma}}, \nu^{\infty}\big{)}\bigg{)}\). A final triangular inequality proves the proposition.