0 = (\p x 0_1, \ldots, \p x 0_{\p N 0})$ be the input vector to the network, and let $\p x l$ for $l > 0$ be the activation of layer $l$. Then a neural network is given by the equations $$\begin{aligned}
        \p x l _i &= \phi(\p h l _i), & \p h l _i &= \sum_{j=1}^N \p w l _{ij} \p x {l-1}_j + \p b l_i
    \end{aligned}$$ where

$\p h l$ is the pre-activation at layer $l$,

$\p w l$ is the weight matrix,

$\p b l$ is the bias vector, and

$\phi$ is a nonlinearity, for example $\tanh$ or ReLU, which is applied coordinatewise to its input.

To lighten up notation, we suppress the explicit layer numbers $l$ and write $$\begin{aligned}
        x_i &= \phi(h_i), & h_i &= \sum_j w_{ij} {\underline}x_j + b_i
    \end{aligned}$$ where $\bullet$ implicitly denotes $\p \bullet l$, and ${\underline}\bullet$ denotes $\p \bullet {l-1}$ (and analogously, ${\overline}\bullet$ denotes $\p \bullet {l+1}$).

A series of papers [@poole_exponential_2016; @raghu_expressive_2016; @schoenholz_deep_2017] investigated the “average behavior” of random neural networks sampled via $\p w l_{ij} \sim {\mathcal{N}}(0, \sigma_w^2/N), \p b l_i \sim {\mathcal{N}}(0, \sigma_b^2)$, for fixed parameters $\sigma_w$ and $\sigma_b$, independent of $l$. Consider the expectation of 