addition has an affine connection given by weights $v$ and biases $a$ from the nonlinearity $\phi(h)$ to the next layer: $$\begin{aligned}
    x_i &= \sum_j v_{ij} \phi( h_j) + {\underline}x_i + a_i, &  h_i &= \sum_j w_{ij} {\underline}x_j + b_i
    \end{aligned}$$

We are interested in the “average behavior” of these network when the weights and biases, $\p w l_{ij}, \p b l_i, \p v l_{ij}$, and $\p a l_i$ are sampled i.i.d. from Gaussian distributions resp. with standard deviations $\sigma_w, \sigma_b, \sigma_v,$ and $\sigma_a$, independent from $l$. Here we take the variance of $\p w l_{ij}$ to be $\sigma_w^2/N$ so that the variance of each $h_i$ is $\sigma_w^2$, assuming each ${\underline}x_j$ is fixed (similarity for $\p v l_{ij}$). Such an initialization scheme is standard in practice.

We make several key “physical assumptions” to make theoretical computations tractable:

\[ass:symAct\] (a) We assume $\la (\p h l _i)^2 \ra = \la (\p h l _j)^2 \ra$ and $\la (\p x 0 _i)^2 \ra = \la (\p x 0 _j)^2 \ra$ for any $i, j, l$. (b) We also assume that the gradient $\pd E/\pd \p x l _i$ with respect to the loss function $E$ satisfies $\la (\pd E/\pd \p x l _i)^2 