A = \f 4 3 \sqrt{\f 2 \pi} \f{\sigma_v^2 \sigma_w}{\sqrt{\sigma_v^2 + \sigma_a^2}}$ in the FRN case (\[thm:dalethExpSqrtTanh\] and \[thm:dalethExpSqrtTanhFullRes\]). The rightmost plot of \[fig:edelta\_plot\] verifies that indeed, for large $l \ge 1000$, this is a very good approximation. This demonstrates that the mean field assumption of independent backpropagation weights is very practical and convenient even for residual networks.

Note that in the FRN case, the constant $\mathcal A$ can be decomposed into $\mathcal A = \f 4 3 \sqrt{\f 2 \pi} \cdot \sigma_v \cdot \sigma_w \cdot (1 + \sigma_a^2/\sigma_v^2)^{-1/2}$. Consider the ratio $\rho := \sigma_a/\sigma_v$. If $\rho \gg 1$, then ${\mathbf{e}}^* \approx 1$ (\[fig:jjj\_vs\_id\_main\]), meaning that the typical network essentially computes a constant function, and thus unexpressive; at the same time, large $\rho$ makes $\mathcal A$ small, and thus ameliorating the gradient explosion problem, making the network more trainable. On the other hand, if $\rho \ll 1$, then ${\mathbf{e}}^* \approx 0$ (\[fig:jjj\_vs\_id\_main\]), the typical network can tease out the finest differences between any two input vectors, and a final linear layer on top of such a network should be able to express a wide variety of functions [@poole_exponential_2016]; at the same time, small $\rho$ increases $\mathcal A$, worsening the gradient 