\pi$. Thus the rate of convergence at its [**slowest**]{} for tanh/FRN is $\delta^* = 1 - \f 2 \pi \approx 0.36338$, where asymptotically the network tends toward a [**chaotic regime**]{} ${\mathbf{e}}^* = 0$, corresponding to a large weight variance and a small bias variance; it at its [**fastest**]{} is $\delta^* = \nicefrac 1 2$, where asymptotically the network tends toward a [**stable regime**]{} ${\mathbf{e}}^* = 1$, corresponding to a large bias variance and small weight variance. We verify $\delta^*$ by comparing $\p {\mathbf{e}}l - \p {\mathbf{e}}{l-1}$ to $l^{-\delta^* - 1}$ in log-log scale. If $\p {\mathbf{e}}l = \Theta(l^{-\delta^*})$, then $\p {\mathbf{e}}l - \p {\mathbf{e}}{l-1} = \Theta(l^{-\delta^* - 1})$ and should obtain the same slope as $l^{-\delta^* - 1}$ as $l \to \infty$. The middle figure of \[fig:edelta\_plot\] ascertains that this is indeed the case, starting around layer number 400.

![Left-to-right: [**(a)**]{} Plots of ${\mathbf{e}}^*$ and $\delta^*$ against $\sigma_a/\sigma_v$. [**(b)**]{} In log-log scale: the dashed line is $l^{-\delta^* - 1}$, and the colored lines are $\p {\mathbf{e}}l - \p {\mathbf{e}}{l-1}$ for different initial conditions $\p {\mathbf{e}}0$. That they become parallel at about $l = 400$ on verifies that $\p {\mathbf{e}}l = \Theta(l^{-\delta^*})$. [^5] [**(c)**]{} In log-log scale: The dashed line is 