$\exp(\Theta(\sqrt l))$, \[thm:dalethExpSqrtTanh\]      $\exp(\Theta(\sqrt l))$, \[thm:dalethRecFull\]       $\exp(\Theta(l))$, \[thm:dalethDynamicsAlphaReLU\]   $\Theta(l^{\f{\alpha^2}{(1-\alpha)(2 \alpha - 1)}})$, \[thm:dalethDynamicsAlphaReLU\]   

  : Summary of Main Dynamics Results. Note that while $\p{{\boldsymbol{\oldchi}}}l$ is exponential for ReLU/FRN, the gradients with respect to weight parameters have norms (${\boldsymbol{\oldchi}}_w$ and ${\boldsymbol{\oldchi}}_v$) constant in $l$ (\[thm:alphaReLUAllGradients\]). Also, the $\p {{\boldsymbol{\oldchi}}}l$ entry for $\alpha$-ReLU is for $\alpha \in (3/4, 1)$ only[]{data-label="tab:dynamics"}

Tanh
----

![Our equations predict the relevant quantities very well in practice. These plots make the comparison between prediction and measurements for the full resnet with tanh activation, with $\sigma_v^2 = 1.5$, $\sigma_a^2 = .5$, $\sigma_w^2 = 1.69$, $\sigma_b^2 = .49$. Left-to-right: [**(a)**]{} $\p {\mathbf{p}}l$ and $\p {\boldsymbol{\oldgamma}}l$ against layer $l$ for 200 layers. [**(b)**]{} $\p {\mathbf{e}}l = \p {\boldsymbol{\oldgamma}}l /\p {\mathbf{p}}l$ against $l$ for 200 layers. Both (a) and (b) trace out curves for different initial conditions. [**(c)**]{} Different gradient quantities against $l$ for 50 layers. From left to right the layer number $l$ decreases, following the direction of backpropagation. Notice that the gradient increases in norm as $l \to 1$. All three figures exhibit smooth curves, which are theoretical estimates, and irregular curves with shades around them, which indicate empirical means and standard deviations (both of which taken in regular scale, 