here, except the full residual network with ReLU, have subexponential gradient dynamics. While tanh residual networks (reduced or full) has $\p {{\boldsymbol{\oldchi}}}0 \approx \exp(\Theta(\sqrt l)) \p {{\boldsymbol{\oldchi}}}l$ (\[thm:dalethExpSqrtTanhFullRes\]), $\alpha$-ReLU residual networks for $\alpha < 1$ have $\p {{\boldsymbol{\oldchi}}}0 \approx {\mathsf{poly}}(l) \p {{\boldsymbol{\oldchi}}}l$ (\[thm:dalethDynamicsAlphaReLU\]). Instead of $\pd E/\pd x_i$, we may also consider the size of gradients of actual trainable parameters. For tanh and $\alpha$-ReLU with $\alpha < 1$, they are still subexponential and polynomial (\[thm:alphaReLUAllGradients\]). On the other hand, while $\p {{\boldsymbol{\oldchi}}}0 = \exp(\Theta(l))\p {{\boldsymbol{\oldchi}}}l$ for a ReLU resnet, its weight gradients have size independent of layer, within $O(1)$ (\[thm:alphaReLUAllGradients\])! This is the only instance in this paper of gradient norm being completely preserved across layers.

The above overviews the theoretical portion of this paper. Through experiments, we discover that we can very accurately predict whether one random initialization leads to better performance than another on the test set, after training, by leveraging this theory we build. Residual networks of different nonlinearities have different [*controlling quantities*]{}: for resnets with tanh, the optimal initialization is obtained by controlling the gradient explosion $\p {{\boldsymbol{\oldchi}}}0 / \p {{\boldsymbol{\oldchi}}}L$; whereas for ReLU and $\alpha$-ReLU, the optimal initialization is obtained by maximizing ${\mathbf{s}}$ without running 