as $\tanh$ [**live on the edge of chaos**]{}, in that the cosine distance of two input vectors will converge to a fixed point at a polynomial rate, rather than an exponential rate, as with vanilla tanh networks. Thus a typical residual network will slowly cross the stable-chaotic boundary with depth, hovering around this boundary for many layers. In addition, for most of the nonlinearities considered here, the mean field estimate of the gradient grows subexponentially with depth. In fact, for $\alpha$-ReLU, the $\alpha$th-power of ReLU, for $\alpha < 1$, the gradient grows only polynomially. These theoretical results provide some theoretical justification for why residual networks work so well in practice. In our experiments, we are also able to predict surprisingly well the relative performances of [*trained*]{} residual networks based only on their initialization hyperparameters, in a variety of settings. In particular, we find that the quality of initialization for tanh resnets is determined by [*trainability*]{} (how much gradient explosion on average) while that for ($\alpha$-)ReLU resnets is determined by expressivity (how far can two different input vectors be pulled apart) (see \[sec:experiments\]). To the best of our knowledge, this is the first time that a quantity other than gradient explosion/vanishing 