\[thm:ReLUSquaredConvergence\], and \[thm:alphaReLUeConvergence\]. This slow convergence preserves geometric information in the input space, and allows a typical residual network to “hover over the edge of chaos”: Even when the cosine distance $\p{\mathbf{e}}l$ converges to 0, corresponding to “chaos”, (resp. 1, corresponding to “stability”), for the number of layers usually seen in practice, $\p {\mathbf{e}}l$ will reside well away from 0 (resp. 1).

Similarly, the quantity ${\mathbf{s}}$ measures the metric geometry of two vectors. The evolution of $\p {\mathbf{s}}l$ with $l$ tells us the ability of the average network to separate two input points in terms of Euclidean distance. Again, for tanh and $\alpha$-ReLU ($\alpha < 1$) nonlinearities, ${\mathbf{s}}$ varies only polynomially with $l$.

On the other hand, $\p {{\boldsymbol{\oldchi}}}l$ measures the size of gradient at layer $l$, and through it we track the dynamics of gradient backpropagation, be it explosion or vanishing. In contrast to vanilla tanh networks, which can experience both of these two phenomenon depending on the initialization variances, typical residual networks cannot have vanishing gradient, in the sense of vanishing $\p {{\boldsymbol{\oldchi}}}l$ as $l \to 1$; see \[thm:dalethRecReduced\] and \[thm:dalethRecFull\]. Furthermore, while vanilla tanh networks exhibit exponentially vanishing or exploding gradients, all of the activation/architecture pairings considered 