the activations in the previous layer.

In what follows, we use this technique to derive recurrence equations governing ${\mathbf{p}}, {\mathbf{q}}, {\boldsymbol{\oldgamma}}, {\boldsymbol{\oldlambda}}, {{\boldsymbol{\oldchi}}}$ for different architectures and different activation functions. We use these equations to investigate the dynamics of ${\mathbf{e}}$ and ${\mathbf{s}}$, the key quantities in the forward pass, and the dynamics of ${{\boldsymbol{\oldchi}}}$, the key quantity in the backward pass.

The cosine distance ${\mathbf{e}}$ in some sense measures the angular geometry of two vectors. If ${\mathbf{e}}= 1$, then the vectors are parallel; if ${\mathbf{e}}= 0$, then they are orthogonal. Just as in [@poole_exponential_2016] and [@schoenholz_deep_2017], we will show that in all of the architectures and activations we consider in this paper, $\p {\mathbf{e}}l$ converges to a fixed point ${\mathbf{e}}^*$ as $l \to \infty$ [^2]. Thus, on the average, as vectors propagate through network, the geometry of the original input space, for example, linear separability, is “forgotten” by residual networks as well as by vanilla networks. But we will prove and verify experimentally that, while @poole_exponential_2016 and [@schoenholz_deep_2017] showed that the convergence rate to ${\mathbf{e}}^*$ is exponential in a vanilla network, the convergence rate is rather only polynomial in residual networks, for tanh and $\alpha$-ReLU (\[defn:alphaReLU\]) nonlinearities; see \[thm:dalethRecReduced\], \[thm:eDynamicsFullResTanh\], 