similar nature, but phrased in the language of reproducing kernel Hilbert spaces, and provided high probability estimates that are meaningful for the case when the width $N$ is finite and the depth is logarithmic in $N$. However, they essentially fixed the variance parameters $\sigma_\bullet$, and furthermore, their framework (for example the notion of a “skeleton”) does not immediately generalize to the residual network case.

In this work, we show that residual networks have very different dynamics from vanilla feedforward networks. In most cases, the cosine distance convergence rate and the gradient growth rate are subexponential in a residual network, and in most cases, these rates may be polynomial.

Preliminaries
=============

Residual networks were first introduced by [@he_deep_2016] and later refined by [@he_identity_2016], and they are now commonplace among deployed neural systems. The key innovation there is the addition of a shortcut connection from the previous layer to the next. We define the following idealized architectures for ease of analysis. Note that we only consider fully-connected affine layers instead of convolutional layers. A [**reduced residual network (RRN)**]{} has the recurrence $$\begin{aligned}
        x_i &= \phi(h_i) + {\underline}x, & h_i &= \sum_j w_{ij} {\underline}x_j + b_i.
    \end{aligned}$$ A [**(full) residual network (FRN)**]{} in 