$\f 1 N \sum_{i=1}^N x^2_i$, the normalized squared length of $x$, over the sampling of $w$ and $b$. @poole_exponential_2016 showed that this quantity converges to a fixed point exponentially fast for sigmoid nonlinearities. Now suppose we propagate two different vectors $\p x 0$ and $(\p x 0)'$ through the network. @poole_exponential_2016 also showed that the expectation of the normalized dot product $\f 1 N \sum_{i=1}^N x_i x'_i$ converges exponentially fast to a fixed point. The ratio between the normalized squared length and the normalized dot product is the cosine distance between $x$ and $x'$. Thus these two exponential convergence results show that the cosine distance converges exponentially fast to a fixed point as well. Intuitively, this means that a vanilla feedforward network “forgets” the geometry of the input space “very quickly,” after only a few layers.

In addition, @schoenholz_deep_2017, under certain independence assumptions, showed that the expected normalized squared norm of the gradient also vanishes or explodes in an exponential fashion with depth, with the ”half-life” controlled by $\sigma_w$ and $\sigma_b$. They verified that this theoretical ”half-life” correlates in practice with the maximal number of layers that are admissible to good performance.

At the same time, @daniely_toward_2016 published work of 