the images of two input vectors. Importantly, we show, theoretically as well as empirically, that common initializations such as the Xavier or the He schemes are not optimal for residual networks, because [*the optimal initialization variances depend on the depth*]{}. Finally, we have made mathematical contributions by deriving several new identities for the kernels of powers of ReLU functions by relating them to the zeroth Bessel function of the second kind.'
author:
- |
    Greg Yang[^1]\
    [Microsoft Research AI]{}\
    `gregyang@microsoft.com`\
    Samuel S. Schoenholz\
    [Google Brain]{}\
    `schsam@google.com`\
bibliography:
- 'neural\_dynamics.bib'
title: 'Mean Field Residual Networks: On the Edge of Chaos'
---

Introduction
============

Previous works [@poole_exponential_2016; @daniely_toward_2016; @schoenholz_deep_2017] have shown that randomly initialized neural networks exhibit a spectrum of behavior with depth, from stable to chaotic, which depends on the variance of the initializations: the cosine distance of two input vectors converges exponentially fast with depth to a fixed point in \[0, 1\]; if this fixed point is 1, then the behavior is stable; if this fixed point is 0, then the behavior is chaotic. It has been argued in many prior works [@bertschinger_real-time_2004; @poole_exponential_2016] that effective computation can only be supported by a dynamical behavior that is on [**the edge 