has been found to control the quality of initialization. We establish theoretically and empirically that the best initialization variances for residual networks depend on the depth of the network (contrary to the feedforward case [@schoenholz_deep_2017]), so that **common initialization schemes like Xavier [@glorot_understanding_2010] or He [@he_delving_2015] cannot be optimal**. In fact, even the rationale of He initialization is incorrect for ReLU residual networks because it tries to control gradient dynamics rather than expressivity. However we want to emphasize that we study a simplified model of residual networks in this work, with no batchnorm or convolutional layers, so that these results are not necessarily indicative of the MSRA residual network used in practice [@he_deep_2016].

In the body of this paper, we give account of general intuition and/or proof strategy when appropriate for our theoretical results, but we relegate all formal statements and proofs to the appendix.

Background {#sec:background}
==========

Consider a vanilla feedforward neural network of $L$ layers, with each layer $l$ having $\p N l$ neurons; here layer 0 is the input layer. For the ease of presentation we assume all hidden layer widths are the same $\p N l = N$ for all $l > 0$. Let $\p x 