Weakness]: (1) There is a large gap in the proof of Theorem 1. (2) Missing discussion of the line of research using random matrix theory to understand the input-output Jacobian [1], which also consider the operator norm of the input-out Jacobian and draws a very similar conclusion, e.g., the squared operator norm must grow linearly with the number of layers; see eq (17) and follow up discussion in [1].
In what follows, I elaborate (1) and (2) since they are related.
The biggest issue I see in the proof is the equation above (A.1) on page 11. The authors mixed the calculation of finite width networks (on the left of the equation) and infinite width network calculation together (on the right). More precisely, the authors exchange the order of the two limits lim w i d t h → ∞ and
lim sup x α → x
. The exchangeability of the two limits is questionable to me. In the order: lim w i d t h → ∞
lim sup x α →
, we need to handle a product of random matrices (if we compute the Jacobian). This is indeed a core contribution of [1], who uses free probability theory to compute the whole spectrum of the singular values of the Jacobian (assuming certain free independence of the matrices). If we swap the limits (we shouldn't do this without justification) to
lim sup x α → x β lim w i d t h →
, the problem itself is reduced to computing the derivative of the composed correlation map, which is much simpler. I think these two limits are not unchangeable in general. E.g., using the order
lim sup x α → x β lim w i d t h →
, both critical gaussian and orthogonal initialization give the same answer. But using the order lim w i d t h → ∞
lim sup x α →
, gaussian and orthogonal initialization can give different answers, see eq (17) vs (22) in [1].
Several Qs: Q1:
How Theorem 1 leads to the four possible cases after it needs more discussion. In addition, what are the new insights quotient the existing ones from the order-chaotic analysis? It seems: the first case corresponds to the chaotic phase, the second case corresponds to the order phase. The third/fourth cases seem to be a finer analysis of the critical regime.
Q2: Remark1 the critical initialization. Several works have already identified the issue of the polynomial rate convergence of the correlation to 1 for Relu and smooth functions; see Proposition 1 in [2]; sec B.3. in [3].
Q3: I can't find places to explain the legends "upper bound", "largest found".
Q4: How does Thm1 imply eq (4.1)? Do you assume the operator norm is bounded by O(1)?
[1] Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, https://arxiv.org/pdf/1711.04735.pdf [2] On the Impact of the Activation Function on Deep Neural Networks Training, https://arxiv.org/abs/1902.06853 [3] Disentangling Trainability and Generalization in Deep Neural Networks, https://arxiv.org/abs/1912.13053
Minors comments: 1.) What is the domain of the inputs? It seems they are lying in the same sphere, not mentioned in the paper.
1.) What is the domain of the inputs? It seems they are lying in the same sphere, not mentioned in the paper.