Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets. Skip connections are extra connections between nodes in different layers of a neural network that skip one or more layers of nonlinear processing. The introduction of skip (or residual) connections has substantially improved the training of very deep neural networks BID8 BID11 Srivastava et al., 2015) . Despite informal intuitions put forward to motivate skip connections, a clear understanding of how these connections improve training has been lacking. Such understanding is invaluable both in its own right and for the possibilities it might offer for further improvements in training very deep neural networks. In this paper, we attempt to shed light on this question. We argue that skip connections improve the training of deep networks partly by eliminating the singularities inherent in the loss landscapes of deep networks. These singularities are caused by the non-identifiability of subsets of parameters when nodes in the network either get eliminated (elimination singularities), collapse into each other (overlap singularities) (Wei et al., 2008) , or become linearly dependent (linear dependence singularities). Saad & Solla (1995) ; BID0 ; Wei et al. (2008) identified the elimination and overlap singularities and showed that they significantly slow down learning in shallow networks; Saxe et al. (2013) showed that linear dependence between nodes arises generically in randomly initialized deep linear networks and becomes more severe with depth. We show that skip connections eliminate these singularities and provide evidence suggesting that they improve training partly by ameliorating the learning slow-down caused by the singularities. In this paper, we proposed a novel explanation for the benefits of skip connections in terms of the elimination of singularities. Our results suggest that elimination of singularities contributes at least partly to the success of skip connections. However, we emphasize that singularity elimination is not the only factor explaining the benefits of skip connections. Even in completely non-degenerate models, other independent factors such as the behavior of gradient norms would affect training performance. Indeed, we presented evidence suggesting that skip connections are also quite effective at dealing with the problem of vanishing gradients and not every form of singularity elimination can be expected to be equally good at dealing with such additional problems that beset the training of deep networks.Alternative explanations: Several of our experiments rule out vanishing gradients as the sole explanation for training difficulties in deep networks and strongly suggest an independent role for the singularities arising from the non-identifiability of the model. (i) In FIG4 , all nets have the exact same plain architecture and similarly vanishing gradients at the beginning of training, yet they have diverging performances correlated with measures of distance from singular manifolds. (ii) Vanishing gradients cannot explain the difference between identity skips and dense orthogonal skips in Figure 7 , because both eliminate vanishing gradients, yet dense orthogonal skips perform better. (iii) In FIG7 , spectrum-equalized non-orthogonal skips often have larger gradient norms, yet worse performance than orthogonal skips. (iv) Vanishing gradients cannot even explain the BiasReg results in FIG5 . The BiasReg and the plain net have almost identical (and vanishing) gradients early on in training (Figure 6a ), yet the former has better performance as predicted by the symmetry-breaking hypothesis. (v) Similar results hold for two-layer shallow networks where the problem of vanishing gradients does not arise (Supplementary Note 7) . In particular, shallow residual nets are less degenerate and have better accuracy than shallow plain nets; moreover, gradient norms and accuracy are strongly correlated with distance from the overlap manifolds in these shallow nets.Our malicious initialization experiment with residual nets FIG5 ) suggests that the benefits of skip connections cannot be explained solely in terms of well-conditioning or improved initialization either. This result reveals a fundamental weakness in purely linear explanations of the benefits of skip connections BID7 BID14 . Unlike in nonlinear nets, improved initialization entirely explains the benefits of skip connections in linear nets (Supplementary Note 5).A recent paper BID2 suggested that the loss of spatial structure in the covariance of the gradients, a phenomenon called "shattered gradients", could be partly responsible for training difficulties in deep nonlinear networks. They argued that skip connections alleviate this problem by essentially making the model "more linear". It is easy to see that the shattered gradients problem is distinct from both the vanishing/exploding gradients problem and the degeneracy problems considered in this paper, since shattered gradients arise only in sufficiently non-linear deep networks (linear networks do not shatter gradients), whereas vanishing/exploding gradients, as well as the degeneracies considered here, arise in linear networks too. The relative contribution of each of these distinct problems to training difficulties in deep networks remains to be determined.
Degenerate manifolds arising from the non-identifiability of the model slow down learning in deep networks; skip connections help by breaking degeneracies. The authors show that elimination singularities and overlap singularities impede learning in deep neural networks, and demonstrate that skip connections can reduce the prevalence of these singularities, speeding up learning. Paper examines the use of skip connections in deep networks as a way of alleviating singularities in the Hessian matrix during training.