for deep linear networks, all local minima of the loss function are global [@NoPoor]. However, no real-world neural networks use the identity function as the activation function $\sigma$, and once $\sigma$ is allowed to be nonlinear, we know very little about the local non-global minima of $L$. Several groups, including [@srabadlocalmin], [@spuriouscommon], and [@ruoyumin], have proven, under various sets of assumptions that are weak enough to include some real-world networks, that deep nonlinear neural networks always have spurious local minima.

The geometry of $L$ near critical points that are neither minima nor maxima is perhaps the least understood aspect of the landscape of the loss function for deep nonlinear networks. Very little is currently known about these other critical points of $L$, beyond the expectation that $L$ has nondegenerate saddle points and the fact that the origin is often a degenerate critical point.

We pause here to recall what kinds of critical points a smooth function $L$ can have, and which of those we might expect gradient-based methods to find. Critical points of $L$ fall into two broad categories — nondegenerate critical points, where all the eigenvalues of the Hessian are nonzero, and degenerate critical points, where one or more 