eigenvalues of the Hessian vanishes. While nondegenerate critical points are easily classified by the number of positive and negative eigenvalues of the Hessian, there is a rich zoo of degenerate critical points.

As to which critical points gradient-based methods might converge to, one does not expect gradient descent from a random initialization to end at a local maximum, and one does expect sometimes to end at a local minimum. The most subtle are saddle points, especially degenerate ones.

In [@jordansaddle], Jordan et. al. showed that gradient-based methods can efficiently escape any saddle point $p$ where at least one eigenvalue of the Hessian is negative. However, beyond that, there is little understanding of which kinds of degenerate critical points gradient based methods can efficiently escape and which they cannot. In this paper, we show that for any deep nonlinear network, overparameterized or not, $L$ has a positive dimensional locus of degenerate critical points which do not satisfy the assumptions of [@jordansaddle], and which to our knowledge no existing results guarantee gradient descent won’t get stuck at.

Given the fact then that $L$ contains degenerate critical points with geometries that haven’t been studied from the point of view of gradient descent, a 