positive-dimensional locus of degenerate critical points.

[**Theorem \[corelocus\]**]{} Every point in $C$ is a degenerate critical point. Furthermore, for any point $p$ in $C$, the Hessian of $L$ at $p$ has one positive eigenvalue and the remaining eigenvalues all vanish.

We go on to compute the dimension of both loci $S$ and $C$, and establish some basic properties of each.

In practice, modern neural networks are essentially always overparameterized, in the sense that the number of parameters $d$ is larger than the number of data points $n$ that the network is being trained on. In this setting, in previous work [@cooper] we showed that the locus $M$ of global minima is generically a smooth manifold of dimension $d-bn$.

Little more is known about neural networks which are overparameterized but to an extent comparable to the extent that networks used in practice are overparameterized. However, there has been substantial recent progress in the theoretical understanding of networks much wider than those used in practice. We call networks where each layer has width $m > n$ very wide, and those where the width $m$ of each layer grows as a polynomial in $n$ extremely wide.

Our best theoretical understanding is for extremely 