identify two families of critical points, the star locus $S$, and the core locus $C$. When$\ell \geq 3$ the star locus is positive dimensional, and once $\ell \geq 4$, the core locus is as well. The core locus consists of critical points with nearly vanishing Hessian — all but one eigenvalue is zero, and the one nonzero eigenvalue is positive. Hence every point in this positive dimensional locus of critical points fails the assumptions of [@jordansaddle], and we cannot from that work conclude that gradient descent will not converge to them.

Next, in Section \[overparam\] we consider all neural networks with more parameters $d$ than data points $n$, and recall from earlier work that the locus of global minima forms a smooth possibly disconnected manifold of dimension $d-bn$.

Finally in Section \[verywide\], we consider neural networks which are not only overparameterized, but where every layer has width greater than $n$. This setting has been of substantial recent theoretical interest, and here we show that there are no isolated local minima or maxima. We go on to give a lower bound on the number of zero eigenvalues of the Hessian at any local minimum or maximum. For saddle points, we give 