would we like to identify all the critical points of $L$, but we’d like to understand the geometry of the set $Crit$ of all critical points of $L$. For example, it would be valuable to understand how many components $Crit$ has, what the dimension of each component is, and what the local geometry of $L$ is near each component of $Crit$. In this paper, we make some of the first steps toward this goal, establishing fundamental facts about several components of the locus of critical points of $L$.

\[minima\]

![The locus of critical points of this function has two components. There’s one isolated local minimum, and then there’s a one dimensional smooth manifold of global minima.](minima_redux.pdf){width="4.2in"}

Previous results
----------------

Not much is currently known about the critical locus of the loss function under assumptions that hold for real-world neural networks. There is a substantial literature on the geometry of the loss function under strong assumptions that rule out many or all real-world networks, for example, that the activation function is the identity, or that the network is unrealistically wide compared to the number of training samples.

We begin by recalling some results about the critical points of $L$ that hold 