global minima, star locus, and core locus, are: $$\begin{aligned}
\dim(M) &= (\ell-2)m^2 + (a+b+\ell-1)m + b(1-n),
\\
\dim(S) &=  (\ell-3)m^2 + (a+1)m,
\\
\dim(C) &=  (\ell-4)m^2 + (a+1)m,\end{aligned}$$ while the dimension of the locus of all critical points is unknown.

Meanwhile,

[**Proposition \[dimzeroeigen\].**]{} Consider a family of feedforward neural networks with hidden layers of increasing width training on a fixed data set. That is, let $a, b,$ and $n$ be fixed while $m$ increases. Then the number of zero eigenvalues of the Hessian of $L$ at any critical point in the core locus, locus of global minima, or any critical point are: $$\begin{aligned}
\begin{cases}
d-1 & \text{for } p \in C,
\\
d-bn & \text{for } p \in M,
\\
\text{at least } (m+1- n) b & \text{for all other critical points } p.
\end{cases} \end{aligned}$$

Outline of paper
----------------

We begin, in Section \[setting\], by establishing the setting in which we will be working. We then study the geometry of $L$ for neural networks of various sizes, starting with underparameterized networks and then working with increasingly overparameterized ones.

In Section \[deepnetwork\] we consider all neural networks of depth $\ell \geq 3$, whether overparameterized or not. In this setting, we 