used in practice. This regime is unrealistic for real world neural networks, but on the other hand, this is the setting in which we have the best understanding of the geometry of $L$ as well as the dynamics of gradient descent on $L$. So this case is also often of interest.

Deep neural networks {#deepnetwork}
====================

In this section, we establishing some basic facts about the loss function $L$ that hold for all deep neural networks, regardless of whether they are overparameterized or not.

Star locus
----------

For any fully connected feedforward network with $L2$ loss and smooth activation function satisfying $\sigma(0) = 0$, the space of all parameters ${\mathbb R}^d$ contains a positive dimensional locus of critical points we call the star locus, which contains a sublocus of degenerate critical points we call the core. In this section we will identify these loci and prove some properties about them.

We begin by recalling some definitions. Given a twice differentiable function $f: {\mathbb R}^d \rightarrow {\mathbb R}$, $p \in {\mathbb R}^d$ is a critical point if $$\left(\frac{\partial}{\partial z_i} f \right)(p) = 0$$ for all $i$.

The Hessian of $f$ is defined as the matrix $$Hess(f) =
\begin{pmatrix}
\frac{\partial^2}{\partial z_1^2} f 