$i^{th}$ layer. Then $d_i = m_i (m_{i-1}+1)$ and $$\begin{aligned}
{\mathbb R}^d = {\mathbb R}^{d_1} \times ... \times {\mathbb R}^{d_\ell}.\end{aligned}$$ Given a parameter vector $p \in {\mathbb R}^d$, we can decompose it as $$\begin{aligned}
p = (p_1, ..., p_\ell).\end{aligned}$$

The loss function $L$ is a function from ${\mathbb R}^d$ to ${\mathbb R}$ defined as $$L(p) = \sum_{i = 1}^{n} (f_{p}(x_i) - y_i)^2,$$ where $p$ is a parameter vector and $f_p$ is the function computed by the neural network with that choice of parameters.

Our interest is in understanding the geometry of the loss function $L$ of overparameterized neural networks, as the majority of modern neural networks deployed in practice today are overparameterized.

Overparameterized regimes
-------------------------

We now outline three regimes of overparameterization. The more overparameterized the neural network, the better our current understanding of the geometry of $L$.

### Mildly overparameterized regime

The most realistic and hence most interesting case is when the neural network is overparameterized in the simplest sense, that the number of parameters $d$ of the network is greater than the number of training points $n$, but not excessively so, i.e. that $d = O(n)$. In this case, the width $m$ of the network is order $\sqrt{n}$. We 