a lower bound on the dimension of certain linear subspaces that contain the saddle point and are contained in the level set containing the saddle point.

We conclude in Section \[discussion\] with a comparison, for networks in which the kinds of critical points discussed in this paper all appear simultaneously, of the relative dimensions of each type, and how they grow with the number of parameters in the network.

Acknowledgements
----------------

We thank Misha Belkin, Nate Bottman, Rong Ge, Felix Janda, Chi Jin, Holden Lee, and Ruoyu Sun for helpful discussions. We thank Quynh Ngyuen for valuable feedback on an early draft of this work.

Setting and notation {#setting}
====================

In this paper, we will consider the following setting. Consider a fully connected feedforward neural network with $L2$ loss, and a monotonically increasing activation function $\sigma$. Further, assume that $\sigma(0)=0$ and $\sigma$ is smooth, meaning it is infinitely differentiable.

Suppose the neural network is training on a data set $D$ consisting of input output pairs $(x_k, y_k)$, $x_k \in {\mathbb R}^a$, $y_k \in {\mathbb R}^b$, and suppose there are $n$ data points in the training set.

We assume that the neural network has $\ell-1$ hidden layers and each layer has 