of various components of the critical locus of $L$.'
author:
- 'Y. Cooper'
bibliography:
- 'next\_paper.bib'
title: The critical locus of overparameterized neural networks
---

Introduction
============

The recent and remarkable success of neural networks is not yet well understood from a theoretical perspective. A fruitful area of study has been the “expressivity” of deep neural networks. That is, given a problem or data set, how large does your neural network need to be so that there exists a function in the corresponding parameter space ${\mathcal{P}}$ that perfectly fits the training data set?

However, even if there exist parameters in ${\mathcal{P}}$ that encode functions with zero training loss, it is far from clear if and when gradient based methods might find such parameters. In most cases, the loss function $L$ is believed to be nonconvex, and under various assumptions, “bad” critical points have been proven to exist. In other words, in most cases, we expect there to be many critical points that gradient descent could get stuck at, and still it appears that empirically gradient descent often finds global minima.

In this work, we are motivated by this remarkable fact. Figure \[minima\] illustrates the long term goal — not only 