Title: Towards a Complete Theory of Neural Networks with Few Neurons. Abstract: Deep learning has seen unprecedented progress thanks to the deployment of models with millions of parameters. 
On the theoretical side, an immense amount of effort has gone to understanding the dynamics of overparameterized networks. 
Although now there is a well-developed theory of networks with infinitely many neurons, the classic problem of understanding how a neural network with a few neurons learns remains unsolved.
To attack this problem, we analytically study the landscapes of neural networks with few neurons. 
We prove for the first time that a student network with one neuron has only one critical point --its global minimum-- when learning from a teacher network with arbitrarily many orthogonal neurons. 
In addition, we prove how a neuron addition mechanism turns a minimum into a line of critical points with transitions from saddles to local minima via non-strict saddles. 
Finally, we discuss how the insights we get from our novel proof techniques may shed light on the dynamics of neural networks with few neurons.