A long-held conventional wisdom states that larger models train more slowly when using gradient descent. This work challenges this widely-held belief, showing that larger models can potentially train faster despite the increasing computational requirements of each training step. In particular, we study the effect of network structure (depth and width) on halting time and show that larger models---wider models in particular---take fewer training steps to converge.

 We design simple experiments to quantitatively characterize the effect of overparametrization on weight space traversal. Results show that halting time improves when growing model's width for three different applications, and the improvement comes from each factor: The distance from initialized weights to converged weights shrinks with a power-law-like relationship, the average step size grows with a power-law-like relationship, and gradient vectors become more aligned with each other during traversal.

Empirically shows that larger models train in fewer training steps, because all factors in weight space traversal improve. This paper shows that wider RNNs improve convergence speed when applied to NLP problems, and by extension the effect of increasing the widths in deep neural networks on the convergence of optimization This paper characterizes the impact of over-parametrization in the number of iterations it takes an algorithm to converge, and presents further empirical observations on the effects of over-parametrization in neural network training.