We show that natural gradient enables us to use a much larger step size, resulting in an even faster convergence rate.
the maximal step size of natural gradient descent is O (1) for (polynomially) wide networks.