SGD has a slow asymptotic rate of convergence due to its inherent variance [@nesterov2013introductory]. Variants of SGD that can reduce the variance of the stochastic gradient estimator, which yield faster convergence, have also been suggested. The use of the information of full gradients to provide variance control for stochastic gradients is addressed in [@johnson2013accelerating; @roux2012stochastic; @shalev2013stochastic]. The works in [@lei2017non; @li2014efficient; @schmidt2017minimizing] further improve the efficiency and complexity of the algorithm by carefully controling the variance.

There is prior work focusing on studying the dynamics of SGD. Neelakantan et. al. [@neelakantan2015adding] propose to add isotropic white noise to the full gradient to study the “structured” variance. The works in [@pmlr-v70-li17f; @mandt2017stochastic; @jastrzkebski2017three] connect SGD with stochastic differential equations to explain the property of converged minima and generalization ability of the model. Smith and Le [@smith2017bayesian] propose an “optimal” mini-batch size which maximizes the test set accuracy by a Bayesian approach. The Stochastic Gradient Langevin Dynamics (SGLD, a variant of SGD) algorithm for non-convex optimization is studied in [@zhang2017hitting; @mou2018generalization].

In most of the prior work about the convergence of SGD, it is assumed that the variance of stochastic gradient estimators is upper-bounded by a linear function of the norm of 