as Entropy-SGD [@Chaudhari2017EntropySGD]. In our approach, we do not attempt to measure geometric property of loss landscape such as flatness with extra computational cost, but instead consider explicit regularization to model parameters.

[**Variance Reduction:**]{} The variance of stochastic gradients is detrimental to SGD, motivating variance reduction techniques [@Roux2012; @johnson2013accelerating; @Chatterji2018OnTT; @zhong2014fast; @shen2016adaptive; @Difan2018svrHMM; @Zhou2019ASim] that aim to reduce the variance incurred due to their stochastic process of estimation, and improve the convergence rate mainly for convex optimization while some are extended to non-convex problems [@allen2016variance; @huo2017asynchronous; @liu2018zeroth]. One of the most practical algorithms for better convergence rates includes momentum [@sutton1986two], modified momentum for accelerated gradient [@nesterov1983method], and stochastic estimation of accelerated gradient (Accelerated-SGD) [@Kidambi2018Acc]. These algorithms are more focused on the efficiency in convergence than the generalization of model for accuracy.

[**Weight-Decay:**]{} is an explicit way of regularization such that a regularization term is added into the energy function. Specifically $L^2$-norm is used as the regularization term in order to penalize large weight values. Different with the other implicit methods, e.g., stochastic update and dropout, one can directly control the regularization effect by the weight-decay coefficient. The weight-decay coefficient is tuned by hand [@simonyan2014very; @chollet2017xception], or learned by Bayesian optimization 