advantages in our problem. First, the Langevin algorithm skips the Metropolis adjustment step existing in most MCMC methods, which saves computational time and avoids duplicated values in the sample. Second, as our theoretical analysis shows, the Langevin algorithm has transparent hyperparameter setting and requires less manual intervention. The downside is the resulting bias of the Langevin algorithm, but the theoretical analysis shows that it does not harm the convergence of SAGD.

The idea to combine SGD with the Langevin algorithm has been seen in articles such as @xie2018cooperative and @han2019divergence, but in these works the Langevin algorithm was merely used as an MCMC-like sampling technique, and its statistical error and impact on the convergence of optimization were ignored. Instead, in SAGD the two ingredients are connected in a coherent way, with a rigorous theoretical analysis. One recent work that is similar to SAGD is @de2019efficient, but the major difference is that they used the overdamped Langevin algorithm for sampling, whose theoretical analysis is very different from SAGD. Another direction of research that combines SGD and the Langevin algorithm is the stochastic gradient Langevin dynamics (SGLD, [@welling2011bayesian]; [@vollmer2016exploration]). However, SGLD utilizes SGD to accelerate the Langevin sampling method, while our work 