sampling algorithm, which are crucial for practical algorithm implementation but are typically missing in other MCMC-based methods. Of course, the preservation of the SGD rate is not without a price. Theorem \[thm:convergence\_convex\] indicates that the number of inner iterations, *i.e.*, $K_{t}$ in Algorithm \[alg:sagd\], needs to increase with $t$. However, the developed error bounds are typically conservative, so for practical use, we advocate the following techniques to speed up SAGD: (1) An educated initial value $\xi_{0}$ can be used to initialize the Langevin algorithm, for example in VAE $\xi_{0}$ is sampled from the trained encoder; (2) A persistent Langevin Markov chain is stored during optimization, motivated by the persistent contrastive divergence [@tieleman2008training]; (3) Some advanced gradient update schemes such as Adam can be used.

More generally, we consider objective functions that are nonconvex but smooth, and assume that $\Theta=\mathbb{R}^{p}$. Theorem \[thm:convergence\_nonconvex\] indicates that with a proper choice of hyperparameters, the algorithm again has a nice convergence property.

\[thm:convergence\_nonconvex\]Suppose that $g(\theta)$ is $G$-Lipschitz continuous in $\theta$, and assume that Assumption \[assu:f\_condition\] and the conditions in Theorem \[thm:finite\_moments\] hold. Let $\delta_{t}=C_{1}t^{-c}$, $K_{t}=C_{2}t^{2c}$, and $\alpha_{t}=\alpha_{0}/t$ for some constants $0<\alpha_{0}<1/(2G)$ and $C_{1},C_{2},c>0$. Then we have $\lim\inf_{t\rightarrow\infty}\mathbb{E}[\Vert g(\theta_{t})\Vert^{2}]=0$.

Theorem \[thm:convergence\_nonconvex\] is an analog to Theorem 