if the distribution $q(u|x)$ matches the true $p(u|x;\theta)$, then $\tilde{\ell}(\theta;x)$ is the genuine log-likelihood function $\ell(\theta;x)$. In this case, the objective function can be optimized via an EM algorithm with a $Q$ function $Q(\theta;\theta_{k})=\mathbb{E}_{U\sim p(u|x;\theta_{k})}[\log p(x|u;\theta)+\log p(u)]$, and we update the current parameter $\theta_{k}$ by a gradient move $$\theta_{k+1}=\theta_{k}+\alpha_{k}\cdot[\partial Q(\theta;\theta_{k})/\partial\theta]|_{\theta=\theta_{k}},$$ with the true expectation replaced by the Langevin-based approximate gradient. The consequence of this refining step is that we are now optimizing the true log-likelihood function $\ell(\theta;x)$ instead of the lower bound $\tilde{\ell}(\theta;x)$, and hence the bias of VAE is removed.

We emphasize that we do not position SAGD as a replacement for VAE; in fact, VAE is computationally more efficient and has a lower variance. Instead, the major virtue of SAGD is its bias-correction capacity that fixes the intrinsic gap between the evidence lower bound of VAE and the true likelihood. Therefore, we suggest using VAE to pre-train models, and then fine-tuning the generative network using SAGD due to its theoretical guarantee.

Numerical Experiments
=====================

EM Algorithm
------------

In this section we use numerical experiments to demonstrate the applications of SAGD in EM algorithm and VAE as discussed in the previous section. First consider a simple model such that 