4.9 of @bottou2018optimization. It is not meant to be the strongest conclusion, but to provide insights on the convergence property of SAGD for nonconvex objective functions.

Applications: EM Algorithm and VAE {#sec:application}
==================================

Automated EM Algorithm
----------------------

The SAGD framework is very useful for implementing an automated version of the EM algorithm [@dempster1977maximum]. EM algorithm is a powerful and indispensable tool to solve missing data problems and latent variable models. Given the data set $X$ that follows a probability distribution with density function $f(x;\theta)$, we are interested in computing the maximum likelihood estimator $\hat{\theta}=\arg\max_{\theta}\,\ell(\theta;x)$ for the unknown parameter vector $\theta$, where $\ell(\theta;x)=\log[f(x;\theta)]$ is the log-likelihood function.

However, in many cases the computation of the marginal distribution $f(x;\theta)$ is intractable, but with an additional random vector $U$, the complete log-likelihood $L(\theta;x,u)=\log[f(x,u;\theta)]$ is simple. This phenomenon typically happens when $U$ represents missing data or latent variables in the model. The EM algorithm computes $\hat{\theta}$ in an iterative way. Given the current value of $\theta$, denoted by $\theta_{k}$, the EM algorithm proceeds by the following two steps:

-   **Expectation Step (E-step)**: Compute the expected value of $L(\theta;x,U)$ with respect to the conditional distribution of $U$ given $X=x$ under the current parameter estimate $\theta_{k}$, 