but for differentiable models the reparameterization technique is typically considerably more efficient [@naesseth; @grep].

STOCHASTIC GRADIENT OPTIMIZATION
--------------------------------

Given estimates for the gradient computed with either method, the optimization problem[^1] is solved by standard gradient descent techniques. In practice all of the automatic variational inference papers have resorted to stochastic gradient descent (SGD) on mini-batches, adaptively tuning the step lengths with the state-of-the-art techniques.

In recent years several more advanced stochastic optimization algorithms have been proposed, such as stochastic average gradients (SAG) [@sage], stochastic variance reduced gradients (SVRG) [@svrg], and SAGA that combines elements of both [@saga]. However, to our knowledge these techniques have not been successfully adapted for automatic variational inference. In Section \[sec:isag\] we will present a new variant of SAG that works also when the gradients are estimated as Monte Carlo approximations, and therefore briefly describe below the basic idea behind stochastic average gradients.

SAG performs gradient updates based on an estimate for the full batch gradient, obtained by summing up gradients stored for individual data points (or for mini-batches to save memory). Whenever a data point is seen again during the optimization the stored gradient for that point is replaced by the gradient evaluated at the 