estimate is more than enough to overcome this.

Stochastic running average (SRA) provides another baseline that down-weights older mini-batches exponentially. Similar to I-SAG, it avoids using mini-batches with badly outdated gradient estimates, by using a simple weighting scheme without transforming the gradients. It outperforms SAG, but converges more slowly than I-SAG. Hence, I-SAG is stable implementation of SAG for variational inference, outperforming the alternative of running averages often considered as a remedy for the issues of SAG.

DISCUSSION
==========

Automatic variational inference using automatically differentiated gradients has in recent years become a feasible technique for inference for a wide class of probabilistic models, extending the scope of variational approximations beyond simple conjugate models towards practical probabilistic inference engines. While standard computational platforms and advances in convex optimization are readily applicable for gradient-based variational inference, the need to use Monte Carlo approximation to estimate the gradients necessarily induces a computational overhead – with very few samples the gradients are noisy whereas the cost grows linearly as a function of the samples.

Our work addressed this central element, discussing ways to speed up the gradient-based inference of variational approximations. By highlighting how the gradient computation separates into two steps we derived an 