of $D = 500$ dimensions; (b) a Bayesian linear regression model with standard normal priors on the weights and a Gamma prior on the precision with $N = 50000$ and $D = 500$; and (c) a Gaussian mixture model with the usual conjugate priors, $N = 10000$, $D = 2$ and $K = 25$ clusters. We used fully factorized variational approximations for all models, with $\operatorname*{Softplus}$ transforms for the positively constrained precision parameters and the stick breaking transform for the mixture weights of the last model. For all three choices the I-SGD algorithm with $t=0.9$ converges to the same optimal solution as SGD, but does so in roughly an order of magnitude faster. For fair comparison the size of the mini-batch, the initial learning rate were chosen for each method to work well for SGD, forcing I-SGD to use the same choices. For both algorithms, we used M = 1 sample to estimate the gradients and Adam [@adam] to adaptively control the learning rate during optimization.

We then compare I-SGD and SGD using the score function estimates on a Poisson likelihood with a single Gamma prior on the rate (d), and Bayesian linear regression models with large (e) and small 