for real variational inference tasks on a range of alternative models and settings.

DIMENSIONALITY OF THE APPROXIMATION {#sec:dimensionality}
-----------------------------------

Figure \[fig:weight-decay\] studies importance weights of approximations factorized at different granularities on a 100-dimensional diagonal multivariate Gaussian. An important observation is that even if importance sampling itself fails for factors of high dimensionality, the I-SGD algorithm degrades gracefully. For low-dimensional factors, up to at least 5-10 dimensions, we can safely take 5-10 steps with each mini-batch while still having accurate gradient estimates. When the dimensionality of individual factors reaches around $25$ the weights tend to zero already after a single gradient step, but the algorithm does not break down. It merely needs to proceed immediately to the next step, reverting back to standard SGD.

[.28]{}

[.28]{}

[.28]{}

[.28]{}

[.28]{}

[.28]{}

IMPORTANCE SAMPLED SGD {#sec:impsamsgd}
----------------------

We show in Figure \[fig:experiments\] how I-SGD consistently bests SGD for a variety of different models, especially for the reparameterization estimate. For these experiments we use fully factorized mean-field approximation,

We first apply the reparameterization estimate for three probabilistic models: (a) a diagonal multivariate Gaussian with a standard normal prior on the mean and Gamma priors on the precisions, trained on $N =
50000$ data points 