the importance sampling changes the weight of the gradient, decreasing it towards zero for the mini-batches evaluated under clearly different parameter settings, and transforms the gradient to better match one that would have been calculated under the current approximation.

This algorithm provides a justified version of SAG for automatic variational inference. The computational cost is higher than for standard SAG since we need to evaluate the importance weights and compute the terms related to the gradient of the transformation for all past mini-batches. There is, however, no additional memory overhead and the amount of evaluations for the gradient of the model itself is the same. This overhead for importance sampling the gradients for other batches is not negligible, but usually still small enough that the resulting algorithm outperforms a naive implementation of SAG because of vastly more accurate gradient estimates, as shown in Section \[sec:isage\]. In case updating the past gradients becomes too costly, a simple remedy is to use only the latest $K$ mini-batches for some reasonable choice of $K$.

EXPERIMENTS
===========

In this section we first demonstrate how the behavior of importance sampling depends on the dimensionality of the approximation. We then empirically compared both I-SGD and I-SAG 