---
abstract: 'Approximate inference in complex probabilistic models such as deep Gaussian processes requires the optimisation of doubly stochastic objective functions. These objectives incorporate randomness both from mini-batch subsampling of the data and from Monte Carlo estimation of expectations. If the gradient variance is high, the stochastic optimisation problem becomes difficult with a slow rate of convergence. Control variates can be used to reduce the variance, but past approaches do not take into account how mini-batch stochasticity affects sampling stochasticity, resulting in sub-optimal variance reduction. We propose a new approach in which we use a recognition network to cheaply approximate the optimal control variate for each mini-batch, with no additional model gradient computations. We illustrate the properties of this proposal and test its performance on logistic regression and deep Gaussian processes.'
author:
- |
    [**Ayman Boustati [^1]**]{}\
    University of Warwick\
    Coventry, UK\
    Prowler.io\
    Cambridge, UK\
    Prowler.io\
    Cambridge, UK\
    Prowler.io\
    Cambridge, UK
bibliography:
- 'references.bib'
title: Amortized variance reduction for doubly stochastic objectives
---

CONCLUSIONS
===========

We introduced a control variate formulation that exploits the structure of doubly stochastic objectives to remove Monte Carlo sampling variance from mini-batch gradient estimators. We proposed three objectives for an amortising recognition network that can learn context aware control variate coefficients. Training the network re-uses the gradients of the model objective and does not require additional passes through the model.

Empirical assessment showed that an approximation to the optimal control variate per mini-batch can be performed during optimisation and reduces the gradient variance in practice compared to a context-free global approach. In our experiments we used linear and quadratic control variates for Gaussian base randomness, but our approach is general and can be applied to other control variate formulae and randomness schemes.

Supplementary Material

[^1]:   Work done while at Prowler.io.
