Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity. Bachman & Precup (2015); Naesseth et al. (2018); Cremer et al. (2017); Domke & Sheldon (2018) show that Importance Weighted Autoencoders (IWAE) (Burda et al., 2015) can be viewed as extending the variational family with auxiliary latent variables. Similarly, we show that this view encompasses many of the recent developments in variational bounds (Maddisonet al., 2017; Naesseth et al., 2018; Le et al., 2017; Yin & Zhou, 2018; Molchanovet al., 2018; Sobolev & Vetrov, 2018). The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model. We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm (Bauer & Mnih, 2018), while being substantially easier to implement. Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation (Jozefowicz et al.,2016; Ma & Collins, 2018) and Contrastive Predictive Coding (Oord et al., 2018). Deep generative models with latent variables have seen a resurgence due to the influential work by BID20 ; BID38 and their success at modeling data such as natural images BID37 BID14 , speech and music time-series BID8 BID13 BID22 , and video BID1 BID15 BID10 . The power of these models lies in the use of auxiliary latent variables to construct complex marginal distributions from tractable conditional distributions. While directly optimizing the marginal likelihood of latent variable models is intractable, we can instead maximize a variational lower bound on the likelihood such as the evidence lower bound (ELBO) BID17 BID5 . The tightness of the bound is determined by the expressiveness of the variational family BID43 .Recently , there have been many advances in constructing tighter variational lower bounds for latent variable models (e.g., BID6 ; BID28 ; BID31 ; BID23 ; BID42 ; BID30 ; BID40 ). Each bound requires a separate derivation and evaluation, however, and the relationship between bounds is unclear.We show that these bounds can be viewed as specific instances of auxiliary variable variational inference BID0 BID36 BID26 . In particular , many partition function estimators can be justified from an auxiliary latent variable or extended state space view (e.g., Sequential Monte Carlo BID12 , Hamiltonian Monte Carlo BID34 , Annealed Importance Sampling BID32 ). Viewed from this perspective, they can be embedded in the variational family as a choice of auxiliary latent variables. Based on the general results for auxiliary latent variables, this immediately gives rise to a variational lower bound with a characterization of the tightness of the bound. Furthermore, this view highlights the implicit (potentially suboptimal) choices made and exposes the reusable components that can be combined to form novel auxiliary latent variable schemes.The success of augmenting variational distributions with auxiliary latent variables motivates investigating a similar augmentation for generative models. When augmenting the variational distribution, the natural target distribution is the intractable posterior over the latent variables in the model. With the generative model, this introduces an extra degree of learnable flexibility (i.e., we can learn the unnormalized potential function). To illustrate this, we develop a latent variable model based on self-normalized importance sampling (Algorithm 1) which can be sampled from exactly and has a tractable lower bound on its log-likelihood. It interpolates between a tractable proposal distribution and an energy model. We show that this model is closely related to ranking NCE BID25 and suggests a principled objective for training the noise distribution in NCE.In summary, our contributions are:1. We view recent tighter variational lower bounds through the lens of auxiliary variable variational inference, unifying their analysis and exposing sub-optimal design choices in algorithms such as IWAE.2. We apply similar ideas to generative models, developing a new model based on selfnormalized importance sampling which can be fit by maximizing a tractable lower bound on its log-likelihood.3. We show that the new model generalizes ranking NCE BID25 and provides a proof that the CPC objective BID35 ) is a lower bound on mutual information.4. We evaluate the proposed model and find it outperforms the recently developed approach in BID4 despite being more computationally efficient and simpler to implement. In this paper, we viewed recent work on improving variational bounds through the lens of auxiliary variable variational inference. This perspective allowed us to expose suboptimal choices in existing algorithms such as IWAE, unify analysis of other methods such as ranking NCE and CPC, and derive new methods for generative modeling such as SNIS. We plan to further develop this view by embedding methods such as Hamiltonian Importance Sampling and Annealed Importance Sampling in generative models which we expect to scale better with dimension of the data space.Published as a workshop paper at ICLR 2019Then, plugging Eqs. (8) and (9) into Eq. (7) with λ = (z 1:K , i) gives log p(x) ≥ E q(z,λ|x) log p(x, z)r(λ|z, x) q(z, λ|x) = E q(λ|x) log p(x, z i )
Monte Carlo Objectives are analyzed using auxiliary variable variational inference, yielding a new analysis of CPC and NCE as well as a new generative model. Proposes a different view on improving variational bounds with auxiliary latent variable models and explores the use of those models in the generative model.