Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that encourages a perfect reconstruction by tying a stochastic autoencoder with a variational autoencoder (VAE). This ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, although our model is fundamentally different to a VAE, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.
This paper introduces a novel generative modelling framework that avoids latent-variable collapse and clarifies the use of certain ad-hoc factors in training Variational Autoencoders. The paper proposes to resolve the issue about a variational auto-encoder ignoring the latent variables. This paper proposes adding a stochastic autoencoder to the original VAE model to address the problem that the LSTM decoder of a language model might be too strong to ignore the latent variable's information. This paper presents AutoGen, which combines a generative variational autoencoder with a high-fidelity reconstruction model based on autoencoder to better utiliza latent representation