the *KL annealing* method (KLA) and *word-dropout* operation (WD) in VAE training to mitigate the *KL-Vanishing problem* in their sentence generating system. And [@zhao2017learning] introduced an additional *bag-of-word loss* which takes the latent variable as input and predicts the words which will appear in the target so that the connection of the latent variable and the outputs are augmented. They applied their *bag-of-word* loss in a CVAE structure for one-to-many text generation tasks and get excellent results in open domain dialogue generation of discourse-level diversity.

The Proposed Method
===================

Analysis of the KL-Vanishing Problem
------------------------------------

Considering VAE’s objective function Equation. \[vae\_elbo\], two facts are observed: (1) The second term $KL(q(z|x)||p(z))$ reaches its global minimum of 0 when $q(z|x)=p(z)$. (2) According to Jensen’s Inequality, $E_{p(z)}[\log{p(x|z)}]\leq\log\sum_{z}[p(x|z)p(z)]=\log{p(x)}$, and the equal sign of the inequation holds when and only when $x$ is independent of $z$. As a consequence, when $x$ and $z$ are independent, the ELBO objective degenerates to original $\log p(x)$ objective under which the decoder learns a plain language model. Thus the encoder $q(z|x) = p(z)$ and decoder $p(x|z) = p(x)$ which fits the dataset as a plain language model constitute a trivial solution of the objective of VAE. At this time, 