diverse responses can be generated by sampling the variable. However, both VAE and CVAE have encountered the KL-vanishing problem that the decoder tends to model the targets without making use of the latent variables.

In this paper, we point out that during optimizing the objective of CVAE, the encoder is gradually pulled to a prior distribution and losing discriminative ability of the targets, while the decoder tends to fit the data even without the help of encoders. Thus KL-vanishing is rooted in the objective of CVAE. Current approaches, either weakening the decoder or strengthening the encoder to make compensation to the objective implicitly in advance, only mitigate this problem and are hard to determine how weak/strong should decoder/encoder be. Orthogonal to these efforts, we propose an explicit optimization objective for the encoder to move towards better expressiveness to fit current decoder. With this novel objective, the latent variable distribution from the encoder has the potential to be appropriately flexible in correspondence with decoder, which naturally coordinates the representative abilities of the encoder and decoder and enhances the utilization of latent variable by decoders. Specifically, an additional module called “labeling network” is used to estimate the “best encoder” for the current 