We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative  Adversarial  Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task. The objective of conditional generative models is learning a mapping function from input to output distributions. Since many conditional distributions are inherently ambiguous (e.g. predicting the future of a video from past observations), the ideal generative model should be able to learn a multi-modal mapping from inputs to outputs. Recently, Conditional Generative Adversarial Networks (cGAN) have been successfully applied to a wide-range of conditional generation tasks, such as image-to-image translation (Isola et al., 2017; Wang et al., 2018; Zhu et al., 2017a) , image inpainting (Pathak et al., 2016; Iizuka et al., 2017 ), text-to-image synthesis (Huang et al., 2017; Hong et al., 2018) , video generation (Villegas et al., 2017) , etc.. In conditional GAN, the generator learns a deterministic mapping from input to output distributions, where the multi-modal nature of the mapping is handled by sampling random latent codes from a prior distribution.However, it has been widely observed that conditional GANs are often suffered from the mode collapse problem (Salimans et al., 2016; , where only small subsets of output distribution are represented by the generator. The problem is especially prevalent for highdimensional input and output, such as images and videos, since the model is likely to observe only one example of input and output pair during training. To resolve such issue, there has been recent attempts to learn multi-modal mapping in conditional generative models (Zhu et al., 2017b; Huang et al., 2018) . However, they are focused on specific conditional generation tasks (e.g. image-toimage translation) and require specific network architectures and objective functions that sometimes are not easy to incorporate into the existing conditional GANs.In this work, we introduce a simple method to regularize the generator in conditional GAN to resolve the mode-collapse problem. Our method is motivated from an observation that the mode-collapse happens when the generator maps a large portion of latent codes to similar outputs. To avoid this, we propose to encourage the generator to produce different outputs depending on the latent code, so as to learn a one-to-one mapping from the latent codes to outputs instead of many-to-one. Despite the simplicity, we show that the proposed method is widely applicable to various cGAN architectures and tasks, and outperforms more complicated methods proposed to achieve multi-modal conditional generation for specific tasks. Additionally, we show that we can control a balance between visual quality and diversity of generator outputs with the proposed formulation. We demonstrate the effectiveness of the proposed method in three representative conditional generation tasks, where most existing cGAN approaches produces deterministic outputs: Image-to-image translation, image inpainting and video prediction. We show that simple addition of the proposed regularization to the existing cGAN models effectively induces stochasticity from the generator outputs. In this paper, we investigate a way to resolve a mode-collapsing in conditional GAN by regularizing generator. The proposed regularization is simple, general, and can be easily integrated into existing conditional GANs with broad classes of loss function, network architecture, and data modality. We apply our regularization for three conditional generation tasks and show that simple addition of our regularization to existing cGAN objective effectively induces the diversity. We believe that achieving an appropriate balance between realism and diversity by learning λ and τ such that the learned distribution matches an actual data distribution would be an interesting future work.
We propose a simple and general approach that avoids a mode collapse problem in various conditional GANs. The paper proposes a regularization term for the conditional GAN objective in order to promote diverse multimodal generation and prevent mode collapse. The paper proposes a method for generating diverse outputs for various conditional GAN frameworks including image-to-image translation, image-inpainting, and video prediction, which can be applied to various conditional synthesis frameworks for various tasks. 