Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution. Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint. In this study, we prove the local stability of optimizing the simple gradient penalty $\mu$-WGAN(SGP $\mu$-WGAN) under suitable assumptions regarding the equilibrium and penalty measure $\mu$. The measure valued differentiation concept is employed to deal with the derivative of the penalty terms, which is helpful for handling abstract singular measures with lower dimensional support. Based on this analysis, we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty. Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results. Deep generative models reached a turning point after generative adversarial networks (GANs) were proposed by BID2 . GANs are capable of modeling data with complex structures. For example, DCGAN can sample realistic images using a convolutional neural network (CNN) structure BID12 . GANs have been implemented in many applications in the field of computer vision with good results, such as super-resolution, image translation, and text-to-image generation BID7 BID6 Zhang et al., 2017; BID13 .However , despite these successes, GANs are affected by training instability and mode collapse problems. GANs often fail to converge, which can result in unrealistic fake samples. Furthermore , even if GANs successfully synthesize realistic data, the fake samples exhibit little variability.A common solution to this instability problem is injecting an instance noise and finding different divergences. The injection of instance noise into real and fake samples during the training procedure was proposed by Sønderby et al. (2017) , where its positive impact on the low dimensional support for the data distribution was shown to be a regularizing factor based on the Wasserstein distance, as demonstrated analytically by . In f -GAN, f -divergence between the target and generator distributions was suggested which generalizes the divergence between two distributions BID11 . In addition, a gradient penalty term which is related with Sobolev IPM(Integral Probability Metric) between data distribution and sample distribution was suggested by BID9 .The Wasserstein GAN (WGAN) is known to resolve the problems of generic GANs by selecting the Wasserstein distance as the divergence . However, WGAN often fails with simple examples because the Lipschitz constraint on discriminator is rarely achieved during the optimization process and weight clipping. Thus, mimicking the Lipschitz constraint on the discriminator by using a gradient penalty was proposed by BID3 .Noise injection and regularizing with a gradient penalty appear to be equivalent. The addition of instance noise in f -GAN can be approximated to adding a zero centered gradient penalty BID14 . Thus, regularizing GAN with a simple gradient penalty term was suggested by BID8 who provided a proof of its stability.Based on a theoretical analysis of the dynamic system, BID10 proved the local exponential stability of the gradient-based optimization dynamics in GANs by treating the simultaneous gradient descent algorithm with a dynamic system approach. These previous studies were useful because they showed that the local behavior of GANs can be explained using dynamic system tools and the related Jacobian's eigenvalues.In this study, we aim to prove the convergence property of the simple gradient penalty µ-Wasserstein GAN(SGP µ-WGAN) dynamic system under general gradient penalty measures µ. To the best of our knowledge , our study is the first theoretical approach to GAN stability analysis which deals with abstract singular penalty measure. In addition, measure valued differentiation BID4 ) is applied to take the derivative on the integral with a parametric measure, which is helpful for handling an abstract measure and its integral in our proof.The main contributions of this study are as follows.• We prove the regularized effect and local stability of the dynamic system for a general penalty measure under suitable assumptions. The assumptions are written as both a tractable strong version and intractable weak version. To prove the main theorem, we also introduce the measure valued differentiation concept to handle the parametric measure.• Based on the proof of the stability , we explain the reason for the success of previous penalty measures. We claim that the support of a penalty measure will be strongly related to the stability, where the weight on the limiting penalty measure might affect the speed of convergence.• We experimentally examined the general convergence results by applying two test penalty measures to several examples. The proposed test measures are unintuitive but they still satisfy the assumptions and similar convergence results were obtained in the experiment. In this study, we proved the local stability of simple gradient penalty µ-WGAN optimization for a general class of finite measure µ. This proof provides insight into the success of regularization with previously proposed penalty measures. We explored previously proposed analyses based on various gradient penalty methods. Furthermore, our theoretical approach was supported by experiments using unintuitive penalty measures. In future research, our works can be extended to alternative gradient descent algorithm and its related optimal hyperparameters. Stability at non-realizable equilibrium points is one of the important topics on stability of GANs. Optimal penalty measure for achieving the best convergence speed can be also investigated using a spectral theory, which provides the mathematical analysis on stability of GAN with a precise information on the convergence theory.
This paper deals with stability of simple gradient penalty $\mu$-WGAN optimization by introducing a concept of measure valued differentiation. WGAN with a squared zero centered gradient penalty term w.r.t. to a general measure is studied. Characterizes the convergence of gradient penalized Wasserstein GAN.