We introduce a new method for training GANs by applying the Wasserstein-2 metric proximal on the generators. 
 The approach is based on the gradient operator induced by optimal transport, which connects the geometry of sample space and parameter space in implicit deep generative models. From this theory, we obtain an easy-to-implement regularizer for the parameter updates. Our experiments demonstrate that this method improves the speed and stability in training GANs in terms of wall-clock time and Fr\'echet Inception Distance (FID) learning curves. Generative Adversarial Networks (GANs) BID11 are a powerful approach to learning generative models. Here, a discriminator tries to tell apart the data generated from a real source and the data generated by a generator, whereas the generator tries to fool the discriminator. This adversarial game is formulated as an optimization problem over an implicit generative model for the generator. An implicit generative model is a parametrized family of functions mapping a noise source to sample space. In trying to fool the discriminator, the generator should try to recreate the density distribution from the real source. The problem of matching a target density can be formulated as the minimization of a discrepancy measure. The Kullback-Leibler (KL) divergence is known to be difficult when the distributions have a low dimensional support set, as is commonly the case in applications with structured data and high dimensional sample spaces. An alternative approach to define a discrepancy measure between densities is optimal transport, a.k.a. Wasserstein distance, or Earth Mover's distance. This has been used recently to define the loss function for learning generative models BID30 BID10 . In particular, the Wasserstein GAN BID4 has attracted much interest in recent years. Besides defining the loss function, optimal transport can also be used to introduce structures serving the optimization itself, in terms of the gradient operator. In full probability space, this is known as the Wasserstein steepest descent flow BID15 BID32 . In this paper we derive the Wasserstein steepest descent flow for deep generative models in GANs. We use the Wasserstein-2 metric function, which allows us to obtain a Riemannian structure and a corresponding natural (i.e., Riemannian) gradient. A well known example of a natural gradient is the Fisher-Rao natural gradient, which is induced by the KL divergence. In learning problems, one often finds that the natural gradients can offer advantages compared to the Euclidean gradient BID1 BID2 . In GANs, because of the low dimensional support sets and the associated difficulties with the KL divergence, the Fisher-Rao natural gradient is problematic. Therefore, we propose to use the gradient operator induced by the Wasserstein-2 metric BID21 b) .We compute the proximal operator for the generators of GANs, where the regularization is the squared constrained Wasserstein-2 distance. In practice, the constrained distance can be approximated by a simple neural network. In implicit generative models, the constrained Wasserstein-2 metric exhibits a simple structure. We generalize the metric and introduce the relaxed proximal operator for generators, which allows us to further simplify the computation. The resulting relaxed proximal operator involves only the difference of outputs, so that the proximal computation has very simple parameter updates. The method can be easily implemented and used as a drop-in regularizer for the generator updates. This paper is organized as follows. In Section 2, we briefly introduce the Wasserstein natural gradient. A Wasserstein proximal method is introduced in Algorithm 1. In Section 3, we demonstrate the effectiveness of the proposed methods in experiments with various types of GANs. Section 4 reviews related work. In this section, we briefly present optimal transport and its proximal operator on a parameter space. We then apply them to the optimization problems of GANs. Optimal transportation defines a class of distance functions between probability densities. Given a pair ρ 0 , ρ 1 ∈ P p (R n ) of probability densities with finite p-th moment, DISPLAYFORM0 where the infimum is over all joint probability densities π(x, y) with marginals ρ 0 (x), ρ 1 (y). In the literature (see BID36 ), W p is referred to as the Wasserstein-p distance. In this paper, we focus on the case p = 2, and further denote W 2 by W .Following BID6 , the Wasserstein-2 distance has a dynamical formulation as a trajectory transporting the initial density ρ 0 to the final density ρ 1 along a trajectory of minimal kinetic energy. The classic theory does not consider the setting where the density path is constrained to lie within a parametrized model. In the following we extend the classic theory to cover parameterized density models. Consider a parameterized probability ρ(θ, x), with parameter space Θ ⊂ R d . Suppose that ρ(θ, x) is locally injective as a mapping from Θ to P 2 (R n ). Then the Wasserstein-2 metric function constrained to the parameter space is given as follows (see BID21 .Theorem 1 (Constrained Wasserstein-2 metric) The constrained Wasserstein-2 metric function d W : Θ × Θ → R + has the following formulation: DISPLAYFORM1 where the infimum is among all feasible Borel potential functions Φ : [0, 1] × R n → R and continuous parameter paths θ : [0, 1] → R d . Here ∇· and ∇ are the divergence and gradient operators over R n .We note that the constrained metric on parameter space can be different from the Wasserstein-2 distance on the full density set. The metric d W can be used to define a steepest descent optimization scheme. This can be formulated in two general ways. One way is in terms of the corresponding Riemannian structure, i.e., an inner product between tangent vectors. A well known example is the Fisher natural gradient BID1 BID2 . The constrained Wasserstein-2 metric allows us to obtain a Riemannian metric structure, from which we obtain the following constrained Wasserstein-2 gradient. We also call it Wasserstein natural gradient. Theorem 2 (Wasserstein natural gradient) Given a loss function F : Θ → R, the Wasserstein gradient operator is given by DISPLAYFORM2 where G(θ) = (G(θ) ij ) 1≤i,j≤d ∈ R d×d is given by DISPLAYFORM3 Here for each i ∈ {1, · · · , d}, Φ i : R n → R is a solution (up to additive constants) of DISPLAYFORM4 Here ∇ W θ represents the natural gradient operator with respect to the constrained Wasserstein metric, ∇ θ represents the ordinary Euclidean gradient operator, and G is the matrix representing the Wasserstein Riemannian metric. The steepest descent flow is given by DISPLAYFORM5 The corresponding gradient descent iteration (forward Euler method) satisfies DISPLAYFORM6 where h > 0 is the step size. Often in practice, the computation of matrix G(θ) −1 is difficult. The second way of obtaining a numerical scheme for equation 2 is in terms of the proximal operator. This is the backward Euler method, also named Jordan-Kinderlehrer-Otto (JKO) scheme BID15 , which is given by DISPLAYFORM7 Here, at each step, the distance of the parameter update acts as a regularization to the original loss function. Computing d W is also often challenging. However, we can approximate the d W distance locally by a second order Taylor expansion. This approximation is particularly tractable within the parameterized setting that we discussed above. This allows us to derive other first order schemes, such as the Semi-Backward Euler method:Proposition 3 (Semi-Backward Euler method) The Semi-Backward Euler method for the gradient flow of loss function F : Θ → R is given by DISPLAYFORM8 where the supremum is taken over Φ : R n → R with sufficient regularity for the integral to be well defined. The Semi-Backward Euler method is often easier to approximate than the forward Euler method, because it does not require computing and inverting G(θ), and it is often simpler than the backward Euler method (JKO), because the constrained optimization over Φ is more tractable than the timedependent constraint involved in computing d W .We implement the Semi-Backward Euler method in implicit generative model as follows. For each parameter θ ∈ R d , let the generator be given by g θ : R m → R n ; z → x = g(θ, z). This takes an input noise prior Z ∼ p(z) ∈ P 2 (R m ) to an output sample with density given by X = g(θ, Z) ∼ ρ(θ, x). Here R d is the parameter space, R m is the latent space, and R n is the sample space. In this case, the update in Proposition 3 forms DISPLAYFORM9 In practice, we apply a neural network to approximate variable Φ. See details in Appendix G. In fact, the constrained Wasserstein-2 metric in implicit generative models allows for yet a simpler formulation. This reformulation allows us to define the relaxed Wasserstein metric, and further introduces a simple algorithm for proximal operator on generators. Proposition 4 (Constrained Wasserstein-2 metric in implicit generative models) DISPLAYFORM0 where the infimum is among all feasible Borel potential functions Φ : [0, 1] × R n → R and continuous parameter paths θ : DISPLAYFORM1 Here the constrained Wasserstein metric requires that the derivative of the generator g w.r.t. θ ∈ R d be a gradient vector field of Φ w.r.t x ∈ R n . In other words, if we denote DISPLAYFORM2 The gradient constraint is satisfied if the sample space is 1 dimensional, i.e., n = 1. In general, this is not true. Here Φ(t, x) is the other function depending on the parameter space Θ. Finding Φ involves computational difficulties. Fitting the gradient constraint is an open problem for the computations of Wasserstein proximal operator. For simple computations, we withdraw the gradient constraint and consider a relaxed Wassersetin metric on parameter space: DISPLAYFORM3 We approximate the relaxed Wasserstein proximal operator based on the new metric d to obtain DISPLAYFORM4 where the infimum is among all feasible continuous parameter path θ : DISPLAYFORM5 In fact, when the sample space is high dimensional, i.e., n > 1, the above update is not exactly the Wasserstein proximal. Instead, it simply regularizes the generator by the expectation of squared difference in sample space. Algorithm 1 Relaxed Wasserstein Proximal, where F ω is a parameterized function to minimize Require: F ω , a parameterized function to minimize (e.g. Wasserstein-1 with a parameterized discriminator). g θ the generator. Require: h proximal step-size, B batch size. Require: Optimizer Fω and Optimizer g θ . Require: max iterations, and generator iterations DISPLAYFORM6 end for end for We present a toy example to illustrate the effectiveness of Wasserstein proximal operator in GANs. Consider a family of distribution with two weighted delta measures. Let Θ = {θ = (a, b) : a < 0, b > 0}, and define DISPLAYFORM0 where α ∈ [0, 1] is a given ratio and δ a (x) is the delta function supported at point a. See FIG1 .3.In this model, for a loss function F : Θ → R, the proximal regularization is given as follows: DISPLAYFORM1 where θ = (a, b) and DISPLAYFORM2 We check the following commonly used statistical distance (divergence) functions d between parameters θ and θ k . 1. Wasserstein-2 distance: DISPLAYFORM3 2. Euclidean distance: DISPLAYFORM4 3. Kullback-Leibler divergence: DISPLAYFORM5 Here the KL divergence and L 2 -distance cannot measure the difference of probability models. The Wasserstein-2 and Euclidean distances still work in these cases. In addition, the Euclidean distance d E does not depend on the structure of model ρ(θ, x), while the constrained Wasserstein-2 metric d W does. Proposition 5 Given θ * = (a * , b * ) ∈ Θ, consider the Wasserstein-1 metric as the loss function, i.e., DISPLAYFORM6 On each step of the update, the solution obtained by Wasserstein proximal decreases the objective function further than the one by Euclidean proximal. Here the proof is based on a simple fact of the shrinkage operator, see details in Appendix B.This example introduces a case that Wasserstein-2 proximal works better than Euclidean proximal for the Wasserstein-1 loss function. Here we present numerical experiments using the Relaxed Wasserstein Proximal (RWP) algorithm and the Semi-Backward Euler (SBE) method in order to perform Wasserstein gradient-descent on various GANs. We find that the Relaxed Wasserstein Proximal provides both better speed (measured by wallclock) and stability in training GANs. The Relaxed Wasserstein Proximal (RWP) algorithm is intended to be an easy-to-implement, drop-in replacement to improve speed and convergence of GAN training. It does this by applying regularization on the generator during training. This is novel as most GAN training focuses on regularizing the discriminator, e.g. with a gradient penalty BID13 BID33 BID16 BID0 BID29 , and there has been limited exploration in regularizing the generator BID8 . Specifically, we modify the update rule for the generator by:• Update for number of iterations before updating the discriminator: DISPLAYFORM0 So two hyperparameters are introduced: the proximal step-size h, and the number of iterations . In some GANs, one may update the discriminator a number of times and then update the generator a number of times, and then repeat; we will call one loop of this update an outer-iteration. A more detailed description of the algorithm is given in Appendix D.We test the Relaxed Wassersteing Proximal regularization on three GAN types:• Standard GANs BID11 ,• WGAN-GP BID12 , and • DRAGAN BID16 .We use the CIFAR-10 dataset BID17 , and the aligned and cropped CelebA dataset BID25 . And we utilize the DCGAN BID34 architecture for the discriminator and generator. To measure the quality of generated samples, we employ the Fréchet Inception Distance (FID) BID14 both to measure performance and to measure convergence of GAN training (lower FID is better); we used 10,000 generated images to measure the FID. For CIFAR-10, we measure the FID every 1000 outer-iterations, and for CelebA we measure the FID every 10,000 outer-iterations. Our particular hyperparameter choices for training are given in Appendix C. Note that since we intend RWP to be a drop-in regularization, the non-RWP hyperparameters (i.e. not h nor ) are chosen to work well before applying RWP.To summarize our results, the Relaxed Wasserstein Proximal regularization improves both the speed (wallclock) and stability of convergence. It is a tricky to compare the result of using RWP, as it performs multiple generator iterations. We thus align the comparison according to wallclock time (this procedure was also used by BID14 . In FIG1 we see that our regularization improves convergence speed (measured in wallclock time), and also obtains a lower FID for all GAN types. In particular, in DRAGAN we see a 20% improvement in sample quality according to the FID. The same results are also found for the CelebA dataset, shown in FIG2 . We note that multiple generator iterations will sometimes prevent Standard GANs on CelebA from learning initially, at which point we restart the algorithm, and once it starts learning then the run is successful. This is practically very easy to detect and provides minimal trouble, so FIG2 focuses on successful runs. We predict this defect will be rectified with a more stable loss function, such as WGAN-GP, or with different h's and 's. We also examine the effect of multiple generator updates compared to discriminator updates. More specifically, in RWP since we update the generator multiple times before updating the discriminator, then it is worth examining the effect of not using the regularization. We see in Figure 4 that even using the most stable GAN type out of the three -WGAN-GP -if we omit regularization then the FID has high variance and even tends to rise in the end. But with RWP, the FID converges with more stability and achieves a lower FID.Samples from the models are provided in Appendix E. We also performed latent space walks BID34 to show RWP regularization does not cause the GAN to memorize. For details see Appendix F. From the three graphs, we see that using the easy-to-implement RWP regularization improves speed as measured by wallclock time, and it also is able to achieve a lower FID. Here we see RWP regularization improves the speed (via wallclock time), and achieves a lower FID. We note multiple generator iterations might cause initial learning to fail, but once it starts then it remains successful. This is practically easy to detect, so we show successful runs. Figure 4: An experiment demonstrating the effect of performing 10 generator iterations per outer-iteration with and without RWP, where an outer-iteration is a single loop of: a number of discriminator iterations, then a number of generator iterations. This experiment goes to 1,000,000 outer-iterations to show long-term behavior. With RWP regularization we obtain convergence, as well as lower FID. Without RWP, the training is highly variable and the FID is even on a rising trend in the end. The effect of the Semi-Backward Euler (SBE) method, on the CIFAR-10 dataset. As we observe, the training is comparable to the standard way of training using the WGAN-GP loss. The experiment was averaged over 5 runs. The bold lines is the average, and the enveloping lines are the minimum and maximum. The training of Semi-Backward Euler (SBE) is a more complicated. Here we attempt to approximate three functions: the usual discriminator and generator, and the potential function Φ p . The algorithm and particular hyperparameter settings are presented in the appendix in Section G. We present our attempts at optimizing over the three networks in FIG3 . Since both the standard WGAN-GP and the SBE on WGAN-GP had the same generator iterations, then we align according to this. As we see, the Semi-Backward Euler method is comparable to norm WGAN-GP. We leave deeper investigation of the Semi-Backward Euler method for future work. In the literature, many different aspects of optimal transport have been applied into machine learning and GANs.1. Loss function. Many studies apply the Wasserstein distance as the loss function. There are mainly two reasons for using the Wasserstein loss function BID10 BID30 . On the one hand, the Wasserstein distance is a statistical distance depending on the metric of the sample space. So it introduces a statistical estimator, named the the minimal Wasserstein estimator BID5 , depending on the geometry of the data. On the other hand, the Wasserstein distance is useful for comparing probability distributions supported on lower dimensional sets. This is often intractable for other divergence functions. In GANs, these properties have been leveraged in Wasserstein GAN BID4 . In this case, the loss function is chosen as the Wasserstein-1 distance function. In its computations, the discriminator, also called the Kantorovich dual variable, needs to satisfy the 1-Lipschitz condition. Many studies work on the regularization of the discriminator in order to satisfy this condition BID13 BID33 .2. Gradient flows in full probability set. The Wasserstein-2 metric provides a metric tensor structure BID26 BID32 BID20 , under which the probability space forms an infinite dimensional Riemannian manifold, named the density manifold BID18 . The gradient flow in the density manifold links with many transport-related partial differential equations BID36 BID31 . A famous example is that the Fokker-Planck equation, the probability transition equation of Langevin dynamics, is the gradient flow of the KL divergence function. In this perspective, two angles have been developed in the learning communities. Firstly, many groups try to leverage the gradient flow structure in probability space supported on the parameter space. They study the stochastic gradient descent by the transition equation in the probability over parameters BID28 . Secondly, many nonparametric models have been studied, such as the Stein gradient descent method BID24 . It can be viewed as the generalization of Wasserstein gradient flow. In addition, BID9 consider an approximate inference method for computing Wasserstein gradient flow in full probability set. Here an approximation towards Kantorovich dual variables is introduced.3. Gradient flow constrained on parameter space. The Wasserstein structure can also be constrained on parameter space. BID7 studied the constrained Wasserstein gradient with fixed mean and variance. Here the density subset is still infinite dimensional. Many approaches also focus on Gaussian families or elliptical distributions BID35 . The Wasserstein gradient flow in Gaussian family has been studied by BID27 .Compared to previous works, our approach applies the Wasserstein gradient to work on general implicit generative models. In this work, we apply the constrained Wasserstein gradient and its relaxations on implicit generative models. Whereas much work has focused on regularizing the discriminator, in this work we focus on regularizing the generator. For Wasserstein GAN (with gradient penalty), we compute the Wasserstein-2 gradient flow of Wasserstein-1 distance on parameter space. Experimentally, the proposed method allows us to obtain a better minimizer in the sense of FID, with faster convergence speeds in wall-clock time. In the full probability set, we consider a metric function W 2 : DISPLAYFORM0 where the infimum is taken among all feasible Borel potential functions Φ : [0, 1] × R n → R and continuous density path ρ : [0, 1] × R n → R + satisfying the continuity equation. The variational formulation in equation 5 introduces a Riemannian structure in density space. Consider the set of smooth and strictly positive probability densities DISPLAYFORM1 Denote F := C ∞ (R n ) the set of smooth real valued functions. The tangent space of P + is given by DISPLAYFORM2 Given Φ ∈ F and ρ ∈ P + , define DISPLAYFORM3 Thus V Φ ∈ T ρ P + . The elliptic operator ∇·(ρ∇) identifies the function Φ modulo additive constants with the tangent vector V Φ of the space of densities. Given ρ ∈ P + , σ i ∈ T ρ P + , i = 1, 2, define DISPLAYFORM4 where DISPLAYFORM5 The inner product g W endows P + with a Riemannian metric tensor. In other words, the variational problem equation 5 is a geometric action energy in (P + , g W ).Given a loss function F : P + → R, the Wasserstein gradient operator in (P + , g W ) is given as follows. DISPLAYFORM6 Thus the gradient flow satisfies DISPLAYFORM7 More analytical results on the Wasserstein-2 gradient flow are provided in BID3 .We next consider Wasserstein-2 metric and gradient operator constrained on statistical models. A statistical model is defined by a triplet (Θ, R n , ρ). For simple presentation of paper, we assume Θ ⊂ R d and ρ : Θ → P(R n ) is a parameterization function. In this case, ρ(Θ) ⊂ P(R n ). We assume that the parameterization map ρ is locally injective and under suitable regularities. We define a Riemannian metric g on ρ(Θ) by pulling back the Wasserstein-2 metric tensor g W .Definition 6 (Wasserstein statistical manifold) Given θ ∈ Θ and σ i ∈ T θ Θ, i = 1, 2, we define DISPLAYFORM8 Here DISPLAYFORM9 In particular, we denote DISPLAYFORM10 where G(θ) = (G(θ) ij ) 1≤i,j≤d ∈ R d×d is the associated metric tensor defined in Theorem 2.Here we assume that G(θ) is smooth and positive definite, so that (Θ, g θ ) forms a smooth Riemannian manifold. In this case, Theorem 2 studies the constrained Wassertein gradient operator in parameter space. Proof of Theorem 1 The distance d W can be written into the action function in Wasserstein statistical manifold. In other words, consider DISPLAYFORM0 where the infimum is taken over θ(t) ∈ C 1 ([0, 1], Θ). Following the definition of metric tensor in definition 6, we haveθ DISPLAYFORM1 with Φ(t, x) satisfying DISPLAYFORM2 We finish the proof. The gradient operator on a Riemannian manifold (Θ, g θ ) is defined as follows. DISPLAYFORM0 Proof of Proposition 3. We next present the derivation of the proposed semi-backward method. DISPLAYFORM1 and DISPLAYFORM2 Proof of Claim. We next prove the claim. Denote the geodesic path θ DISPLAYFORM3 We reparameterize the time of θ * (t) into the time interval [0, h]. Denote τ = ht and θ(τ ) = θ * (ht). DISPLAYFORM4 which proves equation 6.We next prove equation 7. On the L.H.S. of equation 7, DISPLAYFORM5 From the definition of G(θ), DISPLAYFORM6 On the R.H.S. of equation 7, the maximizer Φ * satisfies DISPLAYFORM7 Applying equation 8 into the R.H.S. of equation 7, we have DISPLAYFORM8 Comparing the L.H.S. and R.H.S. of equation 7, we prove the claim. From the claim, DISPLAYFORM9 Thus we derive a consistent numerical method in time, known as the Semi-backward method: DISPLAYFORM10 Proof of Proposition 4. This result is proven in BID23 . We present it here for the completion of paper. The implicit model is given by the following push-forward relation. Denote DISPLAYFORM11 Given the gradient constraint DISPLAYFORM12 we shall show that the probability density transition equation of g(θ(t), z) satisfies the constrained continuity equation DISPLAYFORM13 and DISPLAYFORM14 On the one hand, consider DISPLAYFORM15 where the second equality holds from the push forward relation in equation 9.On the other hand, consider DISPLAYFORM16 where ∇, ∇· are gradient and divergence operators w.r.t. x ∈ R n . The second to last equality holds from the push forward relation equation 9, and the last equality holds using the integration by parts w.r.t. x. Since equation 12 equals equation 13 for any f ∈ C ∞ c (R n ), we prove equation 10.In addition, by the definition of the push forward operator equation 9, we have DISPLAYFORM17 Thus we prove equation 11.Proof of Proposition 5. This example allows us to compute the proximal operator explicitly. On the one hand, we compute the Wasserstein proximal operator explicitly: DISPLAYFORM18 I.e., DISPLAYFORM19 Here DISPLAYFORM20 On the other hand, we calculate the Euclidean proximal operator explicitly: DISPLAYFORM21 I.e., DISPLAYFORM22 Here DISPLAYFORM23 Here we only need to check that for all possible cases, DISPLAYFORM24 and DISPLAYFORM25 . In other cases, the proof follows similarly. We finish the proof. The following hyperparameter settings for the Relaxed Wasserstein Proximal experiments in Section 3.1 are:• A batch size of 64 for all experiments.• For CIFAR-10 with WGAN-GP: The Adam optimizer with learning rate 0.0001, β 1 = 0.5, and β 2 = 0.9 for both the generator and discriminator. We used a latent space dimension of 128, h = 0.1, and = 10 generator iterations.• For CIFAR-10 with Standard and DRAGAN: The Adam optimizer with learning rate 0.0002, β 1 = 0.1, and β 2 = 0.999 for both the generator and discriminator. We used a latent space dimension of 100, h = 0.2, and = 5 generator iterations.• For aligned and cropped CelebA with Standard: The Adam optimizer with learning rate 0.0002, β 1 = 0.5, and β 2 = 0.999 for both the generator and discriminator. We used a latent space dimension of 100, h = 0.2, and = 5 generator iterations. As mentioned in Section 3.1, the Relaxed Wasserstein Proximal is meant to be an easy-to-implement, drop-in regularization. For instructional purposes, we take a specific example to showcase the algorithm: Relaxed Wasserstein Proximal on Standard GANs (with non-saturating gradient for the generator):• Given: DISPLAYFORM0 -Choice of optimizers, Adam ω and Adam θ , -Proximal step-sizes h, and generator iterations , and -Batch size B.Then the algorithm follows: DISPLAYFORM1 , and latent data DISPLAYFORM2 .2. Update the discriminator: DISPLAYFORM3 4. Perform Adam gradient descent number of times: DISPLAYFORM4 for number of times.5. Repeat the above until a chosen stopping condition (e.g. maximum number of iterations).As one can analyze above, the only difference between the standard way of training GANs and using the Relaxed Wasserstein Proximal, are the g θ (z i ) − g θ k−1 (z i ) 2 2 terms and the number of generator iterations . Note that in this paper, we call a single loop of updating a discriminator a number of times and then updating the generator a number of a time, an outer-iteration. In FIG4 , we have samples generated from a Standard GAN with RWP regularization, trained on the CelebA dataset. The FID of these images was 17.105.In FIG5 , we have samples generated from WGAN-GP with RWP , trained on the CIFAR-10 dataset. The FID for these images is 38.3. F LATENT SPACE WALK BID34 suggest that walking in the latent space could detect whether a generator was memorizing. We see in FIG6 and FIG7 that we have smooth transitions, so this is not the case for GANs with RWP regularization. The specific hyperparameter settings used for the Semi-Backward Euler (SBE) on WGAN-GP, trained on CIFAR-10, are:• A batch size of 64.• The DCGAN architecture for the discriminator and generator. A one-hidden-layer fullyconnected network (a.k.a. MLP) for the potential Φ p . We also used layer-normalization (Lei BID19 ) for each layer.• We used the Adam optimizer with learning rate 0.0002, β 1 = 0.1, and β 2 = 0.999 for both the generator, discriminator, and potential Φ p . We used a latent space dimension of 100, and h = 0.2.• Every outer-iteration loop, we updated the discriminator 5 times (as suggested in WGAN-GP), the generator once, and the potential 5 times. Note an outer-iteration is defined as one loop of: updating the discriminator a number of times, updating the potential a number of times, and updating the generator a number of times. .3: DISPLAYFORM0 for s = 0 to phi iterations do Sample latent data {z i } 6: DISPLAYFORM0 end for Sample latent data {z i } 10: DISPLAYFORM0 end for 12: end for
We propose the Wasserstein proximal method for training GANs.  Proposes a new GAN procedure that takes into account points generated in the previous iteration and updates the generator to be carried out l times. Considers natural gradient learning in GAN learning, where the Riemannian structure induced by the Wasserstein-2 distance is employed. The paper intends to utilize natural gradient induced by Wasserstein-2 distance to train the generator in GAN and the authors propose the Wasserstein proximal operator as a regularization.