Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities. Deep Neural Networks (DNNs) have proved successful in various tasks like computer vision, natural language processing, recommendation systems, and autonomous driving. Modern networks are comprised of millions of parameters, requiring significant storage and computational effort. Though accelerators such as GPUs make realtime performance more accessible, compressing networks for faster inference and simpler deployment is an active area of research. Compression techniques have been applied to many networks, reducing memory requirements and improving their performance. Though these approaches do not always harm accuracy, aggressive compression can adversely affect the behavior of the network. Distillation (Schmidhuber, 1991; Hinton et al., 2015) can improve the accuracy of a compressed network by using information from the original, uncompressed network. Generative Adversarial Networks (GANs) (Schmidhuber, 1990; Goodfellow et al., 2014) are a class of DNN that consist of two sub-networks: a generative model and a discriminative model. Their training process aims to achieve a Nash Equilibrium between these two sub-models. GANs have been used in semi-supervised and unsupervised learning areas, such as fake dataset synthesis (Radford et al., 2016; Brock et al., 2019) , style transfer (Zhu et al., 2017b; Azadi et al., 2018) , and image-to-image translation (Zhu et al., 2017a; Choi et al., 2018) . As with networks used in other tasks, GANs have millions of parameters and nontrivial computational requirements. In this work, we explore compressing the generative model of GANs for more efficient deployment. We show that applying standard pruning techniques, with and without distillation, can cause the generator's behavior to no longer achieve the network's goal. Similarly, past work targeted at compressing GANs for simple image synthesis fall short when they are applied to large tasks. In some cases, this result is masked by loss curves that look identical to the original training. By modifying the loss function with a novel combination of the pre-trained discriminator and the original and compressed generators, we can overcome this behavioral degradation and achieve compelling compression rates with little change in the quality of the compressed generator's ouput. We apply our technique to several networks and tasks to show generality. Finally, we study the behavior of compressed generators when pruned with different amounts and types of sparsity, finding that filter pruning, a technique commonly used for accelerating image classification networks, is not trivially applicable to GANs. A complementary method of network compression is quantization. Sharing weight values among a collection of similar weights by hashing (Chen et al., 2015) or clustering (Han et al., 2016) can save storage and bandwidth at runtime. Changing fundamental data types adds the ability to accelerate the arithmetic operations, both in training (Micikevicius et al., 2018) and inference regimes (Jain et al., 2019) . Several techniques have been devised to combat lost accuracy due to compression, since there is always the chance that the behavior of the network may change in undesirable ways when the network is compressed. Using GANs to generate unique training data (Liu et al., 2018b) and extracting knowledge from an uncompressed network, known as distillation (Hinton et al., 2015) , can help keep accuracy high. Since the pruning process involves many hyperparameters, Lin et al. (2019) use a GAN to guide pruning, and Wang et al. (2019a) structure compression as a reinforcement learning problem; both remove some of the burden from the user. In this paper, we propose using a pre-trained discriminator to self-supervise the compression of a generative adversarial network. We show that it is effective and applies to many tasks commonly solved with GANs, unlike traditional compression approaches. Comparing the compressed generators with the baseline models on different tasks, we can conclude that the compression method performs well both in subjective and quantitative evaluations. Advantages of the proposed method include: • The results from the compressed generators are greatly improved over past work. • The self-supervised compression is much shorter than the original GAN training process. It only takes 1%-10% training effort to get an optimal compressed generative model. • It is an end-to-end compression schedule that does not require objective evaluation metrics. • We introduce a single optional hyperparameter (fixed to 0.5 for all our experiments). We use self-supervised GAN compression to show that pruning whole filters, which can work well for image classification models (Li et al., 2017) , may perform poorly for GAN applications. Even pruned at a moderate sparsity (e.g. 25% in Figure 8 ), the generated image has an obvious color shift and does not transfer the photorealistic style. In contrast, the fine-grained compression stategy works well for all tasks we explored. SRGAN seems to be an exception to filter-pruning's poor results; we have to look closely to see differences, and it's not clear which is subjectively better. We have not tried to achieve extremely aggressive compression rates with complicated pruning strategies. Different models may be able to tolerate different amounts of pruning when applied to a task, which we leave to future work. Similarly, we have used network pruning to show the importance and utility of the proposed method, but self-supervised compression is general to other techniques, such as quantization, weight sharing, etc. There are other tasks for which GANs can provide compelling results, and newer networks for tasks we have already explored; future work will extend our self-supervised compression method to these new areas. Finally, self-supervised compression may apply to other network types and tasks if a discriminator is trained alongside the teacher and student networks.
Existing pruning methods fail when applied to GANs tackling complex tasks, so we present a simple and robust method to prune generators that works well for a wide variety of networks and tasks. The authors propose a modification to the classic distillation method for the task of compressing a network to address the failure of previous solutions when applied to generative adversarial networks.