We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets. The datasets include common distributions of points in $R^n$ space and images containing polygons of various shapes and sizes. We find that by and large GANs fail to faithfully recreate point datasets which contain discontinous support or sharp bends with noise. Additionally, on image datasets, we find that GANs do not seem to learn to count the number of objects of the same kind in an image. We also highlight the apparent tension between generalization and learning in GANs. Generative Adversarial Models (GANs) BID4 have been found to produce images of very high quality on some datasets BID6 . However, their results on other datasets, while impressive, still lag behind BID1 . This raises the question whether GANs are indeed the right choice to model some distributions. This paper aims to test the distribution learning ability of GANs by evaluating them on synthetic datasets. In this paper, we present the phenomenon of GANs being unable to count. We support this hypothesis with experiments on synthetic datasets where the count of similar objects in a scene is kept constant while their location is varied. We find that in their current form GANs are unable to learn semantic constraints even in the absence of noise introduced by natural image datasets. We also emphasize the fine line between generalization and good learning outcomes in GANs. Additionally, we conduct experiments on non-image data where we conclude that GANs tend to have difficulty learning discontinuous distributions which might necessitate the usage of mixtures of generators. A thorough evaluation of such an approach is left as future work. Each dataset has associated noise parameters corresponding to the noise parameters in the scikit-learn API. We experiment with varying noise but we find that it does not affect learning outcomes too much.
GANs are evaluated on synthetic datasets