Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter. Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains. We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions.   We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space. We report promising qualitative results for the task of face-to-cartoon translation. The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer. Image-to-image translation -learning to map images from one domain to another -covers several classical computer vision tasks such as style transfer (rendering an image in the style of a given input BID3 ), colorization (mapping grayscale images to color images (Zhang et al., 2016) ), super-resolution (increasing the resolution of an input image BID9 ), or semantic segmentation (inferring pixelwise semantic labeling of a scene BID14 ). In many cases, one can rely on supervision in the form of labels or paired samples. This assumption holds for instance for colorization, where ground-truth pairs are easily obtained by generating grayscale images from colored inputs.Figure 1: On the left, we depict a high-level motivational example for semantic style transfer, the task of adapting an image to the visual appearance of an other domain without altering its semantic content. The proposed XGAN applied on the face-to-cartoon task preserves important face semantics such as hair style or face shape (right).In this work, we consider the task of semantic style transfer: learning to map an image from one domain into the style of another domain without altering its semantic content (see Figure 1) . In that sense, our goal is akin to style transfer: We aim to transfer style while keeping content consistent. The key differences with traditional techniques are that (i) we work with image collections instead of having a single style image, and (ii ) we aim to retain higher-level semantic content in the feature space rather than pixel-level structure. In particular, we experiment on the task of translating faces to cartoons while preserving their various facial attributes (hair color, eye color, etc.). Note that without loss of generality, a photo of a face can be mapped to many valid cartoons, and vice versa. Semantic style transfer is therefore a many-to-many mapping problem, for which obtaining labeled examples is ambiguous and costly. Although this paper specifically focuses on the face-to-cartoon setting, many other examples fall under this category: mapping landscape pictures to paintings (where the different scene objects and their composition describe the input semantics), transforming sketches to images, or even cross-domain tasks such as generating images from text. In this setting, we only rely on two unlabeled training image collections or corpora, one for each domain, with no known image pairings across domains. Hence, we are faced with a double domain shift, first in terms of global domain appearance, and second in terms of the content distribution of the two collections.Recent work BID6 Zhu et al., 2017; Yi et al., 2017; BID1 report good performance using GAN-based models for unsupervised image-to-image translation when the two input domains share similar pixel-level structure (e.g., horses and zebras) but fail for more general transformations (e.g., dogs and cats). Perhaps the best known recent example is CycleGAN (Zhu et al., 2017) . Given two image domains D 1 and D 2 , the model is trained with a pixel-level cycleconsistency loss which ensures that the mapping g 1→2 from D 1 to D 2 followed by its inverse, g 2→1 , yields the identity function; i.e., g 1→2 • g 2→1 = id. However, we argue that such a pixel-level constraint is not sufficient in our case; the category of transformations we are interested in requires a constraint in semantic space even though the transformation occurs in the pixel space.To this end, we propose XGAN ("Cross-GAN"), a dual adversarial autoencoder which learns a shared semantic representation of the two input domains in an unsupervised way, while jointly learning both domain-to-domain translations. In other words, the domain-to-domain translation g 1→2 consists of an encoder e 1 taking inputs in D 1 , followed by a decoder d 2 with outputs in D 2 (and likewise for g 2→1 ) such that e 1 and e 2 , as well as d 1 and d 2 , are partially shared. The main novelty lies in how we constrain the shared embedding using techniques from the domain adaptation literature, as well as a novel semantic consistency loss. The latter ensures that the domain-to-domain translations preserve the semantic representation, i.e., that e 1 ≈ e 2 •g 1→2 and e 2 ≈ e 1 •g 2→1 . Therefore, it acts as a form of self-supervision which alleviates the need for paired examples and preserves semantic featurelevel information rather than pixel-level content. In the following section, we review relevant recent work before discussing the XGAN model in more detail in Section 3. In Section 4, we introduce CARTOONSET, our dataset of cartoon faces for research on semantic style transfer, which we are currently in the process of making publicly available. Finally, in Section 5 we report experimental results of XGAN on the face-to-cartoon task, and discuss various ablation experiments. In this work, we introduced XGAN, a model for unsupervised domain translation applied to the task of semantically-consistent style transfer. In particular, we argue that learning image-to-image translation between two structurally different domains requires passing through a high-level joint semantic representation while discarding local pixel-level dependencies. Additionally, we proposed a semantic consistency loss acting on both domain translations as a form of self-supervision.We reported promising experimental results on the task of mapping the domain of face images to cartoon avatars that clearly outperform the current baseline. We also showed that additional weak supervision, such as a pretrained feature representation, can easily be added to the model in the form of teacher knowledge. While not necessary, it acts as a good regularizer for the learned embeddings and generated samples. This can be particularly useful for natural image data as offthe-shelf pretrained models are abundant. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
XGAN is an unsupervised model for feature-level image-to-image translation applied to semantic style transfer problems such as the face-to-cartoon task, for which we introduce a new dataset. This paper proposes a new GAN-based model for unpaired image-to-image translation similar to DTN