Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain. Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain. To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels. Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process. Image-to-image (I2I) translation refers to the task of mapping an image from a source domain to a target domain, e.g. semantic maps to real images, gray-scale to color images, low-resolution to high-resolution images, and so on. The recent advances in deep learning have greatly improved the quality of I2I translation methods for a number of applications, including super-resolution BID3 , colorization BID33 , inpainting BID26 , attribute transfer BID18 , style transfer BID4 , and domain adaptation BID8 BID22 . Most of these works BID11 BID30 BID35 have been very successful in these cross-domain I2I translation tasks because they rely on large datasets of paired training data as supervision. However, for many tasks it is not easy, or even possible, to obtain such paired data that show how an image in the source domain should be translated to an image in the target domain, e.g. in cross-city street view translation or male-female face translation. For this Figure 2 : The x A to x AB translation procedure of our EGSC-IT framework. 1) Source domain image x A is fed into an encoder E A to compute a shared latent code z A and is further decoded to a common high-level content representation c A . 2) Meanwhile, x A is also fed into a sub-network to compute feature masks m A .3) The target domain exemplar image x B is fed to a sub-network to compute affine parameters γ B and β B for AdaIN . 4) The content representation c A is transferred to the target domain using m A , γ B , β B , and is further decoded to an image x AB by target domain generator G B .unsupervised setting, BID34 proposed to use a cycle-consistency loss, which assumes that a mapping from domain A to B, followed by its reverse operation approximately yields an identity function, that is, F (G(x A )) ≈ x A . BID22 further proposed a shared-latent space constraint, which assumes that a pair of corresponding images (x A , x B ) from domains A and B respectively can be mapped to the same representation z in a shared latent space Z. Note that, all the aforementioned methods assume that there is a deterministic one-to-one mapping between the two domains, i.e. each image in A is translated to only a single image in B. By doing so, they fail to capture the multimodal nature of the image distribution within the target domain, e.g. different color and style of shoes in sketch-to-image translation and different seasons in synthetic-to-real street view translation.In this work, we propose Exemplar Guided & Semantically Consistent I2I Translation (EGSC-IT) to explicitly address this issue. As shown in concurrent works BID6 BID18 , we assume that an image is composed of two disentangled representations. In our case, first a domain-shared representation that models the content in the image, and second a domain-specific representation that contains the style information. However, for a multimodal domain with complex inner-variations, as the ones we target in this paper, e.g. street views of day-and-night or different seasons, it is difficult to have a single static representation which covers all variations in that domain. Moreover, it is unclear which style ( time-of-day/season) to pick during the image translation process. To handle such multimodal I2I translations , some approaches BID0 BID6 BID18 incorporate noise vectors as additional inputs to the generator, but as shown in BID11 BID35 this could lead to mode collapsing issues. Instead, we propose to condition the image translation process on an arbitrary image in the target domain, i.e. an exemplar. By doing so, EGSC-IT does not only enable multimodal (i.e. many-to-many) image translations, but also allows for explicit control over the translation process, since by using different exemplars as guidance we are able to translate an input image into images of different styles within the target domain -see FIG0 .To instantiate this idea, we adopt the weight sharing architecture proposed in UNIT BID22 , but instead of having a single latent space shared by both domains, we propose to decompose the latent space into two components according to the two disentangled representations presented above. That is, a domain-shared component that focuses on the image content, and a domain-specific component that captures the style information associated with the exemplar. In our particular case, the domain-shared content component contains semantic information, such as the objects' category, shape and spatial layout, while the domain-specific style component contains the style information, such as the color and texture, to be translated from a target domain exemplar to an image in the source domain. To realize this translation, we apply adaptive instance normalization (AdaIN) BID9 to the shared content component of the source domain image using the AdaIN parameters computed from the target domain exemplar. However, directly applying AdaIN to the feature maps of the shared content component would mix up all objects and scenes in the image, Published as a conference paper at ICLR 2019 making the image translation prone to failure when an image contains diverse objects and scenes. To tackle this problem, existing works BID5 BID8 BID20 BID24 use semantic labels as an additional form of supervision. However, ground-truth semantic labels are not easy to obtain for most tasks as they require labor-intensive annotations. Instead, to maintain the semantic consistency during image translation without using any semantic labels we propose to compute feature masks. One can think of feature masks as attention modules that approximately decouple different semantic categories in an unsupervised way under the guidance of perceptual losses and adversarial loss. In particular, one feature mask corresponding to a certain semantic category is applied to one feature map of the shared content component, and consequently the AdaIN for that channel is only required to capture and model the style difference for that category, e.g. sky's style in two domains. To the best of our knowledge, this is the first line of work that addresses the semantic consistency issue under this setting. See Fig. 2 for an overview of EGSC-IT.Our contribution is three-fold. i) We propose a novel approach for the I2I translation task, which enables multimodal (i.e. many-to-many) mappings and allows for explicit style control over the translation process. ii) We introduce the concept of feature masks for the unsupervised, multimodal I2I translation task, which provides coarse semantic guidance without using any semantic labels. iii) Evaluation on different datasets show that our method is robust to mode collapse and can generate results with semantic consistency, conditioned on a given exemplar image. Since our method does not use any semantic segmentation labels nor paired data, there are some artifacts in the results for some hard cases. For example, as to the street view translation, day→night and night→day (e.g. Fig. 7 bottom row) are more challenging than day→day (e.g. Fig. 7 top row) . As a result, it is sometimes hard for our model to understand the semantics in such cases. In the future, it would be interesting to extend our method to the semi-supervised setting in order to benefit from the presence of some fully-labeled data. We introduced the EGSC-IT framework to learn a multimodal mapping across domains in an unsupervised way. Under the guidance of an exemplar from the target domain, we showed how to combine AdaIN with feature masks in order to transfer the style of the exemplar to the source image, while retaining semantic consistency at the same time. Numerous quantitative and qualitative results demonstrate the effectiveness of our method in this particular setting.
We propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. Discusses a core failing and need for I2I translation models. The paper explores the idea that an image has two components and applies an attention model where the feature masks that steer the translation process do not require semantic labels