Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.
 Despite strong performances on facial analysis using deep neural networks BID17 BID15 Schroff et al., 2015; Parkhi et al., 2015) , learning a model that generalizes across variations in attributes like ethnicity, gender or age remains a challenge. For example, it is reported by BID5 that commercial engines tend to make mistakes at detecting gender for images of darker-skinned females. Such biases have enormous social consequences, such as conscious or unconscious discrimination in law enforcement, surveillance or security (WIRED, 2018a; b; NYTimes, 2018; GIZMODO, 2018) . A typical solution is to collect and annotate more data along the underrepresented dimension, but such efforts are laborious and time consuming. This paper proposes a novel deep unsupervised domain adaptation approach to overcome such biases in face verification and identification.Deep domain adaptation (Long et al., 2013; BID22 BID12 Sohn et al., 2017; Haeusser et al., 2017; Luo et al., 2017) allows porting a deep neural network to a target domain without extensive labeling efforts. Currently, there are two predominant approaches to deep domain adaptation. The first approach, domain divergence reduction learning, is motivated by the works of BID0 BID1 . It aims to reduce the source-target domain divergence using domain adversarial training BID12 Sohn et al., 2017; BID19 or maximum mean discrepancy minimization BID22 Long et al., 2015; , while leveraging supervised loss from labeled source examples to maintain feature space discriminative power. Since the theoretical basis of this approach BID0 assumes a common task between domains, it is usually applied to a classification problem where the source and target domains share the same label space and task definition. The second approach considers domain adaptation as a semi-supervised learning problem and applies techniques such as entropy minimization (Grandvalet & Bengio, 2005) or self-ensembling (Laine & Aila, 2017; BID18 BID11 on target examples to encourage decisive and consistent predictions.However, neither of those are applicable if the label spaces of source and target domains do not align. As a motivating example, consider a cross-ethnicity generalization of face recognition problem, where the source ethnicity (e.g., Caucasian) contains labeled examples and the target ethnicity (e.g., African-American) contains only unlabeled examples. When it is cast as a classification problem, the tasks of the two domains are different due to disjoint label spaces. Moreover, examples from different ethnicity domains almost certainly belong to different identity classes. To satisfy such additional label constraints, representations of examples from different domains should ideally be distant from each other in the embedding space, which conflicts with the requirements of domain divergence reduction learning as well as entropy minimization on target examples with source domain class labels.In this work, we aim at learning a shared representation space between a source and target domain with disjoint label spaces that not only remains discriminative over both domains but also keep representations of examples from different domains well-separated, when provided with additional label constraints. Firstly, to overcome the limitation of domain adversarial neural network (DANN) BID12 , we propose to convert disjoint classification tasks (i.e., the source and target domains correspond to non-overlapping class labels) into a unified binary verification task. We term adaptation across such source and target domains as cross-domain distance metric adaptation (CD2MA). We demonstrate a generalization of the theory of domain adaptation BID0 to our setup, which bounds the empirical risk for within-domain verification of two examples drawn from the unlabeled target domain. While the theory does not guarantee verification between examples from different domains, we propose approaches that also address such cross-domain verification tasks.To this end, we introduce a Feature Transfer Network (FTN) that separates the target features from the source features while simultaneously aligning them with an auxiliary domain of transformed source features. Specifically, we learn a shared feature extractor that maps examples from different domains to representations far apart. Simultaneously, we learn a feature transfer module that transforms the source representation space to another space used to align with the target representation space through a domain adversarial loss. By forging this alignment, the discriminative power from the augmented source representation space would ideally be transferred to the target representation space. The verification setup also allows us to introduce a novel entropy minimization loss in the form of N -pair metric loss (Sohn, 2016) , termed multi-class entropy minimization (MCEM), to further leverage unlabeled target examples whose label structure is not known. MCEM samples pairs of examples from a discovered label structure within the target domain using an offline hierarchical clustering algorithm such as HDBSCAN BID6 , computes the N -pair metric loss among these examples (Sohn, 2016) , and backpropagates the resulting error derivatives.In experiments, we first perform on a controlled setting by adapting between disjoint sets of digit classes. Specifically, we adapt from 0-4 of MNIST-M BID12 dataset to 5-9 of MNIST dataset and demonstrate the effectiveness of FTN in learning to align and separate domains. Then, we assess the impact of our proposed unsupervised CD2MA method on a challenging cross-ethnicity face recognition task, whose source domain contains face images of Caucasian identities and the target domain of non-Caucasian identities, such as African-American or East-Asian. This is an important problem since existing face recognition datasets show significant label biases towards Caucasian ethnicity, leading to sub-optimal recognition performance for other ethnicities. The proposed method demonstrates significant improvement in face verification and identification compared to a source-only baseline model and a standard DANN. Our proposed method also closely matches the performance upper bounds obtained by training with fully labeled source and target domains. We address the challenge of unsupervised domain adaptation when the source and the target domains have disjoint label spaces by formulating the classification problem into a verification task. We propose a Feature Transfer Network, allowing simultaneous optimization of domain adversarial loss and domain separation loss, as well as a variant of N -pair metric loss for entropy minimization on the target domain where the ground-truth label structure is unknown, to further improve the adaptation quality. Our proposed framework excels at both within-domain and cross-domain verification tasks.As an application, we demonstrate cross-ethnicity face verification that overcomes label biases in training data, achieving high accuracy even for unlabeled ethnicity domains, which we believe is a result with vital social significance.Vinod Nair and Geoffrey E Hinton. Following (Haeusser et al., 2017) , we preprocess the data by subtracting a channel-wise pixel mean and dividing by channel-wise standard deviation of pixel values. For MNIST examples, we also apply color-intensity inversion. All images are resized into 32×32 with 3 channels.Our feature generator module is composed of 6 convolution layers and 3 max-pooling layers followed by 2 fully-connected layers. We use ReLU (Nair & Hinton, 2010) after convolution layers. The output dimension of the feature generator module is 128 and is normalized to have L2-norm of 2. The full description of the generator module is in TAB5 .The feature transfer module maps 128 dimensional vector into the same dimensional vector using two fully-connected layers (128 − 256 − 256 − 128) and residual connection as in Figure 1 (a) . Discriminator architectures are similar to that in Figure 1 (b) but with fully-connected layers whose output dimensions are 128 instead of 256.We use Adam stochastic optimizer with learning rate of 0.0003, λ 1 = 0.3 and λ 2 = 0.03 to train FTN.
A new theory of unsupervised domain adaptation for distance metric learning and its application to face recognition across diverse ethnicity variations. Proposes a novel feature transfer network that optimizes domain adversarial loss and domain separation loss.