# Deep Feature Registration for Unsupervised Domain Adaptation

Youshan Zhang  Computer Science and Artificial Intelligence,
Yeshiva University, NYC, NY
Email<EMAIL_ADDRESS>Brian D. Davison  Computer Science and
Engineering,
Lehigh University, Bethlehem, PA, USA
Email<EMAIL_ADDRESS>
###### Abstract

While unsupervised domain adaptation has been explored to leverage the
knowledge from a labeled source domain to an unlabeled target domain, existing
methods focus on the distribution alignment between two domains. However, how
to better align source and target features is not well addressed. In this
paper, we propose a deep feature registration (DFR) model to generate
registered features that maintain domain invariant features and simultaneously
minimize the domain-dissimilarity of registered features and target features
via histogram matching. We further employ a pseudo label refinement process,
which considers both probabilistic soft selection and center-based hard
selection to improve the quality of pseudo labels in the target domain.
Extensive experiments on multiple UDA benchmarks demonstrate the effectiveness
of our DFR model, resulting in new state-of-the-art performance.

##  I Introduction

The availability of a large amount of labeled training samples is usually the
prerequisite of typical machine learning algorithms. However, a shortage of
labeled data in the domain of interest is not uncommon in many real-world
applications. Therefore, it is of great significance to transfer knowledge
from one label-rich (source) domain to a label-scarce (target) domain.
However, there is a domain gap issue between the two domains, which is caused
by the varied data distributions of different domains.

To alleviate the domain gap issue without manual annotation, unsupervised
domain adaptation (UDA) is well explored to learn transferable representation
from the labeled source domain and unlabeled target domain. Existing domain
adaptation methods assume that the feature distributions of the source and
target domains are different but share the same label space. Early traditional
based methods include subspace learning [1, 2], and distribution alignment [3,
4, 5]. Subspace learning methods transfer the samples of both domains from the
original feature space into a common latent subspace, which consists of the
shared features across the two domains while preserving inherent geometric
data structure via a lower rank or sparse representations [1, 2]. The
traditional distribution alignment usually reduces domain discrepancy via
aligning marginal [6, 3], conditional [7] and joint [8, 4] distributions.
However, traditional DA methods highly depend on features that are extracted
from raw images. The performance of traditional methods is tremendously
improved by using deep features (e.g., ResNet50 [9, 4]) [10, 11].

Recent deep neural networks show superiority in improving UDA performance.
Generally, these deep learning-based DA methods can be roughly categorized as
discrepancy-based methods [12] and adversarial learning-based methods [13].
The former aligns the distributions of source and target domains by directly
minimizing the different distance metrics between the two domains. The latter
methods enforce the feature representations to be indistinguishable by a
domain discriminator and the feature extractor tries to confuse the
discriminator. The domain invariant features are expected to be extracted from
the two domains.

Although existing DA methods have admittedly achieved promising results, most
features are encoded from the neural network without explicitly aligning the
features of the two domains. Feature distribution of two domains is still
difficult to align. Finally, when pseudo labeling is used on the target
domain, noisy labels are problematic. To address aforementioned challenges,
our contributions are three-fold:

  * •

To explicitly align the source and target domain extracted features, we are
the first to impose a feature registration loss to align these two features to
generate registered features, which can maintain both source and target domain
information.

  * •

To further match source and target feature distributions, we enforce a
histogram matching loss, which can reduce the domain discrepancy.

  * •

To suppress noisy pseudo labels in the target domain, we develop an easy-to-
hard refinement process that considers both probabilistic soft and center-
based hard selection. We then form a high-quality pseudo-labeled target domain
so as to jointly optimize the network and improve model performance.

##  II Related work

Given the popularity of deep neural networks, they have also shown great
success in the UDA problem [14, 15, 16]. The discrepancy based method is one
of the most popular deep network models, and it aims to minimize the
discrepancy between the source and target distributions by proposing different
loss functions, such as Maximum Mean Discrepancy (MMD) [17], CORrelation
ALignment [18], Kullback-Leibler divergence [12], Wasserstein distance [19]
and least squares [20]. Recent work included the modification of these
different distance functions. Kang et al. [21] extends MMD to the contrastive
domain discrepancy loss. Rahman et al. [22] proposes a model based on the
alignment of second-order statistics (covariances) as well as maximizing the
mean discrepancy of the source and target data. Sliced Wasserstein discrepancy
(SWD) [16] utilizes the geometric 1-Wasserstein as the discrepancy measure to
obtain the dissimilarity probability of source and target domains.

Adversarial methods are another popular model to reduce domain discrepancy of
different domains by using an adversarial objective with a domain
discriminator. Inspired by GANs [23], adversarial learning has shown its power
in learning domain invariant representations. It consists of a domain
discriminator and a feature extractor. The domain discriminator aims to
distinguish the source domain from the target domain, while the feature
extractor aims to learn domain-invariant representations to fool the domain
discriminator [24, 25, 13]. Domain Adversarial Neural Network (DANN) [26] is a
representative work. Later many adversarial learning methods are proposed.
SymNets [27] maximized the discrepancy between the outputs of the two
classifiers. The cycle-consistent adversarial domain adaptation CyCADA [28]
implemented domain adaptation at both pixel-level and feature-level by using
cycle-consistent adversarial training. TADA [29] was also built upon the
adversarial domain adaptation framework, which added adversarial alignment
constraints on both transferable local regions and global images through two
local/global attention modules.

Pseudo-labeling is another technique to improve the generalizability of the
model in the target domain. Pseudo-labeling typically generates pseudo labels
for the target domain based on the predicted class probability [30, 14, 31].
Therefore, some target domain label information can be included during
training, and further reduce domain divergence. In deep networks, the source
classifier usually generates the pseudo labels (and then uses them as if they
were real labels). Saito et al. [32] proposed an asymmetric tri-training
method for UDA to generate pseudo labels for target samples using two networks
while the third can learn from them to obtain target discriminative
representations. Zhang et al. [30] designed a new criterion to select pseudo-
labeled target samples and developed an incremental model (iCAN), in which
they select samples iteratively and retrain the network using the expanded
training set. PFAN [14] aligns the discriminative features across domains
progressively and develops iterative learning to generate pseudo labels. Zhang
et al. [33] offers a label propagation with augmented anchors (A2LP) method to
improve the label propagation via generation of unlabeled virtual samples with
high confidence label prediction. These methods highly rely on pseudo labels
to compensate for the lack of categorical information in the target domain.
However, they do not check the quality of pseudo-labels, as noisy pseudo-
labeled samples hurt model performance. Our work differs from these approaches
by generating high confidence examples using both probabilistic soft and
center-based hard selection in Sec.III-E.

##  III Methodology

###  III-A Problem

In this work, we consider the unsupervised domain adaptation (UDA)
classification problem in the following setting. There exists a labeled source
domain $\mathcal{D_{S}}=\\{x_{s}^{i},y_{s}^{i}\\}_{i=1}^{n_{s}}$ with $n_{s}$
samples in $C$ categories and an unlabeled target domain
$\mathcal{D_{T}}=\\{x_{t}^{j}\\}_{j=1}^{n_{t}}$ with $n_{t}$ samples in the
same $C$ categories. We assume that the data in the two domains are drawn from
different distributions but share the same label space. The goal of UDA is to
get a well-trained classifier so that domain discrepancy is minimized and
generalization error in the target domain is reduced.

In UDA, existing methods rarely update source domain features except when
learning them from the trained neural network. We instead directly replace the
source domain features with the registered features that can better represent
shared features between both source and target domains. We further align
feature distribution via histogram matching. The domain divergence can be
further reduced by the pseudo labeling refinement process of the target
domain.

Figure 1: The architecture of the DFR model. We first employ $F_{C}$ to
extract coarse features for two domains and then perform feature registration
between source ($F_{C}^{\mathcal{S}}$) and target ($F_{C}^{\mathcal{S}}$)
features. The registered features ($F_{G}^{0}$) and target features will
jointly optimize shared detailed feature extractor $F_{D}$. $\mathcal{L_{R}}$
is the feature registration loss, $\mathcal{L_{S}}$ is the source
classification loss, $\mathcal{L_{H}}$ is the histogram matching loss, and
$\mathcal{L_{T}}$ is the pseudo-labeled target domain classification loss.
$\\{Q(x_{t}^{j}),Q(y_{t}^{j})\\}_{j=1}^{n_{pt}}$ is the high quality pseudo-
labeled target domain after $T$ times pseudo labeling refinement processes
(Orange block: Linear layer, red block: ReLU layer, and cyan block:
BatchNormalization layer).

###  III-B Feature Registration

We first extract coarse features for both source and target images from a
shared coarse feature extractor $F_{C}$, which is a typical one of CNN
backbones. The motivation for feature registration is to find shared invariant
features (registered features $F_{G}^{0}$) between source features and target
features. Given extracted batch-wise source features
$B(F_{C}^{\mathcal{S}})\in\mathbb{R}^{n_{b}\times d}$ and extracted batch-wise
target features $B(F_{C}^{\mathcal{T}})\in\mathbb{R}^{n_{b}\times d}$, where
$n_{b}$ is the number of batch size and $d$ is extracted feature
dimensionality using $F_{C}$, we aim to generate registered features
$F_{G}^{0}$ using gradient descent. To estimate the registered features
$F_{G}^{0}$, we employ a stochastic gradient descent optimizer $\mathcal{N}$
as follows,

$F_{G}^{0}=\mathcal{N}(B(F_{C}^{\mathcal{S}}),B(F_{C}^{\mathcal{T}}),F^{0})$
(1)

where $F^{0}$ is the initial registered features and we initialize it as
$B(F_{C}^{\mathcal{T}})-B(F_{C}^{\mathcal{S}})$ to accelerate convergence.

To ensure the registered features $F_{G}^{0}$ contain shared target features’
information, we first impose a target domain registration loss
$\mathcal{L_{R}^{T}}$.

$\mathcal{L_{R}^{T}}=\sum_{p_{ij}\in\mathcal{W}}|F_{G}^{0}(p_{ij})-B(F_{C}^{\mathcal{T}}(p_{ij}))|,$
(2)

where $p_{ij}$ is the position in batch-wise feature matrix
$\mathcal{W}\in\mathbb{R}^{n_{b}\times d}\
(i\in\\{1,\cdots,n_{b}\\},j\in\\{1,\cdots,d\\}$) (for either $F_{G}^{0}$ or
$B(F_{C}^{\mathcal{T}})$) and $|\cdot|$ takes the absolute value. $F_{G}^{0}$
will be close to batch-wise target features $B(F_{C}^{\mathcal{T}})$ by
minimizing $\mathcal{L_{R}^{T}}$.

In the first iteration, the
$F_{G}^{0}=F^{0}=B(F_{C}^{\mathcal{T}})-B(F_{C}^{\mathcal{S}})$. The target
domain registration loss in Eq. 2 is the pixel-wise difference between
registered features and the batch-wise target domain features, which is able
to align the source and target domain features. However, a single
$\mathcal{L_{R}^{T}}$ might lead to $F_{G}^{0}$ is very similar to
$B(F_{C}^{\mathcal{T}})$, that will lose source domain features’ information.
To improve the quality of $\mathcal{L_{R}^{T}}$, which contains both shared
features between source and target domains, we then design source domain
registration loss:

$\mathcal{L_{R}^{S}}=\sum_{p_{ij}\in\mathcal{W}}|F_{G}^{0}(p_{ij})-B(F_{C}^{\mathcal{S}}(p_{ij}))|.$
(3)

Therefore, in feature registration, we employ the hybrid loss between the
source domain registration loss and the target domain registration loss to get
the registration features in the following equation.

$\displaystyle\mathcal{L_{R}}=\mathcal{L_{R}^{S}}+\alpha\mathcal{L_{R}^{T}}$
$\displaystyle=\sum_{p_{ij}\in\mathcal{W}}|F_{G}^{0}(p_{ij})-B(F_{C}^{\mathcal{S}}(p_{ij}))|+$
(4)
$\displaystyle\alpha\sum_{p_{ij}\in\mathcal{W}}|F_{G}^{0}(p_{ij})-B(F_{C}^{\mathcal{T}}(p_{ij}))|$

where $\alpha$ is a balance factor between the two loss functions. $F_{G}^{0}$
is generated for each pair of source and target mini-batches. The registered
features $F_{G}^{0}$ eventually maintain shared invariant features, which maps
from the source domain to the target domain.

###  III-C Source domain classifier

After feature registration, registered features $F_{G}^{0}$ will be fed into
the shared detailed feature extractor $F_{D}$, which can represent more
detailed features at the categorical level. The task in the labeled source
domain is formulated as:

$\mathcal{L_{S}}=-\frac{1}{n_{s}}\sum_{{\color[rgb]{1,0,0}b=1}}^{n_{s}/n_{b}}\sum_{i=1}^{n_{b}}\mathcal{L}_{ce}(F_{D}(F_{Gb}^{0i}),B_{{\color[rgb]{1,0,0}b}}(y_{s}^{i})),$
(5)

where $\mathcal{L}_{ce}$ is the typical cross-entropy loss. During training of
shared detailed feature extractor $F_{D}$, the registered features $F_{G}^{0}$
will replace batch-wise $B(F_{C}^{\mathcal{S}})$ and keep updating via Eq. 4,
and $b$ is batch-wise data.

###  III-D Histogram matching

To further reduce the domain divergence between source and target domains, we
employ a histogram matching step to align the distributions of the two domains
at the feature level. We assume that the histogram can represent the
distribution of the source and target domain features. Hence, aligning the
distributions of two domains is equivalent to minimizing the difference
between source and target histograms. Histogram matching usually refers to
transforming the histogram of one image so that it looks like another image.
One major limitation of histogram matching using original images for
minimizing domain discrepancy is that it assumes that images have similar
spectral properties when their pixel distributions are aligned. However, this
does not hold, especially when source images are different from target images.
Therefore, we perform the histogram matching at the feature level since
features after the detailed features extracted by $F_{D}$ are already aligned
to a certain extent. The basic principle of feature level histogram matching
is to compute the histogram source and target features individually, then
compute their discrete cumulative distribution functions (CDFs). Let
$Hist(\cdot)$ be histogram generation operation, and we expect the CDF of
registered features and target features are similar to each other. Therefore,
we develop the following histogram matching loss:

$\mathcal{L_{H}}=\sum_{h=1}^{H}|Hist(F_{D}(F_{G}^{0}))_{h}-Hist(F_{D}(B(F_{C}^{\mathcal{T}})))_{h}|,$
(6)

where $H$ is the number of bins of a histogram. By minimizing histogram
matching loss, we can further expect the distributions between two domains to
be aligned.

###  III-E Pseudo labeling refinement

Considering no labels in the target domain, existing methods also try to
generate pseudo labels for the target domain to improve the robustness of
their model. However, the detrimental effects of bad pseudo-labels are still
significant. To mitigate this issue, we employ a $T$ times recurrent easy-to-
hard pseudo-label refinement process to improve the quality of the pseudo-
labels in the target domain, which considers both probabilistic soft selection
and center-based hard selection.

The initial shared $F_{D}$ is optimized by $\mathcal{L_{S}}$ and
$\mathcal{L_{H}}$. For the inference, we can directly get predicted results
for one target domain sample $F_{D}(F_{C}(x_{t}^{j}))$. Let
$\text{Softmax}(F_{D}(F_{C}(x_{t}^{j})))$ be the predicted probability for
each class, and
$y_{pt}^{j}=max(\text{Softmax}(F_{D}(F_{C}(x_{t}^{j}))))_{index}$ be its
dominant class label, where $max(\cdot)_{index}$ return the index of the
maximum probability value. Therefore, for the probabilistic soft selection, a
higher quality pseudo label is defined as
$max(\text{Softmax}(F_{D}(F_{C}(x_{t}^{j}))))>p_{t}$, where $p_{t}$ is a
threshold probability in number of $t$ training. For $T$ times recurrent easy-
to-hard pseudo-label refinement, for easy examples, $p_{t}$ has a higher value
and for hard examples, $p_{t}$ has a lower value, hence
$p_{1}>p_{2}>\cdots>p_{T}$.

For center-based hard selection, let $\\{M_{S}^{c}\in\mathbb{R}^{1\times
C}\\}_{c=1}^{C}$ be the $C$ source cluster centers in the feature space after
$F_{D}$. Let $y_{s}==c$ mean all indices of source label class $c$; then all
source class $c$ features are $F_{D}(F_{C}(x_{s}^{y_{s}==c}))$ and $n_{c}$ is
the number of its features. We define the class center as:
$M_{S}^{c}=\frac{1}{n_{c}}\sum_{ii=1}^{n_{c}}F_{D}(F_{C}(x_{s}^{ii}))$. For
any target sample $x_{t}^{j}$, we compute its distance to each source domain
cluster center $M_{S}^{c}$ as
$dist_{j}=\\{|F_{D}(F_{C}(x_{t}^{j}))-M_{S}^{c}|_{L1}\\}_{c=1}^{C}\in\mathbb{R}^{1\times
C}$, where $|\cdot|_{L1}$ is the L1 norm. Therefore, a higher quality pseudo
label exists if
$min(dist_{j})_{index}==max(\text{Softmax}(F_{D}(F_{C}(x_{t}^{j}))))_{index}$,
where $min(dist_{j})_{index}$ returns the index of the cluster center with
minimum distance. Hence, the center-based hard selection ensures the target
sample has the same prediction target label as the closest source domain
center from the shared $F_{D}$.

In pseudo labeling refinement, we combine probabilistic soft and center-based
hard selection together to form a robust new high quality pseudo-labeled
domain $\\{Q(x_{t}^{j}),Q(y_{t}^{j})\\}_{j=1}^{n_{pt}}$ in the following
equation,

$\displaystyle\\{Q(x_{t}^{j}),Q(y_{t}^{j})\\}_{j=1}^{n_{pt}}\quad\text{if and
only if}$ (7) $\displaystyle(\text{Softmax}(F_{D}(F_{C}(x_{t}^{j})))>p_{t})\
\&(min(dist_{j})_{index}$
$\displaystyle==max(\text{Softmax}(F_{D}(F_{C}(x_{t}^{j}))))_{index}),$

where $Q(x_{t})$ and $Q(y_{t})$ represent the high quality target domain
samples and pseudo labels, $n_{pt}$ is the number of higher quality pseudo
labels for the target domain. We hence can mitigate detrimental effects of bad
pseudo-labels using Eq. 7. Similar to Eq. 5, we define the pseudo-labeled
target domain loss as:

$\mathcal{L_{T}}=-\frac{1}{n_{pt}}\sum_{b=1}^{n_{pt}/n_{b}}\sum_{j=1}^{n_{b}}\mathcal{L}_{ce}(F_{D}(B_{b}(F_{C}(Q(x_{t}^{j})))),B_{b}(Q(y_{t}^{j}))),$
(8)

where $B(\cdot)$ is also the batch-wise data.

###  III-F DFR model

Fig. 1 depicts the overall framework of our proposed DFR model. Considering
Sec. III-B to Sec. III-E, our model minimizes the following objective
function:

$\mathop{\operatorname*{arg\,min}}\
(\mathcal{L_{R}}+\mathcal{L_{S}}+\beta\mathcal{L_{H}}+\sum_{t=1}^{T}\mathcal{L}_{\mathcal{T}}^{t})$
(9)

where $\mathcal{L_{R}}$ is the feature registration loss, $\mathcal{L_{S}}$ is
the source classification loss, $\mathcal{L_{H}}$ is the histogram matching
loss, and $\mathcal{L}_{\mathcal{T}}$ is the pseudo-labeled target domain
classification loss, and we repeat the pseudo labeling refinement process $T$
times. $\beta$ controls the weight of histogram matching loss. The overall
training algorithm of our DFR model is shown in Alg. 1.

Algorithm 1 Deep Feature Registration Network. $B(\cdot)$ denotes the mini-
batch training sets, $I$ is the number of iterations. $T$ is the number of
refinement steps.

1: Input: labeled source samples
$\mathcal{D_{S}}=\\{x_{s}^{i},y_{s}^{i}\\}_{i=1}^{n_{s}}$ and unlabeled target
samples $\mathcal{D_{T}}=\\{x_{t}^{j}\\}_{j=1}^{n_{t}}$

2: Output: predicted target domain labels

3: repeat

4: Derive $B(F_{C}^{\mathcal{T}})$ and $B(F_{C}^{\mathcal{S}})$ sampled from
$\mathcal{D_{S}}$ and $\mathcal{D_{T}}$ after finely tuned $F_{C}$

5: for $iter=1$ to $I$ do

6: for $t=1$ to $T$ do

7: Generate registered features $F_{G}^{0}$ using Eq. (4)

8: Optimize $F_{D}$ using Eq. (5) and Eq. (6)

9: Form pseudo-labeled target domain
$\\{Q(x_{t}^{j}),Q(y_{t}^{j})\\}_{j=1}^{n_{pt}}$ using Eq. (7)

10: Refine $F_{D}$ using Eq. (8)

11: end for

12: end for

13: until converged

14: Make prediction for target domain samples based on trained $F_{C}$ and
$F_{D}$.

##  IV Experiments

###  IV-A Experimental Setup

#### Datasets.

We evaluate our model on three popular benchmark image datasets: Office-31,
Office-Home, and VisDA-2017. Office-31 [34] has 4,110 images from three
domains: Amazon (A), Webcam (W), and DSLR (D) in 31 classes. In experiments,
A$\shortrightarrow$W represents transferring knowledge from domain A to domain
W. Office-Home [35] dataset contains 15,588 images from four domains: Art
(Ar), Clipart (Cl), Product (Pr), and Real-World (Rw) in 65 classes.
VisDA-2017 [36] is a particularly challenging dataset due to a large domain-
shift between the synthetic images (152,397 images from VisDA) and the real
images (55,388 images from COCO) in 12 classes. We test our model on the
setting of synthetic-to-real as the source-to-target domain.

#### Implementation details.

We implement our approach using PyTorch and extract features for the three
datasets from finely tuned ($F_{C}$) ResNet50 (Office-31, Office-Home) and
ResNet101 (VisDA-2017) networks [37]. The 1,000 features are then extracted
from the last fully connected layer for the source and target features. The
outputs of three Linear layers are 512, 256 and $|C|$, respectively, where $C$
is the number of classes in each dataset. Parameters in recurrent pseudo
labeling are $T=3$ and $\\{p_{t}\\}_{t=1}^{3}=[0.9,0.6,0.3]$. Optimizer
(Adam), learning rate ($0.001$), batch size (64), $\alpha=0.6$, $\beta=0.01$
and number of epochs (210) are determined by performance on the source domain.
In histogram matching, 10 bins are used. More detailed parameter analysis is
presented in supplementary material. We compare our results with 20 state-of-
the-art methods. For a fair comparison, we directly report results from
original papers. Experiments are performed with an Nvidia GeForce 1080 Ti.

TABLE I: Accuracy (%) on Office-Home dataset (based on ResNet50) Task |  Ar$\shortrightarrow$Cl |  Ar$\shortrightarrow$Pr |  Ar$\shortrightarrow$Rw |  Cl$\shortrightarrow$Ar |  Cl$\shortrightarrow$Pr |  Cl$\shortrightarrow$Rw |  Pr$\shortrightarrow$Ar |  Pr$\shortrightarrow$Cl |  Pr$\shortrightarrow$Rw |  Rw$\shortrightarrow$Ar |  Rw$\shortrightarrow$Cl |  Rw$\shortrightarrow$Pr | Ave.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
TAT [38] | 51.6 | 69.5 | 75.4 | 59.4 | 69.5 | 68.6 | 59.5 | 50.5 | 76.8 | 70.9 | 56.6 | 81.6 | 65.8
TADA [29] | 53.1 | 72.3 | 77.2 | 59.1 | 71.2 | 72.1 | 59.7 | 53.1 | 78.4 | 72.4 | 60.0 | 82.9 | 67.6
SymNets [27] | 47.7 | 72.9 | 78.5 | 64.2 | 71.3 | 74.2 | 64.2 | 48.8 | 79.5 | 74.5 | 52.6 | 82.7 | 67.6
DMP [39] | 52.3 | 73.0 | 77.3 | 64.3 | 72.0 | 71.8 | 63.6 | 52.7 | 78.5 | 72.0 | 57.7 | 81.6 | 68.1
DCAN [40] | 54.5 | 75.7 | 81.2 | 67.4 | 74.0 | 76.3 | 67.4 | 52.7 | 80.6 | 74.1 | 59.1 | 83.5 | 70.5
SRDC [41] | 52.3 | 76.3 | 81.0 | 69.5 | 76.2 | 78.0 | 68.7 | 53.8 | 81.7 | 76.3 | 57.1 | 85.0 | 71.3
ESD [42] | 53.2 | 75.9 | 82.0 | 68.4 | 79.3 | 79.4 | 69.2 | 54.8 | 81.9 | 74.6 | 56.2 | 83.8 | 71.6
PICSCS [43] | 56.0 | 79.0 | 81.0 | 67.6 | 81.3 | 79.9 | 68.4 | 55.0 | 82.4 | 72.3 | 58.5 | 85.0 | 72.2
DFR | 56.2 | 81.2 | 83.6 | 70.4 | 82.2 | 80.3 | 71.1 | 56.3 | 82.3 | 76.5 | 58.4 | 86.2 | 73.7
TABLE II: Accuracy (%) on VisDA-2017 dataset (based on ResNet101) Task | plane | bcycl | bus | car | horse | knife | mcycl | person | plant | sktbrd | train | truck | Ave.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source-only [37] | 55.1 | 53.3 | 61.9 | 59.1 | 80.6 | 17.9 | 79.7 | 31.2 | 81.0 | 26.5 | 73.5 | 8.5 | 52.4
DANN [26] | 81.9 | 77.7 | 82.8 | 44.3 | 81.2 | 29.5 | 65.1 | 28.6 | 51.9 | 54.6 | 82.8 | 7.8 | 57.4
DAN [44] | 87.1 | 63.0 | 76.5 | 42.0 | 90.3 | 42.9 | 85.9 | 53.1 | 49.7 | 36.3 | 85.8 | 20.7 | 61.1
JAN [9] | 75.7 | 18.7 | 82.3 | 86.3 | 70.2 | 56.9 | 80.5 | 53.8 | 92.5 | 32.2 | 84.5 | 54.5 | 65.7
DMP [39] | 92.1 | 75.0 | 78.9 | 75.5 | 91.2 | 81.9 | 89.0 | 77.2 | 93.3 | 77.4 | 84.8 | 35.1 | 79.3
DADA [15] | 92.9 | 74.2 | 82.5 | 65.0 | 90.9 | 93.8 | 87.2 | 74.2 | 89.9 | 71.5 | 86.5 | 48.7 | 79.8
STAR [45] | 95.0 | 84.0 | 84.6 | 73.0 | 91.6 | 91.8 | 85.9 | 78.4 | 94.4 | 84.7 | 87.0 | 42.2 | 82.7
CAN [21] | 97.9 | 87.2 | 82.5 | 74.3 | 97.8 | 96.2 | 90.8 | 80.7 | 96.6 | 96.3 | 87.5 | 59.9 | 87.2
CDCL [46] | 97.3 | 90.5 | 83.2 | 59.9 | 96.4 | 98.4 | 91.5 | 85.6 | 96.0 | 95.8 | 92.0 | 63.8 | 87.5
DFR | 97.2 | 89.6 | 85.1 | 77.6 | 98.5 | 96.7 | 91.6 | 82.2 | 96.8 | 96.8 | 88.4 | 60.3 | 88.4
TABLE III: Accuracy (%) on Office-31 dataset (based on ResNet50) Task |  A$\shortrightarrow$W |  A$\shortrightarrow$D |  W$\shortrightarrow$A |  W$\shortrightarrow$D |  D$\shortrightarrow$A |  D$\shortrightarrow$W | Ave.
---|---|---|---|---|---|---|---
GSM [2] | 84.8 | 82.7 | 73.5 | 96.6 | 70.9 | 95.0 | 83.9
JAN [9] | 85.4 | 84.7 | 70.0 | 99.8 | 68.6 | 97.4 | 84.3
DMP [39] | 93.0 | 91.0 | 70.2 | 100 | 71.4 | 99.0 | 87.4
TADA [29] | 94.3 | 91.6 | 73.0 | 99.8 | 72.9 | 98.7 | 88.4
SymNets [27] | 90.8 | 93.9 | 72.5 | 100 | 74.6 | 98.8 | 88.4
TAT [38] | 92.5 | 93.2 | 73.1 | 100 | 73.1 | 99.3 | 88.5
CAN [21] | 94.5 | 95.0 | 77.0 | 99.8 | 78.0 | 99.1 | 90.6
SRDC [41] | 95.7 | 95.8 | 77.1 | 100 | 76.7 | 99.2 | 90.8
PICSCS [43] | 93.2 | 93.6 | 78.0 | 100 | 77.1 | 99.1 | 90.2
DFR | 95.9 | 96.3 | 80.6 | 100 | 79.6 | 99.3 | 92.0
TABLE IV: Ablation experiments on Office-31 dataset Task |  A$\shortrightarrow$W |  A$\shortrightarrow$D |  W$\shortrightarrow$A |  W$\shortrightarrow$D |  D$\shortrightarrow$A |  D$\shortrightarrow$W | Ave.
---|---|---|---|---|---|---|---
DFR$-$H/T/R | 85.7 | 89.4 | 75.0 | 97.1 | 75.8 | 95.1 | 86.4
DFR$-$R/T | 86.9 | 90.1 | 75.3 | 97.6 | 76.9 | 96.1 | 87.2
DFR$-$H/R | 91.2 | 90.9 | 77.2 | 98.0 | 77.8 | 96.3 | 88.6
DFR$-$H/T | 91.5 | 91.8 | 77.6 | 98.3 | 78.3 | 96.4 | 89.0
DFR$-$R | 93.5 | 94.1 | 78.2 | 99.0 | 78.5 | 97.0 | 90.1
DFR$-$T | 94.8 | 95.6 | 79.0 | 99.0 | 79.0 | 97.2 | 90.8
DFR$-$H | 95.5 | 96.0 | 79.4 | 99.2 | 79.2 | 98.4 | 91.3
DFR | 95.9 | 96.3 | 80.6 | 100 | 79.6 | 99.3 | 92.0

###  IV-B Results

The results on Office-Home, VisDA-2017, and Office-31 datasets are shown in
Tables I-III. Overall, our DFR model outperforms all methods in terms of
average accuracy and in almost all tasks of the three datasets. Notably, our
DFR model substantially improves classification accuracy on difficult
adaptation tasks (e.g., W$\shortrightarrow$A, D$\shortrightarrow$A task in the
Office-31 dataset and the challenging VisDA-2017 and Office-Home datasets,
which have a larger number of categories and domain discrepancy). For the
Office-31 dataset, the mean accuracy is 92.0%, which achieves state-of-the-art
results. Compared with the best baseline (SRDC), our DFR model provides a 1.2%
improvement. Notably, DFR wins all tasks except for W$\shortrightarrow$A task.
Similarly, in Office-Home and VisDA-2017 datasets, our model is almost ahead
of all tasks except Pr$\shortrightarrow$Cl task and plane category. For the
Office-Home dataset, DFR achieves the highest average, 73.7%, which is 1.5%
higher than the PICSCS method and 2.4% higher than the DCAN model. For the
VisDA-2017 dataset, the DFR model has a 0.9% improvement over the best
baseline (CDCL). Therefore, our proposed feature registration and histogram
matching are useful, and the easy-to-hard refinement process is effective in
improving classification accuracy.

Figure 2: Feature visualization using a 2D t-SNE view of task
D$\shortrightarrow$A in Office-31 dataset. Without (a) and with (b) feature
registration (before $F_{D}$). We highlight obvious changes using black
ellipses. The registered features have less dispersivity after performing
feature registration. (c): after performing histogram matching and (d): after
performing pseudo labeling refinement step. (blue color: source features, red
color: target features, green color: registered features based on source
domain). Best viewed in color. Figure 3: Histogram comparison of one sample in
task Pr$\shortrightarrow$Rw in Office-Home dataset. Our histogram matching
strategy achieves the smallest difference between source and target feature
histograms. (blue color: source domain feature histogram, yellow color: target
feature histogram).

###  IV-C Ablation study

To demonstrate the effects of different loss functions on final classification
accuracy, we conduct an ablation study w.r.t. to each component of our
proposed DFR model in Tab. IV. For this, we reuse ResNet50 on the Office-31
dataset of twelve tasks. R represents feature registration loss
$\mathcal{L_{R}}$, H is the histogram matching loss $\mathcal{L_{H}}$, and T
is the pseudo-labeled target domain loss $\mathcal{L_{T}}$. Notice that source
classification loss $\mathcal{L_{S}}$ is required for UDA. “DFR$-$H/T/R” is
implemented without $\mathcal{L_{H}}$, $\mathcal{L_{T}}$, and
$\mathcal{L}_{\mathcal{R}}$. It is a simple model, which only reduces the
source risk without minimizing the domain discrepancy using $\mathcal{L_{S}}$.
“DFR$-$H/T” only performs the feature registration process. “DFR$-$H” reports
results without performing histogram matching. We can find that with the
increasing number of loss functions, the accuracy of our model keeps
improving. The effective of loss functions on classification accuracy is
ordered as $\mathcal{L}_{\mathcal{R}}>\mathcal{L_{T}}>\mathcal{L_{H}}$.
Therefore, our feature registration, histogram matching, and easy-to-hard
target domain pseudo labeling refinement approaches are effective in
minimizing target domain risk and improving the accuracy.

###  IV-D Discussion

From results in Sec. IV-B, our DFR always achieves the highest average
accuracy, outperforming SOTA methods. There are three compelling reasons.
First, we perform feature registration, which can map the source features to
the target features and minimize the domain discrepancy before $F_{D}$. To
better understand this, we show the 2D t-SNE feature visualization without and
with feature registration in Fig. 2 using the task of D$\shortrightarrow$A in
the Office-31 dataset. Fig. 2(a) is the extracted source and target features
from $F_{C}$ without performing feature registration. After minimizing
$\mathcal{L_{R}}$, we get all registered features based on source domain in
Fig. 2(b), while target features are not changed (red color). Clearly, the
points in the highlighted black ellipses in Fig. 2(b) are closer to each other
than that in Fig. 2(a). Secondly, we propose a histogram matching method to
further align the distribution between two domains. To show the advantages of
our proposed histogram matching loss $\mathcal{L_{H}}$, we compare the
histogram of another two well-known distance-based methods (MMD [44] and CORAL
[18] in Fig. 3. We directly report the value of $\mathcal{L_{H}}$, which can
represent the difference between source and target histograms. A smaller
$\mathcal{L_{H}}$ means a smaller domain discrepancy between two domains. We
find that our histogram matching strategy has the smallest value among all
other methods. Thirdly, we develop an easy-to-hard refinement process to
improve the quality of pseudo labels in the target domain. This strategy
considers both probabilistic soft and center-based hard selection, and it
hence can push the shared $F_{D}$ towards the target domain. As discussed in
Sec. IV-C, the $T$ times easy-to-hard refinement process is effective in
improving the classification accuracy and further reduces the domain
discrepancy. We also show t-SNE feature visualization after histogram matching
and pseudo labeling refinement in Fig. 2(c) and Fig. 2(d). We can find that
Fig. 2(d) is our DFR model, and it can learn highly discriminative features
and keep clear boundaries, especially in the central portion.

##  V Conclusion

We have proposed a novel deep feature registration (DFR) model for UDA. To
align source and target domain features, we develop a feature registration
loss to minimize the intrinsic discrepancy between two domain features. To
further align domain distributions, we design a histogram matching loss. We
also employ an easy-to-hard refinement process, which combines both
probabilistic soft and center-based hard selection and can improve the quality
of pseudo labels. Extensive experiments demonstrate that the proposed DFR
model achieves higher accuracy than state-of-the-art methods.

## References

  * [1] B. Gong, Y. Shi, F. Sha, and K. Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2066–2073. IEEE, 2012.
  * [2] Y. Zhang, S. Xie, and B. D. Davison.  Transductive learning via improved geodesic sampling.  In Proceedings of the 30th British Machine Vision Conference, 2019\.
  * [3] M. Jiang, W. Huang, Z. Huang, and G. G. Yen.  Integration of global and local metrics for domain adaptation learning via dimensionality reduction.  IEEE Transactions on Cybernetics, 47(1):38–51, 2017.
  * [4] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, and P. S. Yu.  Visual domain adaptation with manifold embedded distribution alignment.  In Proceedings of the 26th ACM International Conference on Multimedia, MM ’18, pages 402–410, 2018.
  * [5] Y. Zhang and B. D. Davison.  Deep spherical manifold gaussian kernel for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 4443–4452, 2021.
  * [6] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu.  Transfer joint matching for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1410–1417, 2014.
  * [7] M. Gong, K. Zhang, T. Liu, D. Tao, C. Glymour, and B. Schölkopf.  Domain adaptation with conditional transferable components.  In Proceedings of the International Conference on Machine Learning, pages 2839–2848, 2016.
  * [8] J. Zhang, W. Li, and P. Ogunbona.  Joint geometrical and statistical alignment for visual domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1859–1867, 2017.
  * [9] M. Long, H. Zhu, J. Wang, and M. I. Jordan.  Deep transfer learning with joint adaptation networks.  In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 2208–2217. JMLR.org, 2017.
  * [10] Y. Zhang and B. D. Davison.  Impact of ImageNet model selection on domain adaptation.  In Proceedings of the IEEE Winter Conference on Applications of Computer Vision Workshops, pages 173–182, 2020.
  * [11] Y. Zhang and B. D. Davison.  Efficient pre-trained features and recurrent pseudo-labeling in unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 2719–2728, 2021.
  * [12] Z. Meng, J. Li, Y. Gong, and B. Juang.  Adversarial teacher-student learning for unsupervised domain adaptation.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5949–5953. IEEE, 2018.
  * [13] Y. Zhang, H. Ye, and B. D. Davison.  Adversarial reinforcement learning for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 635–644, 2021.
  * [14] C. Chen, W. Xie, W. Huang, Y. Rong, X. Ding, Y. Huang, T. Xu, and J. Huang.  Progressive feature alignment for unsupervised domain adaptation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pages 627–636, 2019.
  * [15] H. Tang and K. Jia.  Discriminative adversarial domain adaptation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5940–5947, 2020.
  * [16] C. Lee, T. Batra, M. H. Baig, and D. Ulbricht.  Sliced wasserstein discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10285–10295, 2019.
  * [17] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474, 2014.
  * [18] B. Sun and K. Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In Proc. of European Conference on Computer Vision, pages 443–450. Springer, 2016.
  * [19] B. Bhushan Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty.  Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 447–463, 2018.
  * [20] Y. Zhang and B. D. Davison.  Deep least squares alignment for unsupervised domain adaptation.  Proceedings of the 32th British Machine Vision Conference, 2021\.
  * [21] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4893–4902, 2019.
  * [22] M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan.  On minimum discrepancy estimation for deep domain adaptation.  In Domain Adaptation for Visual Understanding, pages 81–94. Springer, 2020.
  * [23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
  * [24] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky.  Domain-adversarial training of neural networks.  The Journal of Machine Learning Research, 17(1):2096–2030, 2016\.
  * [25] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  * [26] M. Ghifary, W. B. Kleijn, and M. Zhang.  Domain adaptive neural networks for object recognition.  In Proceedings of the Pacific Rim International Conference on Artificial Intelligence, pages 898–904. Springer, 2014.
  * [27] Y. Zhang, H. Tang, K. Jia, and M. Tan.  Domain-symmetric networks for adversarial domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5031–5040, 2019.
  * [28] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [29] X. Wang, L. Li, W. Ye, M. Long, and J. Wang.  Transferable attention for domain adaptation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5345–5352, 2019.
  * [30] W. Zhang, W. Ouyang, W. Li, and D. Xu.  Collaborative and adversarial network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3801–3809, 2018.
  * [31] Y. Zhang and B. D. Davison.  Adversarial continuous learning in unsupervised domain adaptation.  In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part II, pages 672–687. Springer International Publishing, 2021.
  * [32] K. Saito, Y. Ushiku, and T. Harada.  Asymmetric tri-training for unsupervised domain adaptation.  arXiv preprint arXiv:1702.08400, 2017.
  * [33] Y. Zhang, B. Deng, K. Jia, and L. Zhang.  Label propagation with augmented anchors: A simple semi-supervised learning baseline for unsupervised domain adaptation.  In European Conference on Computer Vision, pages 781–797. Springer, 2020.
  * [34] K. Saenko, B. Kulis, M. Fritz, and T. Darrell.  Adapting visual category models to new domains.  In Proceedings of the European Conference on Computer Vision, pages 213–226. Springer, 2010.
  * [35] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  * [36] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [38] H. Liu, M. Long, J. Wang, and M. Jordan.  Transferable adversarial training: A general approach to adapting deep classifiers.  In International Conference on Machine Learning, pages 4013–4022, 2019.
  * [39] Y. Luo, C. Ren, D. Dao-Qing, and H. Yan.  Unsupervised domain adaptation via discriminative manifold propagation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [40] S. Li, C. H. Liu, Q. Lin, B. Xie, Z. Ding, G. Huang, and J. Tang.  Domain conditioned adaptation network.  In AAAI, pages 11386–11393, 2020.
  * [41] Hui Tang, Ke Chen, and Kui Jia.  Unsupervised domain adaptation via structurally regularized deep clustering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8725–8735, 2020.
  * [42] Y. Zhang and B. D. Davison.  Enhanced separable disentanglement for unsupervised domain adaptation.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 784–788. IEEE, 2021.
  * [43] Lei Li, Jun Yang, Yulin Ma, and Xuefeng Kong.  Pseudo-labeling integrating centers and samples with consistent selection mechanism for unsupervised domain adaptation.  Information Sciences, 628:50–69, 2023.
  * [44] M. Long, Y. Cao, J. Wang, and M. I. Jordan.  Learning transferable features with deep adaptation networks.  arXiv preprint arXiv:1502.02791, 2015.
  * [45] Z. Lu, Y. Yang, X. Zhu, C. Liu, Y. Song, and T. Xiang.  Stochastic classifiers for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9111–9120, 2020.
  * [46] Xin Luo, Wei Chen, Zhengfa Liang, Chen Li, and Yusong Tan.  Adversarial style discrepancy minimization for unsupervised domain adaptation.  Neural Networks, 157:216–225, 2023.