# Variational Transfer Learning using Cross-Domain Latent Modulation

Jinyong Hou  Jeremiah D. Deng<EMAIL_ADDRESS>Stephen Cranefield
Xuejie Ding  Department of Information Science, University of Otago, 60 Clyde
Street, Dunedin, New Zealand

###### Abstract

To successfully apply trained neural network models to new domains, powerful
transfer learning solutions are essential. We propose to introduce a novel
cross-domain latent modulation mechanism to a variational autoencoder
framework so as to achieve effective transfer learning. Our key idea is to
procure deep representations from one data domain and use it to influence the
reparameterization of the latent variable of another domain. Specifically,
deep representations of the source and target domains are first extracted by a
unified inference model and aligned by employing gradient reversal. The
learned deep representations are then cross-modulated to the latent encoding
of the alternative domain, where consistency constraints are also applied. In
the empirical validation that includes a number of transfer learning benchmark
tasks for unsupervised domain adaptation and image-to-image translation, our
model demonstrates competitive performance, which is also supported by
evidence obtained from visualization.

###### keywords:

variational autoencoders, conditional generation, adversarial learning,
unsupervised domain adaption, image translation

##  1 Introduction

In machine learning, one can rarely directly apply a pre-trained model to a
new dataset or a new task. The performance of the learned model often plunges
significantly, as the new data may have some significant sampling bias, or
simply belong to a very different probability distribution. To tackle this
issue, transfer learning has been intensively studied in recent years, which
aims to utilize the learned knowledge from a previous domain (the “source”) to
improve performance on a related domain or task (the “target”) [1, 2, 3].

From the perspective of probabilistic modelling [4, 5, 6], the key challenge
in achieving cross-domain transfer is to learn a joint distribution of data
from different domains. Once the joint distribution is learned, it can
generate the marginal distribution of the individual domains [7, 4]. Under the
variational inference scenario, an inferred joint distribution is often
applied to the latent space. According to the coupling theory [8, 9],
inferring the joint distribution from the marginal distributions of different
domains is a highly ill-posed problem. To address this problem, UNIT [4]
assumes that there is a shared latent space for the two domains. This can be
achieved by applying the adversarial strategy to the domains’ latent spaces.
Another approach is to use a complex prior to improve the input data’s
representation performance [10, 11, 12]. However, few works have looked into
the role of the generation process for the latent space, which we argue could
help cross-domain transfer scenarios.

This paper presents a novel latent space reparameterization framework and
employs a generative process to cater for the cross-domain transferability.
Specifically, we first propose a transfer latent space (TLS) to build a
general cross-domain latent space. We then incorporate a cross-domain feature
into the reparameterization transformation, which builds the connection
between the variational representations and domain features in a cross-
modulation manner. The generated transfer latent space is further tuned by
domain-level adversarial alignment and inter-class alignment by cross-
consistency between images obtained through reconstruction and generations. We
call our model “Cross-Domain Latent Modulation” (CDLM). The latent modulation
could be regarded as a specific instance of the proposed TLS. In this paper,
we explore three different modulations as examples of transfer latent space
and analyze them in detail. For experiments, the CDLM model is applied to the
homogeneous transfer scenarios including unsupervised domain adaptation and
image-to-image translation, where highly competitive performance is reported.
Further supportive evidence obtained from ablation studies and visualization
is also presented.

This paper is extended from a conference publication [13], where the TLS-based
framework was first presented, and some preliminary experimental results were
provided. Here in this work, we give a detailed presentation of the entire
framework, with the following additional technical and experimental
contributions introduced: (1) We have extended the proposed framework to allow
for both complete and partial modulations, and given some examples for
different scenarios; also new experimental results are obtained for
performance evaluations; (2) We present the graphical model of the CDLM
framework and give a geometric illustration to the operations in the transfer
space in effect; and (3) an analysis on the generalization bound of CDLM is
given. For the empirical study we have also included more state of the art
methods for comparison, with new performance results and visualization outcome
reported.

The rest of the paper is organized as follows. In Section 2, some related work
is briefly reviewed. In Section 3, we outline the overall structure of our
proposed model and develop the learning metrics with defined losses. The
experiments are presented and discussed in Section 4. We conclude our work in
Section 5, indicating our plan of future work.

##  2 Related Work

As discussed above, for a joint distribution, manipulation of the latent space
is common [14, 4, 15] for cross-domain adaptation scenarios, as shown in Fig.
1. One approach focuses on a shared latent space, where the latent encodings
are regarded as common representations for inputs across domains. Some
adversarial strategy is usually used to pull them together so that the
representations are less domain-dependent than their original ones.

Figure 1: Illustration of the latent space manipulation. We can get the latent
space by a deep autoencoder structure ($\mathbf{E}_{u}$ is for encoder and
$\mathbf{D}$ for decoder). Our aim is then to design a transformation to
transfer the latent space ($\bm{z}_{tr}$) for the source and the target. The
source ($\bm{x}_{s},\bm{\hat{}}{x}_{s}$) is in pink and the target
($\bm{x}_{t},\bm{\hat{}}{x}_{t}$) is in blue.

Under the variational inference context, we assume a prior for the estimated
latent space. It is easy to confine the joint distribution into the same space
and apply the adversarial strategy. For example, works [4, 6, 15] assume a
standard Gaussian prior for the transfer. Also, we can do more to manipulate
the variational latent space by the prior. Works in [10, 16, 17] adopt
normalizing flows as complex priors for multimodal latent representations.

In [17], inverse autoregressive flow (IAF) is proposed for the variational
inference, which scales well to high-dimensional latent spaces. IAF is
constructed by the successive transformations based on the autoregressive
neural network. It improves upon diagonal Gaussian approximation posterior
significantly. Reference [16] introduces the normalizing flow for discrete
sequences. It utilizes a VAE-based generative model that jointly learns a
normalizing flow distribution in the latent space and a stochastic mapping to
an observed discrete space. To this end, it is important that the flow-based
distribution be highly multi-modal. Another interesting application is for
cross-domain many-to-many mapping described in [10]. It learns shared joint
representations between images and text, also domain-specific information
separately. The normalizing flow-based prior is used for the domain-specific
information to learn various many-to-many mappings.

Gaussian mixture [18] is also employed as prior of the variational latent
space for a better cluster performance. Reference [19] proposes a variant of
variational autoencoders (VAEs) with a Gaussian mixture prior to improve the
unsupervised clustering performance. In [5], conditional variational
autoencoders [20] (CVAEs) is utilized with a Gaussian mixture and a novel
additive Gaussian prior to create a prior for images that contain different
content simultaneously. It can enhance the image caption generation’s
variability. Reference [21] improves the deep clustering using Gaussian
mixture variational autoencoders(VAEs) with graph embedding. The Gaussian
mixture model (GMM) is used for the local multi-clustering latent space, and
the graph structure is for the global information. The combination can learn
powerful representations for the inputs. In summary, the Gaussian mixture can
be considered as a conditional prior for a complex latent space in these
models.

Another approach is to use disentangled latent representations where the
latent encoding is divided into some defined parts (e.g. style and content
parts), then the model learns separated representations and swaps them for the
transfer [22, 23, 24, 25, 26]. For example, in [25], images are mapped into a
domain-invariant content space and a domain-specific attribute space for each
domain separately. There are then different encoders for the source and the
target, respectively. Each domain has its decoder. However, the latent
encoding is done in a cross-domain manner by swapping the learned content
between domains. The reconstructions with swapped content are re-encoded in a
cross-domain manner for cycle-consistency. During the image mapping, an
adversarial strategy is utilized for the unsupervised alignment.

Reference [22] has a similar strategy. It splits the encodings into similar
and different parts for each domain, extracted by two distinguishing encoders.
The combination of the learned domain-invariant and domain-private information
is fed into a shared decoder for the reconstructions. The model uses a squared
Frobenius norm as a difference regularization, adversarial strategy, and MMD
for the shared information. In [26], a novel network structure is proposed
with a generative adversarial network (GANs) and cross-domain autoencoder. It
aims to separate the internal representation into three parts: shared and
exclusive parts for domains by paralleled variational autoencoders.

The manipulation on the latent space is often interwoven with the homogeneous
image transfer together, such as unsupervised domain adaptation and image
translation [27, 28, 29]. In the domain separation networks [22], separate
encoding modules are employed to extract the invariant representation and
domain-specific representations from the domains respectively, with the
domain-invariant representations being used for the domain adaptation.
References [30, 31, 32] transfer the target images into source-like images for
domain adaptation. Works in [4, 33, 34] map the inputs of different domains to
a single shared latent space, but cycle consistency is required for the
completeness of the latent space. The UFDN [15] utilizes a unified encoder to
extract the multi-domain images to a shared latent space, and the latent
domain-invariant coding is manipulated for image translation between different
domains.

Our method is different from these approaches. In our model, a learned deep
auxiliary representation is used to generate perturbations to the latent space
through a modified reparameterization trick using variational information from
the counterpart domain. It helps generate cross-domain image translation. The
transfer is carried out by a reparameterization transformation, using
statistical moments retaining specific information for one domain and deep
representation providing information from another domain. Our model adopts the
pixel-level adaptation between domains from the cross-domain generation, but
the proposed model can also be used at the feature-level due to the latent
space alignment. Our model also has a unified inference model, but the
consistency is imposed straightforwardly, reducing computational complexity.

##  3 The Proposed Model

Having reviewed the strategies to manipulate the latent space, we will now
present our model in detail. First, we will describe the problem being solved
in this paper. The proposed transfer latent space (TLS) and modulation
background are then described. Following them, we demonstrate our model and
discuss some specific instances of the transfer latent space (TLS). Finally,
our model’s learning process is presented.

######  Problem 3.1.

Let $\mathcal{X}\subset\mathbb{R}^{d}$ be a $d$-dimensional data space, and
$\mathbf{X}=\\{\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{n}\\}\in\mathcal{X}$ the
sample set with marginal distribution $p(\mathbf{X})$. The source domain is
denoted by a tuple $(\mathcal{X}_{s},p(\mathbf{X}_{s}))$, and the target
domain by $(\mathcal{X}_{t},p(\mathbf{X}_{t}))$. In our paper, we consider the
homogeneous transfer with domain shift, i.e.
$\mathcal{X}_{s}\approx\mathcal{X}_{t}$, but $p(\mathbf{X}_{s})\neq
p(\mathbf{X}_{t})$. For the unsupervised pixel-level domain adaptation
scenario, the label set is $\mathbf{Y}=\\{y_{1},y_{2},\ldots
y_{n}\\}\in\mathcal{Y}$ ($\mathcal{Y}$ is the label space), and a task
$\mathcal{T}=p(\mathbf{Y}|\mathbf{X})$ is considered too. However, only the
source domain’s label set $\mathbf{Y}_{s}$ is available during transfer
learning.

###  3.1 Transfer latent space

As an infinite number of joint distributions can yield any given marginal
distribution, we need to build an inference framework with some constraints.
Under the variational autoencoders (VAEs) framework, the latent space is one
of the manipulation targets. We propose the transfer latent space (TLS) as
follows.

######  Definition 3.1 (Transfer Latent Space $\ddot{\bm{Z}}$).

Let $\bm{x}_{s}\in\mathbf{X}_{s}$, $\bm{x}_{t}\in\mathbf{X}_{t}$ be the domain
samples. Let us have a map $f$ that extracts domain information $\bm{\Omega}$
and a deep feature representation $\bm{h}$ given an input $\bm{x}$:

$f\colon\bm{x}\longrightarrow(\bm{\Omega},\bm{h}),\bm{x}\in\mathbf{X}_{s}\cup\mathbf{X}_{t}.$

Suppose we construct a transfer map $\mathcal{G}$ that generates a latent
variable $\bm{\ddot{z}}$ from $\bm{\Omega}$ and $\bm{h}$ with domain
crossovers:

$\bm{\ddot{z}}_{st}=\mathcal{G}(\bm{\Omega}_{s},\bm{h}_{t}),$
$\bm{\ddot{z}}_{ts}=\mathcal{G}(\bm{\Omega}_{t},\bm{h}_{s}).$

The joint space formed by $\bm{\ddot{z}}_{st}$ and $\bm{\ddot{z}}_{ts}$
samples is defined as the transfer latent space, denoted by $\ddot{\bm{Z}}$.

The transfer latent space (TLS) is intended to become a “mixer” for the two
domains, as the resulted latent variables are under cross-domain influences.
Therefore, our transfer latent space can be regarded as a generalization of
the latent space under cross-domain settings.

The TLS definition finds an analogy in telecommunications: modulation, which
is a technique that enables data to be transmitted using a carrier signal that
can propagate through the communication channel. In the case of frequency
modulation, the frequency of the transmission signal is composed by the
carrier frequency $f_{c}$ and the data bit $b$:

$f=f_{c}+b\delta,$

where $\delta$ is a frequency shift. Similarly, our TLS attempts to build a
cross-domain latent space by a “modulation” between the $\Omega$ and $\bm{h}$.
Therefore, we call our model “Cross-Domain Latent Modulation”, or CDLM for
short.

###  3.2 The CDLM framework

####  3.2.1 Overview

Our proposed CDLM framework is shown in Fig. 2. In the framework, we build the
cross-domain generation by a unified inference model
$\mathbf{E}_{\bm{\phi}}(\cdot)$ (as an implementation of the $f$ map) and a
generative model for the desired domain $\mathbf{D}_{\bm{\theta}}(\cdot)$,
e.g., the source domain in our model. A discriminator $\Xi$ is utilized for
the adversarial training. We use the terms “inference model” and “encoder” for
$\mathbf{E}_{\bm{\phi}}(\cdot)$, and “generative model” and “decoder” for
$\mathbf{D}_{\bm{\theta}}(\cdot)$ interchangeably.

Figure 2: Architectural view of the proposed model [13]. It encourages an
image from the target domain (blue hexagon) to be transformed to a
corresponding image in the source domain (black hexagon). The transfer latent
distributions $p(\bm{\ddot{z}}_{ts}|\bm{x}_{t},\bm{h}_{s})$ and
$p(\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{h}_{t})$ are learned and are used to
generate corresponding images by the desired decoder. The deep representations
are integrated into the reparameterization transformation with standard
Gaussian auxiliary noise. Blue lines are for the target domain and black ones
are for the source domain.

As discussed in Section 3.1, under the variational framework, the domain
information $\bm{\Omega}$ (here we remove the domain subscript for simplicity)
is usually the Gaussian distribution parameters pair $(\bm{\mu},\bm{\sigma})$.
Let $\bm{h}^{\prime}$ be the flattened activations of the last convolution
layer in $\mathbf{E}_{\bm{\phi}}$. Then, following the treatment in [35],
$\bm{\mu}$ and $\bm{\sigma}$ can be obtained by
$\bm{\mu}=\mathbf{W}_{\bm{\mu}}\bm{h}^{\prime}+\bm{b}_{\bm{\mu}}$ and
$\bm{\sigma}=\mathbf{W}_{\bm{\sigma}}\bm{h}^{\prime}+\bm{b}_{\bm{\sigma}}$,
where
$\mathbf{W}_{\bm{\mu}},\mathbf{W}_{\bm{\sigma}},\bm{b}_{\bm{\mu}},\bm{b}_{\bm{\sigma}}$
are the weights and biases for $\bm{\mu}$ and $\bm{\sigma}$.

From our observations, both shallow (e.g. PCA features) and deep
representations can be used to obtain domain information $\bm{h}$. In our end-
to-end model we use the latter. We choose the activations of the last
convolutional layer, i.e.,
$\bm{h}=\text{sigmoid}(\mathbf{W}_{\bm{h}}\bm{h}^{\prime}+\bm{b}_{\bm{h}})$ as
the high-level representation [36], where
$\mathbf{W}_{\bm{h}},\bm{b}_{\bm{h}}$ are the weights and biases for the deep
abstractions.

####  3.2.2 Cross-domain modulation

Having obtained the domain information $\bm{\Omega}$ and deep representation
$\bm{h}$, a natural choice for the transfer map $\mathcal{G}$ is through
reparameterization. We now propose a modified reparameterization trick to give
the sampling from the transfer latent space, as follows:

$\bm{\ddot{z}}_{st}=\mathcal{G}((\bm{\mu}_{s},\bm{\sigma}_{s}),\bm{h}_{t})=\bm{\mu}_{s}+\bm{\sigma}_{s}\odot(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon}),$
(1)

and

$\bm{\ddot{z}}_{ts}=\mathcal{G}((\bm{\mu}_{t},\bm{\sigma}_{t}),\bm{h}_{s})=\bm{\mu}_{t}+\bm{\sigma}_{t}\odot(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon}),$
(2)

where $\bm{h}_{s}$ ($\bm{h}_{t}$) is the sample of the deep representation
space $\mathcal{H}_{s}$ ($\mathcal{H}_{t}$); $\bm{\mu}_{s}$ and
$\bm{\sigma}_{s}$ ($\bm{\mu}_{t}$ and $\bm{\sigma}_{t}$) are the mean and
standard deviation of the approximate posterior for the source (target)
domain; $\gamma_{1},\gamma_{2}>0$ are trade-off hyperparameter to weight the
deep feature modulation and the standard Gaussian noise $\bm{\epsilon}$; and
$\odot$ stands for the element-wise product of vectors. Therefore, the
auxiliary noise in VAEs resampling is now a weighted sum of a deep
representation from the other domain and Gaussian noise, different from the
standard VAEs framework. The graphical model of CDLM is shown in Fig. 3.

Figure 3: Illustration of the graphical model of CDLM. The dashed lines are
inference process and the solid lines are generative process.

Now we have obtained the transfer encodings by a unified encoder. Following
the probabilistic encoder analysis [35], a shared inference model confines the
latent variables into the same latent space. However, this cannot guarantee
them to be aligned. To pull the domains close, an adversarial strategy [37,
38] is employed to facilitate the alignment. The gradient reversal layer [37]
is used in our model, by which adversarial learning is carried out to learn
transferable features that are robust to domain shifts. The adversarial
alignment between $\mathcal{H}_{s}$ and $\mathcal{H}_{t}$ is for domain-level.

Furthermore, for better interpretation of the modulated reparameterization,
assume $\bm{h}_{t}\sim
p(\mathcal{H}_{t})=\mathcal{N}(\bm{\mu}_{\bm{h}_{t}},\bm{\sigma}_{\bm{h}_{t}})$.
Then for $\bm{\ddot{z}}_{st}$, we have
$\bm{\ddot{z}}_{st}\sim\mathcal{N}(\bm{\ddot{z}}_{st};\bm{\mu}_{st},\bm{\sigma}_{st}^{2}\mathbf{I})$.
For the $i$-th element of the distribution mean vector is given as follows:

$\begin{split}\mu_{st}^{i}&=\mathbb{E}\left[\mu^{i}_{s}+\sigma_{s}^{i}(\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})\right]\\\
&=\mu_{s}^{i}+\sigma_{s}^{i}\gamma_{1}E\left[h_{t}\right]+\gamma_{2}E\left[\epsilon^{i}\right]\\\
&=\mu_{s}^{i}+\gamma_{1}\sigma_{s}^{i}\mu_{h_{t}}^{i},\end{split}$ (3)

and the $i$-th variation element is obtained as

$\begin{split}\mathbf{Var}(\ddot{z}_{st}^{i})&=\mathbf{Var}[\sigma_{s}^{i}(\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})]\\\
&=\mathbb{E}\left[(\sigma_{s}^{i}(\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})-\gamma_{1}\sigma_{s}^{i}\mu_{h_{t}}^{i})^{2}\right]\\\
&=(\sigma_{s}^{i})^{2}\mathbb{E}\left[((\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})-\gamma_{1}\mu_{h_{t}}^{i})^{2}\right]\\\
&=(\sigma_{s}^{i})^{2}\left(\mathbb{E}\left[((\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i}))^{2}\right]-(\gamma_{1}\mu_{h_{t}}^{i})^{2}\right)\\\
&=(\sigma_{s}^{i})^{2}[\gamma_{1}^{2}(\sigma_{h_{t}}^{i})^{2}+\gamma_{2}^{2}].\end{split}$
(4)

Here, it is reasonable to assume $\bm{\mu}_{\bm{h}_{t}}\approx\bm{\mu}_{t}$
and $\bm{\sigma}_{\bm{h}_{t}}\approx\bm{\sigma}_{t}$ with sufficient training.
With a practical setting of $\gamma_{1}\gg\gamma_{2}$, and in effect
$\sigma_{s}^{i}=1$, Eqs. (3 and 4) can be further simplified to
$\mu_{st}^{i}=\mu_{s}^{i}+\gamma_{1}\mu_{t}^{i}$, and
$\sigma_{st}^{i}=\gamma_{1}\sigma_{s}^{i}\sigma_{t}^{i}$. Then we can see that
$\bm{\mu}_{st}$ can be regarded as a location shift of $\bm{\mu}_{s}$ under
the influence of $\bm{\mu}_{t}$, which helps reduce the domain gap;
$\bm{\sigma}_{st}$ can be taken as a recolouring of $\bm{\sigma}_{s}$ under
the influence from the target. The formulation of $\bm{\ddot{z}}_{ts}$ can be
similarly interpreted. These modulated encodings are hence constructed in a
cross-domain manner. The geometric interpretation of the generation of
$\bm{\mu}_{st}$ from $\bm{\mu}_{s}$ and $\bm{\mu}_{t}$ are given in Fig. 4.

Figure 4: Illustration of the $\bm{\mu}_{st}$. The simplified version is
expressed as vector addition.

####  3.2.3 Partial modulations

We can also obtain some variants of $\mathcal{G}(\cdot)$ in the transfer
latent space (TLS) as follows:

$\bm{\ddot{z}}_{st}=\mathcal{G}((\bm{\mu}_{s},\bm{\sigma}_{t}),\bm{h}_{t})=\bm{\mu}_{s}+\bm{\sigma}_{t}\odot\bm{\epsilon},$
(5)

and

$\bm{\ddot{z}}_{ts}=\mathcal{G}((\bm{\mu}_{t},\bm{\sigma}_{s}),\bm{h}_{s})=\bm{\mu}_{t}+\bm{\sigma}_{s}\odot\bm{\epsilon},$
(6)

which corresponds to a special case of Eqs. 1 and 2 with $\gamma_{1}=0$ and
$\gamma_{2}=1$.It suppress the cross-domain information $\bm{h}$. The
variances are then swap straightforwardly to compensate the vanished cross-
domain information. Under this situation, there are
$\bm{\mu}_{st}=\bm{\mu}_{s}$ and $\bm{\sigma}_{st}=\bm{\sigma}_{t}$. The
“$ts$” counterparts $\bm{\mu}_{ts}$ and $\bm{\sigma}_{ts}$ have similar
interpretations.

We can also obtain a mild variant with linear swapped variance as

$\bm{\ddot{z}}_{st}=\mathcal{G}((\bm{\mu}_{s},\bm{\sigma}_{s},\bm{\sigma}_{t}),\bm{h}_{t})=\bm{\mu}_{s}+\frac{1}{2}(\bm{\sigma}_{s}+\bm{\sigma}_{t})\odot\bm{\epsilon}$
(7)

and

$\bm{\ddot{z}}_{ts}=\mathcal{G}((\bm{\mu}_{t},\bm{\sigma}_{t},\bm{\sigma}_{s}),\bm{h}_{s})=\bm{\mu}_{t}+\frac{1}{2}(\bm{\sigma}_{s}+\bm{\sigma}_{t})\odot\bm{\epsilon},$
(8)

which corresponds to $\gamma_{1}=0$ and $\gamma_{2}=1$. There are
$\bm{\mu}_{st}=\bm{\mu}_{s}$ and
$\bm{\sigma}_{st}=\frac{1}{2}(\bm{\sigma}_{s}+\bm{\sigma}_{t})$. Similarly,
$\bm{\mu}_{ts}=\bm{\mu}_{t}$ and
$\bm{\sigma}_{st}=\frac{1}{2}(\bm{\sigma}_{s}+\bm{\sigma}_{t})$.

The variants in Eq. (5)-(8) only use the second-order moments for modulation,
hence we call them partial modulations.

####  3.2.4 More components

Next, we apply the cross-consistency constraint to the transfer latent space
with modulation for further inter-class alignment. It has been found that
cross-consistency constraints preserve class-discriminative information [34,
39, 6]. For our model, the cross-consistency is applied to the reconstructions
from modulated encodings and the corresponding generations from deep
representations. Let $\mathbf{D}_{\bm{\theta}}(\cdot)$ be the generative model
for domain image generation from the transfer latent space. The cross-
consistency requirements are

$\begin{gathered}\mathbf{D}_{\bm{\theta}}(\bm{\ddot{z}}_{st})=\mathbf{D}_{\bm{\theta}}(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon})\\\
\mathbf{D}_{\bm{\theta}}(\bm{\ddot{z}}_{ts})=\mathbf{D}_{\bm{\theta}}(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon}),\end{gathered}$
(9)

where $\mathbf{D}_{\bm{\theta}}(\bm{\ddot{z}}_{st})$ is the reconstruction of
the source ($\widehat{\bm{x}}_{st}$),
$\mathbf{D}_{\bm{\theta}}(\bm{\ddot{z}}_{ts})$ is for the target
($\widehat{\bm{x}}_{ts}$). $\mathbf{D}_{\bm{\theta}}(\cdot)$ can also function
as a generative model, generating
$\widetilde{\bm{x}}_{s}=\mathbf{D}_{\bm{\theta}}(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon})$
and
$\widetilde{\bm{x}}_{t}=\mathbf{D}_{\bm{\theta}}(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon})$
for the source and the target domain respectively. Also, the consistencies can
guide the encoder to learn the representations from both domains.

Finally, a desired marginalized decoder, e.g., the source decoder, is trained
to map the target images to be source-like. We render the target’s generation
$\widetilde{\bm{x}}_{ts}$ for the test mode. To this end, we do not need the
source to be taken into account for the test. That means a test image from the
target domain $\bm{x}_{t}^{i}$ first passes through the inference model and
obtains its deep feature $\bm{h}_{t}^{i}$. Then it is fed into the generation
model to generate an image with source style while keeping its class. That is
to make the marginal distribution $p(\widetilde{\bm{x}}_{ts}^{i})\approx
p(\bm{x}_{s}^{j})$, but keeping its class $y_{t}^{i}$.

###  3.3 Learning in CDLM

Our goal is to update the variational parameters to learn a joint distribution
and the generation parameters for the desired marginal distribution. Since the
latent variables are generated with inputs from both domains, we have a
modified formulation adapted from the plain VAEs:

$\leavevmode\resizebox{390.25534pt}{}{$\log
p(\bm{x}_{s},\bm{x}_{t})-\text{KL}(q_{\bm{\phi}}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t}))=\mathbb{E}[\log
p(\bm{x}_{s},\bm{x}_{t}|\bm{\ddot{z}})]-\text{KL}(q_{\bm{\phi}}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}}))$},$
(10)

where $\text{KL}(\cdot)$ is the Kullback-Leibler divergence, and the transfer
latent variable $\bm{\ddot{z}}$ can be either $\bm{\ddot{z}}_{st}$ or
$\bm{\ddot{z}}_{ts}$. Minimizing
$\text{KL}(q_{\bm{\phi}}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t}))$
is equivalent to maximizing the variational evidence lower bound (ELBO)
$\mathcal{L}(\bm{\theta},\bm{\phi};\bm{x}_{s},\bm{x}_{t})$:

$\mathcal{L}(\bm{\theta},\bm{\phi};\bm{x}_{s},\bm{x}_{t})=\mathbb{E}_{q_{\bm{\phi}}}[\log
p_{\bm{\theta}}(\bm{x}_{s},\bm{x}_{t}|\bm{\ddot{z}})]-\text{KL}(q_{\bm{\phi}}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}})),$
(11)

where the first term corresponds to the reconstruction cost
($\mathcal{L}_{\text{Rec}}$), and the second term is the KL divergence between
the learned latent probability and the prior (specified as
$\mathcal{N}(0,\bm{I})$) ($\mathcal{L}_{\text{KL}}$). Considering the
reconstruction of $\bm{x}_{s}$, and the KL divergence for both
$\bm{\ddot{z}}_{st}$ and $\bm{\ddot{z}}_{ts}$, we have the following loss:

$\begin{split}\mathcal{L}(\bm{\theta},\bm{\phi};\bm{x}_{s},\bm{x}_{t})&=\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{x}_{t})}[\log
p_{\bm{\theta}}(\bm{x}_{s}|\bm{\ddot{z}}_{st})]\\\ &-\text{KL}(\log
q_{\bm{\phi}}({\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{x}_{t}})\|p(\bm{z}))\\\
&-\text{KL}(\log
q_{\bm{\phi}}({\bm{\ddot{z}}_{ts}|\bm{x}_{t},\bm{x}_{s}})\|p(\bm{z})).\\\
\end{split}$ (12)

To align the source and target domains’ deep representations, we employ an
adversarial strategy to regularize the model. The loss function is given by

$\mathcal{L}_{\text{adv}}=\mathbb{E}_{\bm{h}_{s}\sim
p(\bm{h}_{s}|\bm{x}_{s})}[\log\Xi(\bm{h}_{s})]+\mathbb{E}_{\bm{h}_{t}\sim
p(\bm{h}_{t}|\bm{x}_{t})}[\log(1-\Xi(\bm{h}_{t}))],$ (13)

where $\Xi(\cdot)$ is the discriminator to predict from which domain the deep
representation feature is generated.

We also introduce a pairwise consistency between the reconstruction and the
generation for the source and the target in an unsupervised manner,
respectively. The consistency regularization improves the inter-class
alignment. For the consistency loss $\mathcal{L}_{c}$, both the $L_{1}$ and
$L_{2}$-norm penalty can be used to regularize the decoder. Here we simply use
MSE. Let $\mathcal{L}_{c}^{s}$ and $\mathcal{L}_{c}^{t}$ be the consistency
for the domains respectively. $\mathcal{L}_{c}$ is given as a combination of
these two components, weighted by two coefficients $\beta_{1}$ and
$\beta_{2}$, respectively:

$\displaystyle\mathcal{L}_{c}$
$\displaystyle=\beta_{1}\mathcal{L}_{c}^{s}+\beta_{2}\mathcal{L}_{c}^{t}$ (14)
$\displaystyle=\beta_{1}(\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})}[\log
p(\widehat{\bm{x}}_{s}|\bm{\ddot{z}}_{st})]-\mathbb{E}_{\bm{h}_{s}\sim
p(\bm{h}_{s}|\bm{x}_{s})}[\log p(\widetilde{\bm{x}}_{s}|\bm{h}_{s})])^{2}$
$\displaystyle+\beta_{2}(\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})}[\log
p(\widehat{\bm{x}}_{ts}|\bm{\ddot{z}}_{ts})]-\mathbb{E}_{\bm{h}_{t}\sim
p(\bm{h}_{t}|\bm{x}_{t})}[\log p(\widetilde{\bm{x}}_{ts}|\bm{h}_{t})])^{2}.$

Then, the variational parameters $\bm{\phi}$ and generation parameters
$\bm{\theta}$ are updated by the following rules:

$\begin{gathered}\bm{\phi}\leftarrow\bm{\phi}-\eta_{1}\nabla(\mathcal{L}_{\text{adv}}+\lambda_{1}\mathcal{L}_{\text{KL}}+\lambda_{2}\mathcal{L}_{\text{Rec}})\\\
\bm{\theta}\leftarrow\bm{\theta}-\eta_{2}\nabla(\mathcal{L}_{\text{Rec}}+\mathcal{L}_{c}),\end{gathered}$
(15)

where $\eta_{1},\eta_{2}$ are two learning rates, and $\lambda_{1}$,
$\lambda_{2}$ are weights used to adjust the impact of the discriminator loss
and the reconstruction loss. Note that only data from the desired domain (the
source) are used to train the reconstruction loss. The KL term makes the
transfer latent space to approximate the prior. We summarize our CDLM in Algo
1.

Input: Source: $\mathbf{X}_{s}$, Target:$\mathbf{X}_{t}$

Result: Inference and Generative model of CDLM

1 $\bm{\phi},\bm{\theta}$ $\leftarrow$ initialization

2 for  _iterations of traning_ do

3 $\mathbf{X}_{s},\mathbf{X}_{t}$ $\leftarrow$ sample mini-batch

4 $\bm{\epsilon}$ $\leftarrow$ sample from $\mathcal{N}(0,I)$

5 $\ddot{\bm{z}}_{st},\ddot{\bm{z}}_{ts}$ $\leftarrow$ sample from
$q(\bm{z}_{st}|\mathbf{X}_{s},\bm{h}_{t})$ and
$q(\boldsymbol{z}_{ts}|\mathbf{X}_{t},\bm{h}_{s})$ by Eqs. (1) to (2)

6 Generate
${\widehat{\mathbf{X}}_{st}},{\widehat{\mathbf{X}}_{ts}},{\widetilde{\mathbf{X}}_{s}},{\widetilde{\mathbf{X}}_{t}}$
by feeding $\ddot{\bm{z}}_{st},\ddot{\bm{z}}_{ts}$ through Decoders

7
$\mathcal{L}_{\text{Rec}},\mathcal{L}_{\text{KL}},\mathcal{L}_{\text{adv}},\mathcal{L}_{c}$
$\leftarrow$ calculated by Eqs. (11) to (14)

8 for  _iterations of inference model updating_ do

9 $\bm{\phi}\leftarrow$ –
$\eta_{1}\Delta_{\bm{\phi}}(\mathcal{L}_{\text{adv}}+\lambda_{1}\mathcal{L}_{\text{KL}}+\lambda_{2}\mathcal{L}_{\text{Rec}})$

10

11 end for

12 for  _iterations of generative model updating_ do

13 $\bm{\theta}$ $\leftarrow$
–$\eta_{2}\Delta_{\bm{\theta}}(\mathcal{L}_{\text{Rec}}+\mathcal{L}_{c})$

14

15 end for

16

17 end for

Algorithm 1 Training of the CDLM framework

###  3.4 Generalization bound analysis

Here we give the generalization error analysis using the domain adaptation
theory [40, 41]. The adaptation error on the target, $\epsilon_{t}(f)$, is
bounded by

$\epsilon_{t}(f)\leq\epsilon_{s}(f)+\inf\limits_{f^{*}\in\mathcal{H}}[\epsilon_{s}(f^{*})+\epsilon_{t}(f^{*})]+\hat{d}_{\mathcal{H}}(\mathbf{X}_{s},\mathbf{X}_{t})$
(16)

where $f$ is a mapping function of hypothesis family $\mathcal{H}$, $f^{*}$ is
the ideal joint hypothesis to minimize the combined error on source and target
predictions, $\epsilon_{s}(.)$ and $\epsilon_{t}(.)$ are the prediction error
on the source and target data respectively, and $\hat{d}_{\mathcal{H}}(\cdot)$
is the empirical $\mathcal{H}$-discrepancy between source and target data.

In our model, we use the gradient reversal layer (GRL) to align the high-level
representations. The objective in Eq. (13)), similar to the treatment in DANN
[37], is equivalent to maximizing

$\mathcal{L}_{\text{adv+}}(\Xi(\bm{h}_{i}^{s},\bm{h}_{i}^{t}))=-\frac{1}{n}\sum_{i=1}^{n}d_{i}^{s}\log(\Xi(\bm{h}_{i}^{s}))-\frac{1}{n}\sum_{i=1}^{n}d_{i}^{t}\log(1-\Xi(\bm{h}_{i}^{t})),$
(17)

where $d_{i}^{s}$ and $d_{i}^{t}$ are indicator variables, set to $1$ when the
input $\mathbf{x}_{i}$ is of the source or the target domain respectively.
This helps reduce the upper bound component
$\hat{d}_{\mathcal{H}}(\mathbf{X}_{s},\mathbf{X}_{t})$ in Eq. (16), hence
leading to a better adaptation performance.

Furthermore, the other term in Eq. (16) for the ideal joint hypothesis
$\epsilon_{s}(f^{*})+\epsilon_{t}(f^{*})$ explicitly embodies the notion of
adaptability. In our model, we learn a joint latent space
$p(\bm{\ddot{}}z_{st},\bm{\ddot{}}z_{ts};\bm{x}_{s},\bm{x}_{t})$ to generate
the cross-domain images, that is $p(\mathbf{D}(\bm{\ddot{}}z_{st}))\approx
p(\mathbf{D}(\bm{\ddot{}}z_{ts}))$ with the shared labels. It leads to a
smaller
$\epsilon_{s}(f^{*}(\mathbf{D}(\bm{\ddot{}}z_{st})))+\epsilon_{t}(f^{*}(\mathbf{D}(\bm{\ddot{}}z_{ts})))$
term to further reduce the generalization error.

##  4 Experiments

We have conducted extensive evaluations of CDLM in two homogeneous transfer
scenarios including unsupervised domain adaptation and image-to-image
translation. Our model is implemented using TensorFlow [42]. The structures of
the encoder and the decoder adopt those of UNIT [4] which perform well for
image translation tasks. A two-layer fully connected MLP is used for the
discriminator. SGD with momentum is used for updating the variational
parameters, and Adam for updating generation parameters. The batch size is set
to 64. During the experiments, we set $\gamma_{1}=1.0,\gamma_{2}=0.1$,
$\lambda_{1}=\lambda_{2}=0.0001$, $\beta_{1}=0.1$ and $\beta_{2}=0.01$.

###  4.1 Datasets

We have evaluated our model on a variety of benchmark datasets includingMNIST
[43], MNISTM [37], USPS [44], Fashion-MNIST [45], LineMod [46, 47],
Zap50K-shoes [48] and CelebA [49, 50].

MNIST: The MNIST handwritten dataset [43] is a very popular machine learning
dataset. It has a training set of 60,000 binary images and a test set of
10,000. There are 10 classes in the dataset. In our experiments, we use the
standard split of the dataset. MNISTM [37] is a modified version for the
MNIST, with random RGB background cropped from the Berkeley Segmentation
Dataset111URL
https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/.

USPS: USPS is a handwritten zip digits datasets [44]. It contains 9298 binary
images ($16\times 16$), 7291 used as the training set, while the remaining
2007 are used as the test set. The USPS samples are resized to $28\times 28$,
the same as MNIST.

Fashion: Fashion [45] contains 60,000 images for training, and 10,000 for
testing. All the images are greyscaled, $28\times 28$ in size space. Besides,
following the protocol in [37], we add random noise to the Fashion images to
generate the FashionM dataset, with random RGB background cropped from the
Berkeley Segmentation Dataset.

LineMod 3D images For this scenario, followed the protocol of [30], we render
the LineMod [46, 47] for the adaptation between synthetic 3D images (source)
and real images (target). We note them as L-3D and L-Real in short. The
objects with different poses are located at the centre of the images. The
synthetic 3D images render a black background and a variety of complex indoor
environments for real images. We use the RGB images only, not the depth
images.

CelebA: CelebA [49] is a large celebrities face image dataset. It contains
more than 200K images annotated with 40 facial attributes. We select 50K
images randomly, then transform them to sketch images followed the protocol of
[50]. The original and sketched images are used for translation.

UT-Zap50K-shoes: This dataset [48] contains 50K shoes images with 4 different
classes. During the translation, we get the edges produced by the Canny
detector [51].

###  4.2 Quantitative results on unsupervised domain adaptation

We apply our model to unsupervised domain adaptation, adapting a classifier
trained using labelled samples in the source domain to classify samples in the
target domain. For this scenario, only the labels of the source images were
available during training. We choose DANN [37] as the baseline, and compare
our model with several other state-of-the-art domain adaptation methods,
including Conditional Domain Adaptation Network (CDAN) [52], Pixel-level
Domain Adaptation (PixelDA) [30], Unsupervised Image-to-Image translation
(UNIT) [4], Cycle-Consistent Adversarial Domain Adaptation (CyCADA) [32],
Generate to Adapt (GtA) [31], Transferable Prototypical Networks (TPN) [53],
Domain Symmetric Networks (SymmNets-V2) [54], Instance Level Affinity-based
Networks (ILA-DA) [55], and Deep Adversarial Transition Learning (DATL) [56].
We also use the source-only and target-only training outcome as the lower and
upper bounds following the practice in [30, 37].

CDLM can transfer the target images to be source-like. Then, for the
adaptation tasks, joint or decoupled task learning can be used, and a
supervised classifier trained on the source images along with their labels can
also be applied to recognize the target images. We adopt a joint task
classifier. The parameters of the task classifier $\bm{\theta}_{T}$ are
jointly learned with the update of $\bm{\phi}$ and $\bm{\theta}$. The
performance of domain adaptation for the different tasks is shown in Table 1.
There are 4 scenarios and 7 tasks. Each scenario has bidirectional tasks for
adaptation except LineMod. For LineMod, it is adapted from a synthetic 3D
image to real objects. For the same adaptation task, we cite the accuracy from
the corresponding references; otherwise, some tasks’ accuracy is obtained by
training the open-source code provided by authors with suggested optimal
parameters for a fair comparison.

From Table 1, we can see that our method has a high advantage over the
baseline and the source-only (lower bound) accuracy, and its performance is
only a little lower than the target-only accuracy (upper bound) on both
adaptation directions. In comparison with other models, our model has the best
performance for most tasks. The CDLM has higher adaptation accuracies for the
scenarios with a seemingly large domain gaps, such as MNIST$\rightarrow$MNISTM
and Fashion$\rightarrow$FashionM. For the 3D scenario, our model’s performance
is a little lower than PixelDA [30] and DATL [56] but outperforms all the
other methods. In PixelDA, the input is not only source images but also depth
image pairs. It might be helpful for the generation. Also, the results show
that DATL has better performance for some tasks such as
MNISTM$\rightarrow$MNIST and LineMod3D to Real. This is possibly due to the
better intermediate generation by DATL in these scenarios.

Table 1: Mean classification accuracy comparison. The “source only" row is the
accuracy for target without domain adaptation training only on the source. The
“target only" is the accuracy of the full adaptation training on the target.
For each source-target task the best performance is in bold

Source | MNIST | USPS | MNIST | MNISTM | Fashion | FashionM | LineMod 3D
---|---|---|---|---|---|---|---
Target | USPS | MNIST | MNISTM | MNIST | FashionM | Fashion | LineMod Real
Source Only | 0.634 | 0.625 | 0.561 | 0.633 | 0.527 | 0.612 | 0.632
DATL [56] | 0.961 | 0.956 | 0.890 | 0.983 | 0.853 | 0.917 | 0.998
SymmNets-V2 [54] | 0.948 | 0.968 | - | - | - | - | -
ILA-DA [55] | 0.949 | 0.975 | - | - | - | - | -
DANN [37] | 0.774 | 0.833 | 0.766 | 0.851 | 0.765 | 0.822 | 0.832
CyCADA [32] | 0.956 | 0.965 | 0.921 | 0.943 | 0.874 | 0.915 | 0.960
GtA [31] | 0.953 | 0.908 | 0.917 | 0.932 | 0.855 | 0.893 | 0.930
CDAN [52] | 0.956 | 0.980 | 0.862 | 0.902 | 0.875 | 0.891 | 0.936
PixelDA [30] | 0.959 | 0.942 | 0.982 | 0.922 | 0.805 | 0.762 | 0.998
TPN [53] | 0.921 | 0.941 | - | - | - | - | -
UNIT [4] | 0.960 | 0.951 | 0.920 | 0.932 | 0.796 | 0.805 | 0.964
CDLM ($\widetilde{\bm{x}}_{ts}$) | 0.961 | 0.983 | 0.987 | 0.962 | 0.913 | 0.922 | 0.984
Target Only | 0.980 | 0.985 | 0.983 | 0.985 | 0.920 | 0.942 | 0.998

###  4.3 Qualitative results on unsupervised domain adaptation

For this part, we first present visual assessment of the domain adaptation
outcomes. Fig. 5 is the visualization for the digits and Fashion adaptation,
respectively. The generation for the task USPS$\rightarrow$MNIST is shown in
Fig. 5(a), where we can see target MNIST images are transferred to the USPS
style while its semantic content (i.e., the class). For example, the digit ‘1’
in MNIST becomes more leaned than its original look, and ‘9’ becomes
flattened. In Fig. 5(b), the target USPS digits gain the MNIST style. For
scenarios in Fig. 5(d) and Fig. 5(c), colour backgrounds are effectively added
or removed through adaptation.

For the Fashion scenario, fashion items have a variety of different textures
and content. Besides, the noisy backgrounds pollute the items randomly. For
example, different parts of cloth may have different colours. Fig. 5(e) is for
the task Fashion $\rightarrow$ FashionM, where the noisy background is removed
while the content is kept. On the other hand, Fig. 5(f) shows that the
original target Fashion images are added with a similar noisy background as
the source. All these indicate very promising adaptation outcome.

(a) USPS$\rightarrow$MNIST

(b) MNIST$\rightarrow$USPS

(c) MNIST$\rightarrow$MNISTM

(d) MNISTM$\rightarrow$MNIST

(e) Fashion$\rightarrow$FashionM

(f) FashionM$\rightarrow$Fashion

Figure 5: Visualization for the adaptations. 6 different tasks are
illustrated. For each task, the first row shows target images and the second
row shows the adapted images with source-like style.

For the scenario of LineMod3D, the images of the real objects with different
backgrounds are transferred to the source (synthetic 3D) images with a black
background. We only use the image’s RGB channel, and the depth image is not
included in the inputs during the evaluation. Our CDLM can generate realistic
synthetic images for the adaptation. Due to the 3D style, the generation of
the target gives different poses. For example, in Fig.6, different poses of
the iron object are obtained from different trials.

Figure 6: Visualization of task LineMod 3D Synthetic$\rightarrow$Real. For a
query image (on the left), different adaptation images (to the right) with
various poses can be generated.

Next, we investigate the visualization of the latent embedding of the high-
dimensional representations using the t-SNE visualization [57] of deep
representations ($\bm{h}_{s},\bm{h}_{t}$) and latent encodings
($\bm{\ddot{z}}_{st},\bm{\ddot{z}}_{ts}$) w.r.t source and target,
respectively. Fig. 7(a) \- 7(b) are the visualizations for the task MNISTM-
MNIST and 7(c) \- 7(d) for the task MNIST-USPS. Both visualizations
demonstrate a good alignment between the source and the target. The t-SNE
embeddings of $\bm{h}$ vectors also show that the high-level deep
representations are aligned with data distribution information, supporting our
algorithm design.

(a) t-SNE visualization for $\bm{h}_{t},\bm{h}_{s}$ of task
MNISTM$\rightarrow$MNIST: $\bm{h}_{s}$ in blue, $\bm{h}_{t}$ in red.

(b) t-SNE visualization for $\bm{\ddot{z}}_{st},\bm{\ddot{z}}_{ts}$ of task
MNISTM$\rightarrow$MNIST: $\bm{\ddot{z}}_{st}$ in blue, $\bm{\ddot{z}}_{ts}$
in red.

(c) t-SNE visualization for $\bm{h}_{t},\bm{h}_{s}$ of task
MNIST$\rightarrow$USPS: $\bm{h}_{s}$ in blue, $\bm{h}_{t}$ in red.

(d) t-SNE visualization for $\bm{\ddot{z}}_{st},\bm{\ddot{z}}_{ts}$ of task
MNIST$\rightarrow$USPS: $\bm{\ddot{z}}_{st}$ in blue, $\bm{\ddot{z}}_{ts}$ in
red.

Figure 7: t-SNE visualization of deep representations and modulated latent
encodings. Both embeddings align well between the source and the target. The
deep representations is demonstrated with data distribution information to
support the modulations.

###  4.4 Results on cross-domain image translation

The proposed model also can be used for cross-domain image translation. Fig. 8
gives the demonstration for the mapping experiments. Specifically, Fig. 8(a)
and 8(b) are for shoes and edges; Fig. 8(c) and 8(d) are for faces and pencil
sketches. We can see that the proposed model can effectively translate the
edges to their counterpart for shoes and edges. The translation is multi-
modal, which means the edges can generate different colour shoes with
different tests. For the challenging face and pencil sketch task, the proposed
model can also map one to another on both directions. The generations have
some variations compared with the original images, but in general our method
can generate realistic, translated images.

We have found that the reverse task seems harder than the task to convert a
sketch to a real image. For example, when the face image is given, the
generated sketch may lose some details. The reason may be that the low-level
features are neglected when the deep feature acts as the condition. Another
observation is the blurriness in images. It is caused by the intrinsic
character of VAEs that employs a $L_{2}$-based regularizer for the marginal
evidence.

For further evaluation, quantitative performance is evaluated for image
translation. SSIM [58], MSE, and PSNR are used for the evaluation. The results
are shown in Table 2. We can see that our model outperforms E-CDRD [50], which
learns a disentangled latent encoding for the source and the target for domain
adaptation. Meanwhile, it matches the performance of StarGAN [29], which is
designed for multi-domain image translation.

Table 2: Performance for the image mapping.

Models | “Sketch” to “Face”
---|---
SSIM | MSE | PSNR
E-CDRD [50] | 0.6229 | 0.0207 | 16.86
StarGAN [29] | 0.8026 | 0.0142 | 19.04
CDLM | 0.7961 | 0.0140 | 19.89

In addition, some classification experiments are carried out to evaluate the
translation performance. We take “shoes” as an example, which are labelled
into four different classes. Our proposed model’s recognition accuracy for
task shoes$\rightarrow$edge is 0.953, which is higher than the results of
PixelDA (0.921) and UNIT (0.916), respectively.

(a) “edge” to “shoes”

(b) “shoes” to “edge”

(c) “sketch” to “face”

(d) “face” to “sketch”

Figure 8: Visualization of cross-domain image mapping.

###  4.5 Ablation studies

In our model, two unsupervised consistency metrics are added for achieving
good generation quality. Ablation studies are then carried out, where
adaptation accuracy is used for evaluation. Table 3 gives the results for four
different tasks. The performance without $\mathcal{L}_{c}$ drops down because
the decoder cannot generate realistic cross-domain images.
$\mathcal{L}_{c}^{t}$ connects outputs generated from $\bm{h}_{t}$ and
$\ddot{\bm{z}}_{ts}$ only for the target, which improves the performance
slightly. Meanwhile, we can see that the $\mathcal{L}_{c}^{s}$ loss boosts the
accuracy for adaptation significantly, which connects the two domains with the
generations by $\bm{h}$. Finally, the scenario with both $\mathcal{L}_{c}^{s}$
and $\mathcal{L}_{c}^{t}$ gives the best performance in all four tasks. It
bridges both $\bm{h}$ and $\ddot{\bm{z}}$ between the two domains.

Table 3: Evaluation on the effect of unsupervised consistency metrics. The
recognition accuracy is shown for four tasks in the unsupervised domain
adaptation scenario. Our model is on the last row with both the
$\mathcal{L}_{c}^{s}$ and $\mathcal{L}_{c}^{t}$, which achieves the best
performance.

Model/Tasks | MNIST$\rightarrow$USPS | USPS$\rightarrow$MNIST | Fashion$\rightarrow$FashionM | FashionM$\rightarrow$Fashion
---|---|---|---|---
CDLM w/o $\mathcal{L}_{c}$ | 0.635 | 0.683 | 0.646 | 0.672
CDLM+$\mathcal{L}_{c}^{t}$ | 0.689 | 0.695 | 0.682 | 0.691
CDLM+$\mathcal{L}_{c}^{s}$ | 0.951 | 0.980 | 0.912 | 0.915
CDLM+$\mathcal{L}_{c}^{s}$+$\mathcal{L}_{c}^{t}$ | 0.961 | 0.983 | 0.913 | 0.922

###  4.6 Model analysis

The effect of encoder settings – depth and different $\gamma_{1},\gamma_{2}$:
In our model, the deep features are utilized to cross-modulate the transfer
latent encoding. They play an essential role in our framework, but may be
influenced by the encoder’s depth. To look into this We use MNIST
$\rightarrow$ USPS and Fashion $\rightarrow$ FashionM for evaluation. For the
first task, they have different contents, but with the same background. The
second task is different, images have the same content but different
backgrounds. The outputs of different encoder layers (Conv4, Conv5 and
Conv${}_{l}ast$) are used for the experiments.

Table 4: Adaptation accuracy with different layer depth. Tasks MNIST
$\rightarrow$ USPS and Fashion $\rightarrow$ FashionM are considered.

Tasks/Layers | Conv4 | Conv5 | Convlast
---|---|---|---
MNIST $\rightarrow$ USPS | 0.954 | 0.956 | 0.961
Fashion $\rightarrow$ FashionM | 0.890 | 0.905 | 0.913

The result (Table 4) shows that higher accuracies are achieved when more
layers are used to extract the deep representations. The accuracy gain of the
task MNIST $\rightarrow$ USPS is lower than that of Fashion $\rightarrow$
FashionM. It is expected that features extracted by higher layers would
naturally eliminate lower-level variations between domains, such as background
and illumination changes in the images.

To experiment with $\gamma_{1},\gamma_{2}$, we fix the last convolutional
layer for the deep representations and evaluate the different values. From
Table 5, we can see that the performance is dropped down significantly with a
smaller $\gamma_{1}$ compared with $\gamma_{2}$, and increased with a larger
$\gamma_{1}$. The performance seems to be stabilized when $\gamma_{1}$ is
greater than 0.9, while $\gamma_{2}$ remains 0.1. Following the standard VAE,
we keep the noise $\bm{\epsilon}$ ($\gamma_{2}\neq 0$) in evaluation.
Meanwhile, our model works well even when $\gamma_{2}=0$. The results suggest
that deep representation plays a crucial role in cross-domain modulation.

Table 5: Adaptation accuracy with different ($\gamma_{1},\gamma_{2}$). Tasks
MNIST $\rightarrow$ USPS and Fashion $\rightarrow$ FashionM are considered.

Tasks / ($\gamma_{1},\gamma_{2}$) | (0.1,1.0) | (0.5,0.5) | (0.9,0.1) | (1.0,0.1) | (1.0, 0)
---|---|---|---|---|---
MNIST $\rightarrow$ USPS | 0.320 | 0.723 | 0.961 | 0.961 | 0.961
Fashion $\rightarrow$ FashionM | 0.226 | 0.513 | 0.912 | 0.913 | 0.913

Performance of partial modulation: The proposed transfer latent space is a
generalization of the transfer space. Several transfer functions can be
satisfied for this purpose. We give the performance of the two partial
modulation variants described in Section 3.2, as shown in Table 6. The results
show that the CDLM has a better performance than the two variants. The partial
modulation causes that only the image’s second-order moment contributes to the
transfer and cross-consistency.

Table 6: Performance of the partial modulation variants.

Model | MNIST$\rightarrow$MNISTM | MNIST$\rightarrow$USPS
---|---|---
Variant 1 | 0.967 | 0.953
Variant 2 | 0.959 | 0.950
CDLM | 0.987 | 0.961

$\mathcal{A}$-Distance: $\mathcal{A}$-distance can be used as a measure for
domain discrepancy [40]. As the exact $\mathcal{A}$-distance is intractable, a
proxy is defined as $\hat{d}_{\mathcal{A}}=2(1-2\epsilon)$, where $\epsilon$
is the generalization error of a binary classifier (e.g., kernel SVM) trained
to distinguish the input’s domain (source or target). Following the protocol
of [59, 60], we calculate the $\mathcal{A}$-distance on four adaptation tasks
under the scenarios of the original images (“Raw”), and the adapted features
in DANN and CDLM. The results are shown in Fig. 9. We observe that both DANN
and CDLM reduce the domain discrepancy compared with the Raw images scenario,
and the $\mathcal{A}$-distance of CDLM is smaller than DANN’s. This suggests
the domain discrepancy between the source and the target generations gets most
reduced by CDLM.

Figure 9: $\mathcal{A}$-distances comparison for four tasks.

Convergence: We also conduct the convergence experiment with training error on
task MNIST-USPS to evaluate our model. Fig. 10 shows that our model has better
convergence than DANN, though there are some oscillations at the beginning of
the training. The error of CDLM quickly drops lower than DANN after about 5000
iterations, indicating its better adaptation performance. This is consistent
with the adaptation performance given in Table 1.

Figure 10: Convergence of CDLM compared with DANN.

##  5 Conclusion

In this paper, we have presented a novel variational cross-domain transfer
learning model with cross modulation of deep representations from different
domains. A shared transfer latent space is introduced, and the
reparameterization transformation is modified to implement cross-domain
modulations. Evaluations carried out in unsupervised domain adaptation and
image translation tasks demonstrate our model’s competitive performance. Its
effectiveness is also clearly shown in visual assessment of the adapted
images, as well as in the alignment of the latent information as revealed by
visualization using t-SNE. Overall, competitive performance has been achieved
by our model despite its relative simplicity compared with the-state-of-the-
art.

We have considered only the one-on-one transfer scenario for both domain
adaptation and image translation tasks. Our variational cross-domain
mechanism, however, seems promising for extension to multi-domain transfer
learning tasks. An interesting, and plausible application will be to translate
images to multiple target styles through the cross-modulation of multiple
style priors. For future work, we intend to further improve our variational
transfer learning framework and use it for heterogeneous, multi-domain
transfer tasks.

## References

  * [1] S. J. Pan, Q. Yang, A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering 22 (10) (2010) 1345–1359.
  * [2] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, C. Liu, A survey on deep transfer learning, in: V. Kurková, Y. Manolopoulos, B. Hammer, L. S. Iliadis, I. Maglogiannis (Eds.), International Conference on Artificial Neural Networks (ICANN), 2018, pp. 270–279.
  * [3] K. R. Weiss, T. M. Khoshgoftaar, D. Wang, A survey of transfer learning, Journal of Big Data 3 (9) (2016) 1–40.
  * [4] M.-Y. Y. Liu, T. Breuel, J. Kautz, Unsupervised image-to-image translation networks, in: Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 700–708.
  * [5] L. Wang, A. G. Schwing, S. Lazebnik, Diverse and accurate image description using a variational auto-encoder with an additive Gaussian encoding space, in: Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5757–5767.
  * [6] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrell, Z. Akata, Generalized zero-and few-shot learning via aligned variational autoencoders, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8247–8255.
  * [7] D. P. Kingma, Fast gradient-based inference with continuous latent variable models in auxiliary form, arXiv:1306.0733 [cs.LG].
  * [8] T. Lindvall, Lectures on the Coupling Method, Dover, 2002.
  * [9] F. d. Hollander, Probability theory: The coupling method, Leiden University, Lectures Notes-Mathematical Institute (2012) 31.
  * [10] S. Mahajan, I. Gurevych, S. Roth, Latent normalizing flows for many-to-many cross-domain mappings, in: International Conference on Learning Representations (ICLR), 2020.
  * [11] J. M. Tomczak, M. Welling, VAE with a vampprior, in: A. Storkey, F. Perez-Cruz (Eds.), International Conference on Artificial Intelligence and Statistics (AISTATS), Vol. 84 of PMLR, 2018, pp. 1214–1223.
  * [12] M. D. Hoffman, M. J. Johnson, ELBO surgery: yet another way to carve up the variational evidence lower bound, in: Advances in Neural Information Processing Systems Workshop (NeurIPSW), 2016.
  * [13] J. Hou, J. D. Deng, S. Cranefield, X. Ding, Cross-domain latent modulation for variational transfer learning, in: IEEE Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 3149–3158.
  * [14] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, O. Winther, Autoencoding beyond pixels using a learned similarity metric, in: M. F. Balcan, K. Q. Weinberger (Eds.), International Conference on Machine Learning (ICML), Vol. 48 of PMLR, 2016, pp. 1558–1566.
  * [15] A. H. Liu, Y. C. Liu, Y. Y. Yeh, Y. C. F. Wang, A unified feature disentangler for multi-domain image translation and manipulation, in: Advances in Neural Information Processing Systems (NeurIPS), 2018, pp. 2590–2599.
  * [16] Z. M. Ziegler, A. M. Rush, Latent normalizing flows for discrete sequences, in: R. S. Kamalika Chaudhuri (Ed.), International Conference on Machine Learning (ICML), Vol. 97 of PMLR, 2019, pp. 7673–7682.
  * [17] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling, Improved variational inference with inverse autoregressive flow, in: Advances in Neural Information Processing Systems (NeurIPS), 2016, pp. 4743–4751.
  * [18] C. M. Bishop, Pattern Recognition and Machine Learning, Information Science and Statistics, Springer, 2006.
  * [19] N. Dilokthanakul, P. A. M. Mediano, M. Garnelo, M. C. H. Lee, H. Salimbeni, K. Arulkumaran, M. Shanahan, Deep unsupervised clustering with Gaussian mixture variational autoencoders, arXiv:1611.02648 [cs.LG].
  * [20] K. Sohn, X. Yan, H. Lee, Learning structured output representation using deep conditional generative models, in: Advances in Neural Information Processing Systems (NeurIPS), 2015, pp. 3483–3491.
  * [21] L. Yang, N. M. Cheung, J. Li, J. Fang, Deep clustering by Gaussian mixture variational autoencoders with graph embedding, in: IEEE International Conference on Computer Vision (ICCV), 2019, pp. 6439–6448.
  * [22] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, D. Erhan, Domain separation networks, in: Advances in Neural Information Processing Systems (NeurIPS), 2016, pp. 343–351.
  * [23] J. Zhang, Y. Huang, Y. Li, W. Zhao, L. Zhang, Multi-attribute transfer via disentangled representation, in: AAAI Conference on Artificial Intelligence (AAAI), Vol. 33, 2019, pp. 9195–9202.
  * [24] Z. Feng, A. Zeng, X. Wang, D. Tao, C. Ke, M. Song, Dual swap disentangling, in: Advances in Neural Information Processing Systems (NeurIPS), 2018, pp. 5894–5904.
  * [25] H. Y. Lee, H. Y. Tseng, J. B. Huang, M. Singh, M. H. Yang, Diverse image-to-image translation via disentangled representations, in: European Conference on Computer Vision (ECCV), 2018, pp. 36–52.
  * [26] A. Gonzalez-Garcia, J. van de Weijer, Y. Bengio, Image-to-image translation for cross-domain disentanglement, in: Advances in Neural Information Processing Systems (NeurIPS), 2018, pp. 1294–1305.
  * [27] M. Naseer, S. H. Khan, H. Khan, F. S. Khan, F. Porikli, Cross-domain transferability of adversarial perturbations, in: Advances in Neural Information Processing Systems (NeurIPS), 2019.
  * [28] A. Noguchi, T. Harada, Image generation from small datasets via batch statistics adaptation, in: IEEE International Conference on Computer Vision (ICCV), 2019, pp. 2750–2758.
  * [29] Y. Choi, M. Choi, M. Kim, J. W. Ha, S. Kim, J. Choo, StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8789–8797.
  * [30] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, D. Krishnan, Unsupervised pixel-level domain adaptation with generative adversarial networks, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 95–104.
  * [31] S. Sankaranarayanan, Y. Balaji, C. D. Castillo, R. Chellappa, Generate to adapt: Aligning domains using generative adversarial networks, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8503–8512.
  * [32] J. Hoffman, E. Tzeng, T. Park, J.-Y. Y. Zhu, P. Isola, K. Saenko, A. A. Efros, T. Darrell, CyCADA: Cycle-consistent adversarial domain adaptation, in: J. G. Dy, A. Krause (Eds.), International Conference on Machine Learning (ICML), Vol. 80 of PMLR, 2018, pp. 3162–3174.
  * [33] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242–2251.
  * [34] T. Kim, M. Cha, H. Kim, J. K. Lee, J. Kim, Learning to discover cross-domain relations with generative adversarial networks, in: Y. W. T. Doina Precup (Ed.), International Conference on Machine Learning (ICML), Vol. 70 of PMLR, 2017, pp. 1857–1865.
  * [35] D. P. Kingma, M. Welling, Auto-encoding variational Bayes, in: International Conference on Learning Representations (ICLR), 2014.
  * [36] J. Yosinski, J. Clune, Y. Bengio, H. Lipson, How transferable are features in deep neural networks?, in: Advances in Neural Information Processing Systems (NeurIPS), 2014, pp. 3320–3328.
  * [37] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, V. S. Lempitsky, Domain-adversarial training of neural networks, Journal of Machine Learning Research 17 (59) (2016) 1–35.
  * [38] E. Tzeng, J. Hoffman, K. Saenko, T. Darrell, Adversarial discriminative domain adaptation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7167–7176.
  * [39] M. Ghifary, W. B. Kleijn, M. Zhang, D. Balduzzi, W. Li, Deep reconstruction-classification networks for unsupervised domain adaptation, in: European Conference on Computer Vision (ECCV), 2016, pp. 597–613.
  * [40] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira, Analysis of representations for domain adaptation, in: Advances in Neural Information Processing Systems (NeurIPS), 2007, pp. 137–144.
  * [41] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, J. W. Vaughan, A theory of learning from different domains, Machine Learning 79 (2010) 151–175.
  * [42] M. Abadi, et al., TensorFlow: Large-scale machine learning on heterogeneous distributed systems, arXiv:1603.04467 [cs.DC].
  * [43] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  * [44] Y. Le Cun, L. D. Jackel, B. Boser, J. S. Denker, H. P. Graf, I. Guyon, D. Henderson, R. E. Howard, W. Hubbard, Handwritten digit recognition: Applications of neural network chips and automatic learning, IEEE Communications Magazine 27 (11) (1989) 41–46.
  * [45] H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms, arXiv:1708.07747 [cs.LG].
  * [46] S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, P. Fua, V. Lepetit, Gradient response maps for real-time detection of texture-less objects, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (5) (2012) 876–888.
  * [47] P. Wohlhart, V. Lepetit, Learning descriptors for object recognition and 3d pose estimation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3109–3118.
  * [48] A. Yu, K. Grauman, Semantic jitter: Dense supervision for visual comparisons via synthetic images, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5570–5579.
  * [49] Z. Liu, P. Luo, X. Wang, X. Tang, Deep learning face attributes in the wild, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 3730–3738.
  * [50] Y. C. Liu, Y. Y. Yeh, T. C. Fu, S. D. Wang, W. C. Chiu, Y. C. F. Wang, Detach and adapt: Learning cross-domain disentangled deep representation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8867–8876.
  * [51] J. F. Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 8 (6) (1986) 679–698.
  * [52] M. Long, Z. Cao, J. Wang, M. I. Jordan, Conditional adversarial domain adaptation, in: Advances in Neural Information Processing Systems (NeurIPS), 2018, pp. 1640–1650.
  * [53] Y. Pan, T. Yao, Y. Li, Y. Wang, C. Ngo, T. Mei, Transferrable prototypical networks for unsupervised domain adaptation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2239–2247.
  * [54] Y. Zhang, B. Deng, H. Tang, L. Zhang, K. Jia, Unsupervised multi-class domain adaptation: Theory, algorithms, and practice, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (5) (2020) 2775–2792.
  * [55] A. Sharma, T. Kalluri, M. Chandraker, Instance level affinity-based transfer for unsupervised domain adaptation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5361–5371.
  * [56] J. Hou, X. Ding, J. D. Deng, S. Cranefield, Deep adversarial transition learning using cross-grafted generative stacks, Neural Networks 149 (2022) 172–183.
  * [57] L. van der Maaten, G. Hinton, Visualizing data using t-SNE, Journal of Machine Learning Research 9 (2008) 2579–2605.
  * [58] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing 13 (4) (2004) 600–612.
  * [59] M. Long, Y. Cao, J. Wang, M. I. Jordan, Learning transferable features with deep adaptation networks, in: F. Bach, D. Blei (Eds.), International Conference on Machine Learning (ICML), Vol. 37 of PMLR, 2015, pp. 97–105.
  * [60] X. Peng, Z. Huang, Y. Zhu, K. Saenko, Federated adversarial domain adaptation, in: International Conference on Learning Representations (ICLR), 2020.