# CycleGANAS: Differentiable Neural Architecture Search for CycleGAN

Taegun An
Korea University
Department of Computer Science
<EMAIL_ADDRESS>Changhee Joo
Korea University
Department of Computer Science
<EMAIL_ADDRESS>
###### Abstract

We develop a Neural Architecture Search (NAS) framework for CycleGAN that
carries out unpaired image-to-image translation task. Extending previous NAS
techniques for Generative Adversarial Networks (GANs) to CycleGAN is not
straightforward due to the task difference and greater search space. We design
architectures that consist of a stack of simple ResNet-based cells and develop
a search method that effectively explore the large search space. We show that
our framework, called CycleGANAS, not only effectively discovers high-
performance architectures that either match or surpass the performance of the
original CycleGAN, but also successfully address the data imbalance by
individual architecture search for each translation direction. To our best
knowledge, it is the first NAS result for CycleGAN and shed light on NAS for
more complex structures.

##  1 Introduction

Generative Adversarial Networks [7] (GANs) is a unsupervised generative
modeling for diverse and realistic data generation with a generator and
discriminator trained in an adversarial manner. It can deal with the domain
with insufficient amounts of data or high labeling costs and also has a great
advantage over the domain where the generation of diverse and high-quality
data is critical. In consequence, many variants and extensions of GANs have
been proposed for a variety of tasks such as conditional generation, style
transfer, machine translation, and anomaly detection.

CycleGAN is one of the powerful extensions of GANs and was developed for the
image-to-image translation _without image pairing_ [44]. Successfully removing
the expensive laboring cost of building paired datasets, CycleGAN has been
intensively applied to many translation applications, e.g., style transfer,
medical diagnosis, voice conversion, etc. However, CycleGAN requires thorough
fine-tuning of multiple neural networks, which has to consider the task
objectives and possible imbalances in the dataset. Without the fine-tuning, it
is likely to suffer from instability as GANs [21, 22]. This often becomes an
obstacle when applying CycleGAN to new translation tasks [37] or designing a
versatile architecture suitable for multiple tasks.

As a result, for CycleGAN, it is common to have a manual optimization process
to tailor its architecture to specific applications [18, 8, 13]. Such manual
optimization is not only very costly but also arbitrarily restricts the scope
of the architecture search.

To our best knowledge, there is no priori NAS work for CycleGAN. However,
there have been several interesting NAS works for unconditional GANs including
AGAN [33], AutoGAN [6], and AdversarialNAS [4]. AGAN and AutoGAN adopted
Reinforcement Learning (RL) to explore architectures through trial-and-error.
AdversarialNAS has a more complex cell structure than the others, which
motivates it to exploit the gradient-based approach introduced in DARTs [20]
to effectively search on a large search space $(10^{38})$. Although they are
an effective NAS method for GANs, their extension to CycleGAN is neither
straightforward nor fruitful due to essential differences in the task and
structure. In general, an image-to-image translation task is more complex than
an unconditional image generation task, and CycleGAN has twice as many neural
networks as GANs. Further, unlike unconditional GANs that can be trained with
any random inputs, CycleGAN work only with inputs from finite unpaired
datasets and thus should be more sample efficient. The larger search space and
the limited inputs are one of the key factors that differentiate the NAS for
CycleGAN and the NAS for GANs, and motivate us to develop a novel NAS
framework for CycleGAN.

In this paper, we propose CycleGANAS, a multi-network architecture search
framework for CycleGAN under data imbalance. We design a simple cell inspired
by the design of residual cells and build the supernetworks of generators and
discriminators of CycleGAN by stacking many simple cells. Accounting for the
task nature of CycleGAN and the required search efficiency, we optimized the
architecture and neural network weights with CycleGAN objectives
_simultaneously_ , which is different from the previous NAS for GANs that
takes an iterative bi-level optimization method. Finally, during the search,
we let the two generators have a different architecture, allowing them to be
better tailored to each subtask and dataset. Through the experiments on
various unpaired datasets, we show that CycleGANAS searches for good
architectures not only efficiently but also in a stable manner, even under the
data imbalance.

Our contributions can be summarized as follows.

  * •

We develop CycleGANAS, a novel framework of multi-network architecture search
for CycleGAN under data imbalance. We stack up many simple ResNet-based cells
and take the gradient-based approach along with single-level joint
optimization of neural network architecture and weights. Our framework admits
asymmetric architectures that take into account the data imbalance.

  * •

We show the performance of CycleGANAS through extensive experiments with
various unpaired datasets. We investigate the effect of CycleGAN architecture
and model size, demonstrating that manual balancing or naive asymmetric models
are not effective for data imbalance. In contrast, we observe that CycleGANAS
successfully searches for good architectures in a stable manner and achieve
high performance under the data imbalance.

##  2 Related Works

###  2.1 Neural Architecture Search (NAS)

Neural architecture search (NAS) is a big branch of automated machine learning
(Auto-ML) to search for the best neural network design, taking into
consideration the task and dataset. NAS studies are commonly classified using
three criteria: the search space, search method, and evaluation method [9, 3].
Among these, the search method is the key element for identifying various NAS
algorithms. In the following, we provide a concise overview of NAS algorithms
in the literature from the perspective of the search method.

Several search methods have been developed for NAS including RL, evolutionary
algorithms (EA), differentiable methods, and other optimization techniques.
Since RL was successfully adopted for NAS in [46], it has been exploited to
optimize many architectures including CNNs [1, 2, 26]. Although they
outperform the handcrafted neural networks in performance, they demand a
significant amount of computation. Although recent NAS frameworks with
Bayesian optimization (BO) [12, 35, 43, 29, 30] or evolutionary algorithms
(EA) [27, 25, 19] substantially reduce the computation cost, their
applications are still limited to a problem with small search space. Gradient-
based NAS method, first appeared in DARTS [20], enables NAS with a large
search space through continuous relaxation of architectures. It has been
reported that in certain problems, the architecture converges to the optimal
one under the gradient-based NAS [16].

###  2.2 Generative Adversarial Networks (GANs)

Since the GANs framework has been developed for the unconditional image
generation via adversarial training of generator and discriminator, the
framework has been extended to many computer vision tasks including super-
resolution [41, 34], image inpainting [38], natural language processing (NLP)
[39, 5], etc.

CycleGAN is also an extension of the GANs framework to the multi-network
system for image-to-image translation task. Pix2pix [10] is the first work
that performs the translation task with two pairs of GANs and paired data.
CycleGAN [44] and DiscoGAN [14] remove the requirement of the dataset pairing
by introducing the cycle-consistency objective, and enable the translation
with _unpaired data_. The technique has been now widely used for medical image
translation [42], frame prediction [15], inter-domain translation [28], and
multi-modal learning [45].

###  2.3 NAS for GANs

Since the success of NAS for convolutional neural networks (CNNs), NAS for
GANs has attracted much attention. As in CNNs, RL-based search methods were
adopted in NAS for GANs [33, 6, 40], in which the architecture of GANs is
divided into a predetermined number of cells and optimized through an LSTM
agent. A common challenge of these RL-based approaches is the computational
complexity to obtain Inception Score (IS) or Frechet inception distance (FID)
score, which is used as the reward feedback. Some works introduce score
predictors to mitigate the computation burden [36].

The most relevant to our work among previous NAS frameworks for GANs is
AdversarialNAS [4] that adopts the gradient-based, differentiable architecture
search method. Using the adversarial loss, AdversarialNAS could achieve state-
of-the-art performance on a large search space at the cost of $1$ GPU day.
However, the extension of AdversarialNAS to CycleGAN is not straightforward,
since it has a large and complicated cell structure, which makes it hard to
scale, and its bi-level optimization is not sufficiently efficient for the
task with a small amount of data, i.e., the image-to-image translation task of
CycleGAN [31].

While not directly related to NAS, there are a few studies that focus on
compressing the architecture of CycleGAN via combinatorial optimization [17]
or evolutionary algorithm [32]. Given good reference architectures, they find
small-size architectures with comparable performance. Although they are doing
a sort of architecture search, their approach is quite different from NAS
since they have clear reference models.

##  3 CycleGANAS – NAS for CycleGAN

###  3.1 Preliminaries

Figure 1: Overall process of CycleGAN and its losses. CycleGAN uses two
generators $G_{A},G_{B}$ to translate image $a\in\mathcal{A}$ to image
$b^{\prime}\in\mathcal{B}$ and image $b\in\mathcal{B}$ to image
$a^{\prime}\in\mathcal{A}$, and two discriminators $D_{A},D_{B}$ to
distinguish the generated images. It computes several losses using the images
from different sources.

CycleGAN has two generators ($G_{A},G_{B}$) and two discriminators
($D_{A},D_{B}$) as shown in Fig. 1, and translate images in dataset, which
consists of two subdatasets, each from a different domain denoted by
$\mathcal{A},\mathcal{B}$. We slightly abuse the notation and also denote the
subdataset of each domain by $\mathcal{A}$ and $\mathcal{B}$, respectively.
CycleGAN aims to learn the mapping of optimal generators
$G^{*}_{A}:\mathcal{A}\rightarrow\mathcal{B}$,
$G^{*}_{B}:\mathcal{B}\rightarrow\mathcal{A}$, and optimal discriminators
$D^{*}_{A}:\Omega\rightarrow\\{0,1\\},D^{*}_{B}:\Omega\rightarrow\\{0,1\\}$
that distinguish between translated samples and real samples, where $\Omega$
denotes the set of all possible image samples. For CycleGAN, we relax the
output of discriminators to a real number in the range $[0,1]$, and consider
$G_{A},G_{B},D_{A},D_{B}$ as a function that takes an input image and outputs
an image or a number in $[0,1]$. The objective of CycleGAN consists of three
loss functions, known as the adversarial, cycle-consistency, and identity loss
[44].

  * •

The adversarial objective makes each pair of generator and discriminator
engage in a two-player mini-max game. The _adversarial loss_
$L_{adv}(G_{A},D_{B})$ for $G_{A},D_{B}$ can be written as

$\begin{split}L_{adv}(G_{A},D_{B})&=\mathbb{E}_{b\sim\mathcal{B}}[\log
D_{B}(b)]\\\
&+\mathbb{E}_{a\sim\mathcal{A}}[\log(1-D_{B}(G_{A}(a)))].\end{split}$ (1)

The adversarial loss $L_{adv}(G_{B},D_{A})$ for $G_{B}$ and $D_{A}$ can be
defined similarly.

  * •

It is claimed that CycleGAN should be cycle-consistent, i.e.,
$G_{B}(G_{A}(a))\approx a$ for $a\in\mathcal{A}$ and $G_{A}(G_{B}(b))\approx
b$ for $b\in\mathcal{B}$. The cycle consistency objective is imposed only on
the generators and couples them under the cooperative framework to generate
better output images for each other. The _cycle-consistency loss_
$L_{cyc}(G_{A},G_{B})$ can be written as

$\begin{split}L_{cyc}(G_{A},G_{B})&=\mathbb{E}_{a\sim\mathcal{A}}[|G_{B}(G_{A}(a))-a|]\\\
&+\mathbb{E}_{b\sim\mathcal{B}}[|G_{A}(G_{B}(b))-b|].\end{split}$ (2)

  * •

It has been shown that, if the input image does not belong to the target
domain, the identity objective helps the generator to preserve the identity of
an input image, e.g., color composition. The _identity loss_ $L_{idt}(G_{A})$
for $G_{A}$ can be written as

$L_{idt}(G_{A})=\mathbb{E}_{b\sim\mathcal{B}}[|G_{A}(b)-b|].$ (3)

The identity loss for $G_{B}$ can be written similarly.

The full objective of CycleGAN is formed by linearly combining the three loss
functions as

$\begin{split}&L(G_{A},G_{B},D_{A},D_{B})\\\
&=\lambda_{1}L_{adv}(G_{A},D_{B})+\lambda_{2}L_{adv}(G_{B},D_{A})\\\
&+\lambda_{3}L_{cyc}(G_{A},G_{B})+\lambda_{4}L_{idt}(G_{A})+\lambda_{5}L_{idt}(G_{B}),\end{split}$
(4)

where $\lambda_{1},\dots,\lambda_{5}$ are a weight. Through this work, we set
$\lambda=(\lambda_{1},\dots,\lambda_{5})=(1,1,10,5,5)$ unless otherwise
stated. Note that the discriminators are involved only in the adversarial
losses.

###  3.2 Search space

We develop CycleGANAS that simultaneously searches neural network
architectures for $G_{A},G_{B},D_{A},D_{B}$. Considering the significant
challenge in conducting architecture search from scratch, many NAS approaches
restrict their search space to a combination of operations referred to as a
_cell_ , and construct a neural network architecture by stacking a few cells
[6, 4]. We also follow the cell-based NAS approach, but different from the
previous works, we use a much simpler cell structure and stack many of them to
construct an architecture.

Motivated by the neural network architectures of CycleGAN, we design a ResNet-
based cell for CycleGANAS. Our cell has only two operations, each of which
followed by a normalization layer and an activation layer. We restrict the
cell’s operations by _cell type_ , which can be either encoding $e$, residual
$r$, or decoding $d$. Letting $\mathcal{S}^{T}$ be the set of possible
operations for the cell type $T\in\\{e,r,d\\}$, we define the operation sets
as

  * •

$\mathcal{S}^{e}=\\{$max pooling, avg pooling, Conv3x3, Conv4x4, Conv5x5,
Conv7x7, DilConv3x3, DilConv5x5$\\}$,

  * •

$\mathcal{S}^{r}=\\{$Conv3x3, Conv5x5, Conv7x7, DilConv3x3, DilConv5x5$\\}$,

  * •

$\mathcal{S}^{d}=\\{$Nearest neighbor interpolation, Bi-linear interpolation,
Transposed Conv3x3$\\}$.

A cell $C^{T}$ of type $T$ has two operations from $\mathcal{S}^{T}$.

We build a generator $G$ of $N$ cells; starting from one encoding cell,
followed by $N-2$ residual cells, and ending with one decoding cell. Thus, we
can represent a generator as a sequence of cells:
$G=(C^{e}_{1},C^{r}_{2},\dots,C^{r}_{N-1},C^{d}_{N})$. For a decoder $D$, we
have it with a much simpler structure of $2$ encoding cells:
$D=(C^{e}_{1},C^{e}_{2})$. Note that a generator has the search space of size
$|\mathcal{S}^{e}|^{2}\cdot|\mathcal{S}^{r}|^{2(N-2)}\cdot|\mathcal{S}^{d}|^{2}$,
where $|\cdot|$ denotes the cardinality. We will use $N=11$ as a default
setting for each generator, resulting in $8^{2}\times 5^{18}\times
3^{2}\approx 2.2\times 10^{15}$. Similarly, each discriminator has the search
space of size $8^{4}$. Since there are two generators and two discriminators,
the total search space size of CycleGANAS is $8.1\times 10^{37}$, which is
comparably large considering those of AutoGAN ($10^{5}$) and AdversarialNAS
($10^{38}$).

Figure 2: A super-network cell with two mixed operations
$\bar{o}^{1},\bar{o}^{2}$. The weights $\alpha$ of the mixed operations are
trained by the gradient descent. After the architecture search, each cell is
converted to an ordinary cell with two discrete operations.

Element-wise searching in the huge space will demand prohibitively large
computational cost. To avoid excessive computation, we adopt the idea of
_mixed operation_ from DARTS [20] and take the approach of _differentiable
neural architecture search_. Basically, we convert the discrete search space
into a continuous one by replacing an operation with a mixed one of multiple
operations that admits the gradient-based search. To elaborate, consider a
cell $\bar{C}^{T}$ that has two mixed operations $\bar{o}^{1},\bar{o}^{2}$ in
order, where each $\bar{o}^{i}$ with $i\in{1,2}$ is a combination of all
possible operations, i.e.,

$\bar{o}^{i}=\sum_{o\in\mathcal{S}^{T}}\beta^{i}_{o}\cdot o,$

where
$\beta^{i}_{o}=\frac{\exp(\alpha^{i}_{o})}{\sum_{p\in\mathcal{S}^{T}}\exp(\alpha^{i}_{p})}$
and $\alpha^{i}_{o}$ is the weight of operation $o\in\mathcal{S}^{T}$, as
shown in Fig. 2. Also, suppose that we build a super-network by stacking the
cells with mixed operations. Then we can optimize all $\alpha^{i}$’s of the
super-network’s cells through gradient descent. When the search finishes, we
construct a discrete architecture by selecting the highest-weight operations
in the super-network, i.e., each cell
$\bar{C}^{T}=\\{\bar{o}^{1},\bar{o}^{2}\\}$ of the super-network is converted
to an ordinary cell $C^{T}=\\{o^{1},o^{2}\\}$, where
$o^{i}=\arg\max_{o\in\mathcal{S}^{T}}\alpha^{i}_{o}$.

Architecture search with the super-network demands a substantial amount of
memory and time to take into account all the possible operations, which can be
burdensome in practice. To this end, it is common that, for the search, one
uses the super-network with reduced hidden dimension, and after the search,
builds the discrete architecture with restored hidden dimension [4]. We also
apply the technique of the _hidden dimension reduction_ to CycleGANAS for the
architecture search. For example, during the search, a (super-network)
generator of our CycleGANAS takes an image input of $256\times 256\times 3$,
and encodes it to a tensor of $64\times 64\times 64$, whose hidden dimension
size is smaller than that of the original CycleGAN’s encoder output ($64\times
64\times 256$). Passing through the residual blocks, the generator decodes it
back to the shape of $256\times 256\times 3$. Once the search completes, from
the trained super-network, we construct a discrete architecture, whose encoder
output has the shape of $64\times 64\times H$, where the hidden dimension $H$
is a hyperparameter.

###  3.3 Optimization process

Algorithm 1 One-step CycleGANAS.

Initialize weights:
$\theta_{G_{A}},\theta_{G_{B}},\theta_{D_{A}},\theta_{D_{B}}$
Initialize differentiable architecture: $G,G,D,D$
Input: Unpaired dataset $\mathcal{A,B}$
Output: architecture of $G_{A},G_{B},D_{A},D_{B}$

1: for each search epoch do

2: for each iteration do

3: sample $a\in\mathcal{A}$ and $b\in\mathcal{B}$

4: /* Forward path */

5: Compute $L_{adv}(G_{A},D_{B}),L_{adv}(G_{B},D_{A})$

6: Compute $L_{cyc}(G_{A},G_{B}),L_{cyc}(G_{B},G_{A})$

7: Compute $L_{idt}(G_{A}),L_{idt}(G_{B})$

8: Compute $L(G_{A},G_{B},D_{A},D_{B})$

9: /* Backward path */

10: Update $\theta_{G_{A}},\theta_{G_{B}}$ from $L(\cdot)$

11: Update $\theta_{D_{A}},\theta_{D_{B}}$ from corresponding $L_{adv}(\cdot)$

12: end for

13: end for

14: return architecture of $G_{A},G_{B},D_{A},D_{B}$

Let $\alpha$ denote the vector of all architecture weights, and let $w$ denote
the vector of neural network weights. Note that if $\alpha$ changes, the
optimal $w$ also changes, and vice versa. Typically, a NAS for GANs takes a
bi-level optimization process to optimize $\alpha$ and $w$, which is
equivalent to iteratively apply the following two equations in sequence,

$\begin{split}\alpha^{*}&=\textstyle\arg\min_{\alpha}L_{val}(\alpha,w^{*}),\\\
w^{*}&=\textstyle\arg\min_{w}L_{train}(\alpha^{*},w),\end{split}$ (5)

where $L_{val}$ is a loss with the validation dataset and $L_{train}$ is a
loss with the training dataset. For example, AutoGAN has the FID score for
$L_{val}$ and the adversarial loss for $L_{train}$, and AdversarialNAS has the
adversarial loss for both $L_{val}$ and $L_{train}$.

We highlight that the bi-level optimization (5) requires the computation of
$L_{train}$ and $L_{val}$ from two exclusive datasets for stability. AutoGAN
and AdversarialNAS can easily accomplish this since the input to the GANs
generator is a random sample from Gaussian noise. For CycleGAN, however, the
input to a generator is not a random sample but an image from a subdataset,
and thus each input subdataset has to be divided into two for the bi-level
optimization. Further, rotating the role of divided subdatasets will be
necessary; otherwise, the architecture optimizer and weight optimizer will
only observe a portion of the entire subdataset, leading to a performance
degradation. As a result, the bi-level optimization process of CycleGANAS
requires a more intricate approach to dividing the dataset and rotating their
roles.

We simplify the optimization process by developing a single-level optimization
process of $\alpha$ and $w$. We use the entire input subdataset for the joint
optimization of

$\textstyle(\alpha^{*},w^{*})=\arg\min_{\alpha,w}~{}L_{train}(\alpha,w),$ (6)

where $L_{train}=L(G_{A},G_{B},D_{A},D_{B})$ is the CycleGAN loss (4). Note
that the single-level optimization not only eliminates the need for dividing
the input subdataset but also reduces the number of optimization steps by
half, in comparison to the bi-level optimization. This results in an
accelerated learning process. We denote CycleGANAS with the single-level
optimization by _one-step_ CycleGANAS, whose detailed process is shown in
Algorithm 1, where $\theta=(\alpha,w)$.

##  4 Experiments

We evaluate CycleGANAS with several unpaired datasets, e.g. maps, facades,
apple2orange, horse2zebra, summer2winter, and iphone2dslr-flower.Each dataset
has two subdatasets, each of which may have a different number of images, and
all the images are of the same shape $3\times 256\times 256$. For certain
datasets, there exists an imbalance in the data, and the image translations
between the two subdatasets have different levels of difficulty. For example,
in the horse2zebra dataset, the number of pixels taken by zebra is more than
twice as many as those taken by horse [23]. This data imbalance demands
asymmetric capability of neural networks, in particular, of the two
generators. In general, a more challenging translation task, e.g., zebra-to-
horse, is likely to necessitate a larger generator model111We remark that the
original CycleGAN is symmetric – its two generators have the same
architecture, and the same for the two discriminators.. We demonstrate that
CycleGANAS not only successfully search high-performance architectures, but
also, in the presence of dataset imbalance, naturally adopts asymmetric
architectures, effectively mitigating the issue.

Throughout our experiments, we mostly use the configuration of CycleGAN [44],
e.g., batch size $1$ and instance normalization. For one-step CycleGANAS of
Algorithm 1, we use Adam optimizer with learning rate of $\alpha=0.0002$,
$\beta_{1}=0.5$, $\beta_{2}=0.999$, and set the maximum search epoch to $400$.
All our experiments are based on Python 3.8, CUDA 11.3, CuDNN 8.2.0, and the
learning frameworks are implemented with PyTorch. To evaluate architectures,
we use the FID score estimation provided by the clean-fid project [24]. Since
the FID score is stochastic for an architecture [11], we repeat the weight
training several times and select the one with the best (lowest) FID score.

We present the impact of the optimization method on performance, and evaluate
the performance of CycleGANAS using diverse datasets and illustrate its
response to data imbalance.

###  4.1 Bi-level vs single-level optimization

Figure 3: Performance of the four variants of CycleGANAS with three datasets,
in terms of the generator model size and FID scores. The model sizes are
normalized w.r.t. that of the original CycleGAN’s generator, and the FIDs
w.r.t. the average FIDs of one-step CycleGANAS (OF). For each architecture
outcome, we repeat the weight training $3$ times, and mark the FIDs by cross
and the average by blue dash.

Figure 4: Performance of CycleGAN (red) and the architecture outcomes of
CycleGANAS (green) with different hidden dimension sizes. We evaluate each
architecture $3$ times. The mean (thick line) and range (light shade) of FIDs
are shown. The outcomes of CycleGANAS achieve comparable performance or even
outperforms CycleGAN.

Figure 5: Performance of CycleGAN (red) and the architecture outcomes of
CycleGANAS (green). Two CycleGAN architectures with $H=32$ ($C32$) and $H=64$
($C64$) are used. We also use two versions of CycleGANAS architecture outcomes
($N32$ and $N64$), such that their total model sizes are roughly the same as
$C32$ and $C64$, respectively. Comparing their performance over 6 datasets, we
can observe that the architectures searched by CycleGANAS outperform CycleGAN,
in particular, for the larger model size.

Besides the single-level optimization, we can incorporate our CycleGANAS
framework with the previous bi-level optimization, which is denoted by _two-
step_ CycleGANAS. Numerous variations of the two-step CycleGANAS can emerge
based on how the subdataset is divided and rotated. We consider the following
three variants of two-step CycleGANAS.

  * •

(TF) Two-step CycleGANAS with the full subdataset: Without dividing the
subdatasets, we use the bi-level optimization of $w$ using $\mathcal{A}$ and
$\alpha$ using $\mathcal{B}$.

  * •

(TH) Two-step CycleGANAS with evenly halved subdatasets: For $\mathcal{A}$, we
assign all its images to $\mathcal{A}_{1}$ or $\mathcal{A}_{2}$ at random such
that $|\mathcal{A}_{1}|=|\mathcal{A}_{2}|$. Similarly, we have
$\mathcal{B}\rightarrow\mathcal{B}_{1},\mathcal{B}_{2}$. Then, we use the bi-
level optimization of $w$ using $\mathcal{A}_{1},\mathcal{B}_{1}$ and $\alpha$
using $\mathcal{A}_{2},\mathcal{B}_{2}$ throughout the search.

  * •

(THS) Two-step CycleGANAS with evenly halved subdatasets and swapping: It is
similar to Two-half, except that after certain epoch, we swap the role of the
halved subdatasets. Our intention is for the optimizers of $w$ and $\alpha$ to
have visibility over all images within the entire subdatasets. In our
experiments, we do the bi-level optimization of $w$ using
$\mathcal{A}_{1},\mathcal{B}_{1}$ and $\alpha$ using
$\mathcal{A}_{2},\mathcal{B}_{2}$ for the first $200$ epochs, and then do the
optimization of $w$ using $\mathcal{A}_{2},\mathcal{B}_{2}$ and $\alpha$ using
$\mathcal{A}_{1},\mathcal{B}_{1}$ afterward.

Unlike the two-step CycleGANAS, our one-step CycleGANAS features a simpler
optimization process and necessitates only half the iterations.

  * •

(OF) One-step CycleGANAS with the full subdatasets. We jointly optimize $w$
and $\alpha$ using $\mathcal{A},\mathcal{B}$.

We search CycleGAN architectures using the above four schemes with three
different datasets of facades, horse2zebra, and iphone2dslr. Fig. 3 shows the
model size of the generators and FIDs of their searched architectures, which
are normalized by the model size of the original CycleGAN’s generator, and by
the minimum FIDs of one-step OF CycleGANAS, respectively.

In the facades dataset, the model sizes are all similar and the best (lowest)
FID is achieved by two-step THS. On the other hand, one-step OF achieves the
least variation of FIDs and the lowest average, which implies that one-step OF
performs well in a stable manner. In horse2zebra with data imbalance, all
schemes effectively discover asymmetric architectures, with a focus on
enhancing the generator for zebra-to-horse translation, although the extent of
asymmetry varies among the schemes. Also we can observe that two-step THS and
one-step OF outperform two-step TH. We conjecture that it is due to the
limited visibility of two-step TH’s optimizers to the subdatasets. In
iphone2dslr, CycleGAN is asked to increase or decrease the image resolution,
and it is clear that the resolution-increasing task is more difficult than the
other. Similar to the horse2bebra case, most schemes except two-step TH
provide asymmetric architectures enhancing the generator for iphone-to-dslr
translation. Further, one-step OF achieves the best FID scores. Another
interesting observation is that the reversely asymmetric architectures
searched by two-step TH also achieve comparable performance.

Overall, two-step TF/TH/THS schemes exhibit a high variation depending on the
translation task, while one-step OF achieves a competitive, close-to-best
result in a stable manner. Further it is worth highlighting that one-step OF
completes the optimization process in half the iterations of two-step methods.
Given its performance and simplicity, we establish it as the default choice
for CycleGANAS. Henceforth, when referring to CycleGANAS, we are referring to
one-step CycleGANAS.

###  4.2 Performance evaluation

We demonstrate the search capability of CycleGANAS. Over $6$ datasets, we
evaluate the architectures searched by CycleGANAS in comparison with the
original CycleGAN. We vary the hidden dimension size $H$ for both CycleGAN and
the search outcomes of CycleGANAS to see their achievable performance. For
each architecture, we repeat its evaluation $3$ times.

Fig. 4 shows the mean (thick line) and range (light shade) of the FID scores
of CycleGAN (red) and the architecture outcomes of CycleGANAS (green).
Overall, in terms of the lowest FIDs, the outcomes of CycleGANAS demonstrate
comparable performance (for maps and iphone2dslr) or even outperform CycleGAN
(for facades, apple2orange, horse2zebra, and summer2winter). Another
interesting observation is that a larger $H$ does not imply a better
performance, and there is an optimal value, usually between $32$ and $96$,
depending on the dataset and architecture.

Remark: the architecture of CycleGAN and the outcomes of CycleGANAS have
convolutions of different filter sizes, leading to differences in their model
size in bytes even when they have the same hidden dimension size $H$.

Next we conduct a more direct performance comparison between CycleGANAS
outcomes and CycleGAN by ensuring their model sizes are equal. From the
previous experiment results, we train two CycleGAN architectures with $H=32$
and $64$, denoted by $C32$ and $C64$, respectively. For the architectures
searched by CycleGANAS, we adjust the hidden dimension sizes accordingly such
that the total model size roughly equals that of $C32$ or $C64$. Note that we
configure the two generators searched by CycleGANAS to have the same hidden
dimension size $H$, and as a result, depending on their chosen convolution
operations, they will have a different model size, leading to asymmetric
architectures.

Fig. 5 shows the experiment results over $6$ datasets in terms of generator
model sizes and FIDs. All the model sizes are normalized with respect to that
of $C32$ generator ($11.378$ MB), and the FIDs with respect to $C32$’s mean
FIDs. $N32$ and $N64$ denote the architectures searched by CycleGANAS with
normalized $H$ with respect to $C32$ and $C64$, respectively. Overall,
CycleGANAS successfully finds good architectures for most datasets (except
maps). Its architectures have comparable (for summer2winter and iphone2dslr)
or better performance (for facades, apple2orange, horse2zebra) than the
CycleGAN counterpart. In particular, with the larger model size, the outcomes
of CycleGANAS outperform the original CycleGAN, by up to $30$% in
$\mathcal{A}$-to-$\mathcal{B}$ and $10\%$ in $\mathcal{B}$-to-$\mathcal{A}$.
Further we can also observe that, for the datasets with imbalance (horse2zebra
and iphone2dslr), CycleGANAS provides asymmetric generator architectures
accordingly, and for the others, it yields the generators of similar model
size.

Figure 6: Performance of the original CycleGAN with $H=64$ ($C64$), its
asymmetric variant by scaling the hidden dimension ($C^{\prime}64$), and the
architecture outcome of CycleGANAS ($N64$). Scaling the hidden dimension does
not improve the performance, while the architecture search does.

###  4.3 Architecture search vs hidden dimension scaling

The two generators of the original CycleGAN have the same architecture. In
contrast, the architectures searched by CycleGANAS may have asymmetric
structure that naturally comes from the dataset when it selects appropriate
operations in $\mathcal{S}^{e},\mathcal{S}^{r},\mathcal{S}^{d}$ for each of
the two generators. One may argue that the crucial factor to the performance
is the asymmetric model size of the two generators, rather than the selection
of their operations. We show that this is not the case, and the operation
selection (i.e., architecture search) is of great importance.

For the datasets of horse2zebra and iphone2dslr, we consider the symmetric
architectures of $C64$ (CycleGAN) and the asymmetric architectures $N64$
searched by CycleGANAS. Note that their total model sizes are the same. Then
we build another asymmetric architectures $C^{\prime}64$ by scaling the hidden
dimension of $C64$ generators. To elaborate, we scale the hidden dimensions of
two generators of $C64$ such that their model sizes equal to those of $N64$’s
two generators, respectively. As a result, each generator of $C^{\prime}64$
has the same operations as the $C64$ counterpart and the same model size as
the $N64$ counterpart.

Fig. 6 shows the model sizes and the FIDs of $5$ evaluations, normalized
w.r.t. the generator model size and mean FIDs of $C64$. It is confirmed that
the generators of $C^{\prime}64$ have the same model sizes as those of $N64$.
We can observe that the mean FIDs ($\mathcal{A}$-to-$\mathcal{B}$,
$\mathcal{B}$-to-$\mathcal{A}$) of $C^{\prime}64$ are ($45.19,132.87$) for
horse2zebra and ($84.03,93.41$) for iphone2dslr-flower, which are similar to
$C64$’s ($45.78,133.42$) and ($84.12,91.70$), respectively. The searched
asymmetric architectures of $N64$ achieve the lowest FIDs of ($38.05,131.44$)
and ($80.07,89.75$), respectively. This suggests that, when dealing with data
imbalance, simply scaling the hidden dimension might not yield substantial
benefits; instead, identifying suitable operations becomes of paramount
importance.

##  5 Conclusion

We develop a NAS framework for CycleGAN that carries out unpaired image-to-
image translation task. Compared to NAS for GANs, NAS for CycleGAN is more
challenging due to the limited samples from dataset, multiple neural networks
and their involvement in learning, and data imbalance.

We design a framework, called CycleGANAS, that can search CycleGAN
architectures of two generators and two discriminators, simultaneously. For
flexible and practical search, we build architectures by stacking many simple
ResNet-based cells, take the approach of differentiable search through super-
networks, and apply the hidden dimension reduction. We further reduce the
computational complexity and stabilize the search with the single-level
optimization, enabling CycleGANAS to effectively explore a vast search space
of size $8.1\times{10}^{36}$.

Our experiments demonstrate that CycleGANAS effectively discovers high-
performance architectures that either match or surpass the performance of the
original CycleGAN. Furthermore, CycleGANAS successfully addresses the data
imbalance by individually searching for distinct generator architectures for
each translation direction, thereby regulating the model ratio.

## References

  * Baker et al. [2017] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar.  Designing neural network architectures using reinforcement learning.  In _International Conference on Learning Representations_ , 2017.
  * Cai et al. [2018] Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang.  Efficient architecture search by network transformation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), 2018.
  * Elsken et al. [2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  _Journal of Machine Learning Research_ , 20(55):1–21, 2019.
  * Gao et al. [2020] Chen Gao, Yunpeng Chen, Si Liu, Zhenxiong Tan, and Shuicheng Yan.  Adversarialnas: Adversarial neural architecture search for gans.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Gao et al. [2018] Yang Gao, Rita Singh, and Bhiksha Raj.  Voice impersonation using generative adversarial networks.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2506–2510, 2018.
  * Gong et al. [2019] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang.  Autogan: Neural architecture search for generative adversarial networks.  In _The IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2014.
  * Haws and Cui [2019] David Haws and Xiaodong Cui.  Cyclegan bandwidth extension acoustic modeling for automatic speech recognition.  In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6780–6784, 2019.
  * He et al. [2021] Xin He, Kaiyong Zhao, and Xiaowen Chu.  AutoML: A survey of the state-of-the-art.  _Knowledge-Based Systems_ , 212:106622, 2021.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Jo and Joo [2023] Haesung Jo and Changhee Joo.  Autogan-dsp: Stabilizing gan architecture search with deterministic score predictors.  Forthcoming, 2023.
  * Kandasamy et al. [2018] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing.  Neural architecture search with bayesian optimisation and optimal transport.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2018.
  * Kaneko et al. [2019] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo.  Cyclegan-vc2: Improved cyclegan-based non-parallel voice conversion.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019.
  * Kim et al. [2017] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon. Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , page 1857–1865. JMLR.org, 2017.
  * Kwon and Park [2019] Yong-Hoon Kwon and Min-Gyu Park.  Predicting future frames using retrospective cycle gan.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Li et al. [2021] Liam Li, Mikhail Khodak, Nina Balcan, and Ameet Talwalkar.  Geometry-aware gradient algorithms for neural architecture search.  In _International Conference on Learning Representations_ , 2021.
  * Li et al. [2020] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han.  Gan compression: Efficient architectures for interactive conditional gans.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Li et al. [2019] Yu Li, Sheng Tang, Rui Zhang, Yongdong Zhang, Jintao Li, and Shuicheng Yan.  Asymmetric gan for unpaired image-to-image translation.  _IEEE Transactions on Image Processing_ , 28(12):5881–5896, 2019.
  * Liang et al. [2021] Tingting Liang, Yongtao Wang, Zhi Tang, Guosheng Hu, and Haibin Ling.  Opanas: One-shot path aggregation network architecture search for object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10195–10203, 2021.
  * Liu et al. [2019] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable architecture search.  In _International Conference on Learning Representations_ , 2019.
  * Mescheder et al. [2017] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  The numerics of gans.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Mescheder et al. [2018] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for GANs do actually converge?  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 3481–3490. PMLR, 2018.
  * Park [2021] Taesung Park.  _Machine Learning for Deep Image Synthesis_.  PhD thesis, EECS Department, University of California, Berkeley, 2021.
  * Parmar et al. [2022] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  In _CVPR_ , 2022.
  * Peng et al. [2020] Wei Peng, Xiaopeng Hong, Haoyu Chen, and Guoying Zhao.  Learning graph convolutional network for skeleton-based human action recognition by neural searching.  _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI_ , 2020.
  * Pham et al. [2018] Hieu Pham, Melody Y. Guan, Barret Zoph, and Jeff Le, Quoc V. andDean.  Efficient neural architecture search via parameter sharing.  In _ICML_ , 2018.
  * Piergiovanni et al. [2019] AJ Piergiovanni, Anelia Angelova, Alexander Toshev, and Michael S. Ryoo.  Evolving space-time neural architectures for videos.  In _ICCV_ , 2019.
  * Qiao et al. [2019] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  Mirrorgan: Learning text-to-image generation by redescription.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Ru et al. [2021a] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne.  Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels.  In _International Conference on Learning Representations_ , 2021a.
  * Ru et al. [2021b] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne.  Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels.  In _International Conference on Learning Representations_ , 2021b.
  * Saxena et al. [2023] Divya Saxena, Jiannong Cao, Jiahao Xu, and Tarun Kulshrestha.  Re-gan: Data-efficient gans training via architectural reconfiguration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16230–16240, 2023.
  * Shu et al. [2019] Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu.  Co-evolutionary compression for unpaired image translation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Wang and Huan [2019] Hanchao Wang and Jun Huan.  Agan: Towards automated design of generative adversarial networks.  _ArXiv_ , abs/1906.11080, 2019.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018.
  * White et al. [2021] Colin White, Willie Neiswanger, and Yash Savani.  Bananas: Bayesian optimization with neural architectures for neural architecture search.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(12):10293–10301, 2021.
  * Yi et al. [2023] Yun Yi, Haokui Zhang, Wenze Hu, Nannan Wang, and Xiaoyu Wang.  Nar-former: Neural architecture representation learning towards holistic attributes prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7715–7724, 2023.
  * You et al. [2020] H. You, Y. Cheng, T. Cheng, C. Li, and P. Zhou.  Bayesian cycle-consistent generative adversarial networks via marginalizing latent sampling.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–15, 2020.
  * Yu et al. [2018] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Generative image inpainting with contextual attention.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Yu et al. [2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 31(1), 2017.
  * Yuan et al. [2020] Tian Yuan, Wang Qin, Huang Zhiwu, Li Wen, Dai Dengxin, Yang Minghao, Wang Jun, and Fink Olga.  Off-policy reinforcement learning for efficient and effective gan architecture search.  In _The European Conference on Computer Vision (ECCV)_ , 2020.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * Zhang et al. [2018] Zizhao Zhang, Lin Yang, and Yefeng Zheng.  Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9242–9251, 2018.
  * Zhou et al. [2019] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan.  BayesNAS: A Bayesian approach for neural architecture search.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 7603–7613. PMLR, 2019.
  * Zhu et al. [2017a] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017a.
  * Zhu et al. [2017b] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Toward multimodal image-to-image translation.  In _Advances in Neural Information Processing Systems 30_ , pages 465–476. Curran Associates, Inc., 2017b.
  * Zoph and Le [2017] Barret Zoph and Quoc Le.  Neural architecture search with reinforcement learning.  In _International Conference on Learning Representations_ , 2017.


HorizontalFlip | - | 0.5
GaussianBlur | $\sigma\sim U[0.1,2]$ | 0.5
Solarization | - | 0.4
Normalization | - | -

We evaluate the proposed method and two AM-SSL baselines as mentioned in the
main text, namely, Barlow Twins (Zbontar et al., 2021) and BYOL (Grill et al.,
2020) by feeding the learned representations to a linear classifier. We report
both the Top-1 linear classification accuracy and the KNN (with $k=5$)
accuracy. The results are shown in Tab. LABEL:tab:cifar10. One can see that
different methods achieve comparable results. The proposed method and BYOL
attain essentially the same accuracy. The t-SNE (Van der Maaten & Hinton,
2008) visualizations of the test set is plotted in Fig. G.4. One can see that
all methods extract “identity-revealing” information to a certain extent, as
in the MNIST case.

###### Remark 1

We should remark that the experiment results suggest that latent correlation
maximization (or latent component matching) used in many AM-SSL and DCCA
methods works towards the same ultimate goal under our generative model in
(5). However, this does not suggest that different SSL and DCCA methods are
essentially the same in practice—one should not expect that. In fact, there
are many factors that affect DCCA and AM-SSL methods’ results, e.g., model
mismatches, optimization procedure, network construction, and the detailed
designs in their loss functions. The difference between the methods normally
are more articulated with larger data sets or more complex problems.
Nonetheless, our interest lies in theoretical understanding of their common
properties, other than the differences in practical implementations. From this
perspective, the results in this section support our theoretical analysis in
Theorem 1.

Table G.8: Evaluation using CIFAR10. | BYOL | Barlow Twins | Proposed
---|---|---|---
Classification Acc. (%) | 84.2 | 82.8 | 84.2
KNN ($k=5$) Acc. (%) | 80.5 | 78.7 | 81.0
Figure G.4: t-SNE of learned representations for CIFAR10.

###  G.5 Real Data - More on Validating Theorem 2

Cars3D Data for Cross-sample Generation. In this subsection, we provide more
detailed settings and results of the Cars3D experiment. To create a multiview
dataset, we assume that given the car type that is captured by shared
variables $\bm{z}$ and the azimuths that are captured by $\bm{c}^{(q)}$, the
generation mappings $\bm{g}^{(1)}$ and $\bm{g}^{(2)}$ produce car images with
low elevations and high elevations, respectively (so they must be different
mappings).

We split the car images as follows. We treat the same car model (e.g., a red
convertible) with lower and higher elevations as $\bm{x}^{(1)}_{\ell}$ and
$\bm{x}^{(2)}_{\ell}$, respectively. The azimuths are randomly shuffled with
different pairs of $\bm{x}^{(1)}_{\ell}$ and $\bm{x}^{(2)}_{\ell}$. This way,
if our generative model holds, $\bm{z}$, $\bm{c}^{(q)}$ and $\bm{g}^{(q)}$ are
responsible for ‘type’, ‘azimuth’ and ‘elevation’, respectively. Some samples
are shown in Fig. G.5.

Figure G.5: Samples of the lower elevations (view1) and higher elevations
(view2).

Under our splitting, each view has $2\times 183\times 24=8784$ RGB images that
all have a size of $64\times 64\times 3$. We model the ‘type’ information
$\bm{z}$ using $D=10$ latent dimensions since many different factors (e.g.,
color and shape) together give rise to a ‘type’. On the other hand, we set
$D_{1}=D_{2}=2$ to model the ‘azimuth’ information.

Tab. G.9 shows network structures for the encoders and decoders of our
formulation in (F.6). In the table, FC denotes fully connected layer, Conv
denotes convolutional layer and Conv_Trans denotes 2D transposed convolutional
layer. As before, the structures of $\bm{\phi}^{(q)}$ and $\bm{\tau}^{(q)}$
are MLPs with two-hidden-layer and 256 neurons for each layer. For
hyperparameters, we set batch size to be $|\mathcal{B}_{1}|=100$ and
$|\mathcal{B}_{2}|=800$, $\beta=0.1$, $\lambda=1.0$. For this real data
experiment, we also use Adam (Kingma & Ba, 2014) as the optimizer with initial
learning rate $0.001$ for $\bm{\theta}$ and $1.0$ for $\bm{\eta}$. We add
$\|\bm{\eta}\|_{2}^{2}$ for regularization with parameter $0.0001$. We limit
the inner loops for solving the $\bm{\theta}$ and $\bm{\eta}$ subproblems to
10 epochs as well.

Table G.9: Network structures for the Cars3D experiment. Encoders | Decoders
---|---
input: $\bm{x}^{(q)}_{\ell}\in\mathbb{R}^{64\times 64\times 3}$ | input: $\bm{f}^{(q)}(\bm{x}^{(q)}_{\ell})\in\mathbb{R}^{10+2}$
$4\times 4$ Conv, 32 ReLU, stride 2 | FC 256, ReLU
$4\times 4$ Conv, 32 ReLU, stride 2 | FC $4\times 4\times 64$, ReLU
$4\times 4$ Conv, 64 ReLU, stride 2 |  $4\times 4$ Conv_Trans, 64 ReLU, stride 2
$4\times 4$ Conv, 64 ReLU, stride 2 |  $4\times 4$ Conv_Trans, 32 ReLU, stride 2
FC 256, ReLU |  $4\times 4$ Conv_Trans, 32 ReLU, stride 2
FC 10+2 |  $4\times 4$ Conv_Trans, 3, stride 2

Figs. G.6 and G.7 show more results under the same setting as in Fig. 2 in the
main text.

Figure G.6: Generated samples by fixing $\widehat{\bm{z}}_{\ell}$ and varying
$\widehat{\bm{c}}_{j}^{(q)}$; rows in blue boxes are w/ $\mathcal{R}$; rows in
green boxes are w/o $\mathcal{R}$. Figure G.7: Generated samples by fixing
$\widehat{\bm{c}}_{j}^{(q)}$ and varying $\widehat{\bm{z}}_{l}$; rows in blue
boxes are w/ $\mathcal{R}$; rows in green boxes are w/o $\mathcal{R}$.

dSprites Data for Cross-sample data Generation. We present the results on an
additional dataset, i.e., dSprites (Higgins et al., 2017). In the dSprites
dataset, $64\times 64$ images are generated based on five factors: 3 shapes
(square, ellipse, heart), 6 scales, 40 orientations, and 32 different
horizontal and vertical coordinates.

In particular, we take a subset that contains squares and hearts as the two
views of data. In this subset, all the data samples are with the same scale.
We assume the generating functions $\bm{g}^{(1)}$ and $\bm{g}^{(2)}$ are
responsible for the shape of square and heart, respectively. We treat the
orientation and horizontal positions as the shared information, i.e.,
$\bm{z}$, and the vertical position as the private information $\bm{c}^{(q)}$.
The vertical coordinates are random and not matched between different pairs of
$\bm{x}^{(1)}_{\ell}$ and $\bm{x}^{(2)}_{\ell}$. Overall, we have $40,960$
samples for each view. We set $D=2$ and $D_{1}=D_{2}=1$.

We use the same neural network structure as in Tab. G.9. The only differences
lie in the input and latent dimensions. The network structure for
$\bm{\phi}^{(q)}$ and $\bm{\tau}^{(q)}$ are MLPs with two hidden layers of 128
neurons, and we set $|\mathcal{B}_{1}|=100$, $|\mathcal{B}_{2}|=500$,
$\beta=0.1$, and $\lambda=100.0$. Similar as before, we use the Adam (Kingma &
Ba, 2014) optimizer with initial learning rate $0.001$ for $\bm{\theta}$. As
before, we add a squared $\ell_{2}$ norm regularization on the network
parameters $\bm{\eta}$, and set the regularization parameter to $0.1$. We let
the inner loop stochastic optimizers run for 10 epochs.

We conduct the same cross-sample data generation experiment as in the Cars3D
case. Fig. G.8 shows the results. To be specific, we extract
$\widehat{\bm{z}}_{\ell}^{(q)}=\widehat{\bm{f}}_{\rm S}(\bm{x}_{\ell}^{(q)})$
that represents the rotation and horizontal coordinate information and
$\widehat{\bm{c}}_{j}^{(q)}=\widehat{\bm{f}}_{\rm P}(\bm{x}_{\ell}^{(q)})$.
And we combine this information together to generate synthetic samples
$\widehat{\bm{x}}_{\ell,j}^{(q)}$ with the learned reconstruction network
$\bm{r}^{(q)}$, i.e.,
$\widehat{\bm{x}}_{\ell,j}^{(q)}=\widehat{\bm{r}}^{(q)}([(\widehat{\bm{z}}_{\ell}^{(q)})^{\\!\top\\!},(\bm{c}_{j}^{(q)})^{\\!\top\\!}]^{\\!\top\\!})$.

The observations are similar to that in the Cars3D experiments. Ideally, the
generated samples should have the rotation and horizontal position of
$\bm{x}_{\ell}^{(q)}$ (contained in $\widehat{\bm{z}}^{(q)}_{\ell}$) while the
vertical position of $\bm{x}^{(q)}_{j}$ (contained in
${\widehat{\bm{c}}^{(q)}_{j}}$). One can see that without using the
independence regularizer $\mathcal{R}$, the generated samples may have
rotation change, shape deformation compared to $\bm{x}_{\ell}^{(q)}$ or simply
the vertical position is not exactly replicated from the sample
$\bm{x}_{j}^{(q)}$. However, with the ${\cal R}$ regularization, the results
are exactly what one expects to see. This again verifies our claim in Theorem
2.

Figure G.8: Generated samples by fixing $\widehat{\bm{z}}_{\ell}$ (rotation
and horizontal position) and varying $\widehat{\bm{c}}_{j}^{(q)}$ (vertical
position). Top: the square view; bottom: the heart view; rows in blue boxes
are w/ $\mathcal{R}$; rows in green boxes are w/o $\mathcal{R}$. Figure G.9:
Cross-view generation from $\bm{x}_{\ell}^{(2)}$ to $\bm{x}_{\ell}^{(1)}$.

Multiview MNIST Data for Cross-view Generation. Using the multiview MNIST
data, we also show the cross-view generation results in Fig. G.9. Here, we
extract $\widehat{\bm{z}}^{(2)}_{\ell}=\widehat{\bm{f}}_{\rm
S}^{(2)}(\bm{x}^{(2)}_{\ell})$ from the second view and
${\widehat{\bm{c}}^{(1)}_{j}}=\widehat{\bm{f}}_{\rm
P}^{(1)}(\bm{x}^{(1)}_{j})$ from the first. Then, we generate
$\widehat{\bm{x}}_{\ell,j}^{(1)}=\widehat{\bm{r}}^{(1)}([(\widehat{\bm{z}}_{\ell}^{(2)})^{\\!\top\\!},(\bm{c}_{j}^{(1)})^{\\!\top\\!}]^{\\!\top\\!})$
shown in the blue and green boxes. Ideally, the generated samples should have
the digit information of $\bm{x}_{\ell}^{(2)}$ (contained in
$\widehat{\bm{z}}^{(2)}_{\ell}$) while the style information of
$\bm{x}^{(1)}_{j}$ (contained in ${\widehat{\bm{c}}^{(1)}_{j}}$). Clearly,
using ${\cal R}$ attains the desired results. Note that this dataset is
challenging as the noise in view 2 is very high, making it hard to achieve
perfect matching and reconstruction—but our result is still plausible.

###### Remark 2

We would like to mention that multiview data and pertinent learning tasks are
pervasive in the real world. For example, acoustic features and articulatory
recordings are two views of speech signals, and multiview based representation
learning can be used to enhance speech recognition (Arora & Livescu, 2013;
Wang et al., 2015). Another example is cross-media information retrieval (Gong
et al., 2014). There, a data entity has a text view and an image view, and the
task is to retrieve one view from another. This task can be efficiently done
in the learned shared domain. In natural language processing, multilingual
word embedding can also be formulated as a CCA-type shared information
learning problem; see (Socher & Fei-Fei, 2010; Dhillon et al., 2012). In
computer vision, there are a number of important tasks such as image style
translation (e.g., sketch to picture and picture to cartoon) (Zhu et al.,
2017; Huang et al., 2018; Lee et al., 2018) and super-resolution (Ledig et
al., 2017) can be considered as multiview learning problems. In particular,
image style translation will benefit from our method’s guaranteed shared
(content) and private (style) disentanglement.

##  Appendix H Additional Notes on Shared-Private Modeling

Regarding the generative model in (5), some remarks are as follows. The
intuition that multiview data consists of shared and private components are
widely used; see, e.g., (Huang et al., 2018; Lee et al., 2018; Wang et al.,
2016; Gundersen et al., 2019). However, explicit generative models were only
considered in limited theory-oriented works.

The model in (5) can be understood as a nonlinear generalization of the linear
CCA model in (Ibrahim & Sidiropoulos, 2020), where the views are modeled as

$\bm{x}_{\ell}^{(q)}=\bm{A}^{(q)}[\bm{z}_{\ell}^{\\!\top\\!},(\bm{c}_{\ell}^{(q)})^{\\!\top\\!}]^{\\!\top\\!}$

for $q=1,2$. In (Lyu & Fu, 2020), a special type of nonlinear model, namely,
the post-nonlinear mixture model, was analyzed. There, the model is

$\bm{x}_{\ell}^{(q)}=\bm{g}^{(q)}(\bm{A}^{(q)}[\bm{z}_{\ell}^{\\!\top\\!},(\bm{c}_{\ell}^{(q)})^{\\!\top\\!}]^{\\!\top\\!}),$

where $\bm{g}^{(q)}(\bm{y})$ applies a nonlinear distortion to each element of
$\bm{y}$ individually. However, post-nonlinear models are much less general
compared to our model in (5)—where $\bm{g}^{(q)}(\bm{y})$ nonlinearly distorts
all elements of $\bm{y}$ jointly in an unknown way. More recently, under the
context of AM-SSL, the work in (von Kügelgen et al., 2021) considered a
multiview generative model that is similar to our model, but the views share
the same generative nonlinear function, i.e.,

$\bm{x}_{\ell}^{(q)}=\bm{g}([\bm{z}_{\ell}^{\\!\top\\!},(\bm{c}_{\ell}^{(q)})^{\\!\top\\!}]^{\\!\top\\!}).$

This assumption restricts the applicability of the model to scenarios where
the two views are generated using exactly the same nonlinear distortions,
which may be less flexible. Our model in (5) subsumes the models in (Ibrahim &
Sidiropoulos, 2020; Lyu & Fu, 2020; von Kügelgen et al., 2021) as its special
cases.

##  Appendix I Avoiding Reconstruction Using Entropy Regularization

###  I.1 Entropy Regularization and Shared Component Identifiability

If we ignore private information extraction, our formulation for shared
information extraction is as follows:

$\displaystyle\operatorname*{\textrm{minimize}}_{\bm{f}^{(1)},\bm{f}^{(2)}}$
$\displaystyle~{}\mathbb{E}\left[\left\|\bm{f}_{\rm
S}^{(1)}\left(\bm{x}^{(1)}\right)-\bm{f}_{\rm
S}^{(2)}\left(\bm{x}^{(2)}\right)\right\|^{2}\right]$ (I.1a) subject to
$\displaystyle~{}\bm{f}^{(q)}\text{~{}for $q=1,2$ are invertible},$ (I.1b)
$\displaystyle~{}\mathbb{E}\left[\bm{f}_{\rm
S}^{(q)}\left(\bm{x}^{(q)}\right)\bm{f}^{(q)}_{\rm
S}\left(\bm{x}^{(q)}\right)^{\top}\right]=\bm{I},~{}\mathbb{E}\left[\bm{f}_{\rm
S}\left(\bm{x}^{(q)}\right)\right]=\bm{0},~{}q=1,2,$ (I.1c)

with the latent variables satisfying:

$p(\bm{z},\bm{c}^{(1)},\bm{c}^{(2)})=p(\bm{z})p(\bm{c}^{(1)})p(\bm{c}^{(2)}).$

We hope to encourage invertibility of $\bm{f}^{(q)}$ without using a decoder
reconstruction network. To this end, we generalize the idea in Theorem 4.4 in
(von Kügelgen et al., 2021). Note that (von Kügelgen et al., 2021) deals with
the case where only one $\bm{f}$ is learned (i.e.,
$\bm{f}^{(1)}=\bm{f}^{(2)}$). Here, we show that this idea can be used under
our case as well. To see this, let us consider the following formulation:

$\displaystyle\operatorname*{\textrm{minimize}}_{\bm{f}_{\rm
S}^{(1)},\bm{f}_{\rm S}^{(2)}}$
$\displaystyle~{}\mathbb{E}\left[\left\|\bm{f}_{\rm
S}^{(1)}\left(\bm{x}^{(1)}\right)-\bm{f}_{\rm
S}^{(2)}\left(\bm{x}^{(2)}\right)\right\|^{2}\right]-H\left(\bm{f}_{\rm
S}^{(1)}\left(\bm{x}^{(1)}\right)\right)$ (I.2a) subject to
$\displaystyle~{}\bm{f}_{\rm S}^{(q)}:\mathbb{R}^{M_{q}}\rightarrow(0,1)^{D}.$
(I.2b)

where $H(\cdot)$ computes the differential entropy of its argument. The
formulation still aims to match the latent representations of the two views,
but at the same time maximizes the entropy of the learned features of a the
first view. The proof of this case consists of three major steps.

Step 1. It is straightforward to see that the optimal solution of (I.2) is

$\displaystyle\widehat{\bm{z}}={\bm{f}}^{(1)}_{\rm
S}(\bm{x}^{(1)})={\bm{f}}^{(2)}_{\rm
S}(\bm{x}^{(2)}),~{}\widehat{\bm{z}}\sim{\rm Uniform}(0,1)^{D}$

since the first term has optimal value 0 when two view are perfectly matched,
and the differential entropy of a random variable is maximized when the
distribution on $(0,1)^{D}$ is uniform (Cover, 1999). Next, following the idea
in (von Kügelgen et al., 2021), by the Darmois construction (Darmois, 1951),
there exists $\bm{d}(\cdot):\mathcal{Z}\rightarrow(0,1)^{D}$ which maps the
ground-truth $\bm{z}$ to a uniform random variable on $(0,1)^{D}$. Thus, one
can construct an optimal solution of (I.2) as:

$\displaystyle{\bm{f}}^{(q)}_{\rm
S}=\bm{d}\circ\left[\left(\bm{g}^{(q)}\right)^{-1}\right]_{1:D}$

where the first $D$ dimensions of the output of
$\left(\bm{g}^{(q)}\right)^{-1}$ are fed to $\bm{d}(\cdot)$.

Step 2. By using our proof technique in Theorem 1, employing the equation
$\widehat{\bm{z}}={\bm{f}}^{(1)}_{\rm S}(\bm{x}^{(1)})={\bm{f}}^{(2)}_{\rm
S}(\bm{x}^{(2)})$, it can be shown that $\widehat{\bm{z}}$ only depends on the
shared component $\bm{z}$ but does not depend on either $\bm{c}^{(1)}$ or
$\bm{c}^{(2)}$, which we denote as $\widehat{\bm{z}}=\bm{\gamma}(\bm{z})$.
Note that the proof of this part holds since it only uses the latent
correlation maximization (or $\bm{f}_{\rm S}^{(q)}$ matching). Using the same
derivation as in Theorem 1, we have the Jacobian of $\bm{h}^{(1)}$ as follows:

$\displaystyle\bm{J}^{(1)}=\begin{bmatrix}\bm{J}^{(1)}_{11}&\bm{H}_{\rm
S}^{(1)}\\\
\bm{J}^{(1)}_{21}&\bm{J}^{(1)}_{22}\end{bmatrix}=\begin{bmatrix}\bm{J}^{(1)}_{11}&\bm{0}_{D\times
D_{1}}\\\ \bm{J}^{(1)}_{21}&\bm{J}^{(1)}_{22}\end{bmatrix},$

which indicates that $\widehat{\bm{z}}$ only depends on $\bm{z}$ but not
$\bm{c}^{(1)}$. The above also holds for the second view. Note that the
possibility of $\bm{f}^{(q)}_{\rm S}$ being a trivial constant solution (and
thus making $\bm{H}_{\rm S}^{(1)}=\bm{0}$) is ruled out since
$\bm{f}^{(q)}_{\rm S}$’s entropy is maximized.

Step 3. The last step in Theorem 1 is to use ${\rm
rank}(\bm{J}^{(1)})=D+D_{1}$ to show that
$\bm{J}_{11}^{(1)}\in\mathbb{R}^{D\times D}$ has full rank. There, ${\rm
rank}(\bm{J}^{(1)})=D+D_{1}$ is natural since $\bm{f}^{(1)}$ is constructed to
be invertible using an autoencoder (and thus $\bm{f}^{(1)}\circ\bm{g}^{(1)}$
is also invertible). Here, we could not use this argument. However, similar to
Theorem 4.4 in (von Kügelgen et al., 2021), by applying Proposition 5 of
(Zimmermann et al., 2021), one can show that
$\widehat{\bm{z}}=\bm{\gamma}(\bm{z})$ where $\bm{\gamma}(\cdot)$ is an
invertible function, if $p(\bm{z})$ is a regular density, i.e.,
$0<p(\bm{z})<\infty$ everywhere. Note that under our generative model,
$\bm{f}^{(1)}(\bm{x}^{(1)})=\bm{f}^{(2)}(\bm{x}^{(2)})$ for all
$\bm{x}^{(q)}$. Hence, the above derivations can be repeated for
$\bm{f}^{(2)}$. This concludes the proof.

###  I.2 Realization and Connection to Contrastive SSL

To implement the formulation (I.2), following the idea in (von Kügelgen et
al., 2021), one can use the idea of InfoNCE (Gutmann & Hyvärinen, 2010; Oord
et al., 2018), which it has interesting connections to contrastive SSL (Wang &
Isola, 2020). In particular, the formulation of InfoNCE is as follows:

$\displaystyle\mathbb{E}_{\\{\bm{x}^{(1)}_{\ell},\bm{x}^{(2)}_{\ell}\\}_{\ell=1}^{K}\sim
p(\bm{x}^{(1)},\bm{x}^{(2)})}\left[-\sum_{i=1}^{K}\log\frac{\exp\\{\text{sim}(\widehat{\bm{z}}_{i},\widehat{\bm{z}}^{\prime}_{i})/\tau\\}}{\sum_{j=1}^{K}\exp\\{\text{sim}(\widehat{\bm{z}}_{i},\widehat{\bm{z}}^{\prime}_{j})/\tau\\}}\right]$
(I.3)

where $\widehat{\bm{z}}_{\ell}$ and $\widehat{\bm{z}}_{\ell}^{\prime}$ are the
learned representations of the two corresponding samples $\bm{x}^{(1)}_{\ell}$
and $\bm{x}^{(2)}_{\ell}$, respectively, $\text{sim}(a,b)$ computes the
similarity of its arguments, $\tau$ is a temperature hyperparameter and there
are $K$ samples of each batch where $K-1$ of them are negative.

Note that in (von Kügelgen et al., 2021), only one generative function
$\bm{g}(\cdot)$ is considered. In their implementation, given sample pairs
$\\{\bm{x}^{(1)}_{\ell},\bm{x}^{(2)}_{\ell}\\}_{\ell=1}^{K}$, the above
InfoNCE objective can be rewritten with $\tau=1$ and
$\text{sim}(\bm{a},\bm{b})=-\|\bm{a}-\bm{b}\|_{2}^{2}$ as

$\displaystyle\mathbb{E}_{\\{\bm{x}^{(1)}_{\ell},\bm{x}^{(2)}_{\ell}\\}_{\ell=1}^{K}\sim
p(\bm{x}^{(1)},\bm{x}^{(2)})}\left[\sum_{i=1}^{K}\left\\{\left\|\bm{f}(\bm{x}^{(1)}_{i})-\bm{f}(\bm{x}^{(2)}_{i})\right\|_{2}^{2}+\log{\sum_{j=1}^{K}\exp\left\\{-\left\|\bm{f}(\bm{x}^{(1)}_{i})-\bm{f}(\bm{x}^{(2)}_{j})\right\|_{2}^{2}\right\\}}\right\\}\right].$
(I.4)

The second term is a non-parametric entropy estimator of the representation as
$K\rightarrow\infty$ (Wang & Isola, 2020). The above nicely connects AM-SSL
with contrastive learning when $\bm{g}^{(1)}=\bm{g}^{(2)}$ and only one
encoder is used, i.e., $\bm{f}^{(1)}=\bm{f}^{(2)}$.

However, in our problem the generative functions are different in each view.
Hence, the formulation above is not directly applicable. Nonetheless, one can
use the slack variable based design as in (10). Then, the problem can be
reformulated as

$\displaystyle\mathbb{E}_{\\{\bm{x}^{(1)}_{\ell},\bm{x}^{(2)}_{\ell}\\}_{\ell=1}^{K}\sim
p(\bm{x}^{(1)},\bm{x}^{(2)})}\left[\sum_{i=1}^{K}\left\\{\sum_{q=1}^{2}\left\|\bm{u}_{i}-\bm{f}^{(q)}(\bm{x}_{i}^{(q)})\right\|^{2}+\log{\sum_{j=1}^{K}\exp\\{-\|\bm{u}_{i}-\bm{u}_{j}\|_{2}^{2}\\}}\right\\}\right].$
(I.5)

Note that the entropy regularization is imposed on the slack variable
$\bm{u}$—which indirectly promotes high entropy of $\bm{f}^{(q)}$’s. This way,
one can handle $Q$ views with different generative functions $\bm{g}^{(q)}$’s.