# Bi-level Feature Alignment for Versatile Image Translation and Manipulation

Fangneng Zhan∗, Yingchen Yu∗, Rongliang Wu, Kaiwen Cui, Aoran Xiao,
Shijian Lu§, Ling Shao, Fellow, IEEE  F. Zhan, Y. Yu, R. Wu, K. Cui, A. Xiao,
and S. Lu are with the School of Computer Science and Engineering, Nanyang
Technological University, Singapore, 639798. L. Shao is with the Inception
Institute of Artificial Intelligence, UAE, 111999. * denotes equal
contribution. $\S$ denotes corresponding author, E-mail:
<EMAIL_ADDRESS>
###### Abstract

Generative adversarial networks (GANs) have achieved great success in image
translation and manipulation. However, high-fidelity image generation with
faithful style control remains a grand challenge in computer vision. This
paper presents a versatile image translation and manipulation framework that
achieves accurate semantic and style guidance in image generation by
explicitly building a correspondence. To handle the quadratic complexity
incurred by building the dense correspondences, we introduce a bi-level
feature alignment strategy that adopts a top-$k$ operation to rank block-wise
features followed by dense attention between block features which reduces
memory cost substantially. As the top-$k$ operation involves index swapping
which precludes the gradient propagation, we propose to approximate the non-
differentiable top-$k$ operation with a regularized earth mover’s problem so
that its gradient can be effectively back-propagated. In addition, we design a
novel semantic position encoding mechanism that builds up coordinate for each
individual semantic region to preserve texture structures while building
correspondences. Further, we design a novel confidence feature injection
module which mitigates mismatch problem by fusing features adaptively
according to the reliability of built correspondences. Extensive experiments
show that our method achieves superior performance qualitatively and
quantitatively as compared with the state-of-the-art. The code is available at
https://github.com/fnzhan/RABIT.

###### Index Terms:

Image Translation, Image Manipulation, Image Correspondence, Generative
Adversarial Networks.

##  1 Introduction

Image translation and manipulation aim to generate and edit photo-realistic
images conditioning on certain inputs such as semantic segmentation [54, 65],
key points [60, 11] and layout [35]. It has been studied intensively in recent
years thanks to its wide spectrum of applications in various tasks [56, 51,
63]. However, achieving high fidelity image translation and manipulation with
faithful style control remains a grand challenge due to the high complexity of
natural image styles. A typical approach to control image styles is to encode
image features into a latent space with certain regularization (e.g., Gaussian
distribution) on the latent feature distribution. For example, Park _et
al._[54] utilize VAE [10] to regularize the distribution of encoded features
for faithful style control. However, VAE struggles to encode the complex
distribution of natural image styles and often suffers from posterior collapse
[45] which leads to degraded style control performance. Another strategy is to
encode reference images into style codes to provide style guidance in image
generation. Choi _et al._ [5] employ a style encoder to extract the style code
from a given reference image and achieve diverse image synthesis over multiple
domains. Zhu _et al._ [95] further propose to extract style codes for each
semantic region individually and achieve flexible style control within each
semantic region. However, style codes often capture the overall image style or
local region style without an explicit style guidance in spatial domain.

Recently, building dense correspondences between images has attracted
increasing interest in image translation and manipulation thanks to its
impressive image generation performance. Prior studies have explored to build
correspondences between images of the same domain for exemplar-based image
colorization [17, 85]. Zhang _et al._ [87] further explore to build cross-
domain correspondences with Cosine similarity to achieve exemplar-based image
translation. However, Zhang _et al._ construct the semantic correspondences
based on Cosine similarity that often leads to many-to-one matching (i.e.
multiple conditional input features match to the same exemplar feature). Zhan
_et al._ [78] thus propose to build the correspondence with optimal transport
which facilitates the mass preserving property to mitigate the many-to-one
matching. On the other hand, building dense correspondences has quadratic
complexity which incurs high memory costs and struggles while scaling up to
high-resolution images. To achieve high-resolution image translation, Zhou _et
al._ [93] propose a GRU-assisted Patch-Match [1] method to build high-
resolution correspondences efficiently. Zheng _et al._ [90] tackle the high-
resolution correspondences via sparse attention with applications to semantic
image manipulation. However, all above methods aim to build correspondences
based on their semantic coherence without consideration of their structure
coherence. As textures within a semantic region share identical semantic
information, the texture structure information tends to be lost while building
pure semantic correspondence. Warping exemplars with such pure semantic
correspondence will further cause destroyed texture patterns in the warped
exemplars which provide inaccurate guidance for image generation.

Figure 1:  Bi-level feature alignment via ranking and attention scheme: With a
query block from the Conditional Input, we first retrieve the top-$k$ most
similar blocks from the Exemplar Image through a differentiable ranking
operation, and then compute dense attention between features in query block
and features in retrieved top-$k$ blocks. Such bi-level alignment reduces the
computational cost greatly, and it also allows to build high-resolution
correspondences which leads to more realistic translation with finer details.

This paper presents RABIT, a Ranking and Attention scheme with Bi-level
feature alignment for versatile Image Translation and manipulation. RABIT
consists of an alignment network and a generation network that are optimized
jointly. The alignment network establishes feature correspondences between a
conditional input (semantic guidance) and an exemplar (style guidance). With
the built correspondences, the exemplar is warped to be aligned with the
conditional input to provide accurate style guidance for the generation
network. However, building dense correspondence incurs quadratic computational
complexity which struggles with high-resolution correspondences. We design a
bi-level alignment strategy with a Ranking and Attention Scheme (RAS) which
builds feature correspondences efficiently at two levels: 1) a top-$k$ ranking
operation for dynamically generating block-wise ranking matrices; 2) a dense
attention module that achieves dense correspondences between features within
blocks as illustrated in Fig. 1. RAS enables to build high-resolution
correspondences and reduces the memory cost from $\mathcal{O}(L^{2})$ to
$\mathcal{O}(N^{2}+b^{2})$ ($L$ is the number of features for alignment, $b$
is block size, and $N=\frac{L}{b}$). However, the top-$k$ operation involves
index swapping whose gradient cannot be propagated in networks. To address
this issue, we approximate the top-$k$ ranking operation to regularized earth
mover’s problem by imposing entropy regularization to earth mover’s distance.
Then the regularized earth mover’s problem can be solved with a Sinkhorn
iteration [8] (in a differentiable manner) which enables gradient back-
propagation effectively.

As in [87, 93], building correspondences based on semantic information only
often leads to the losing of texture structures and patterns in warped
exemplars. Thus, the spatial information should also be incorporated to
preserve the texture structures and patterns and yield more accurate feature
correspondences. A vanilla method to encode the position information is
concatenating the semantic features with the corresponding feature coordinates
via coordconv [40]. However, the vanilla position encoding builds a single
coordinate system for the whole image which ignores the position information
within each semantic region. Instead, we design a semantic position encoding
(SPE) mechanism that builds a dedicated coordinate system for each semantic
region which outperforms the vanilla position encoding significantly.

In addition, conditional inputs and exemplars are seldom perfectly matched,
e.g., conditional inputs could contain several semantic classes that do not
exist in exemplar images. Under such circumstance, the built correspondences
often contain errors which lead to inaccurate exemplar warping and further
deteriorated image generation. We tackle this problem by designing a
CONfidence Feature Injection (CONFI) module that fuses features of conditional
inputs and warped exemplars according to the reliability of the built
correspondences. Although the warped exemplar may not be reliable, the
conditional input always provides accurate semantic guidance in image
generation. The CONFI module thus assigns higher weights to the conditional
input when the built correspondence (or warped exemplar) is unreliable.
Experiments show that CONFI helps to generate faithful yet high-fidelity
images consistently by assigning adaptive weights (to the conditional input)
based on the reliability of the built correspondence.

The contributions of this work can be summarized in four aspects. First, we
propose a versatile image translation and manipulation framework which
introduces a bi-level feature alignment strategy that greatly reduces the
memory cost while building the correspondence between conditional inputs and
exemplars. Second, we approximate non-differentiable top-$k$ ranking to a
regularized earth mover’s problem, which enables effective gradient
propagation for end-to-end network training. Third, we introduce a semantic
position encoding mechanism that encodes region-level position information to
preserve texture structures and patterns. Fourth, we design a confidence
feature injection module that provides reliable feature guidance in image
translation and manipulation.

##  2 Related Work

###  2.1 Image-to-Image Translation

Image translation has achieved remarkable progress in learning the mapping
among images of different domains. It could be applied in different tasks such
as style transfer [22, 13, 36], image super-resolution [32, 38, 31, 86],
domain adaptation [57, 51, 19, 62, 77, 80], image synthesis [84, 7, 74, 73,
75, 82, 79, 81, 83, 76], image inpainting [71, 72, 39, 66], etc. To achieve
high-fidelity and flexible translation, existing work uses different
conditional inputs such as semantic segmentation [25, 65, 54], scene layouts
[59, 89, 35], key points [48, 50, 11], edge maps [25, 12], etc. However,
effective style control remains a challenging task in image translation.

Style control has attracted increasing attention in image translation and
generation. Earlier works such as [30] regularize the latent feature
distribution to control the generation outcome. However, they struggle to
capture the complex textures of natural images. Style encoding has been
studied to address this issue. For example, [23] and [47] transfer style codes
from exemplars to source images via adaptive instance normalization (AdaIN)
[22]. [5] employs a style encoder for style consistency between exemplars and
translated images. [95] designs semantic region-adaptive normalization (SEAN)
to control the style of each semantic region individually. Wang _et al._ [64]
demonstrate the feasibility of exemplar-guided style control by directly
concatenating exemplar image and condition as input for image translation.
However, encoding style exemplars tends to capture the overall image style and
ignores the texture details in local regions. To achieve accurate style
guidance for each local region, Zhang _et al._ [87] build dense semantic
correspondences between conditional inputs and exemplars with Cosine
similarity to capture accurate exemplar details. To mitigate the issue of
many-to-one matching in Zhang _et al._ [87], Zhan _et al._ [78] further
propose to utilize the mass preserving property of optimal transport to build
the correspondence. On the other hand, above methods usually work with low-
resolution correspondences due to the quadratic complexity in correspondence
computation. To build correspondence in high resolution, Zhou _et al._ [93]
introduce the GRU-assisted Patch-Match to efficiently establish the high-
resolution correspondence. Zheng _et al._ [90] tackle the high-resolution
correspondences through a sparse attention module with applications to
semantic image manipulation. However, all these methods only utilize semantic
information for building correspondence, which often leads to destroyed
texture structures and patterns in the warped exemplar. In this work, we
propose a bi-level alignment strategy that allows to build correspondence
efficiently and design a semantic position encoding to preserve the texture
structures and patterns.

###  2.2 Semantic Image Editing

The arise of generative adversarial network (GANs) brings revolutionary
advance to image editing [94, 20, 52, 4, 55, 68, 67]. As one of the most
intuitive representation in image editing, semantic information has been
extensively investigated in conditional image synthesis. For example, Isola
_et al._ [25] achieve label-to-pixel generation by training an encoder-decoder
network with a conditional adversarial objective. Wang _et al._ [65] further
achieve high-resolution image manipulation by editing the pixel-wise semantic
labels. Park _et al._ [54] introduce spatially-adaptive normalization (SPADE)
to inject guided features in image generation. MaskGAN [33] exploits a dual-
editing consistency as auxiliary supervision for robust face image
manipulation. Gu _et al._ [14] learn facial embeddings for different face
components to enable local facial editing. Chen _et al._ [3] propose a mask
re-targeting strategy for identity-preserved face animation. Xia _et al._ [69]
map images into the latent space of a pre-trained network to facilitate
editing. Instead of directly learning a label-to-pixel mapping, Hong _et al._
[20] propose a semantic manipulation framework HIM that generates images
guided by a predicted semantic layout. Upon this work, Ntavelis _et al._ [52]
propose SESAME which requires only local semantic maps to achieve image
manipulation. However, the aforementioned methods either only learn a global
feature without local focus (e.g., MaskGAN [33]) or ignore the features in the
editing regions of the original image (e.g., HIM [20], SESAME [52]). To better
utilize the fine features in the original image, Zheng _et al._ [90] adapt
exemplar-based image synthesis framework CoCosNet [87] for semantic image
manipulation by building a high-resolution correspondence between the original
image and the edited semantic map. However, it may inherit the issue of
texture pattern losing from [87], which can be effectively ameliorated by the
proposed semantic positional encoding mechanism.

###  2.3 Feature Correspondence

Early studies determine feature correspondence by focusing on sparse
correspondence [44] or dense correspondences between nearby view of the same
objects only [21, 53]. Differently, semantics correspondences establish the
dense correlation between different instances of the same semantic object. For
example, [2, 24, 29] focus on matching hand-crafted features. Leveraging the
power of convolutional neural networks (CNNs) in learning high-level semantic
features, Long _et al._ [43] first employ CNNs to establish semantic
correspondences between images. Later efforts further improve correspondence
quality by including additional annotations [92, 6, 15, 16], adopting coarse-
to-fine strategy [37], extending to cross-domain images [87], etc. However,
most existing studies only work with low-resolution correspondences as
constrained by the heavy computation cost. We design a bi-level alignment
strategy that greatly improves computation efficiency and allows to compute
dense correspondences at higher resolution.

Figure 2:  The framework of the proposed RABIT: Conditional Input and Exemplar
are fed to feature extractors $F_{X}$ and $F_{Z}$ to extract feature vectors
$X$ and $Z$ where $b$ local features form a feature block. In the first level,
each block from the conditional input serves as the query to retrieve top-$k$
similar blocks from the exemplar through a differentiable ranking operation.
In the second level, dense Attention is then built between the $b$ features in
query block and $b*k$ features in the retrieved blocks. The built Ranking
Matrices and Attention Matrices are combined to warp the exemplar to be
aligned with the conditional input as in Warped Exemplar, which serves as a
style guidance to generate the final result through a generation network.

##  3 Proposed Method

The proposed RABIT consists of an alignment network and a generation network
that are inter-connected as shown in Fig. 2. The alignment network learns the
correspondence between a conditional input and an exemplar for warping the
exemplar to be aligned with the conditional input. The generation network
produces the final generation under the guidance of the warped exemplar and
the conditional input. RABIT is typically applicable in the task of
conditional image translation with extra exemplar as style guidance. It is
also applicable to the task of image manipulation by treating the exemplars as
the original images for editing and the conditional inputs as the edited
semantic.

###  3.1 Alignment Network

The alignment network aims to build the correspondence between conditional
inputs and exemplars, and accordingly provide accurate style guidance by
warping the exemplars to be aligned with the conditional inputs. As shown in
Fig. 2, conditional input and exemplar are fed to feature extractors $F_{X}$
and $F_{Z}$ to extract two sets of feature vectors
$X=[x_{1},\cdots,x_{L}]\in\mathbb{R}^{d}$ and
$Z=[z_{1},\cdots,z_{L}]\in\mathbb{R}^{d}$, where $L$ and $d$ denote the number
and dimension of feature vectors, respectively. Most existing methods [87, 17,
85] align $X$ and $Z$ by building a $L\times L$ dense correspondence matrix
where each entry denotes the Cosine similarity between the corresponding
feature vectors in $X$ and $Z$. However, such correspondence computation has
quadratic complexity which incurs large memory and computation costs. Most
existing studies thus work with low-resolution exemplar images (e.g. 64
$\times$ 64 in CoCosNet [87]) which often struggle in generating realistic
images with fine texture details.

In this work, we propose a bi-level alignment strategy via a novel ranking and
attention scheme (RAS) that greatly reduces computational costs and allows to
build correspondences with high-resolution images as shown in Fig. 4. Instead
of building correspondences between features directly, the bi-level alignment
strategy builds the correspondences at two levels, including the first level
that introduces top-$k$ ranking to generate block-wise ranking matrices
dynamically and the second level that achieves dense attention between the
features within blocks. As Fig. 2 shows, $b$ local features are grouped into a
block, thus the features of conditional input and exemplar are partitioned
into $N$ blocks ($N=L/b$) as denoted by
$X=[X_{1},\cdots,X_{N}]\in\mathbb{R}^{bd}$ and
$Z=[Z_{1},\cdots,Z_{N}]\in\mathbb{R}^{bd}$. In the first level of top-$k$
ranking, each block feature of the conditional input serves as a query to
retrieve top-$k$ block features from the exemplar according to the Cosine
similarity between blocks. In the second level of local attention, the
features in each query block further attends to the features in the top-$k$
retrieved blocks to build up local attention matrices within block features.
The correspondence between the exemplar and conditional input can thus be
built much more efficiently by combine such inter-block ranking and inner-
block attention.

Semantic Position Encoding. Existing works [87, 93] mainly rely on semantic
features to establish the correspondences. However, as all textures within a
semantic region share the same semantic feature, the pure semantic
correspondence fails to preserve the texture structures or patterns within
each semantic region. For example, the building regions of conditional inputs
in Fig. 4 will establish correspondence with the building regions in the
exemplars without consideration of building textures, which will result in
warped exemplars with messy textures as shown in the Baseline (64). Thus, the
position information of features should also be facilitated to preserve the
texture structures and patterns. A vanilla method to encode the position
information is employing a simple coordconv [40] to build a global coordinate
for the full image. However, this vanilla position encoding mechanism builds a
single coordinate system for the whole image, ignoring region-wise semantic
differences. To preserve the fine texture pattern within each semantic region,
we design a semantic position encoding (SPE) mechanism that builds a dedicated
coordinate for each semantic region as shown in Fig. 3. Specifically, SPE
treats the center of each semantic region as the origin of coordinate, and the
coordinates within each semantic region are normalized to [-1, 1]. The
proposed SPE outperforms the vanilla position encoding significantly as shown
in Fig. 4 and to be evaluated in experiments.

Figure 3:  The comparison of vanilla position encoding and the proposed
semantic position encoding (SPE). Red dots denote the coordinate origin. The
proposed SPE builds dedicated coordinate for each semantic region.  Figure 4:
Warped exemplars with different methods: ‘64’ and ‘128’ mean to build
correspondences at resolutions $64\times 64$ and $128\times 128$. CoCosNet
[87] tends to lose texture details and structures, while CoCosNet v2 [93]
tends to generate messy warping. The Baseline denotes building correspondences
with Cosine similarity, which tends to lose textures details and structures.
The proposed ranking and attention scheme (RAS) allows efficient image warping
at high resolutions, the proposed semantic position encoding (SPE) can better
preserve texture structures. The combination of the two as denoted by SPE+RAS
achieves the best warping performance with high resolution and preserved
texture structures.

###  3.2 Differentiable Top-k Ranking

The core of the ranking and attention scheme lies with a top-$k$ operation
that ranks the correlative blocks. However, the original top-$k$ operation
involves index swapping whose gradient cannot be computed and so cannot be
integrated into end-to-end network training. We tackle this issue by
formulating the top-$k$ ranking as a regularized earth mover’s problem which
allows gradient computation via implicit differentiation [46, 70].

####  3.2.1 Top-k Ranking Formulation

We first show that a specific form of earth mover’s problem is essentially
equivalent to a top-$k$ element ranking problem. Earth mover’s problem [27]
aims to find a transport plan that minimizes the total cost to transform one
distribution to another. Consider two discrete distributions
$U=[\mu_{1},\dots,\mu_{N}]^{\top}$ and $V=[\nu_{1},\dots,\nu_{M}]^{\top}$
defined on supports $\mathcal{A}=[a_{1},\cdots,a_{N}]$ and
$\mathcal{B}=[b_{1},\cdots,b_{M}]$, with probability (or amount of earth)
$\mathbb{P}(a_{i})=\mu_{i}$ and $\mathbb{P}(b_{j})=\nu_{j}$. We define
$C\in\mathbb{R}^{N\times M}$ as the cost matrix where $C_{ij}$ denotes the
cost of transportation between $a_{i}$ and $b_{j}$, and $T$ as a transport
plan where $T_{ij}$ denotes the amount of earth transported between $\mu_{i}$
and $\nu_{j}$. An earth mover’s (EM) problem can be formulated by:

$\displaystyle{\rm EM}=\mathop{\min}\limits_{T}\langle C,T\rangle,\quad{\rm
s.t.}\ T\vec{1}_{M}=U,\;T^{\top}\vec{1}_{N}=V.$ (1)

where $\vec{1}$ denotes a vector of ones, $\langle\rangle$ denotes inner
product.

Figure 5:  Illustration of the earth mover’s problem in top-$k$ retrieval.
Earth mover’s problem is conducted between distributions $U$ and $V$ which is
defined on supports $\mathcal{A}=[a_{1},\cdots,a_{N}]$ and
$\mathcal{B}=[b_{1},b_{2}]$. Transport Plan $T_{i2}$ indicates the retrieved
top-$k$ elements.

We then derive the earth mover’s form of top-$k$ operator. With a query block
from the conditional input and $N$ blocks from the exemplar, their correlation
scores $\mathcal{A}=[a_{1},\cdots,a_{N}],a_{i}\in[-1,1]$ can be obtained based
on their Cosine similarity. The top-$k$ operation aims to retrieve $k$ most
similar elements from $\mathcal{A}$. We define another set
$\mathcal{B}=\\{-1,1\\}$, and consider two discrete distributions
$U=[\mu_{1},\cdots,\mu_{N}]$ and $V=[\nu_{1},\nu_{2}]$ defined on supports
sets $\mathcal{A}$ and $\mathcal{B}$ with $\mu_{i}=1/N,i\in[1,N]$,
$\nu_{1}=(N-k)/N$ and $\nu_{2}=k/N$. The cost $C$ is defined to be the squared
Euclidean distance, i.e., $C_{i1}=(a_{i}+1)^{2}$ and $C_{i2}=(a_{i}-1)^{2}$,
$i\in[1,N]$. The earth mover’s distance between $U$ and $V$ can thus be
formulated as:

$\begin{split}\mathop{\min}\limits_{T}\langle
C,T\rangle=&\mathop{\min}\limits_{T}\left\\{\sum_{i=1}^{N}\Big{[}(a_{i}+1)^{2}T_{i1}+(a_{i}-1)^{2}T_{i2}\Big{]}\right\\}\\\
=&\mathop{\min}\limits_{T}\left\\{2\sum_{i=1}^{N}a_{i}(T_{i1}-T_{i2})+\frac{1}{N}\sum_{i=1}^{N}a_{i}^{2}+1\right\\}\end{split}$

Therefore minimizing $\langle C,T\rangle$ suffices to minimize
$\sum_{i=1}^{N}a_{i}(T_{i1}-T_{i2})$. It is obvious that
$\sum_{i=1}^{N}T_{i2}=\frac{k}{N}$, $\sum_{i=1}^{N}T_{i1}=\frac{N-k}{N}$ and
$T_{i1}+T_{i2}=\frac{1}{N}$. Hence, minimizing
$\sum_{i=1}^{N}a_{i}(T_{i1}-T_{i2})$ essentially aims to select the largest
$K$ elements from $\mathcal{A}=[a_{1},\cdots,a_{N}]$ as implied in the
transport plan $T$:

$\displaystyle T_{i1}=\begin{cases}0&a_{i}\in{\rm top-}k\\\
\frac{1}{N}&a_{1}\not\in{\rm top-}k\end{cases},\
T_{i2}=\begin{cases}\frac{1}{N}&a_{i}\in{\rm top-}k\\\ 0&a_{1}\not\in{\rm
top-}k\end{cases}$

where $T_{i2}$ indicates the retrieved top-$k$ elements. Fig. 5 illustrates
the earth mover’s problem and transport plan, where the earth from the $k$
closest points is transported to $1$, and meanwhile the earth from the $N-k$
remaining points is transported to $-1$. Therefore, the transport plan $T$
exactly indicates the top-$k$ elements.

####  3.2.2 Differentiable Optimization

The top-$k$ operation has been formulated as an earth mover’s problem, while
the standard earth mover’s problem cannot be solved in a differentiable way.
We introduce a regularized earth mover’s distance which serves as a smoothed
approximation to the standard top-$k$ operator, and enables effective gradient
propagation. The regularized earth mover’s problem in Eq. (1) is defined as:

$\begin{split}&{\rm EM_{\epsilon}}=\mathop{\min}\limits_{T}\langle
C,T\rangle+\epsilon H(T)\\\ &{\rm s.t.}\quad T\vec{1}_{M}=U,\
T^{\top}\vec{1}_{N}=V\\\ \end{split}$ (2)

where $H(T)=\sum_{i,j}T_{ij}\log T_{ij}$ is the regularization term,
$\epsilon$ is the regularization coefficient. The optimal transport plan
$T_{\epsilon}$ of the regularized earth mover’s problem thus becomes a
smoothed version of the standard top-$k$ operator.

The regularized earth mover’s distance can be efficiently computed via the
Sinkhorn algorithm [8]. Specifically, an exponential kernel is applied to the
cost matrix $C$ which yields $C^{\prime}=\exp(-\lambda C)$. Then $C^{\prime}$
is converted iteratively towards a doubly stochastic matrix through a Sinkhorn
operation $\mathcal{S}(\cdot)$ as denoted by:

$\mathcal{S}^{m}(C^{\prime})=\left\\{\begin{matrix}C^{\prime}&m=0\\\
\mathcal{N}^{c}(\mathcal{N}^{r}(S^{m-1}(C^{\prime})))&otherwise\end{matrix}\right.$

where $m$ denotes the iteration number, $\mathcal{N}^{r}(\cdot)$ and
$\mathcal{N}^{c}(\cdot)$ are row and column normalization which can be denoted
by:

$\mathcal{N}_{ij}^{r}(C^{\prime})=\frac{C^{\prime}_{ij}}{\sum_{k=1}^{2}C^{\prime}_{ik}},\quad\mathcal{N}_{ij}^{c}(C^{\prime})=\frac{C^{\prime}_{ij}}{\sum_{k=1}^{N}C^{\prime}_{kj}},$

where $C^{\prime}_{ij}$ represents an element in $C^{\prime}$. Then the
partial derivatives for the iteration (taking $m=1$ as the example) can be
derived by:

$\frac{\partial\mathcal{S}^{1}}{\partial
c^{\prime}_{st}}=\frac{\partial\mathcal{N}^{c}_{st}}{\partial\mathcal{N}^{r}_{st}}\cdot\sum_{j=1}^{2}\left[\frac{\mathbb{I}_{j}^{t}}{\sum_{k=1}^{2}c^{\prime}_{sk}}-\frac{c^{\prime}_{sj}}{(\sum_{k=1}^{2}c^{\prime}_{sk})^{2}}\right]$

where $s$ and $t,k,j$ represent the indices of the row and columns in
$C^{\prime}$, $\mathbb{I}_{j}^{t}=\begin{cases}1&{\rm if\ j=t}\\\ 0&{\rm
otherwise}\end{cases}$ represents an indication function. Thus, the Sinkhorn
operation is differentiable and its gradient can be calculated by unrolling
the sequence of the row and column normalization operations. When iterations
converge, the transport plan $T_{\epsilon}=\mathcal{S}^{m}(C^{\prime})$
indicating the top-$k$ elements can be obtained.

####  3.2.3 Complexity Analysis

The vanilla dense correspondence has a self-attention memory complexity
$\mathcal{O}(L^{2})$ where $L$ is the input sequence length. For our bi-level
alignment strategy, the memory complexity of building the block ranking
matrices and local attention matrices are $\mathcal{O}(N^{2})$ and
$\mathcal{O}(b*(kb))$, where $b$, $N$ ($N=L/b$) and $k$ are block size, block
number and the number of top-$k$ selection. Thus, the overall memory
complexity of the proposed bi-level alignment strategy is
$\mathcal{O}(N^{2}+b*(kb))$.

Figure 6:  Illustration of confidence feature injection: Conditional input and warped exemplar are initially fused with a confidence map (CMAP) of size $1\times H\times W$. A multi-channel confidence map (Multi-CMAP) of size $C\times H\times W$ is then obtained from the initial fusion which further fuses the conditional input and warped exemplar in multiple channels.  TABLE I:  Comparing RABIT with state-of-the-art image translation methods over four translation tasks with FID, SWD and LPIPS as the evaluation metrics.  | ADE20K | CelebA-HQ (Semantic) | DeepFashion | CelebA-HQ (Edge)
---|---|---|---|---
Methods | FID $\downarrow$ | SWD $\downarrow$ | LPIPS $\uparrow$ | FID $\downarrow$ | SWD $\downarrow$ | LPIPS $\uparrow$ | FID $\downarrow$ | SWD $\downarrow$ | LPIPS $\uparrow$ | FID $\downarrow$ | SWD $\downarrow$ | LPIPS $\uparrow$
Pix2pixHD[65] | 81.80 | 35.70 | N/A | 43.69 | 34.82 | N/A | 25.20 | 16.40 | N/A | 42.70 | 33.30 | N/A
StarGAN v2[5] | 98.72 | 65.47 | 0.551 | 53.20 | 41.87 | 0.324 | 43.29 | 30.87 | 0.296 | 48.63 | 41.96 | 0.214
SPADE[54] | 33.90 | 19.70 | 0.344 | 39.17 | 29.78 | 0.254 | 36.20 | 27.80 | 0.231 | 31.50 | 26.90 | 0.207
SelectionGAN[61] | 35.10 | 21.82 | 0.382 | 42.41 | 30.32 | 0.277 | 38.31 | 28.21 | 0.223 | 34.67 | 27.34 | 0.191
SMIS[96] | 42.17 | 22.67 | 0.476 | 28.21 | 24.65 | 0.301 | 22.23 | 23.73 | 0.240 | 23.71 | 22.23 | 0.201
SEAN[95] | 24.84 | 10.42 | 0.499 | 17.66 | 14.13 | 0.285 | 16.28 | 17.52 | 0.251 | 16.84 | 14.94 | 0.203
CoCosNet[87] | 26.40 | 10.50 | 0.580 | 21.83 | 12.13 | 0.292 | 14.40 | 17.20 | 0.272 | 14.30 | 15.30 | 0.208
CoCosNet v2[93] | 25.20 | 9.900 | 0.557 | 20.64 | 11.21 | 0.303 | 13.04 | 16.65 | 0.270 | 13.21 | 14.01 | 0.216
RABIT | 24.35 | 9.893 | 0.571 | 20.44 | 11.18 | 0.307 | 12.58 | 16.03 | 0.284 | 11.67 | 14.22 | 0.219
TABLE II:  Comparing RABIT with state-of-the-art image translation methods in semantic consistency and style consistency (on ADE20K [91]).  | Semantic Consistency | Style Consistency
---|---|---
Methods |  ${\rm VGG_{42}}$ $\uparrow$ |  ${\rm VGG_{52}}$ $\uparrow$ |  ${\rm VGG_{M}}$ $\uparrow$ |  ${\rm VGG_{V}}$ $\uparrow$
SPADE [54] |  0.861 |  0.772 |  0.934 |  0.884
StarGAN v2 [5] |  0.741 |  0.718 |  0.919 |  0.907
SelectionGAN [61] |  0.843 |  0.785 |  0.951 |  0.912
SMIS [96] |  0.862 |  0.787 |  0.951 |  0.933
SEAN [95] |  0.868 |  0.791 |  0.962 |  0.942
CoCosNet [87] |  0.878 |  0.790 |  0.986 |  0.965
CoCosNetv2 [93] |  0.889 |  0.800 |  0.994 |  0.972
RABIT |  0.891 |  0.812 |  0.993 |  0.977

###  3.3 Generation Network

The generation network aims to synthesize images under the semantic guidance
of conditional inputs and style guidance of exemplars. As the exemplars are
warped by the alignment network to be semantically matched with the
conditional inputs, the warped exemplar can serve as accurate style guidance
for each image region in the generation network. The overall architecture of
the generation network is similar to SPADE [54]. Please refer to supplementary
material for details of the network structure.

State-of-the-art approach [87] simply concatenates the warped exemplar and
conditional input to guide the image generation process. However, conditional
input and warped exemplar are from different domains with different
distributions and a naive concatenation of them is often sub-optimal [9]. In
addition, the warped input image and edited semantic map could be structurally
aligned but semantically different especially when they have severe semantic
discrepancy. Such unreliably warped exemplars could serve as false guidance
for the generation network and heavily deteriorate the generation performance.
Therefore, a mechanism is required to identify the semantic reliability of
warped exemplar to provide reliable guidance for the generation network. To
this end, we propose a CONfidence Feature Injection (CONFI) module that
adaptively weights the features of conditional input and warped exemplar
according to the reliability of feature matching.

Confidence Feature Injection. Intuitively, in the case of lower reliability of
the feature correspondence, we should assign a relatively lower weight to the
warped exemplar which provides unreliable style guidance and a higher weight
to the conditional input which consistently provides accurate semantic
guidance.

As illustrated in Fig. 6, the proposed CONFI fuses the features of the
conditional input and warped exemplar based on a confidence map (CMAP) that
captures the reliability of the feature correspondence. To derive the
confidence map, we first obtain a block-wise correlation map of size $N\times
N$ by computing element-wise Cosine distance between $X=[X_{i},\cdots,X_{N}]$
and $Z=[Z_{i},\cdots,Z_{N}]$. For a block $X_{i}$, the correlation score with
$Z$ is denoted by $\mathcal{A}=[a_{1},\cdots,a_{N}]$. As higher correlation
scores indicate more reliable feature matching, we treat the peak value of
$\mathcal{A}$ as the confidence score of $X_{i}$. Similar for other blocks, we
can obtain the confidence map (CMAP) of size $1\times H\times W$ ($N=H*W$)
which captures the semantic reliability of all blocks. The features of the
conditional input and exemplar (both of size $C\times H\times W$ after passing
through convolution layers) can thus be fused via weighted sum based on the
confidence map CMAP:

$\displaystyle F=X*(1-{\rm CMAP})+(T\cdot Z)*{\rm CMAP}$

where $T$ is the built correspondence matrix.

As the confidence map contains only one channel ($1\times H\times W$), the
above feature fusion is conducted in $H\times W$ but ignores that in $C$
channel. To achieve thorough feature fusion in all channels, we feed the
initial fusion $F$ to convolution layers to generate a multi-channel
confidence map (Multi-CMAP) of size $C\times H\times W$. The conditional input
and warped exemplar are then thoroughly fused via a full channel-weighted
summation according to the Multi-CMAP. The final fused feature is further
injected to the generation process via spatial de-normalization [54] to
provide accurate semantic and style guidance.

Figure 7:  Qualitative comparison of the proposed RABIT and state-of-the-art
methods over four types of conditional image translation tasks.

###  3.4 Loss Functions

The alignment network and generation network are jointly optimized. For
clarity, we still denote the conditional input and exemplar as $X$ and $Z$,
the ground truth as $X^{\prime}$, the generated image as $Y$, the feature
extractors for conditional input and exemplar as $E_{X}$ and $E_{Z}$, the
generator and discriminator in the generation network as $G$ and $D$.

Alignment Network. First, the warping should be cycle consistent, i.e. the
exemplar should be recoverable from the warped warped. We thus employ a cycle-
consistency loss as follows:

$\displaystyle\mathcal{L}_{cyc}=||T^{\top}\cdot T\cdot Z-Z||_{1}$

where $T$ is the correspondence matrix. The feature extractors $F_{X}$ and
$F_{Z}$ aim to extract invariant semantic information across domains, i.e. the
extracted features from $X$ and $X^{\prime}$ should be consistent. A feature
consistency loss can thus be formulated as follows:

$\displaystyle\mathcal{L}_{cst}=||F_{X}(X)-F_{Z}(X^{\prime})||_{1}$

Generation Network. The generation network employs several losses for high-
fidelity synthesis with consistent style with the exemplar and consistent
semantic with the conditional input. As the generated image $Y$ should be
semantically consistent with the ground truth $X^{\prime}$, we employ a
perceptual loss $\mathcal{L}_{perc}$ [26] to penalize their semantic
discrepancy as below:

$\mathcal{L}_{perc}=||\phi_{l}(Y)-\phi(X^{\prime})||_{1}$ (3)

where $\phi_{l}$ is the activation of layer $l$ in pre-trained VGG-19 [58]
model. To ensure the statistical consistency between the generated image $Y$
and the exemplar $Z$, a contextual loss [49] is adopted:

$\mathcal{L}_{cxt}=-\log(\sum_{i}\mathop{\max}\limits_{j}CX_{ij}(\phi_{l}^{i}(Z),\phi_{l}^{j}(Y)))$
(4)

where $i$ and $j$ are the indexes of the feature map in layer $\phi_{l}$.
Besides, a pseudo pairs loss $\mathcal{L}_{pse}$ as described in [87] is
included in training.

The discriminator $D$ is employed to drive adversarial generation with an
adversarial loss $\mathcal{L}_{adv}$ [25]. The full network is thus optimized
with the following objective:

$\begin{split}\mathcal{L}=&\mathop{\min}\limits_{F_{X},F_{Z},G}\mathop{\max}\limits_{D}(\lambda_{1}\mathcal{L}_{cyc}+\lambda_{2}\mathcal{L}_{cst}+\lambda_{3}\mathcal{L}_{perc}\\\
&+\lambda_{4}\mathcal{L}_{cxt}+\lambda_{5}\mathcal{L}_{pse}+\lambda_{6}\mathcal{L}_{adv})\\\
\end{split}$ (5)

where the weights $\lambda$ balance the losses in the objective.

##  4 Experiments

###  4.1 Experimental Settings

Datasets: We evaluate and benchmark our method over multiple datasets for
image translation & manipulation tasks.

$\bullet$ ADE20K [91] has 20k training images each of which is associated with
a 150-class segmentation mask. We use its semantic segmentation as conditional
inputs in image translation experiments, and 2k test images for evaluations.
For image manipulation, we apply object-level affine transformations on the
test set to acquire paired data (150 images) for evaluations as in [90].

$\bullet$ CelebA-HQ [42] has 30,000 high-quality face images. We conduct two
translation tasks by using face semantics and face edges as conditional
inputs. In addition, we also conduct image manipulation experiments on this
dataset by editing the face semantics. We use 2993 face images for translation
evaluations as in [87], and manually edit 100 semantic maps which is randomly
selected for image manipulation evaluations.

$\bullet$ DeepFashion [41] has 52,712 person images of different appearance
and poses. We use its key points as conditional inputs for image translation,
and select 4993 images for evaluations as in [87].

Evaluation Metrics: For image translation, we adopt Fréchet Inception Score
(FID) [18] and Sliced Wasserstein distance (SWD) [28] to evaluate the
perceptual quality of translated images. We adopt Learned Perceptual Image
Patch Similarity (LPIPS) [88] to evaluate the translation diversity with
different exemplars.

For image manipulation, we adopted FID, SWD and LPIPS to evaluate perceptual
quality of manipulated images. We also adopted L1 distance, peak signal-to-
noise ratio (PSNR) and structural similarity index (SSIM) as low-level
evaluation metrics. Note LPIPS evaluates image translation diversity by
measuring the distance between translated images, while it evaluates the image
manipulation quality by measuring the distance between manipulated images and
ground truth.

Similar to [87], we design $\rm VGG_{42}$ and $\rm VGG_{52}$ metrics to
evaluate semantic consistency and $\rm VGG_{M}$ and $\rm VGG_{V}$ metrics to
evaluate style consistency. For semantic consistency, we apply a pre-trained
VGG model [58] to extract high-level features ($relu4\\_2$ and $relu5\\_2$) of
the ground truth and generated images. The semantic consistency ($\rm
VGG_{42}$ and $\rm VGG_{52}$) is defined by the distance between the extracted
high-level features as computed by Cosine similarity. For style consistency,
we extract low-level style features ($relu1\\_2$) from the generated images
and exemplars. The style consistency ($\rm VGG_{M}$ and $\rm VGG_{V}$) is
defined by the distance of channel-wise mean and standard deviation as
computed by Cosine similarity.

Implementation Details: The alignment and generation networks are jointly
optimized with learning rates 1$e$-4 and 4$e$-4 for the generator and
discriminator, respectively. We adopted Adam solver with $\beta_{1}=0$ and
$\beta_{2}=0.999$. All experiments were conducted on 4 32GB Tesla V100 GPUs
with synchronized BatchNorm. The default size for our correspondence
computation is $128\times 128$ with a block size of $2\times 2$. The number
$k$ in top-$k$ ranking is set at 3 by default in our experiments. The size of
generated images is $256\times 256$ in all generation tasks.

###  4.2 Image Translation Experiments

We compare RABIT with eight state-of-the-art image translation methods: 1)
Pix2pixHD [65] on supervised image translation; 2) StarGAN v2[5] on multi-
modal translation with support for style encoding from reference images; 3)
SPADE [54] on supervised translation that supports style injection from
exemplar images; 4) SelectionGAN [61] on guided translation with cascaded
semantic guidance; 5) SMIS [96] on semantically multi-modal synthesis with all
group convolutions; 6) SEAN [95] on conditional generation that can control
the style of each individual semantic region; 7) CoCosNet [87] on exemplar-
based image translation that builds cross-domain correspondences; and 8)
CoCosNet v2 [93] on building high-resolution correspondences for image
translation. Note CoCosNet adopts a default correspondence size of $64\times
64$ in this work as constrained by high memory costs, whereas CoCosNet v2 and
RABIT adopt a default correspondence size of $128\times 128$.

Figure 8:  Illustration of generation diversity of the proposed RABIT: With
the same conditional input, RABIT can generate a variety of images that have
consistent styles with the provided exemplars. It works for different types of
conditional inputs consistently.

Quantitative Results. In quantitative experiments, all methods translate
images with the same exemplars except Pix2PixHD [65] which doesn’t support
style injection from exemplars. LPIPS is calculated by comparing the generated
images with randomly selected exemplars. All compared methods adopt three
exemplars for each conditional input and the final LPIPS is obtained by
averaging the LPIPS between any two generated images.

Table I shows experimental results. It can be seen that RABIT outperforms all
compared methods over most metrics and tasks consistently. By building
explicit yet accurate correspondences between conditional inputs and
exemplars, RABIT enables direct and accurate guidance from the exemplar and
achieves better translation quality (in FID and SWD) and diversity (in LPIPS)
as compared with the regularization-based methods such as SPADE [54] and SMIS
[96], and style-encoding methods such as StarGAN v2 [5] and SEAN [95].
Compared with correspondence-based method CoCosNet [87], the proposed bi-level
alignment allows RABIT to build correspondences and warp exemplars at higher
resolutions (e.g. $128\times 128$) which offers more detailed guidance in the
generation process and helps to achieve better FID and SWD. While compared
with CoCosNet v2 [93], the proposed semantic position encoding enables to
preserve the texture structures and patterns, thus yielding more accurate
warped exemplars as guidance. In addition, the proposed confidence feature
injection module fuses conditional inputs and warped exemplars adaptively
based on the matching confidence, which provides more reliable guidance and
improves FID and SWD. Besides generation quality, RABIT achieves the best
generation diversity in LPIPS except StarGAN v2 [5] which sacrifices the
generation quality with much lower FID and SWD.

TABLE III:  Comparing RABIT with state-of-the-art image manipulation methods on ADE20K [91] with evaluation metrics FID, SWD, PSNR, and SSIM.  Models | FID $\downarrow$ | SWD $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---|---
SPADE [54] | 120.2 | 41.62 | 13.11 | 0.334
HIM [20] | 59.89 | 22.23 | 18.23 | 0.667
SESAME [52] | 52.51 | 29.40 | 18.67 | 0.691
CoCosNet [87] | 41.03 | 23.08 | 20.30 | 0.744
CoCosNetv2 [93] | 34.31 | 19.55 | 21.75 | 0.797
RABIT | 26.61 | 15.05 | 23.08 | 0.823
TABLE IV:  Comparing RABIT with state-of-the-art image manipulation methods on CelebA-HQ [42] with evaluation metrics FID, SWD, SSIM.  Models |  FID $\downarrow$ |  SWD $\downarrow$ |  LPIPS $\downarrow$
---|---|---|---
SPADE [54] |  105.1 |  41.90 |  0.376
SEAN [95] |  96.31 |  35.90 |  0.351
MaskGAN [33] |  80.89 |  23.86 |  0.271
CoCosNet [87] |  68.70 |  22.90 |  0.224
CoCosNetv2 [93] |  62.53 |  21.11 |  0.190
RABIT |  60.87 |  21.07 |  0.176
TABLE V:  Comparing RABIT with state-of-the-art image manipulation methods in semantic consistency and style consistency (on ADE20K [91]).  | Semantic Consistency | Style Consistency
---|---|---
Methods |  ${\rm VGG_{42}}$ $\uparrow$ |  ${\rm VGG_{52}}$ $\uparrow$ |  ${\rm VGG_{M}}$ $\uparrow$ |  ${\rm VGG_{V}}$ $\uparrow$
SPADE [54] |  0.853 |  0.766 |  0.929 |  0.876
HIM [20] |  0.865 |  0.773 |  0.934 |  0.884
SESAME [52] |  0.870 |  0.779 |  0.969 |  0.947
CoCosNet [87] |  0.878 |  0.790 |  0.986 |  0.965
CoCosNet v2 [93] |  0.889 |  0.804 |  0.985 |  0.967
RABIT |  0.889 |  0.802 |  0.992 |  0.975

We also evaluated the generated images by measuring their semantic consistency
with the conditional inputs and their style consistency with the exemplars. As
shown in Table II, the proposed RABIT achieves the best style consistency
thanks to the bi-level feature alignment for building high-resolution
correspondences and the semantic position encoding for preservation of texture
patterns. It also achieves the best semantic consistency due to the confidence
feature injection that offers reliable fusion of semantic and style features.

Figure 9:  Qualitative illustration of RABIT and state-of-the-art image
manipulation methods on the augmented test set of ADE20K with ground truth as
described in [90]: The edited regions of the semantic maps are highlighted by
white boxes. The artifacts generated by CoCosNet and CoCosNet v2 are
highlighted by orange boxes. The proposed RABIT is capable of generating high-
fidelity editing results without undesired artifacts.  Figure 10:  The
comparison of image manipulation by MaskGAN [33] and the proposed RABIT over
dataset CelebA-HQ [42].

Qualitative Evaluations. Fig. 7 shows qualitative comparisons. It can be seen
that RABIT achieves the best visual quality with faithful styles as exemplars.
SPADE [54], SMIS [96] and StarGAN v2 [5] adopt single latent code to encode
image styles, which tend to capture global styles but miss local details. SEAN
[95] employs multiple latent codes but struggles in preserving faithful
exemplar styles. CoCosNet [87] builds low-resolution correspondences which
leads to missing details, while CoCosNet v2 [93] builds high-resolution
correspondence without position encoding which leads to destroyed texture
patterns. RABIT excels with its RAS that offers accurate feature alignment at
high resolution and SPE that preserves texture structures and patterns.

RABIT also demonstrates superior diversity in image translation as illustrated
in Fig. 8. It can be observed that RABIT is capable of synthesizing various
high-fidelity images with faithful styles as various exemplars.

###  4.3 Image Manipulation Experiment

The proposed RABIT manipulates images by treating input images as exemplars
and edited semantic guidance as conditional inputs. We compare RABIT with
several state-of-the-art image manipulation methods including 1) SPADE [54],
which supports semantic manipulation with style injection from input images;
2) SEAN [95] which supports semantic manipulation with style control of each
individual semantic region; 3) MaskGAN [34], a geometry-oriented face
manipulation framework with semantic masks as an intermediate representation
for manipulation. 4) Hierarchical Image Manipulation (HIM) [20], a
hierarchical framework for semantic image manipulation. 5) SESAME [52], a
semantic image editing method covering the operation of adding, manipulating,
and erasing. 6) CoCosNet [87], a leading exemplar-based image generation
framework that enables manipulation by building cross-domain correspondences.
7) CoCosNet v2 [93], which builds high-resolution correspondences ($128\times
128$) for image generation.

Figure 11:  Various image editing by the proposed RABIT: With input images as
the exemplars and edited semantic maps as the conditional input, RABIT
generates new images with faithful semantics and high-fidelity textures with
little artifacts.  Figure 12:  AMT (Amazon Mechanical Turk) user studies of
different image translation and image manipulation methods in terms of the
visual quality and style consistency of the generated images.

Quantitative Results: In quantitative experiments, all compared methods
manipulate images with the same input image and edited semantic label map.
Table III shows experimental results over the synthesized test set of ADE20K
[91]. It can be observed that RABIT outperforms state-of-the-art methods over
all evaluation metrics consistently. Table IV shows experimental results over
the CelebA-HQ dataset with manual edited semantic maps. It can be observed
that RABIT outperforms the state-of-the-art methods by large margins in all
perceptual quality metrics. The superior generation quality of RABIT is
largely attributed to the ranking and attention scheme for building high-
resolution correspondences and the semantic position encoding for preserving
rich texture details of input images.

Besides the quality of manipulated images, we also evaluate their semantic
consistency and style consistency as shown in Table V. It can be seen that
RABIT achieves the best semantic consistency and style consistency as compared
with state-of-the-art image manipulation methods. The outstanding performance
can be explained by the proposed ranking and attention scheme for building
high-resolution correspondence, the semantic position encoding for texture
pattern preservation as well as the confidence feature injection for reliable
image generation.

Qualitative Evaluation: Fig. 9 shows visual comparisons with state-of-art
manipulation methods on ADE20K. HIM [20] and SESAME [52] produce unrealistic
texture and artifacts for drastic semantic changes due to the lack of texture
details after masking. CoCosNet [87] can preserve certain details, but it
adopts Cosine similarity to align low-resolution features which often lead to
missing details as demonstrated by blurry textures and artifacts. RABIT
achieves superior fidelity due to its bi-level feature alignment for building
high-resolution correspondences, semantic position encoding for the
preservation of texture patterns and confidence feature injection for reliable
guidance in image generation. Fig. 11 shows the editing capacity of RABIT with
various types of manipulation on semantic labels. It can be seen that the
RABIT manipulation results faithfully aligns with the edited semantic maps and
produces realistic details. With the proposed bi-level feature alignment
strategy and semantic position encoding, RABIT accurately matches features for
the edited semantics and minimizes undesired changes outside the editing
regions.

We also compare RABIT with MaskGAN [33] on CelebA-HQ [34] in Fig. 10. MaskGAN
tends to introduce undesired changes in the edited images such as the skin
color (columns 1 and 3) and the missing hand (column 2). RABIT achieves better
editing with little change in other regions due to the accurate
correspondences built between input images and edited semantic maps.

###  4.4 User Study

We conduct crowdsourcing user studies through Amazon Mechanical Turk (AMT) to
evaluate the image translation & manipulation in terms of generation quality
and style consistency. The code of the AMT user studies is available at
111https://github.com/fnzhan/AMT. Specifically, each compared method generates
100 images with the same conditional inputs and exemplars. Then the generated
images together with the conditional inputs and exemplars were presented to 10
users for assessment. For the evaluation of image quality, the users were
instructed to pick the best-quality images. For the evaluation of style
consistency, the users were instructed to select the images with best style
relevance to the exemplar. The final AMT score is the averaged number of the
methods to be selected as the best quality and the best style relevance.

Fig. 12 shows AMT results on multiple datasets. It can be observed that RABIT
outperforms state-of-the-art methods consistently in image quality and style
consistency on both image translation & image manipulation tasks.

TABLE VI:  Ablation studies on image translation and image manipulation tasks (both on ADE20K [91]): COS refer to Cosine similarity for building correspondence. RAS and CONFI denote the proposed ranking and attention scheme for building correspondence and confidence feature injection module in the generation network, respectively. PE and SPE refer to vanilla position encoding and the proposed semantic position encoding, respectively. Model in the last row is the standard RABIT.  | Image Translation | Image Manipulation
---|---|---
Models | FID $\downarrow$ | SWD $\downarrow$ | LPIPS $\uparrow$ |  $\rm VGG_{52}$ $\uparrow$ |  $\rm VGG_{V}$ $\uparrow$ | FID $\downarrow$ | SWD $\downarrow$ | PSNR $\uparrow$ |  $\rm VGG_{52}$ $\uparrow$ |  $\rm VGG_{V}$ $\uparrow$
SPADE | 33.90 | 19.70 | 0.344 | 0.772 | 0.884 | 120.2 | 41.62 | 13.11 | 0.766 | 0.876
SPADE+COS | 27.72 | 14.98 | 0.556 | 0.787 | 0.941 | 42.02 | 23.23 | 19.92 | 0.772 | 0.926
CONFI+COS | 26.58 | 12.33 | 0.529 | 0.801 | 0.958 | 40.56 | 22.87 | 20.73 | 0.782 | 0.959
CONFI+RAS | 25.51 | 11.94 | 0.548 | 0.807 | 0.966 | 33.32 | 18.89 | 21.97 | 0.785 | 0.964
CONFI+RAS+PE | 24.93 | 10.20 | 0.578 | 0.791 | 0.974 | 28.32 | 16.48 | 22.79 | 0.797 | 0.971
CONFI+RAS+SPE* | 24.35 | 9.893 | 0.596 | 0.812 | 0.977 | 26.61 | 15.05 | 23.08 | 0.802 | 0.975
TABLE VII:  Ablation study of correspondence accuracy and memory cost with different correspondence size ($32\times 32$, $64\times 64$, and $128\times 128$), block size ($b$=4, 16, and 64) and top-$k$ selection ($k$=1, 2, and 3) on DeepFashion [41] for image translation. The correspondence accuracy is evaluated by comparing the warped images and the ground truth with evaluation metrics L1, SSIM and PSNR. MC denotes the memory cost in gigabyte (GB). ‘*’ denotes the default setting of RAS.  | $\mathbf{32\times 32}$ | $\mathbf{64\times 64}$ | $\mathbf{128\times 128}*$
---|---|---|---
Models | FID $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | MC $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | MC $\downarrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | MC $\downarrow$
CoCosNet [87] | 102.9 | 0.402 | 0.621 | 6.179 | 64.43 | 0.298 | 0.630 | 11.23 | 54.87 | 0.273 | 0.657 | 21.73
CoCosNet v2 [93] | 132.3 | 0.458 | 0.614 | 5.101 | 73.14 | 0.339 | 0.625 | 9.065 | 58.66 | 0.283 | 0.641 | 14.94
RAS ($k$=1, b=64) | 144.1 | 0.470 | 0.608 | 4.912 | 97.79 | 0.367 | 0.618 | 8.986 | 76.47 | 0.333 | 0.622 | 14.35
RAS ($k$=1, b=16) | 129.8 | 0.431 | 0.615 | 4.963 | 81.07 | 0.344 | 0.622 | 9.012 | 66.22 | 0.306 | 0.625 | 15.16
RAS ($k$=1, b=4) | 102.4 | 0.382 | 0.620 | 5.093 | 70.12 | 0.321 | 0.624 | 9.082 | 59.97 | 0.292 | 0.634 | 15.95
RAS ($k$=2, b=4) | 97.82 | 0.379 | 0.619 | 5.126 | 65.39 | 0.312 | 0.624 | 9.114 | 57.54 | 0.281 | 0.638 | 16.06
RAS ($k$=3, b=4)* | 95.93 | 0.357 | 0.623 | 5.157 | 63.84 | 0.302 | 0.628 | 9.136 | 54.15 | 0.268 | 0.644 | 16.35
TABLE VIII:  Ablation studies of loss functions over ADE20K [91]. $\mathcal{L}_{cyc}$ and $\mathcal{L}_{cst}$ denote the cycle-consistency loss and feature consistency loss in the alignment network. $\mathcal{L}_{per}$, $\mathcal{L}_{cxt}$ and $\mathcal{L}_{pse}$ denote perceptual loss, contextual loss and pseudo pairs loss in the generation network.  Models |  FID $\downarrow$ |  $\rm\mathbf{VGG_{52}}$ $\uparrow$ |  $\rm\mathbf{VGG_{V}}$ $\uparrow$
---|---|---|---
w/o $\mathcal{L}_{cyc}$ |  28.17 |  0.794 |  0.958
w/o $\mathcal{L}_{cst}$ |  29.27 |  0.809 |  0.962
w/o $\mathcal{L}_{per}$ |  45.16 |  0.738 |  0.861
w/o $\mathcal{L}_{cxt}$ |  35.05 |  0.798 |  0.853
w/o $\mathcal{L}_{pse}$ |  25.43 |  0.807 |  0.972
$\mathcal{L}_{full}$ |  24.35 |  0.812 |  0.977

###  4.5 Ablation Study

We conduct extensive ablation studies to evaluate our technical designs on
image translation and image manipulation tasks. Table VI shows experimental
results on ADE20K. SPADE [54] is selected as the baseline which achieves image
translation & manipulation without feature alignment. The performance is
clearly improved when Cosine similarity is included to align features as
denoted by (SPADE+COS). By replacing SPADE with the proposed CONFI for feature
injection, the FID score is improved to 15.97. In addition, the translation is
further improved by large margins when the proposed RAS is included for
building high-resolution correspondences. By including vanilla position
encoding (PE), FID score presents some improvement but $VGG_{52}$ scores
(semantic consistency) is affected severely. The proposed semantic position
encoding improves FID scores and semantic consistency consistently.

As the correspondence quality is critical to correspondence-based generation,
we analyze the accuracy, memory costs and parameters (e.g., resolution, block
size) in correspondence construction in different methods. The experiment was
conducted on DeepFashion dataset [41] (with paired images) where the warped
exemplars and the ground truth (resized to $256\times 256$) are compared in
L1, SSIM and PSNR metrics to evaluate the accuracy of built correspondence.
The memory cost is evaluated through the memory footprint in GPU. In
experiments, we compare Cosine similarity, Patch match and the proposed RAS
over sizes of $64\times 64$, $128\times 128$ and $128\times 128$,
respectively. As shown in Table VII, RAS(k=3, b=4) outperforms Cosine
similarity in CoCosNet [87] and Patch match in CoCosNet v2 [93] in L1, SSIM
and PSNR. In addition, RAS(k=1,b=64) reduces memory costs consistently under
different image resolutions as compared with CoCosNet [87] and CoCosNet v2
[93]. We also study the correspondence resolution ( $32\times 32$, $64\times
64$, $128\times 128$), top-$k$ number ($k$=1,2,3) and block size (b=4, 16, 64)
in RAS. As Fig. VII shows, the accuracy of the built correspondences keeps
improving and the memory cost keeps increasing when image resolution or the
top-k selection increases and the block size decreases. Compared with CoCosNet
and CoCosNet v2, RAS reduces memory more clearly with the increase of
correspondence resolution. With a trade-off between correspondence accuracy
and memory cost, we select $k$=3, b=4 and correspondence resolution of
$128\times 128$ as the default setting of RAS.

In addition, we perform several ablation studies to examine the contribution
of each loss by removing it from the overall objective. Table VIII show
experimental results on the image translation task over ADE20K. As shown in
Table VIII, all involved losses contribute to the image translation in
different manners and significance. Specially, the image quality as indicated
by FID drops clearly without the perceptual loss $\mathcal{L}_{perc}$, and the
style consistency as indicated by $VGG_{v}$ decreases significantly with the
removal of contextual loss $\mathcal{L}_{cxt}$.

##  5 Conclusions

This paper presents RABIT, a versatile conditional image translation &
manipulation framework that adopts a novel bi-level alignment strategy with a
ranking and attention scheme (RAS) to align the features between conditional
inputs and exemplars efficiently. As the ranking operation precludes the
gradient propagation in model training, we approximate it with a regularized
earth mover’s formulation which enables differentiable optimization of the
ranking operation. A semantic position encoding mechanism is designed to
facilitate semantic-level position information and preserve the texture
patterns in the exemplars. To handle the semantic mismatching between the
conditional inputs and warped exemplars, a novel confidence feature injection
module is proposed to achieve multi-channel feature fusion based on the
matching reliability of warped exemplars. Quantitative and qualitative
experiments over multiple datasets show that RABIT is capable of achieving
high-fidelity image translation and manipulation while preserving consistent
semantics with the conditional input and faithful styles with the exemplar.

The current exemplar-based image translation still requires the conditional
input and the exemplar to be semantically similar in building meaningful
correspondences, and this constrains the generalization of this translation
approach. A possible solution is to further relax the constraint of exemplar
selection. In this work, we propose the confidence feature injection module to
mitigate the semantic discrepancy between conditional inputs and exemplars by
assigning higher weights to the conditional input when the exemplar features
are misaligned. However, adjusting fusion weights only mitigates the
misalignment and the misaligned features still tend to mislead the generation
process more or less. Instead of adjusting the fusion weights, we could
rectify the misaligned features directly based on a pre-built feature bank
with well-aligned features. These related issues will be studied in our future
research.

##  6 Acknowledgement

This study is supported under the RIE2020 Industry Alignment Fund – Industry
Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-
kind contribution from the industry partner(s).

## References

  * [1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [2] H. Bristow, J. Valmadre, and S. Lucey.  Dense semantic correspondence where every pixel is a classifier.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4024–4031, 2015.
  * [3] Z. Chen, C. Wang, B. Yuan, and D. Tao.  Puppeteergan: Arbitrary portrait animation with semantic-aware appearance transformation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13518–13527, 2020.
  * [4] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8789–8797, 2018.
  * [5] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188–8197, 2020.
  * [6] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker.  Universal correspondence network.  arXiv preprint arXiv:1606.03558, 2016.
  * [7] K. Cui, G. Zhang, F. Zhan, and S. Lu.  Fbc-gan: Diverse and flexible image synthesis via foreground-background composition.  arXiv preprint, 2021.
  * [8] M. Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  In Advances in neural information processing systems, pages 2292–2300, 2013.
  * [9] Y. Dai, F. Gieseke, S. Oehmcke, Y. Wu, and K. Barnard.  Attentional feature fusion.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3560–3569, 2021.
  * [10] C. Doersch.  Tutorial on variational autoencoders.  arXiv preprint arXiv:1606.05908, 2016.
  * [11] H. Dong, X. Liang, K. Gong, H. Lai, J. Zhu, and J. Yin.  Soft-gated warping-gan for pose-guided person image synthesis.  arXiv preprint arXiv:1810.11610, 2018.
  * [12] Y. Fu, J. Ma, L. Ma, and X. Guo.  Edit: Exemplar-domain aware image-to-image translation.  arXiv preprint arXiv:1911.10520, 2019.
  * [13] L. A. Gatys, A. S. Ecker, and M. Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [14] S. Gu, J. Bao, H. Yang, D. Chen, F. Wen, and L. Yuan.  Mask-guided portrait editing with conditional gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3436–3445, 2019.
  * [15] B. Ham, M. Cho, C. Schmid, and J. Ponce.  Proposal flow: Semantic correspondences from object proposals.  IEEE transactions on pattern analysis and machine intelligence, 40(7):1711–1725, 2017.
  * [16] K. Han, R. S. Rezende, B. Ham, K.-Y. K. Wong, M. Cho, C. Schmid, and J. Ponce.  Scnet: Learning semantic correspondence.  In Proceedings of the IEEE international conference on computer vision, pages 1831–1840, 2017.
  * [17] M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan.  Deep exemplar-based colorization.  ACM Transactions on Graphics (TOG), 37(4):1–16, 2018.
  * [18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [19] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [20] S. Hong, X. Yan, T. Huang, and H. Lee.  Learning hierarchical semantic image manipulation through structured representations.  In NIPS, 2018.
  * [21] A. Hosni, C. Rhemann, M. Bleyer, C. Rother, and M. Gelautz.  Fast cost-volume filtering for visual correspondence and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2):504–511, 2012.
  * [22] X. Huang and S. Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510, 2017.
  * [23] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 172–189, 2018.
  * [24] J. Hur, H. Lim, C. Park, and S. Chul Ahn.  Generalized deformable spatial pyramid: Geometry-preserving dense correspondence estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1392–1400, 2015.
  * [25] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [26] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [27] L. V. Kantorovich.  Mathematical methods of organizing and planning production.  Management science, 6(4):366–422, 1960.
  * [28] T. Karras, T. Aila, S. Laine, and J. Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [29] J. Kim, C. Liu, F. Sha, and K. Grauman.  Deformable spatial pyramid matching for fast dense correspondences.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2307–2314, 2013.
  * [30] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [31] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 624–632, 2017.
  * [32] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  * [33] C.-H. Lee, Z. Liu, L. Wu, and P. Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In CVPR, pages 5549–5558, 2020.
  * [34] C.-H. Lee, Z. Liu, L. Wu, and P. Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [35] Y. Li, Y. Cheng, Z. Gan, L. Yu, L. Wang, and J. Liu.  Bachgan: High-resolution image synthesis from salient object layout.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [36] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang.  Universal style transfer via feature transforms.  arXiv preprint arXiv:1705.08086, 2017.
  * [37] J. Liao, Y. Yao, L. Yuan, G. Hua, and S. B. Kang.  Visual attribute transfer through deep image analogy.  arXiv preprint arXiv:1705.01088, 2017.
  * [38] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [39] H. Liu, B. Jiang, Y. Song, W. Huang, and C. Yang.  Rethinking image inpainting via a mutual encoder-decoder with feature equalizations.  In ECCV, pages 725–741, 2020.
  * [40] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski.  An intriguing failing of convolutional neural networks and the coordconv solution.  In Advances in Neural Information Processing Systems, 2018.
  * [41] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1096–1104, 2016.
  * [42] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.
  * [43] J. Long, N. Zhang, and T. Darrell.  Do convnets learn correspondence?  arXiv preprint arXiv:1411.1091, 2014.
  * [44] D. G. Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [45] J. Lucas, G. Tucker, R. Grosse, and M. Norouzi.  Don’t blame the elbo! a linear vae perspective on posterior collapse.  arXiv preprint arXiv:1911.02469, 2019.
  * [46] G. Luise, A. Rudi, M. Pontil, and C. Ciliberto.  Differential properties of sinkhorn approximation for learning with wasserstein distance.  In NIPS 2018-Advances in Neural Information Processing Systems, pages 5864–5874, 2018.
  * [47] L. Ma, X. Jia, S. Georgoulis, T. Tuytelaars, and L. Van Gool.  Exemplar guided unsupervised image-to-image translation with semantic consistency.  In International Conference on Learning Representations, 2018.
  * [48] L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool.  Pose guided person image generation.  In Advances in neural information processing systems, pages 406–416, 2017.
  * [49] R. Mechrez, I. Talmi, and L. Zelnik-Manor.  The contextual loss for image transformation with non-aligned data.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 768–783, 2018.
  * [50] Y. Men, Y. Mao, Y. Jiang, W.-Y. Ma, and Z. Lian.  Controllable person image synthesis with attribute-decomposed gan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5084–5093, 2020.
  * [51] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim.  Image to image translation for domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  * [52] E. Ntavelis, A. Romero, I. Kastanis, L. Van Gool, and R. Timofte.  Sesame: Semantic editing of scenes by adding, manipulating or erasing objects.  In ECCV, pages 394–411. Springer, 2020.
  * [53] M. Okutomi and T. Kanade.  A multiple-baseline stereo.  IEEE Transactions on pattern analysis and machine intelligence, 15(4):353–363, 1993.
  * [54] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [55] A. Pumarola, A. Agudo, A. M. Martinez, A. Sanfeliu, and F. Moreno-Noguer.  Ganimation: Anatomically-aware facial animation from a single image.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 818–833, 2018.
  * [56] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb.  Learning from simulated and unsupervised images through adversarial training.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2107–2116, 2017.
  * [57] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb.  Learning from simulated and unsupervised images through adversarial training.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2107–2116, 2017.
  * [58] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [59] W. Sun and T. Wu.  Image synthesis from reconfigurable layout and style.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10531–10540, 2019.
  * [60] H. Tang, D. Xu, G. Liu, W. Wang, N. Sebe, and Y. Yan.  Cycle in cycle generative adversarial networks for keypoint-guided image generation.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 2052–2060, 2019.
  * [61] H. Tang, D. Xu, N. Sebe, Y. Wang, J. J. Corso, and Y. Yan.  Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2417–2426, 2019.
  * [62] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker.  Learning to adapt structured output space for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7472–7481, 2018.
  * [63] Z. Wan, B. Zhang, D. Chen, P. Zhang, D. Chen, J. Liao, and F. Wen.  Bringing old photos back to life.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2747–2757, 2020.
  * [64] M. Wang, G.-Y. Yang, R. Li, R.-Z. Liang, S.-H. Zhang, P. M. Hall, and S.-M. Hu.  Example-guided style-consistent image synthesis from semantic labeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1495–1504, 2019.
  * [65] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [66] Y. Wang, X. Tao, X. Qi, X. Shen, and J. Jia.  Image inpainting via generative multi-column convolutional neural networks.  In Advances in Neural Information Processing Systems, pages 331–340, 2018.
  * [67] R. Wu and S. Lu.  Leed: Label-free expression editing via disentanglement.  In European Conference on Computer Vision, pages 781–798. Springer, 2020.
  * [68] R. Wu, G. Zhang, S. Lu, and T. Chen.  Cascade ef-gan: Progressive facial expression editing with local focuses.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5021–5030, 2020.
  * [69] W. Xia, Y. Yang, J.-H. Xue, and B. Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2256–2265, 2021.
  * [70] Y. Xie, H. Dai, M. Chen, B. Dai, T. Zhao, H. Zha, W. Wei, and T. Pfister.  Differentiable top-k with optimal transport.  Advances in Neural Information Processing Systems, 33, 2020.
  * [71] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang.  Free-form image inpainting with gated convolution.  In ICCV, pages 4471–4480, 2019.
  * [72] Y. Yu, F. Zhan, R. Wu, J. Pan, K. Cui, S. Lu, F. Ma, X. Xie, and C. Miao.  Diverse image inpainting with bidirectional and autoregressive transformers.  arXiv preprint arXiv:2104.12335, 2021.
  * [73] F. Zhan and S. Lu.  Esir: End-to-end scene text recognition via iterative image rectification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2059–2068, 2019.
  * [74] F. Zhan, S. Lu, and C. Xue.  Verisimilar image synthesis for accurate detection and recognition of texts in scenes.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 249–266, 2018.
  * [75] F. Zhan, S. Lu, C. Zhang, F. Ma, and X. Xie.  Adversarial image composition with auxiliary illumination.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [76] F. Zhan, S. Lu, C. Zhang, F. Ma, and X. Xie.  Towards realistic 3d embedding via view alignment.  arXiv preprint arXiv:2007.07066, 2020.
  * [77] F. Zhan, C. Xue, and S. Lu.  Ga-dan: Geometry-aware domain adaptation network for scene text detection and recognition.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9105–9115, 2019.
  * [78] F. Zhan, Y. Yu, K. Cui, G. Zhang, S. Lu, J. Pan, C. Zhang, F. Ma, X. Xie, and C. Miao.  Unbalanced feature transport for exemplar-based image translation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [79] F. Zhan, Y. Yu, R. Wu, C. Zhang, S. Lu, L. Shao, F. Ma, and X. Xie.  Gmlight: Lighting estimation via geometric distribution approximation.  arXiv preprint arXiv:2102.10244, 2021.
  * [80] F. Zhan and C. Zhang.  Spatial-aware gan for unsupervised person re-identification.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6889–6896. IEEE, 2021.
  * [81] F. Zhan, C. Zhang, W. Hu, S. Lu, F. Ma, X. Xie, and L. Shao.  Sparse needlets for lighting estimation with spherical transport loss.  arXiv preprint arXiv:2106.13090, 2021.
  * [82] F. Zhan, C. Zhang, Y. Yu, Y. Chang, S. Lu, F. Ma, and X. Xie.  Emlight: Lighting estimation via spherical distribution approximation.  arXiv preprint arXiv:2012.11116, 2020.
  * [83] F. Zhan, H. Zhu, and S. Lu.  Scene text synthesis for efficient and effective deep network training.  arXiv preprint arXiv:1901.09193, 2019.
  * [84] F. Zhan, H. Zhu, and S. Lu.  Spatial fusion gan for image synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3653–3662, 2019.
  * [85] B. Zhang, M. He, J. Liao, P. V. Sander, L. Yuan, A. Bermak, and D. Chen.  Deep exemplar-based video colorization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8052–8061, 2019.
  * [86] J. Zhang, S. Lu, F. Zhan, and Y. Yu.  Blind image super-resolution via contrastive representation learning.  arXiv preprint arXiv:2107.00708, 2021.
  * [87] P. Zhang, B. Zhang, D. Chen, L. Yuan, and F. Wen.  Cross-domain correspondence learning for exemplar-based image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5143–5153, 2020.
  * [88] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [89] B. Zhao, L. Meng, W. Yin, and L. Sigal.  Image generation from layout.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8584–8593, 2019.
  * [90] H. Zheng, Z. Lin, J. Lu, S. Cohen, J. Zhang, N. Xu, and J. Luo.  Semantic layout manipulation with high-resolution sparse attention.  arXiv preprint arXiv:2012.07288, 2020.
  * [91] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
  * [92] T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros.  Learning dense correspondence via 3d-guided cycle consistency.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 117–126, 2016.
  * [93] X. Zhou, B. Zhang, T. Zhang, P. Zhang, J. Bao, D. Chen, Z. Zhang, and F. Wen.  Cocosnet v2: Full-resolution correspondence learning for image translation.  In CVPR, 2021.
  * [94] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros.  Generative visual manipulation on the natural image manifold.  In ECCV, pages 597–613. Springer, 2016.
  * [95] P. Zhu, R. Abdal, Y. Qin, and P. Wonka.  Sean: Image synthesis with semantic region-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5104–5113, 2020.
  * [96] Z. Zhu, Z. Xu, A. You, and X. Bai.  Semantically multi-modal image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5467–5476, 2020.

Fangneng Zhan received the B.E. degree in Communication Engineering and Ph.D.
degree in Computer Science & Engineering from the University of Electronic
Science and Technology of China and Nanyang Technological University,
respectively. He will join the Inception Institute of Artificial Intelligence
(IIAI) as a research associate. His research interests include deep generative
models, image synthesis and manipulation. He contributed to the research field
by publishing more than 10 articles in prestigious conferences. He also served
as a reviewer or program committee member for top conferences including ICLR,
NeurIPS, CVPR, ICCV, ECCV, AAAI.
---
Yingchen Yu obtained the B.E. degree in Electrical & Electronic Engineering at
Nanyang Technological University, and M.S. degree in Computer Science at
National University of Singapore. He is currently pursuing the Ph.D. degree at
School of Computer Science and Engineering, Nanyang Technological University
under Alibaba Talent Programme. His research interests include computer vision
and machine learning, specifically for image synthesis and manipulation.
---
Rongliang Wu received the B.E. degree in Information Engineering from South
China University of Technology, and M.S. degree in Electrical and Computer
Engineering from National University of Singapore. He is currently pursuing
the Ph.D. degree at School of Computer Science and Engineering, Nanyang
Technological University. His research interests include computer vision and
deep learning, specifically for facial expression analysis and generation.
---
Kaiwen Cui obtained the B.E. degree in Electrical Engineering at National
University of Singapore, and M.S. degree in Electrical Engineering at National
University of Singapore. He is currently pursuing the Ph.D. degree at School
of Computer Science and Engineering, Nanyang Technological University. His
research interests include computer vision and machine learning, specifically
for image synthesis.
---
Aoran Xiao received the BS and MS degrees from Wuhan University in 2016 and
2019. He is currently pursuing the PhD degree at School of Computer Science
and Engineering, Nanyang Technological University. His research interests
include deep learning, point cloud processing and computer vision.
---
Shijian Lu is an Assistant Professor in the School of Computer Science and
Engineering, Nanyang Technological University. He received his PhD in
Electrical and Computer Engineering from the National University of Singapore.
His research interests include computer vision and deep learning. He has
published more than 100 internationally refereed journal and conference
papers. Dr Lu is currently an Associate Editor for the journals of Pattern
Recognition and Neurocomputing.
---
Ling Shao is the CEO and the Chief Scientist of the Inception Institute of
Artificial Intelligence (IIAI), Abu Dhabi, United Arab Emirates. He was also
the initiator and the Founding Provost and Executive Vice President of the
Mohamed bin Zayed University of Artificial Intelligence (the world’s first AI
University), UAE. His research interests include computer vision, deep
learning, medical imaging and vision and language. He is a fellow of the IEEE,
the IAPR, the IET, and the BCS.
---