# Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative
Adversarial Network with Graph Representation Learning

Xingqun Qi∗, Muyi Sun∗, Qi Li, , Caifeng Shan Xingqun Qi is with the School of
Automation, Beijing University of Posts and Telecommunications, Beijing
100876, China (e-mail: xingqunqi@bupt.edu.cn).Muyi Sun and Qi Li are with the
Center for Research on Intelligent Perception and Computing, National
Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of
Sciences, Beijing 100190, China (e-mail<EMAIL_ADDRESS>qli@nlpr.ia.ac.cn). Caifeng Shan is with the College of Electrical Engineering
and Automation, Shandong University of Science and Technology, Qingdao 266590,
China, and also with the Artificial Intelligence Research, Chinese Academy of
Sciences, Beijing 100190, China (e-mail: caifeng.shan@gmail.com).${\ast}$
These authors contributed equally to this work. This work was done when
Xingqun Qi was an intern in the Center for Research on Intelligent Perception
and Computing, National Laboratory of Pattern Recognition, Institute of
Automation, Chinese Academy of Sciences.Corresponding author: Caifeng Shan.

###### Abstract

In recent years, significant progress has been achieved in biphasic face
photo-sketch synthesis with the development of Generative Adversarial Network
(GAN). Biphasic face photo-sketch synthesis could be applied in wide-ranging
fields such as digital entertainment and law enforcement. However, generating
realistic photos and distinct sketches suffers from great challenges due to
the low quality of sketches and complex photo variations in the real scenes.
To this end, we propose a novel Semantic-Driven Generative Adversarial Network
to address the above issues, cooperating with the Graph Representation
Learning. Specifically, we inject class-wise semantic layouts into the
generator to provide style-based spatial supervision for synthesized face
photos and sketches. In addition, to improve the fidelity of the generated
results, we leverage the semantic layouts to construct two types of
Representational Graphs which indicate the intra-class semantic features and
inter-class structural features of the synthesized images. Furthermore, we
design two types of constraints based on the proposed Representational Graphs
which facilitate the preservation of the details in generated face photos and
sketches. Moreover, to further enhance the perceptual quality of synthesized
images, we propose a novel biphasic training strategy which is dedicated to
refine the generated results through Iterative Cycle Training. Extensive
experiments are conducted on CUFS and CUFSF datasets to demonstrate the
prominent ability of our proposed method which achieves the state-of-the-art
performance.

###### Index Terms:

Generative adversarial network, face photo-sketch synthesis, graph
representation learning, intra-class and inter-class, iterative cycle
training.

##  I Introduction

Biphasic face photo-sketch synthesis refers to generating sketches from face
photos and generating photos from face sketches, which has attracted plentiful
attention in recent years. The wide-ranging application fields of the biphasic
face photo-sketch synthesis include digital entertainment, law enforcement,
and criminal case judgment. Face sketch is one of the most popular and
fundamental portrait painting styles in the scopes of digital entertainment
[1]. However, it requires vast time and efforts to create distinct face
sketches by professional artists. In law enforcement and criminal case
judgment, police commonly just hold the sketches of suspects drawn from the
description of the witnesses. Face photos synthesized from these sketches with
clear identities and manifest features could provide a feasible way to promote
the efficiency of the justice criminal cases [2]. Due to the significant
practical value, it is especially essential to automatically synthesize face
photos and sketches with realistic effects and consistent identity
preservation. Thus far, numerous approaches have been proposed to facilitate
face sketch synthesis and photo synthesis which could be unified together
because of the inseparable relevance between these two tasks.

Figure 1: Illustrations of samples with facial prior knowledge and parsing
layout in different databases: (a) CUHK database, (b) AR database, (c) XM2VTS
database, (d) CUFSF database. For each database, from top to bottom are face
photos, parsing layout of photos, saliency detection result of photos,
sketches, parsing layout of sketches, and saliency detection result of
sketches, sequentially.

In the previous research studies on biphasic face photo-sketch synthesis,
there are three types of methods, exemplar-based approaches [3], linear-
regression based approaches [4], and model-based approaches [5].

In the scope of solving biphasic face photo-sketch synthesis, exemplar-based
approaches are the primitive pioneering mainstream methods [1], [3], [6], [7].
The essential of the exemplar-based approaches is to bridge the mapping
relationship from exemplar face images to test face images which belong to the
dataset with strict alignment of photo and sketch. Concretely, these kinds of
approaches mostly employ image patches for modeling, with the assumption that
patches can be more easily matched for face alignment than the integral images
[8]. Then the final output sketches (or photos) are constructed directly by
blending the exemplar patches that corresponding to the photo-sketch pairs.
Despite exemplar-based approaches obtain considerable performance in early
researches, these approaches have obvious shortcomings. There is less
personality identity of the synthesized sketches (or photos), which means the
artifacts and over-smoothing phenomenon in the final images. Furthermore,
exemplar-based approaches need massive computational cost and inference time
in the patch-wise matching process. Otherwise, exemplar-based approaches are
highly dependent on exemplars and have defective effects on the results
generated by low-quality images which impair the generalization ability of
these approaches in the real scenes.

Linear-regression based approaches engage in building the linear mapping
between face photos and sketches, which relies on the hypothesis that they
have similar structures in low-dimensional manifolds [9], [10]. In order to
ensure that the synthesized sketches and photos retain sufficient personalized
characteristics, the low-dimensional manifolds of the input images need to be
learned from a large amount of training data. Therefore, three types of
methods have been proposed to solve the problem of insufficient training data,
which could be classified as: the Locally Linear Embedding (LLE) based method
[10], [11], the Markov Random Field (MRF) based method [12], and the Bayesian-
based method [13]. However, the mapping function might not be accurately
formulated, resulting in the poor quality of the generated images. Moreover,
although linear-regression based approaches are computationally inexpensive,
it is difficult to guarantee the identity-specific consistency between results
and input images, such as hairstyles, the contour of the ears and other
details [6].

Recently, the image-to-image translation task has witnessed impressive
progress with the development of deep learning, especially with Generative
Adversarial Network (GAN) [14]. A large number of researchers are motivated to
employ GAN on biphasic face photo-sketch synthesis tasks [15-18]. These
researches directly model the mapping from face photos to sketches or vice
versa, which avoids the problem of insufficient training data in the low-
dimensional manifold hypothesis. Nevertheless, there are distortion and noises
in the images generated by these methods, due to the low quality of sketches
and complex photo variation in the real scenes. Therefore, many recent studies
are dedicated to providing prior information and more supervisions for the
generated images [19], [20]. However, these methods are often short of the
realistic detailed depiction of the synthesized images. They generate sketches
or face photos from the holistic perspective and ignore the local specificity
of different regions in the face.

Motivated by the above studies, we propose a novel Semantic-Driven Generative
Adversarial Network with Graph Representation Learning for biphasic face
photo-sketch synthesis. To begin with, we leverage the pre-trained facial
saliency detection network U2-Net [21] to obtain the prior information of
overall facial texture structure. Besides, we inject class-wise semantic
layouts as two modulation parameters into the generator to provide style-based
spatial supervision for synthesized face photos and sketches. To improve the
fidelity of the details of the generated images, we exploit semantic layouts
to construct two types of Representational Graphs which could constrain the
intra-class and inter-class features of the synthesized images. Furthermore,
based on the observation of the inseparability of the symmetrical tasks of
face sketch synthesis and photo synthesis, we propose a novel biphasic
training strategy which is dedicated to refine the generated results through
Iterative Cycle Training.

The main contributions of this study are summarized as:

$\bullet$ A novel Semantic-Driven Generative Adversarial Network with Graph
Representation Learning is proposed for generating realistic photos and
distinct sketches.

$\bullet$ We construct two types of Representational Graphs and design
corresponding constraints which facilitate the preservation of the details in
generated face photos and sketches.

$\bullet$ We propose a novel biphasic training strategy to improve the
perceptual quality of the synthesized images through the Iterative Cycle
Training.

$\bullet$ Extensive comparison experiments are conducted on CUFS and CUFSF
datasets, which show that our method obtains state-of-the-art performance.

Compared to our preliminary work in [22], this article makes the following
extensions and improvements: 1) We consult the previous intra-class feature
supervision to propose a novel region-wise inter-class structural graph which
facilitates the preservation of the details and personal identities in
generated face photos and sketches; 2) A novel biphasic training strategy is
proposed to refine the generated results through Iterative Cycle Training; 3)
We conduct the further extension experiments on both sketch synthesis and face
photo synthesis tasks.

##  II Related Work

###  II-A Biphasic Face Photo-Sketch Synthesis

Biphasic face photo-sketch synthesis has developed rapidly in the last few
decades. Massive works have been proposed to solve this problem, which
includes two closely related subtasks: face sketch synthesis and face photo
synthesis. Therefore, researchers often analyze and discuss these two subtasks
in a unified manner.

Deep neural network based approaches are the mainstream routine of biphasic
photo-sketch synthesis in recent years, which have gradually emerged with the
boost of Generative Adversarial Networks. Ji _et al._ [16] employed multi-
domain adversarial methods to construct a mapping from photo-domain to sketch-
domain. Zhu _et al._ [19] borrowed knowledge from transfer learning and
proposed a lightweight network supervised by a high-performance larger
network. Zhang _et al._ [23] embedded the photo parsing priors and designed a
parametric sigmoid activation function in GAN based framework to facilitate
robust sketch synthesis. Recently, Yu _et al._ [20] decomposed the face
parsing layouts into multiple compositions and encoded them into conditional
GAN (cGAN) for biphasic face photo-sketch synthesis which achieved state-of-
the-art performance. Moreover, Duan _et al._ [18] introduced the gradient-
based self-attention mechanism to combine the global residual connection and
local residual connection in the proposed network which achieved better
results. Besides, Zhang _et al._ [24] designed a dual transfer strategy to
promote the biphasic face photo-sketch synthesis. Lin _et al._ [25] proposed
the feature injection module to preserve the identity of synthesized sketches
and photos. However, the sketches and photos synthesized by these methods are
short of realistic detailed depiction.

Inspired by previous works, we inject the statistics information into the
generator of our proposed network. However, our injection module is different
from previous works which embedded the multi-level identity feature into the
network. In contrast, we aim at providing class-wise style-based spatial
supervision for synthesized face photos and sketches. Furthermore, the
previous work [25] did not follow the common experiment setting which ignored
the influence of background on the generated results.

Figure 2: The pipeline of our semantic-driven network. (a) The over
architecture of network, which exploits Pix2Pix as backbone. The decoder
contains several Statistics Injection (SI) ResBlocks with upsampling layers.
(b) Details of Statistics Injection (SI) ResBlocks. (c) Details of Statistics
Injection Module. In each Statistics Injection Module, the semantic map is
convoluted to produce pixel-level normalization parameters $\gamma$ and
$\beta$.

###  II-B Paired Image-to-Image Style Transfer

Paired image-to-image style transfer tasks can be regarded as a combination of
image-to-image translation tasks and style transfer tasks. The image-to-image
translation is often formulated as pixel-wise image generation tasks applied
with paired images like biphasic face photo-sketch synthesis. Isala _et al._
[26] proposed a cGAN architecture to solve the image-to-image translation task
with paired input and output named Pix2Pix. Due to the eminent performance of
the Pix2Pix on the paired dataset, researchers have made numerous improvements
based on Pix2Pix and applied them to a wide range of research fields [27],
[28]. By combining Pix2Pix and residual blocks, Wang _et al._ [29] proposed a
novel network architecture to generate high-resolution images named pix2pixHD.
Moreover, Park _et al._ [30] introduced the semantic layouts as spatial
supervision injected in the pix2pixHD for synthesizing photorealistic images.
Motivated by previous researches, we exploit cGAN like Pix2Pix as our backbone
network.

Biphasic face photo-sketch synthesis task could be treated as an image style
transfer task between realistic photo and vivid sketch portraits. Gatys _et
al._ [31] successfully applied pre-trained CNNs to the image style transfer
task. Furthermore, Ulyanov _et al._ [32], [33] optimized the style transfer
process by manipulating the Batch Normalization (BN) layers and Instance
Normalization (IN) layers. Dumoulin _et al._ [34] utilized a group of
parameters to realize the transfer of various image styles. Consecutively,
Huang _et al._ [35] proposed the adaptive instance normalization (AdaIN)
layers which could perform arbitrary style transfer without training
repeatedly. Recently, Park _et al._ [13] put forward the spatially-adaptive
normalization (SPADE) layers that inject the image style from the semantic
layouts to obtain photorealistic images. Motivated by the previous research,
we inject the style-based statistic information into the network to generate
images with more distinct characteristics.

###  II-C Graph Representation Learning

Graph representation learning plays a significant role in the computer vision
tasks which could encode each node represented by a low-dimensional dense
embedding in the graph structure [36]. Perozzi _et al._ [37] effectively
embedded information of nodes by randomly sampling the graph structure.
Furthermore, Defferrard _at al._[38] designed a novel pooling strategy to
rearrange the nodes for preserving more useful information of graph. Li _et
al._ [39] constructed the structural graph of actions to capture the high-
order dependencies between successive skeletons in the proposed actional-
structural graph convolution network for skeleton-based action recognition.
Besides, Ren _et al._ [40] adopted the graph generator to build the
connections among spatial parts and construct the feature graph of these nodes
representation for biometrics. Yang _et al._ [41] utilized a novel graph
representation specifically designed for sketches by bridging the structural
hierarchical relationship of sketches. This work has significantly improved
the recognition accuracy of sketches. As for biphasic face photo-sketch
synthesis task, Zhu _et al._ [7] combined graphical exemplar-based features
with deep neural networks to synthesize high-quality sketches. The results
synthesized by [7] are robust against lighting variations and clutter
backgrounds. However, they still suffer from over-smoothing and lacking
specific-identify characters in the synthesized photos and sketches. On the
contrary, our graph representation learning algorithms can make the restraint
of intra-class features and inter-class features which help keeping the
realistic personal details of synthesized photos and sketches.

Figure 3: Illustration of our proposed intra-class semantic graph and inter-
class structural graph. The semantic masks are produced by pre-trained face
parsing network from the input photo. Besides, both the intra-class semantic
graph and inter-class structural graph consist of $\mu$ and $\nu$ as graph
nodes which denote the mean center and variance center of each semantic class.

##  III Method

In this section, details about our proposed Semantic-Driven Generative
Adversarial Network with Graph Representation Learning are presented. To begin
with, we introduce the preliminaries and motivations of our method.
Afterwards, our network architecture is described. Subsequently, we elaborate
the graph representation learning algorithms and biphasic training strategy of
our method. Finally, the objective functions of our model are introduced.

###  III-A Preliminaries

Our semantic-driven network aims to construct a biphasic mapping from paired
photo to sketch and sketch to photo by utilizing class-wise semantic layouts
as guidance. Previous researches synthesize sketches or photos directly
without taking advantage of semantic information. Inherently, the translations
from face to sketch and sketch to photo are paired, and the maintenance of
semantic information is extremely significant. Our network adopts semantic
layouts to guide the generation of the sketches and photos, especially the
detailed class-wise regions. Moreover, we construct two types of semantic
related graph representation algorithms which are dedicate to apply the
constraints on the intra-class and inter-class feature distributions of the
synthesized images. Furthermore, we design a novel biphasic training strategy
to refine the perceptual quality of the synthesized images through the
Iterative Cycle Training. Besides, we conduct facial saliency detection on the
input images to provide overall facial texture structure. In the following
part, we take the sketch synthesis task as prototype to introduce our network.

Given paired photo-sketch training samples
$\begin{Bmatrix}\left(x_{i},y_{i}\right)|x_{i}\in X,y_{i}\in
Y\end{Bmatrix}_{i=1}^{N}$, where $x_{i}$ represents photo and $y_{i}$
represents sketch. The purpose of face sketch synthesis is to construct a
mapping from source photo domain $X$ to target sketch domain $Y$. As
illustrated in Fig. 1, we find that there are complex variations in the source
domain which could result in severe impacts on the identity and fidelity of
the generated sketches as depicted in Fig. 5 (c), (d) and Fig. 6 (c), (d).
Conversely, in the photo synthesis task, the low quality sketches of the
source domain will also affect the clarity of the generated image. Face
saliency detection could effectively capture the structured texture of the
input photos or sketches. With regards to this, to handle these challenging
issues, we first utilize face saliency detection results as prior information
to provide the overall facial texture structure. We concatenate the texture
structure $M$ and the $X$ as the input to the generator. Besides, we also
employ the pre-trained face parsing network to acquire semantic layouts $S$.
Then the face semantic information $S$ is injected into our network to produce
the final synthesized result. Therefore, the overall mapping can be formulated
as $\begin{Bmatrix}X,M,S\end{Bmatrix}\rightarrow Y$.

###  III-B Semantic Driven Network Architecture

Fig. 2 illustrates the overall architecture of our network. Specifically, we
concatenate the paired $M$ and $X$ as inputs to supply the overall facial
texture structure. Pix2Pix [32] is exploited as the backbone which contains 7
convolutional and downsampling layers in the encoder part. In order to further
strengthen the conditional semantic information in the forwarding process, we
also design 7 Statistics Injection (SI) ResBlocks in the decoder, which is
motivated by [36]. As shown in Fig. 2 (b), the SI ResBlock consists of two
convolutional layers, two ReLU layers, and two Statistics Injection (SI)
modules.

As shown in Fig. 2 (c), each SI module takes two inputs: the forward
activation features after Batch Normalization layer and semantic masks $S$,
which are obtained by pre-trained BiSeNet [42]. In order to prevent semantic
ambiguity, we divide the face into 12 classes which are closely related to
sketches: two eyes, two eyebrows, two ears, glasses, upper and lower lips,
inner mouth, hair, nose, skin, neck, cloth, and background. Therefore, we have
$S=\begin{Bmatrix}s^{(1)},\cdots,s^{(c)}\end{Bmatrix}\in\mathbb{R}^{h\times
w\times c}$, where $c\in\begin{bmatrix}1,2,\cdots,12\end{bmatrix}$,
$s^{(c)}\in\begin{bmatrix}0,1\end{bmatrix}$, $h$ and $w$ denote the height and
width of the feature maps. In order to inject the semantic information into
the SI module, we perform the convolutional operation on $S$ to produce the
modulation parameters $\gamma$ and $\beta$ to normalize the final output. The
produced $\gamma$ and $\beta$ encode sufficient spatial layout information
which are multiplied and added to the normalized activation through an
element-wise way as shown in Fig. 2 (c). In fact, the modulation parameters
$\gamma$ and $\beta$ could provide a kind of spatial supervision from $X$ to
$Y$ through $S$ which are robust to the low quality of sketches introduced by
different artists and complex variations of photos. Finally, the network
structure of the discriminator keeps the same settings as Pix2Pix.

###  III-C Graph Representation Learning

In previous studies, researchers always impose overall supervision and
constraints on the entire generated sketches or photos, which leads to
defective performances in facial details. In this paper, we construct two
types of graph representations which are named intra-class semantic graph and
inter-class structural graph to facilitate the preservation of the details in
generated face photos and sketches as depicted in Fig. 3. Meanwhile, we design
an Adaptive Re-weighting algorithm which could extract the semantic graph
nodes for two types of graphs. Besides, the Adaptive Re-weighting algorithm
could balance the contributions of different semantic classes, especially the
detailed parts.

####  III-C1 Intra-class Semantic Graph Representation

As illustrated in Fig. 3, the synthesized sketch is represented as $F\in
R^{h_{f}\times w_{f}\times c_{f}}$, where $h_{f}$, $w_{f}$ and $c_{f}$ denote
the height, width and channel of the sketch, respectively. The $F$ could be
treated as 12 semantic class-regions. Then we enforce the element-wise
multiplication between $F$ and 12 semantic masks $S$ to extract the Interest-
Region (IR) for each semantic class node. Consequently, each node of intra-
class semantic graph could be formulated by Adaptive Re-weighting algorithm as
follows:

$\displaystyle\mu\begin{pmatrix}c\end{pmatrix}=\frac{1}{\begin{vmatrix}S\begin{pmatrix}:,:,c\end{pmatrix}\end{vmatrix}}\sum_{i=1}^{h_{f}}\sum_{j=1}^{w_{f}}S\begin{pmatrix}i,j,c\end{pmatrix}F\begin{pmatrix}i,j\end{pmatrix}.$
(1)
$\displaystyle\nu\begin{pmatrix}c\end{pmatrix}=\frac{1}{\begin{vmatrix}S\begin{pmatrix}:,:,c\end{pmatrix}\end{vmatrix}}\sum_{i=1}^{h_{f}}\sum_{j=1}^{w_{f}}\left\\{S\begin{pmatrix}i,j,c\end{pmatrix}F\begin{pmatrix}i,j\end{pmatrix}-\mu\begin{pmatrix}c\end{pmatrix}\right\\}^{2}.$
(2)

where ${\begin{vmatrix}S\begin{pmatrix}:,:,c\end{pmatrix}\end{vmatrix}}$
represents the summation of pixel numbers in each node with the same semantic
class $c$. Obviously, this strategy normalizes each node with
${\begin{vmatrix}S\begin{pmatrix}:,:,c\end{pmatrix}\end{vmatrix}}$ which could
balance the contributions of different semantic classes. Moreover, $\mu(c)$
also could be considered as the mean center of all pixels in the c-th semantic
category. Furthermore, the modulation variance $\nu(c)$ can faithfully react
to the semantic variation of the intra-class feature distribution. Note that
both $\mu$ and $\nu$ are tensors in practice. As depicted in Fig. 3, each
circle in the Graph Nodes Space represents different semantic classes by
distinct colors. Furthermore, we calculate the Cosine similarity between the
synthesized sketch $F$ and each semantic node $\mu$ (and $\nu$) to construct
the intra-class semantic graph. Formally, the computation is listed as
follows:

$\displaystyle\mathbb{C}_{1}=$
$\displaystyle\frac{F\cdot\mu}{\left\|F\right\|_{2}\cdot\left\|\mu\right\|_{2}}$
$\displaystyle\mathbb{C}_{2}=$
$\displaystyle\frac{F\cdot\nu}{\left\|F\right\|_{2}\cdot\left\|\nu\right\|_{2}}.$
(3)

In this way, we construct the intra-class semantic graph for the synthesized
sketch and the target sketch.

####  III-C2 Inter-class Structural Graph Representation

Subsequently, we construct the inter-class structural graphs which could be
defined as:

$\displaystyle\mathbb{G}_{1}=\langle\mu,\mathbb{E}_{1}\rangle$
$\displaystyle\mathbb{G}_{2}=\langle\nu,\mathbb{E}_{2}\rangle$ (4)

where the $\mu$ and $\nu$ represent nodes in the structural graph, and the
edge $\mathbb{E}$ between every two pairs of nodes is used to represent the
relevant information of the set of nodes. As illustrated in Fig. 3, we utilize
the Euclidean Distance to measure relevant information between each paired
nodes which could be calculated as follows:

$\displaystyle\mathbb{E}(c_{1},c_{2})=\mathcal{E}(\mu(c_{1}),\mu(c_{2}))$
$\displaystyle\mathbb{E}(c_{1},c_{2})=\mathcal{E}(\nu(c_{1}),\nu(c_{2}))$ (5)

where the $c\in\begin{bmatrix}1,2,\cdots,12\end{bmatrix}$ denotes different
nodes and $\mathcal{E}$ represents the Euclidean Distance. And we conduct the
constraint of inter-class structural graphs between the synthesized sketches
and target sketches.

Accordingly, we construct two types of representational graphs which engage in
providing the intra-class and inter-class supervisions from the region-wise
semantic perception. These supervisions could enhance the quality of the
detailed regions in the synthesized images.

Figure 4: The Pipeline of our proposed Iterative Training Strategy. We take
the sketch synthesis task as prototype.

###  III-D Iterative Training Strategy

Inspired by the symmetry between the two tasks of sketch synthesis and face
photo synthesis, we propose a biphasic iterative cycle training algorithm
which could improve the perceptual quality of the synthesized results as
illustrated in Fig. 4.

There are multi-stages in our iterative training strategy. We name the
generator and discriminator as $G_{k}^{i}$, $D_{k}^{i}$, $G_{o}^{i}$ and
$D_{o}^{i}$, where $i\in\left\\{0,1,\cdots,n\right\\}$ denote the iterative
stages; $k$, $o$ represent sketch synthesis and face photo synthesis,
sequentially. To begin with, we train the $G_{k}^{0}$ and $D_{k}^{0}$ from
scratch for the sketch synthesis task; $G_{o}^{0}$ and $D_{o}^{0}$ for the
face photo synthesis task, respectively. Afterwards, for face sketch
synthesis, we take the pre-trained $G_{o}^{0}$ as knowledge extractor to boost
the $G_{k}^{1}$ and $D_{k}^{1}$ which are trained from scratch. Concretely, we
take the output fake sketch from $G_{k}^{1}$ as $\widehat{Y}$ that concat with
saliency detection result $M_{k}$ (produced from real sketch) feed to the pre-
trained $G_{o}^{0}$ to acquire the $\widehat{X}$. Meanwhile, we exploit the
corresponding real sketch $Y$ and $M_{k}$ feed to the $G_{o}^{0}$ that acquire
the $\widetilde{X}$. We make the distillation between the $\widehat{X}$ and
$\widetilde{X}$. In addition, the pre-trained $G_{o}^{0}$ is regarded as a
knowledge extractor to obtain higher-level perceptual information. We impose
distillations on multi-scale feature maps to further improve $G_{k}^{1}$ and
$D_{k}^{1}$. Similarly, we utilize the pre-trained $G_{k}^{0}$ to further
boost the $G_{o}^{1}$ and $D_{o}^{1}$ from scratch. Note that this iterative
cycle training strategy could be conducted on multi-stages until we obtain the
optimal models. The detailed procedure of the optimization is presented in
Algorithm 1.

Algorithm 1 The iterative training strategy of sketch synthesis

1:

2:Input photo: $X$; saliency detection map: $M_{o}$; parsing mask: $S_{o}$;

3:Target sketch: $Y$; saliency detection map: $M_{k}$; parsing mask: $S_{k}$;

4:The number of iteration: $i$; max iteration: $T=4$;

5:

6:The optimal models $G_{k}^{i}$ and $D_{k}^{i}$;

7:Step1: Initialize training $G_{k}^{0}$, $D_{k}^{0}$, $G_{o}^{0}$ and
$D_{o}^{0}$ from scratch;

8:Step2: Iterative training;

9:for $i=0$ to T do

10: Feed the $X$, $M_{o}$ and $S_{o}$ to train the $G_{k}^{i+1}$ and
$D_{k}^{i+1}$ which obtain fake sketch $\widehat{Y}$;

11: Feed the $\widehat{Y}$, $M_{k}$ and $S_{k}$ to the pre-trained $G_{o}^{i}$
to obtain corresponding $\widehat{X}$; Extract multi-level feature maps from
$G_{o}^{i}$ as $\widehat{F_{o}}$;

12: Feed the $Y$, $M_{k}$ and $S_{k}$ to the pre-trained $G_{o}^{i}$ to obtain
corresponding $\widetilde{X}$; Extract multi-level feature maps from
$G_{o}^{i}$ as $\widetilde{F_{o}}$;

13: Compute the loss between $\widehat{X}$ and $\widetilde{X}$,
$\widehat{F_{o}}$ and $\widetilde{F_{o}}$; Back-propagate the gradients;

14: Update the $G_{k}^{i+1}$ and $D_{k}^{i+1}$;

15:end for

Figure 5: Ablation studies of synthesized sketches and photos on the CUFS
dataset. From the top to bottom, the examples are selected from XM2VTS
database, CUHK database, and AR database. (a) Source images, (b) Target
Images, (c) CycleGAN, (d) Pix2Pix, (e) w/ $M$ , (f) w/ $M$ \+ $S$, (g) w/ $M$
\+ $S$ \+ Perceptual Loss, (h) w/ $M$ \+ $S$ \+ Perceptual Loss + BCE Loss,
(i) w/ $M$ \+ $S$ \+ Perceptual Loss + BCE Loss + IAG Loss, (j) w/ $M$ \+ $S$
\+ Perceptual Loss + BCE Loss + IAG Loss + ITG Loss, (k) w/ $M$ \+ $S$ \+
Perceptual Loss + BCE Loss + IAG Loss + ITG Loss + ICT Loss. Note that the
_(i)_ column represents the results of our preliminary work SDGAN.

###  III-E Objective Function

The overall objective of our model includes seven loss functions:
$\mathcal{L}_{GAN}$, $\mathcal{L}_{content}$, $\mathcal{L}_{IAG}$,
$\mathcal{L}_{perceptual}$, $\mathcal{L}_{BCE}$, $\mathcal{L}_{ITG}$ and
$\mathcal{L}_{ICT}$. Here, we adopt the sketch synthesis task as prototype to
elaborate the objective functions.

####  III-E1 Adversarial Loss.

The adversarial loss is leveraged to correctly distinguish the real sketches
or generated sketches. Follows the setting of [32], the adversarial loss is
formulated as:

$\displaystyle\mathcal{L}_{GAN}$ $\displaystyle=\mathit{E}_{X,M,Y}\left[\log
D\left(X,M,Y\right)\right]$
$\displaystyle+{E}_{X,M}\left[\log\left(1-D\left(X,M,G\left(X,M\right)\right)\right)\right].$
(6)

where $X$, $Y$ and $M$ denote the source photos, target sketches and saliency
detection maps.

####  III-E2 Content Loss.

In addition, we utilize the normalized $L_{1}$ distance to represent content
loss since it causes less blurring than $L_{2}$ distance.

$\displaystyle\mathcal{L}_{content}\left(G\right)=\mathit{E}_{X,M,Y}\left[\left\|Y-G\left(X,M\right)\right\|_{1}\right].$
(7)

####  III-E3 Perceptual Loss.

In order to ensure that the generated sketch and the target sketch have more
similar specificity, we employ pre-trained VGG-19 net [43] as feature
extractor to obtain high-level representations. We compare the features from
VGG-19 after pool1 and pool2 layers. The perceptual loss engages to make
training procedure more stable.

$\displaystyle\mathcal{L}_{perceptual}=\sum_{l=1}^{2}\left\|\omega^{l}\left(Y\right)-\omega^{l}\left(G\left(X,M\right)\right)\right\|_{2}^{2}.$
(8)

where $\omega^{l}\left(\cdot\right)$ represents the output features of VGG-19
net and $l$ denotes the selected pool1 and pool2 layers.

####  III-E4 Binary Cross-Entropy Parsing Loss.

Moreover, we introduce the Binary Cross-Entropy (BCE) loss to further refine
the synthesized sketch in semantic level. We contrast the semantic mask of
synthesized sketch and target sketch produced by pre-trained parsing network
[42].

$\displaystyle\mathcal{L}_{BCE}=\left(\mathbb{P}\left(Y\right),\mathbb{P}\left(G\left(X,M\right)\right)\right).$
(9)

where $\mathbb{P}$ denotes the inference process of parsing network.

####  III-E5 Intra-class Semantic Graph Loss.

In practice, we extract the intra-class semantic graphs from target sketch and
synthesized sketch, respectively. These semantic graphs contain comprehensive
knowledge between sketch and class-wise graph nodes space which represent the
intra-class feature distribution. Furthermore, in the process of calculating
graph nodes, the contribution of each semantic class is adaptively re-
weighted. Consequently, we reinforce the supervision on these intra-class
semantic graphs to restrain the generated sketch matching the feature
distribution of the target domain. The IntrA-class Graph (IAG) loss is
formulated as:

$\displaystyle\mathcal{L}_{IAG}\left(\mathbb{C}^{target},\mathbb{C}^{fake}\right)$
$\displaystyle=$ (10)
$\displaystyle\sum_{r=1}^{2}\sum_{c=1}^{12}\left\|\mathbb{C}_{r}^{target}\left(c\right)-\mathbb{C}_{r}^{fake}\left(c\right)\right\|_{2}^{2}.$

where the $r\in\begin{bmatrix}1,2\end{bmatrix}$ denotes two types of intra-
class semantic graphs.

TABLE I: Results on face sketch synthesis in the CUFS and the CUFSF datasets.
$\uparrow$ indicates the higher is better, $\downarrow$ indicates the lower is
better. Our method reaches the optimal and _sub-optimal_ results in the CUFS
dataset and the CUFSF dataset.

Model | CycleGAN | Pix2Pix | MDAL | KT | Col-cGAN | KD+ | MSG-SARL | SCAGAN | DP-GAN | DIR-MFP | SDGAN | ours
---|---|---|---|---|---|---|---|---|---|---|---|---
CUFS | LPIPS(alex)$\downarrow$ | 0.2776 | 0.1654 | - | 0.2297 | - | 0.1971 | - | - | - | - | 0.1444 | 0.1432
LPIPS(squeeze)$\downarrow$ | 0.1863 | 0.1156 | - | 0.1688 | - | 0.1471 | - | - | - | - | 0.1017 | 0.0986
LPIPS(vgg-16)$\downarrow$ | 0.3815 | 0.3059 | - | 0.3483 | - | 0.3052 | - | - | - | - | 0.2767 | 0.2646
FSIM$\uparrow$ | 0.6829 | 0.7356 | 0.7275 | 0.7373 | - | 0.7350 | 0.7594 | 0.716 | 0.7345 | 0.7378 | 0.7446 | 0.7494
SSIM$\uparrow$ | 0.4638 | 0.5172 | 0.5280 | - | 0.5244 | - | 0.5288 | - | - | 0.5703 | 0.5360 | 0.5493
FID$\downarrow$ | 58.394 | 44.272 | - | - | - | - | 46.39 | 34.2 | - | - | 33.408 | 33.256
CUFSF | LPIPS(alex)$\downarrow$ | 0.2234 | 0.1932 | - | 0.2522 | - | 0.2368 | - | - | - | - | 0.1906 | 0.1867
LPIPS(squeeze)$\downarrow$ | 0.1617 | 0.1422 | - | 0.1740 | - | 0.1619 | - | - | - | - | 0.1370 | 0.1341
LPIPS(vgg-16)$\downarrow$ | 0.3787 | 0.3551 | - | 0.3743 | - | 0.3550 | - | - | - | - | 0.3358 | 0.3341
FSIM $\uparrow$ | 0.7011 | 0.7284 | 0.7076 | 0.7311 | - | 0.7171 | 0.7316 | 0.729 | 0.7080 | 0.7200 | 0.7328 | 0.7332
SSIM$\uparrow$ | 0.3753 | 0.4204 | 0.3818 | - | 0.4224 | - | 0.4230 | - | - | 0.4215 | 0.4339 | 0.4407
FID$\downarrow$ | 31.262 | 30.984 | - | - | - | - | 38.25 | 18.2 | - | - | 30.594 | 24.577

####  III-E6 Inter-class Structural Graph Loss.

Based on our preliminary work [22], we employ the inter-class structural graph
to facilitate the detailed identity-specific preservation of synthesized
sketches. These inter-class structural graphs consist of nodes ($\mu$ and
$\nu$) for each semantic class and structural information represented by graph
edges. Consequently, we apply the constraints between the InTer-class
structural Graphs (ITG) of synthesized sketches and target sketches named ITG
Loss which could be formulated as:

$\displaystyle\mathcal{L}_{ITG}\left(\mathbb{G}^{target},\mathbb{G}^{fake}\right)$
$\displaystyle=$ (11)
$\displaystyle\sum_{r=1}^{2}\sum_{c=1}^{12}\left\|\mathbb{G}_{r}^{target}\left(c\right)-\mathbb{G}_{r}^{fake}\left(c\right)\right\|_{2}^{2}.$

where the $r\in\begin{bmatrix}1,2\end{bmatrix}$ denotes two types of inter-
class structural graphs.

####  III-E7 Iterative Cycle Training Loss.

Finally, we design the Iterative Cycle Training (ICT) Loss for iterative
training strategy. Four iterations of iterative cycle training are conducted
in the experiments to reach the optimal solution. We express the ICT Loss as:

$\displaystyle\mathcal{L}_{ICT}=\sum_{l=1}^{5}\left\|G_{o}^{l,i}\left(Y\right)-G_{o}^{l,i}\left(G_{k}^{i}\left(X,M\right)\right)\right\|_{1}.$
(12)

where $G_{o}^{l,i}\left(\cdot\right)$ denotes the symmetric network from face
sketch to photo and $i$ represents the times of iteration. We select the
feature maps which could extract multi-level identity-specific information
represented by $l$.

Figure 6: Ablation studies of synthesized sketches and photos on the CUFSF
dataset. (a) Source images, (b) Target Images, (c) CycleGAN, (d) Pix2Pix, (e)
w/ $S$, (f) w/ $S$ \+ Perceptual Loss, (g) w/ $S$ \+ Perceptual Loss + BCE
Loss, (h) w/ $S$ \+ Perceptual Loss + BCE Loss + IAG Loss, (i) w/ $S$ \+
Perceptual Loss + BCE Loss + IAG Loss + ITG Loss, (j) w/ $S$ \+ Perceptual
Loss + BCE Loss + IAG Loss + ITG Loss + ICT Loss. Note that the _(h)_ column
represents the results of our preliminary work SDGAN.

####  III-E8 Full Objective.

Eventually, we combine all loss functions to achieve overall supervision:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{GAN}$
$\displaystyle+\alpha\mathcal{L}_{content}+\lambda\mathcal{L}_{perceptual}$
$\displaystyle+\delta\mathcal{L}_{BCE}+\eta\mathcal{L}_{IAG}$
$\displaystyle+\tau\mathcal{L}_{ITG}+\xi\mathcal{L}_{ICT}.$ (13)

where the $\alpha$, $\lambda$, $\delta$, $\eta$, $\tau$ and $\xi$ are
weighting factors. Furthermore, the generator $G$ and the discriminator $D$
could be optimized by the following formulation:

$\displaystyle\min_{G}\max_{D}\mathcal{L}_{total}$ (14)

##  IV Experiments

In this section, we introduce the implementation details of our method,
firstly. Afterward, we describe the biphasic photo-sketch datasets and
evaluation criterion which is applied in the experiments. Next, the
experimental results are presented from both quantitative and qualitative
perspectives, which show the effectiveness of our proposed method. Finally, we
conduct the ablation study to verify the utility of each module we proposed.

###  IV-A Implementation Details

We train the proposed model from scratch. Both the generator and discriminator
are implemented on the platform Pytorch [44] with a single NVIDIA GeForce
Titan X GPU. We leverage the Adam optimizer with $\beta_{1}=0.5$ and
$\beta_{2}=0.999$. The total training epochs are 200, then the initial
learning rate is set to 0.0002 for the first 100 epochs and decay linearly in
the last 100 epochs. Additionally, we utilize the Instance Normalization [45],
and set the batchsize = 1. Meanwhile, the weighting factors are set as
$\alpha=100$, $\lambda=10$, $\delta=15$, $\eta=100$, $\tau=100$ and $\xi=5$,
respectively.

TABLE II: Results on face photo synthesis in the CUFS and CUFSF datasets. $\uparrow$ indicates the higher is better, $\downarrow$ indicates the lower is better. Our method reaches the optimal and _sub-optimal_ results in the CUFS dataset and the CUFSF dataset. Note that the results of PS2MAN model is directly borrowed from [19]. Model | CycleGAN | Pix2Pix | KT | KD+ | MSG-SARL | SCGAN | PS2MAN | SDGAN | ours
---|---|---|---|---|---|---|---|---|---
CUFS | LPIPS(alex) $\downarrow$ | 0.2898 | 0.1687 | 0.1919 | 0.1717 | - | - | 0.2464 | 0.1674 | 0.1497
LPIPS(squeeze) $\downarrow$ | 0.2509 | 0.1433 | 0.1747 | 0.1474 | - | - | 0.2158 | 0.1370 | 0.1225
LPIPS(vgg-16) $\downarrow$ | 0.4383 | 0.3031 | 0.3208 | 0.2806 | - | - | 0.3254 | 0.2640 | 0.2367
FSIM $\uparrow$ | 0.7270 | 0.7723 | 0.7851 | 0.7819 | 0.7866 | 0.795 | 0.7819 | 0.7845 | 0.8001
SSIM $\uparrow$ | 0.4461 | 0.6086 | - | - | 0.6242 | - | - | 0.6543 | 0.6822
FID $\downarrow$ | 124.540 | 86.996 | - | - | 66.17 | 40.3 | - | 63.937 | 49.925
CUFSF | LPIPS(alex) $\downarrow$ | 0.2271 | 0.2115 | 0.2440 | 0.2322 | - | - | 0.3145 | 0.2011 | 0.1998
LPIPS(squeeze) $\downarrow$ | 0.1725 | 0.1669 | 0.2023 | 0.1791 | - | - | 0.2853 | 0.1581 | 0.1556
LPIPS(vgg-16) $\downarrow$ | 0.3690 | 0.3579 | 0.3758 | 0.3565 | - | - | 0.4237 | 0.3422 | 0.3376
FSIM $\uparrow$ | 0.7544 | 0.7855 | 0.7931 | 0.7789 | 0.7734 | 0.845 | 0.7812 | 0.7902 | 0.7955
SSIM $\uparrow$ | 0.5594 | 0.6194 | - | - | 0.6114 | - | - | 0.6305 | 0.6441
FID $\downarrow$ | 29.584 | 60.286 | - | - | 59.61 | 20.6 | - | 38.776 | 38.372

###  IV-B Datasets and Evaluation Criteria

In this article, we conduct extensive experiments on the CUHK Face Sketch
Dataset (CUFS) [1] and the CUHK Face Sketch FERET Dataset (CUFSF) [46]. In
CUFS dataset, there are 606 faces, of which 188 faces from the Chinese
University of Hong Kong (CUHK) student database [3], 123 faces from the AR
database [47], and 295 faces from the XM2VTS database [48]. For each sample,
there are paired face photo and sketch drawn by the artist in natural lighting
conditions. The CUFSF dataset contains 1194 face photos with paired sketches.
However, all the photos in the CUFSF dataset are acquired under complex
illumination variations as illustrated in Fig. 1 (d). For the CUFS dataset and
CUFSF dataset, we exploit the geometrical alignment strategy between the
photos and sketches, based on the points of two eye centers and the mouth
centers. The geometric landmark coordinates of these points are released.
Concretely, the aligned photo-sketch is cropped to $200\times 250$ following
[49]. Meanwhile, we adopt the reshaping and padding conventions in [20] to
expand the input image size to $256\times 256$.

The experimental performance is measured by multiple metrics. We employ the
Feature Similarity Index Metric (FSIM) [50] to evaluate the feature quality of
synthesized sketches. FSIM is commonly utilized to measure the low-level
similarity between the paired images, which extracts the phase congruence (PC)
and the image gradient magnitude (GM) as features to index the quality.
Consequently, blurring and noise of the generated images are evaluated by
FSIM, ordinarily. In addition, we apply the Structural Similarity Index Metric
(SSIM) [51] to demonstrate the perceptual similarity between synthesized
results and ground-truth images, which follows the visibility of humans.
However, some works point that SSIM tends to favor over-smoothed images and
ignores the texture of the results, which is not completely consistent with
human perception [52]. To this end, we introduce the Learned Perceptual Image
Patch Similarity (LPIPS) [53] combined with SSIM to measure the perceptual
visibility of synthesized results. LPIPS calculates the distance of embedding
features between the generated images and target images. In this paper, LPIPS
is exploited by three classification networks which are SqueezeNet [54],
AlexNet [55], and VGGNet [43].

TABLE III: Comparison between the preliminary work SDGAN and our current network named Ours in the table. All the performances of our current model are BETTER than preliminary work. $\uparrow$ indicates the higher is better, $\downarrow$ indicates the lower is better. Task |  | Sketch
---
Synthesis
| Photo
---
Synthesis
Model | SDGAN | Ours | Gap | SDGAN | Ours | Gap
CUFS |  | LPLPS
---
(alex)$\downarrow$
0.1444 | 0.1432 | 0.0012 | 0.1674 | 0.1497 | 0.0177
| LPIPS
---
(squeeze)$\downarrow$
0.1017 | 0.0986 | 0.0031 | 0.1370 | 0.1225 | 0.0145
| LPIPS
---
(vgg-16)$\downarrow$
0.2767 | 0.2646 | 0.0021 | 0.2640 | 0.2367 | 0.0273
FSIM $\uparrow$ | 0.7446 | 0.7494 | 0.0048 | 0.7845 | 0.8001 | 0.0156
SSIM $\uparrow$ | 0.5360 | 0.5493 | 0.0133 | 0.6543 | 0.6822 | 0.0279
FID $\downarrow$ | 33.408 | 33.256 | 0.152 | 63.937 | 49.925 | 14.012
FVR $\uparrow$ | 86.179 | 87.542 | 1.363 | 79.040 | 81.780 | 2.74
CUFSF |  | LPIPS
---
(alex)$\downarrow$
0.1906 | 0.1867 | 0.0039 | 0.2011 | 0.1998 | 0.0013
| LPIPS
---
(squeeze)$\downarrow$
0.1370 | 0.1341 | 0.0029 | 0.1581 | 0.1556 | 0.0025
| LPIPS
---
(vgg-16)$\downarrow$
0.3358 | 0.3341 | 0.0017 | 0.3422 | 0.3376 | 0.0046
FSIM $\uparrow$ | 0.7328 | 0.7332 | 0.0004 | 0.7902 | 0.7955 | 0.0053
SSIM $\uparrow$ | 0.4339 | 0.4407 | 0.0068 | 0.6305 | 0.6441 | 0.0136
FID $\downarrow$ | 30.594 | 24.577 | 6.017 | 38.776 | 38.372 | 0.404
FVR $\uparrow$ | 86.758 | 87.109 | 0.351 | 62.763 | 63.344 | 0.581
TABLE IV: The ablation study on CUFS dataset. $\uparrow$ indicates the higher
is better, $\downarrow$ indicates the lower is better. Note that our
preliminary work SDGAN is indicated as the last _3rd_ row from up to bottom in
each task.

Model Variations | Criterion
---|---
| Backbone |  | Salicency
---
Mask $M$
| Parsing
---
Layouts $S$
| Perceptual
---
Loss
| BCE
---
Loss
| IAG
---
Loss
| ITG
---
Loss
| ICT
---
Loss
| LPIPS
---
(alex)$\downarrow$
| LPIPS
---
(squeeeze)$\downarrow$
| LPIPS
---
(vgg-16)$\downarrow$
FSIM$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | FVR$\uparrow$
Sketch Synthesis | $\checkmark$ | - | - | - | - | - | - | - | 0.1654 | 0.1156 | 0.3059 | 0.7356 | 0.5172 | 44.272 | 84.252
$\checkmark$ | $\checkmark$ | - | - | - | - | - | - | 0.1581 | 0.1141 | 0.3080 | 0.7376 | 0.5145 | 42.367 | 84.504
$\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | - | - | 0.1570 | 0.1108 | 0.2965 | 0.7411 | 0.5286 | 41.048 | 84.698
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | - | 0.1644 | 0.1123 | 0.2918 | 0.7424 | 0.5248 | 34.854 | 83.893
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | 0.1642 | 0.1138 | 0.2918 | 0.7433 | 0.5311 | 34.638 | 84.892
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | 0.1444 | 0.1017 | 0.2767 | 0.7446 | 0.5360 | 33.408 | 86.179
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 0.1446 | 0.0995 | 0.2727 | 0.7466 | 0.5482 | 33.166 | 86.648
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.1432 | 0.0986 | 0.2646 | 0.7494 | 0.5493 | 33.256 | 87.542
Photo Synthesis | $\checkmark$ | - | - | - | - | - | - | - | 0.1687 | 0.1433 | 0.3031 | 0.7723 | 0.6086 | 86.996 | 73.527
$\checkmark$ | $\checkmark$ | - | - | - | - | - | - | 0.1660 | 0.1449 | 0.3018 | 0.7742 | 0.6093 | 93.204 | 74.046
$\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | - | - | 0.1589 | 0.1341 | 0.2788 | 0.7759 | 0.6266 | 66.176 | 75.260
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | - | 0.1609 | 0.1341 | 0.2799 | 0.7777 | 0.6303 | 64.495 | 76.427
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | 0.1562 | 0.1326 | 0.2760 | 0.7788 | 0.6320 | 65.729 | 76.778
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | 0.1674 | 0.1370 | 0.2640 | 0.7845 | 0.6543 | 63.937 | 79.040
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 0.1571 | 0.1306 | 0.2588 | 0.7892 | 0.6627 | 61.726 | 78.438
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.1497 | 0.1225 | 0.2367 | 0.8001 | 0.6822 | 49.925 | 81.780

TABLE V: The ablation study on CUFSF dataset. $\uparrow$ indicates the higher
is better, $\downarrow$ indicates the lower is better. Note that our
preliminary work SDGAN is indicated as the last _3rd_ row from up to bottom in
each task.

Model Variations | Criterion
---|---
| Backbone |  | Salicency
---
Mask $M$
| Parsing
---
Layouts $S$
| Perceptual
---
Loss
| BCE
---
Loss
| IAG
---
Loss
| ITG
---
Loss
| ICT
---
Loss
| LPIPS
---
(alex)$\downarrow$
| LPIPS
---
(squeeeze)$\downarrow$
| LPIPS
---
(vgg-16)$\downarrow$
FSIM$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | FVR$\uparrow$
Sketch Synthesis | $\checkmark$ | - | - | - | - | - | - | - | 0.1932 | 0.1422 | 0.3551 | 0.7284 | 0.4204 | 30.984 | 86.027
$\checkmark$ | $\checkmark$ | - | - | - | - | - | - | 0.1939 | 0.1429 | 0.3562 | 0.7275 | 0.4193 | 29.765 | 85.772
$\checkmark$ | - | $\checkmark$ | - | - | - | - | - | 0.2037 | 0.1443 | 0.3454 | 0.7290 | 0.4297 | 30.970 | 86.052
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | - | - | - | - | 0.2070 | 0.1423 | 0.3545 | 0.7299 | 0.4270 | 28.639 | 86.099
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | 0.1983 | 0.1420 | 0.3460 | 0.7300 | 0.4282 | 25.723 | 86.512
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | 0.1906 | 0.1370 | 0.3358 | 0.7328 | 0.4339 | 30.594 | 86.758
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 0.1896 | 0.1370 | 0.3349 | 0.7321 | 0.4355 | 28.995 | 87.023
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.1867 | 0.1341 | 0.3341 | 0.7332 | 0.4407 | 24.577 | 87.109
Photo Synthesis | $\checkmark$ | - | - | - | - | - | - | - | 0.2115 | 0.1669 | 0.3579 | 0.7855 | 0.6194 | 60.286 | 62.074
$\checkmark$ | - | $\checkmark$ | - | - | - | - | - | 0.2137 | 0.1653 | 0.3546 | 0.7875 | 0.6243 | 42.517 | 61.322
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | - | - | - | - | 0.2106 | 0.1672 | 0.3537 | 0.7882 | 0.6219 | 51.397 | 62.174
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | - | 0.2091 | 0.1650 | 0.3497 | 0.7890 | 0.6269 | 49.787 | 62.026
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | 0.2011 | 0.1581 | 0.3422 | 0.7902 | 0.6305 | 38.776 | 62.763
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 0.2030 | 0.1593 | 0.3403 | 0.7929 | 0.6426 | 37.352 | 63.052
$\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.1998 | 0.1556 | 0.3376 | 0.7955 | 0.6441 | 38.372 | 63.344

Besides, we adopt the Fréchet Inception Distance (FID) [56] to compute the
Earth-mover distance (EMD) of distributions between the target domain and the
synthesized image domain. Specifically, a pre-trained Inception-v3 network
[57] is raised to measure the 2048-dimension features between the two contrast
domains. FID is widely used in biphasic photo-sketch synthesis tasks and
present high confidence in images realism and quality.

To evaluate the preservation of personal identity characteristics of the
generated images, we introduce the face verification rate (FVR) which is
implemented by Face++ API [58]. Previously, other researchers often use Null-
space Linear Discriminant Analysis (NLDA) to measure identity-specific.
However, NLDA is found to be seriously affected by image texture and
deformations so that it could not make the accurate assessment of identity-
specific characteristics [19]. We utilize the Face++ APIs which the threshold
is set as 73.975@FAR=1e-5 in our identity preservation experiments.

To demonstrate the ascendancy of our method, we adopt several benchmark
approaches (such as CycleGAN [59], Pix2Pix [26], MDAL [16], KT [60], Col-cGAN
[61], KD+ [19], MSG-SARL [18], SCAGAN [20], DP-GAN [62], DIP-MFP [63] and
PS2MAN [64]) for comparison. In addition, to further demonstrate the
effectiveness of the proposed method, we also conduct comparison experiments
between our preliminary work SDGAN [22] and our current model.

###  IV-C Results on face sketch synthesis task

We demonstrate the performance of our method on the face sketch synthesis task
with the CUFS and the CUFSF datasets. The experiment results are shown in
Table. I. Our method achieves the best capability on the indicators of LPIPS
(alex), LPIPS (squeeze), LPIPS (vgg-16), and FID in the CUFS dataset. The
proposed method decreases the previous best FID from 34.2 to 33.256. In
addition, our method realizes the best performance on all indicators in the
CUFSF dataset except FID. Obviously, the LIPIS (alex, squeeze, vgg) are
significantly decreased from the large margin in both datasets which implies
the higher-fidelity and realistic of the sketches generated by our method. On
the other hand, MSG-SARL [18], DIR-MFP [63], and SCAGAN [20] reach better
results in limited indicators including FSIM, SSIM, and FID of different
datasets. These might be caused by the GAN-based approaches tend to produce
more blurring and noise. Additionally, the images generated by our method
contain more texture information introduced by saliency detection map $M$
which is utilized in the CUFS datasets. These may lead to a negative impact on
SSIM which prefers over-smoothed images. (We discard the $M$ in the CUFSF
dataset, the detailed analysis is presented in the section of ablation study.)
However, as a whole, our method has the most balanced performance on the task
of face sketch synthesis. Besides, our method still achieves sub-optimal
results on indicators that lag behind other methods.

As illustrated in the upper three rows of Fig. 5 (k) and upper two rows of
Fig. 6 (j), we randomly sample face sketches synthesized from our network and
other benchmarks. Our network could integrally construct the details of each
sketch with various backgrounds. The sketches we generated have prolific
identify features, especially in facial details.

###  IV-D Results on face photo synthesis task

Compared with face sketch synthesis, there are fewer previous methods focus on
the photo synthesis task. In this experiment, our method obtains the best
performance with LPIPS (alex, squeeze, vgg) and SSIM on both datasets.
According to Table. II, the SCAGAN [20] achieves better results than our
method with FSIM and FID because of its stacking architecture. In contrast,
our network is end-to-end which has stronger practical significance on edge
devices. Moreover, it is more challenging to synthesize photos from the CUFSF
dataset. As shown in Fig. 1 (d), there are illumination variations in the
CUFSF dataset. Meanwhile, all photos are grayscale and sketches are drawn with
deformation. However, our network achieves prodigious performance on the CUFSF
dataset. As depicted in the lower three rows of Fig. 5 (k) and lower two rows
of Fig. 6 (j), the photos synthesized by our method maintain much personal
identity and perspicuity.

###  IV-E Comparison with preliminary work

To evaluate the effectiveness gap between our current model and preliminary
work SDGAN, we conduct the performance comparison from the quantitative and
qualitative perspectives as reported in Table. III.

As for SDAGN, in view of its capability which could supervise the synthesis of
sketches and photos in the intra-class semantic space, the generated sketches
and photos always have abundant detailed features. Nevertheless, SDGAN does
not effectively extract the structured knowledge in the facial images, which
could be represented as the inter-class semantic graph supervision in our
current model. To be specific, as for our current model, we take advantage of
both intra-class and inter-class semantic spaces that allow the synthesized
sketches and photos to be more clearly structured and contain distinctive
features, as illustrated in columns (i-j) of Fig. 5 and columns (h-i) of Fig.
6. Furthermore, the SDGAN is designed for unidirectional photo-sketch
synthesis which neglects the domain consistency of photo domain and sketch
domain in the biphasic task. On the contrary, with the aid of the Iterative
Cycle Training strategy, it could enhance the identity-preserving ability of
our current model which is essential in the image-to-image translation task
with geometrical alignment datasets. As reported in Table. IV and Table. V,
the Face verification Rate (FVR) significantly increased after applying the
Iterative Cycle Training strategy. Besides, as illustrated in Table. III, our
current model performance has been significantly raised compared with
preliminary SDGAN, especially in the photo synthesis task. From a more
intuitive point of view, the sketches and photos synthesized by the current
model give the expression of more fidelity and consistency than SDGAN as
depicted in columns (i-k) of Fig. 5 and column (h-j) of Fig. 6.

TABLE VI: Ablation Study of Biphsic Iterative Training Strategy. $\uparrow$ indicates the higher is better, $\downarrow$ indicates the lower is better.  | Sketch Synthesis | Face Synthesis
---|---|---
Iteration | Initial-0 | Stage-1 | Stage-2 | Stage-3 | Stage-4 | Initial-0 | Stage-1 | Stage-2 | Stage-3 | Stage-4
CUFS | LPIPS(alex) $\downarrow$ | 0.1446 | 0.1447 | 0.1406 | 0.1432 | 0.1483 | 0.1571 | 0.1497 | 0.1563 | 0.1497 | 0.1518
LPIPS(squeeze) $\downarrow$ | 0.0995 | 0.0991 | 0.0988 | 0.0986 | 0.1007 | 0.1306 | 0.1237 | 0.1306 | 0.1225 | 0.1248
LPIPS(vgg-16) $\downarrow$ | 0.2727 | 0.2665 | 0.2650 | 0.2646 | 0.2656 | 0.2588 | 0.2449 | 0.2442 | 0.2367 | 0.2392
FSIM $\uparrow$ | 0.7466 | 0.7468 | 0.7482 | 0.7494 | 0.7474 | 0.7892 | 0.7918 | 0.7973 | 0.8001 | 0.7980
SSIM $\uparrow$ | 0.5482 | 0.5419 | 0.5472 | 0.5493 | 0.5424 | 0.6627 | 0.6704 | 0.6778 | 0.6822 | 0.6767
FID $\downarrow$ | 33.166 | 30.640 | 30.092 | 33.256 | 34.490 | 61.726 | 55.804 | 54.303 | 49.925 | 51.149
FVR $\uparrow$ | 86.648 | 87.382 | 87.523 | 87.542 | 87.359 | 78.438 | 79.922 | 80.423 | 81.790 | 80.249
CUFSF | LPIPS(alex) $\downarrow$ | 0.1896 | 0.1883 | 0.1904 | 0.1867 | 0.1912 | 0.2030 | 0.2040 | 0.2038 | 0.1998 | 0.2011
LPIPS(squeeze) $\downarrow$ | 0.1370 | 0.1361 | 0.1351 | 0.1341 | 0.1352 | 0.1593 | 0.1593 | 0.1585 | 0.1556 | 0.1565
LPIPS(vgg-16) $\downarrow$ | 0.3349 | 0.3372 | 0.3347 | 0.3341 | 0.3377 | 0.3403 | 0.3424 | 0.3419 | 0.3376 | 0.3393
FSIM $\uparrow$ | 0.7321 | 0.7323 | 0.7322 | 0.7332 | 0.7330 | 0.7929 | 0.7940 | 0.7941 | 0.7955 | 0.7947
SSIM $\uparrow$ | 0.4355 | 0.4371 | 0.4384 | 0.4407 | 0.4375 | 0.6426 | 0.6410 | 0.6430 | 0.6441 | 0.6411
FID $\downarrow$ | 28.995 | 28.462 | 25.690 | 24.577 | 26.031 | 37.352 | 46.418 | 43.447 | 38.372 | 40.534
FVR $\uparrow$ | 87.023 | 87.099 | 87.041 | 87.109 | 86.980 | 63.052 | 63.228 | 63.191 | 63.344 | 62.774

###  IV-F Ablation Study

In this section, we conduct extensive ablation experiments to verify the
effectiveness of each module and the loss function we proposed.

####  IV-F1 Saliency Detection Map $M$

As reported in Table. IV, in our method we concatenate the saliency detection
map $M$ with input images to provide structural texture prior knowledge. We
observe that the SSIM is affected after the introduction of M because SSIM is
more sensitive to texture information. However, as a whole, the performance of
the model is improved especially the FSIM. In the CUFSF dataset, we find that
the saliency detection is influenced by the low quality and lighting
variations of the source images which results in the negative impact on the
performance. As shown in Table. V, we discard the M in the following
experiments after the injection of $S$.

####  IV-F2 Parsing Layouts Injection

Furthermore, we adopt the semantic parsing layouts as two modulation
parameters with spatial dimensions injected in the decoder of our network. The
semantic layouts aim at providing a kind of spatial supervision of synthesized
images. As depicted in Fig. 5 (f) and Fig. 6 (e), the details of the
synthesized images are more realistic and vivid in class-region. The
experiment results also show the effectiveness of parsing layouts injection as
shown in Table. IV and Table. V.

####  IV-F3 Perceptual Loss and BCE Loss

In addition, we apply perceptual loss to improve the high-frequency quality of
synthesized images. As displayed in Table. IV and Table. V, the perceptual
loss could significantly reduce the value of FID while increasing FSIM in the
face sketch synthesis task. Besides, the Binary Cross-Entropy loss is
implemented by BiSeNet [42] to refine the structural layouts of synthesized
images. As illustrated in Fig. 5 (h) and Fig. 6 (g), the synthesized sketches
and photos retain more distinct facial contours.

####  IV-F4 Intra-class Semantic Graph Loss

Moreover, we propose a novel IntrA-class semantic Graph (IAG) loss to
restraint the generated images with ground truth images. The IAG loss could
force the synthesized images to hold more semantic intra-class knowledge. In
other words, the details of the sketches and photos we generated are more
similar to ground truth, such as the texture of the hair, the contour of the
ears, the position of the eyebrows and the eyes are more consistent as
represented in Fig. 5 (i) and Fig. 6 (h). More than that, the IAG Loss could
raise multiple indicators considerably such as LPIPS (alex, squeeze, vgg) as
expressed in Table. IV and Table. V.

####  IV-F5 Inter-class Structural Graph Loss

Combined with the IAG loss we proposed, we bring forward a novel InTer-class
structural Graph (ITG) loss which could remarkably enhance the performance of
our network. As reflected in Table. IV and Table. V, the SSIM is increased in
whole experiments especially on the sketch synthesis task in the CUFS dataset.
The ITG loss allows the synthesized sketches to become more obviously
constructed. It could be observed that in Fig. 5 (j) and Fig. 6 (i), the
generated sketches are more vivid and distinct.

####  IV-F6 Iterative Cycle Training Loss

Eventually, we design a novel biphasic Iterative Cycle Training (ICT) loss to
boost the training of sketch and photo synthesis. There are multi-stages in
this training strategy. To examine the comprehensive training procedure, we
express the results in Table. VI. After conducting three times iterations, the
network tends to converge and reach its optimum at the third iteration. As
depicted in Fig. 5 (k) and Fig. 6 (j), the synthesized sketches and photos
contain more personal characteristics. Obviously, the details of the face
images are more meticulous without less blurring and noises such as the
eyelashes, hairstyles, and teeth.

###  IV-G Heterogeneous Face Verification Rate

Meanwhile, we exploit the face verification rate (FVR), deployed by the Face++
API, to report the identity preservation ability of our multi-variation
models, as highlighted in Table. IV and Table. V. Our method is dedicated to
synthesis the face photos and sketches from the heterogeneous domains.
Comprehensively, in the CUFS dataset, the FVR increases distinctly from the
backbone to final model by 3.332 (84.252$\rightarrow$87.542) for sketch
synthesis and 8.253 (73.527$\rightarrow$81.780) for face photo synthesis.
Similarly, the proposed method improves the FVR considerably despite the
deformation between the paired sketch photo in the CUFSF dataset. To be more
specific, the FVR increases significantly after we exploit the intra-class and
inter-class representational graph loss functions, especially in the CUFS
dataset. Although there are deformations between the photos and sketches in
the CUFSF dataset, the proposed graph loss functions improve the FVR by 0.511
( 86.512$\rightarrow$87.023) for sketch synthesis and 1.026
(62.026$\rightarrow$63.052) for photo synthesis. This highlights the
robustness of our proposed graph representation algorithms.

##  V Conclusion

In this paper, we propose a Semantic-Driven Generative Adversarial Network
with Graph Representation Learning for biphasic face photo-sketch synthesis by
utilizing saliency detection and face parsing layouts as prior information. In
particular, parsing layouts are employed to construct two types of
representational graphs that restraint the intra-class and inter-class
features of the synthesized images. These graphs could enforce our network to
generate the considerate face structure and details. In addition, based on the
observation that the symmetry of the biphasic face photo-sketch synthesis, we
design a novel iterative cycle training strategy to refine the high-frequency
quality of the synthesized images. Extension experiments are conducted to
verify the effectiveness of proposed each module. Although the proposed method
achieves state-of-the-art performance, there are still limitations in our
work. The photos generated in the CUFSF dataset are not vivid and realistic
leading to the low face verification rate of the synthesized photos. These
issues are caused by the deformation between photos and sketches in the
dataset. In addition, in terms of few indicators, our method is still not
optimal which imply there are much improvements can be done. In the future, we
will pay more attention to the biphasic face photo-sketch synthesis task.

## References

  * [1] X. Wang and X. Tang, ”Face photo-sketch synthesis and recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 31, no. 11, pp. 1955-1967, 2008.
  * [2] N. Wang, D. Tao, X. Gao, X. Li, and J. Li, ”A comprehensive survey to face hallucination,” _Int. J. Comput. Vis._ , vol. 106, no. 1, pp. 9-30, 2014.
  * [3] X. Tang and X. Wang, ”Face sketch synthesis and recognition,” in _Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV)_ , Oct. 2003, pp. 687-694.
  * [4] M. Zhu and N. Wang, ”A simple and fast method for face sketch synthesis,” in _Proc 16th ACM Int. Conf. Int. Multimedia. Comput. Serv. (ICMCS)_ , Aug. 2016, pp. 168-171.
  * [5] L.Jonathan, E. Shelhamer, and T. Darrell, ”Fully convolutional networks for semantic segmentation,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , 2015, pp. 3431-3440.
  * [6] M. Zhang, N. Wang, Y. Li, and X. Gao, ”Deep latent low-rank representation for face sketch synthesis,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 30, no. 10, pp. 3109-3123, Oct. 2019.
  * [7] M. Zhu, N. Wang, X. Gao, and J. Li, ”Deep graphical feature learning for face sketch synthesis,” in _Proc. 26th Int. Joint Conf. Artif. Intell. (IJCAI)_ , Aug. 2017, pp. 3574-3580.
  * [8] C. Peng, N. Wang, J. Li, and X. Gao, ”Universal Face Photo-Sketch Style Transfer via Multiview Domain Translation,” _IEEE Trans. Image Process._ , vol. 29, pp. 8519-8534, Aug. 2020.
  * [9] N. Wang, D. Tao, X. Gao, X. Li, and J. Li, ”Transductive face sketch-photo synthesis,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 24, no. 9, pp. 1364-1376, Sept. 2013.
  * [10] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, ”A nonlinear approach for face sketch synthesis and recognition,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2005, pp. 1005-1010.
  * [11] S. T. Roweis and L. K. Saul, ”Nonlinear dimensionality reduction by locally linear embedding,” _science._ , vol. 290, no. 5500, pp. 2323-2326, 2000.
  * [12] H. Zhou, Z. Kuang, and K. K. Wong., ”Markov weight fields for face sketch synthesis,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2012, pp. 1091-1097.
  * [13] N. Wang, X. Gao, L. Sun, and J. Li, ”Bayesian face sketch synthesis,” _IEEE Trans. Image Process._ , vol. 26, no. 3, pp. 1264-1274, Jan. 2017.
  * [14] I. Goodfellow _et al._ , “Generative adversarial nets,” in _Proc. Adv. Neural Inf. Process. Syst. (NeurIPS)_ , 2014, pp. 2672-2680.
  * [15] L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, ” End-to-end photo-sketch generation via fully convolutional representation learning,” in _Proc. 5th ACM Int. Conf. Multimedia Retr. (ICMR)_ , Jun. 2015, pp. 627-634.
  * [16] S. Zhang, R. Ji, J. Hu, X. Lu, and X. Li, ”Face sketch synthesis by multidomain adversarial learning,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 30, no. 5, pp. 1419-1428, May. 2019.
  * [17] H. Bi, N. Li, H. Guan, D. Lu, and L. Yang., ”A Multi-Scale Conditional Generative Adversarial Network for Face Sketch Synthesis,” in _Proc. IEEE Int. Conf. Image Process. (ICIP)_ , Sept. 2019, pp. 3876-3880.
  * [18] S. Duan, Z. Chen, Q. M. J. Wu, L. Cai, and D. Lu, ”Multi-Scale Gradients Self-Attention Residual Learning for Face Photo-Sketch Transformation,” _IEEE Trans. Inf. Forensics Security,_ vol. 16, pp. 1218-1230, 2021.
  * [19] M. Zhu, J. Li, N. Wang, and X. Gao, ”Knowledge Distillation for Face Photo-Sketch Synthesis,” _IEEE Trans. Neural Netw. Learn. Syst._ , pp. 1-14, Oct. 2020.
  * [20] J. Yu, X. Xu, F. Gao, S. Shi, M. Wang, D. Tao, and Q. Huang, ”Toward realistic face photo-sketch synthesis via composition-aided GANs,” _IEEE Trans Cybern_ , pp. 1-13, Mar. 2020.
  * [21] X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, and M. Jagersand, ”U2-Net: Going deeper with nested U-structure for salient object detection,” _Pattern Recognit,_ vol. 106, May. 2020.
  * [22] X. Qi, M. Sun, W. Wang, X. Dong, Q. Li, and C. Shan, ”Face Sketch Synthesis via Semantic-Driven Generative Adversarial Network,” in _Proc. IEEE Int. Joint Conf. Biometrics (IJCB)_ , Aug. 2021, pp. 1-8.
  * [23] S. Zhang, R. Ji, J. Hu, Y. Gao, and C. W. Lin, ”Robust Face Sketch Synthesis via Generative Adversarial Fusion of Priors and Parametric Sigmoid,” in _Proc. 27th Int. Joint Conf. Artif. Intell. (IJCAI),_ Jul. 2018, pp. 1163-1169.
  * [24] M. Zhang, R. Wang, X. Gao, J. Li, and D. Tao, ”Dual-transfer face sketch–photo synthesis,” _IEEE Trans. Image Process._ , vol. 28, no. 2, pp. 642-657, Feb. 2019.
  * [25] Y. Lin, K. Fu, S. Ling, J. Wang, and P. Cheng., ”Toward Identity Preserving Face Synthesis between Sketches and Photos Using Deep Feature Injection,” _IEEE Trans Ind. Inform._ , April. 2021.
  * [26] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, ”Image-to-image translation with conditional adversarial networks,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , July. 2017, pp. 1125-1134.
  * [27] Y. Qu, Y. Chen, J. Huang, and Y. Xie, ”Enhanced pix2pix dehazing network,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2019, pp. 8160-8168.
  * [28] X. Wang, H. Yan, C. Huo, J. Yu, and C. Pant, ”Enhancing Pix2Pix for remote sensing image classification,” in _Proc. 24th Int. Conf. Pattern Recognit. (ICPR),_ Aug. 2018, pp. 2332–2336.
  * [29] T. C. Wang, M. Y. Liu, J. Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, ”High-resolution image synthesis and semantic manipulation with conditional gans,’ in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2018. pp. 8798-8807.
  * [30] T. Park, M. Y. Liu, T. C. Wang, and J. Y. Zhu, ”Semantic image synthesis with spatially-adaptive normalization,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2019, pp. 2337-2346.
  * [31] L. A. Gatys, A. S. Ecker, and M. Bethge, ”Image style transfer using convolutional neural networks,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2016, pp. 2414-2423.
  * [32] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky, ”Texture networks: Feed-forward synthesis of textures and stylized images,” in _Proc. 33rd. Int. Conf. Mach. Learn. (ICML),_ Jun. 2016, vol. 1, No. 2, p. 4.
  * [33] D. Ulyanov, A. Vedaldi, and V. Lempitsky, ”Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , July. 2017, pp. 6924-6932.
  * [34] V. Dumoulin, J. Shlens, and M. Kudlur, ”A learned representation for artistic style,” 2016, _arXiv:1610.07629._ [Online]. Available: https://arxiv.org/abs/1610.07629
  * [35] X. Huang and S. Belongie, ”Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV)_ , Oct. 2017, pp. 1501-1510.
  * [36] S. Cao, W. Lu, and Q. Xu, ”Deep Neural Networks for Learning Graph Representations,” in _Proc. 30th AAAI Conf. Artif. Intell_ , vol. 30, no. 1, Feb. 2016.
  * [37] B. Perozzi, R. Al-Rfou, and S. Skiena, ”Deepwalk: Online learning of social representations,” in _Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD)_ , Aug. 2014, pp. 701-710.
  * [38] M. Defferrard, X. Bresson, and P. Van der Gheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in _Proc. Adv. Neural Inf. Process. Syst. (NeurIPS),_ 2016, pp. 3844–3852.
  * [39] M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian, ”Actional-structural graph convolutional networks for skeleton-based action recognition,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2019, pp. 3595-3603.
  * [40] M. Ren, Y. Wang, Z. Sun, and T. Tan, ”Dynamic graph representation for occlusion handling in biometrics,” in _Proc. 34th AAAI Conf. Artif. Intell._ , April. 2020, vol. 34, No. 07, pp. 11940-11947.
  * [41] L. Yang, A. Sain, L. Li, Y. Qi, H. Zhang, and Y. Z. Song, ”S3Net:Graph Representational Network For Sketch Recognition,” in _Proc. IEEE Int. Conf. Multimedia Expo (ICME)_ , July. 2020, pp. 1-6.
  * [42] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, ”Bisenet: Bilateral segmentation network for real-time semantic segmentation,” in _Proc. Eur. Conf. Comput. Vis. (ECCV)_ , Sept. 2018, pp. 325-341.
  * [43] K. Simonyan and A. Zisserman, ”Very deep convolutional networks for large-scale image recognition,” 2014, _arXiv:1409.1556._ [Online]. Available: https://arxiv.org/abs/1409.1556.
  * [44] A. Paszke et al. ”PyTorch: An imperative style, high-performance deep learning library,” in _Proc. Adv. Neural Inf. Process. Syst._ , 2019, pp. 8026–8037.
  * [45] D. Ulyanov, A. Vedaldi, and V. Lempitsky, ”Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , July. 2017, pp. 6924-6932.
  * [46] W.Zhang, X. Wang, and X. Tang, ”Coupled information-theoretic encoding for face photo-sketch recognition,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , June. 2011, pp. 513-520.
  * [47] A. Martinez and R. Benavente, “The AR face database,” Comput. Vis. Center, Barcelona, Spain, CVC Tech. Rep. 24, 1998.
  * [48] K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, “XM2VTSDB: The extended M2VTS database,” in _Proc. Int. Conf. Audio-Video-Based Biometric Person Authentication,_ 1999, pp. 72-77.
  * [49] N. Wang, X. Gao, and J. Li, “Random sampling for fast face sketch synthesis,” _Pattern Recognit._ , vol. 76, pp. 215-227, Apr. 2018.
  * [50] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” _IEEE Trans. Image Process._ , vol. 20, no. 8, pp. 2378-2386, Aug. 2011.
  * [51] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600-612, Apr. 2004.
  * [52] N. Wang, X. Gao, J. Li, B. Song, and Z. Li, “Evaluation on synthesized face sketches,” _Neurocomputing_ , vol. 214, pp. 991-1000, Nov. 2016.
  * [53] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , Jun. 2018, pp. 586-595.
  * [54] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and $<$ 0.5 MB model size,” 2016, _arXiv:1602.07360._ [Online]. Available: https://arxiv.org/abs/1602.07360
  * [55] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in _Proc. Adv. Neural Inf. Process. Syst. (NIPS)_ , 2012, pp. 1097-1105.
  * [56] M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet, “Are GANs created equal? A large-scale study,” 2017, _arXiv:1706.08500,_ [Online]. Available: https://arxiv.org/abs/1711.10337
  * [57] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proc. IEEE/CVF Comput. Vis. Pattern Recognit. (CVPR)_ , June. 2016, pp. 2818-2826.
  * [58] Megvii Inc. _Face++ Research Toolkit._ Accessed: July. 14, 2021. [Online]. Available: http://www.faceplusplus.com
  * [59] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, ”Unpaired image-to-image translation using cycle-consistent adversarial networks,” _Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV)_ , Oct. 2017, pp. 2223-2232.
  * [60] M. Zhu, N. Wang, X. Gao, J. Li, and Z. Li., ”Face photo-sketch synthesis via knowledge transfer,” in _Proc 28th Int. Joint Conf. Artif. Intell. (IJCAI)_ , 2019, pp. 1048-1054.
  * [61] M. Zhu, J. Li, N. Wang, and X. Gao, ”A deep collaborative framework for face photo-sketch synthesis,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 30, no. 10, pp. 3096-3108. Jan. 2019.
  * [62] W. Wan, Y. Yang, and H. J. Lee, ”Generative adversarial learning for detail-preserving face sketch synthesis, _Neurocomputing_ , vol. 438, pp. 107-121, Jan. 2021.
  * [63] D. Zhou, N. Wang, C. Peng, Y. Yu, X. Yang, and X. Gao, ”Towards Multi-domain Face Synthesis via Domain-Invariant Representations and Multi-level Feature Parts,” _IEEE Transactions on Multimedia._ , July. 2021.
  * [64] L. Wang, V. Sindagi, and V. Patel, “High-quality facial photo-sketch synthesis using multi-adversarial networks,” _in Proc. 13th IEEE Int. Conf. Autom. Face Gesture Recognit.,_ May. 2018, pp. 83-90.