# Semantic Encoder Guided Generative Adversarial Face Ultra-Resolution Network

Xiang Wang, Yimin Yang, Qixiang Pang, Xiao Lu, and Yu Liu X. Wang is with the
Dept. of Computer Science, Lakehead University, Thunder Bay, ON, Canada,
e-mail<EMAIL_ADDRESS>Yang is with the Dept. of electrical and
computer engineering, Western University, London, ON, Canada, e-mail:
<EMAIL_ADDRESS>Pang is with the Okanagan College, Kelowna, BC, Canada,
e-mail<EMAIL_ADDRESS>Lu is with the College of Engineering and
Design, Hunan Normal University, Hunan, China, e-mail<EMAIL_ADDRESS>Liu
is with the School of Microelectronics, Tianjin University, Tianjin, China,
e-mail<EMAIL_ADDRESS>Du is with the Dept. of Comp. Sci., Math, Physics &
Statistics, The University of British Columbia Okanagan, BC, Canada, e-mail:
<EMAIL_ADDRESS>
###### Abstract

Face super-resolution is a domain-specific image super-resolution, which aims
to generate High-Resolution (HR) face images from their Low-Resolution (LR)
counterparts. In this paper, we propose a novel face super-resolution method,
namely Semantic Encoder guided Generative Adversarial Face Ultra-Resolution
Network (SEGA-FURN) to ultra-resolve an unaligned tiny LR face image to its HR
counterpart with multiple ultra-upscaling factors (e.g., 4× and 8×). The
proposed network is composed of a novel semantic encoder that has the ability
to capture the embedded semantics to guide adversarial learning and a novel
generator that uses a hierarchical architecture named Residual in Internal
Dense Block (RIDB). Moreover, we propose a joint discriminator which
discriminates both image data and embedded semantics. The joint discriminator
learns the joint probability distribution of the image space and latent space.
We also use a Relativistic average Least Squares loss (RaLS) as the
adversarial loss to alleviate the gradient vanishing problem and enhance the
stability of the training procedure. Extensive experiments on large face
datasets have proved that the proposed method can achieve superior super-
resolution results and significantly outperform other state-of-the-art methods
in both qualitative and quantitative comparisons.

###### Index Terms:

Face Super-Resolution, Semantic Encoder, Embedded Semantics, Adversarial
Learning, Discriminator.

Figure 1: Proposed SEGA-FURN and its components: Semantic Encoder $E$,
Generator $G$, Joint Discriminator $D$ and Feature Extractor $\phi$. For $D$,
ESLDSN represents the Embedded Semantics-Level Discriminative Sub-Net, ILDSN
represents the Image-Level Discriminative Sub-Net, and FCM denotes the Fully
Connected Module. As for the generator $G$, there are three stages: Shallow
Feature Module (SFM), Multi-level Residual Dense Module (MRDM), and Upsampling
Module (UM). $I^{HR}$ and $I^{LR}$ denote HR face images and LR face images
respectively. $I^{SR}$ is SR images from $G$. Furthermore, $E(\cdot)$ denotes
the embedded semantics obtained from $E$. $D(\cdot)$ represents the output
probability of $D$. $\phi(I^{HR})$ and $\phi(I^{SR})$ describe the features
learned by $\phi$.

##  I Introduction

Face Super-Resolution (FSR) has been a promising computer vision topic in
recent years. It is widely applied to the face applications such as face
surveillance [1], identification [2], and medical diagnosis [3]. However,
there are many particular challenges in the research, such as when faces have
complex facial expressions and when most of the facial information has been
lost. To address these problems, many FSR methods have been proposed.

The deep learning-based SR methods enhance the effect of super-resolution [4,
5, 6, 7, 8]. The first deep learning-based SR end-to-end manner model, SRCNN,
was proposed by Dong $et$ $al.$ [4]. To enhance the super-resolution
performance, Zhu $et$ $al.$ [6] presented the deep cascaded bi-network
incorporating face hallucination and dense correspondence field estimation.
Recently, some methods resort to applying facial prior knowledge (e.g., facial
landmarks, component maps, etc.) to super resolution process. By incorporating
the face recovery model and landmark estimation model, Ma $et\ al.$ [7] used
prior landmark maps to explore facial key structures. In [8], Hu $et\ al.$
proposed a 3D facial prior guided FSR method, where it utilizes 3D facial
prior to estimating the facial spatial information. However, a large amount of
prior knowledge is required in these prior knowledge based methods, which can
cost huge computational resources and increase the time of the training
procedure. Thus, they are not suitable for real-time applications.

Another common issue that occurred in the above-mentioned methods is that
because of utilizing reconstruction objective loss $L_{1}$ or $L_{2}$ in the
learning process, the super-resolved image tends to be overly smooth and
incompatible with the HR image perceptually. To address the existing problem,
Generative Adversarial Network (GAN) -based FSR methods have been investigated
[9, 10, 11, 12, 13, 14, 15]. It is proved that by introducing perceptual loss
in the optimization process, these methods are capable of producing a visually
plausible super-resolved image that is faithful to the HR version. As pioneer
GAN-based SR methods, SRGAN [9], ESRGAN [10] composed of a generator and
discriminator incorporates perceptual loss with adversarial loss to produce
realistic images. Yu $et$ $al.$ involved GAN manner into FSR area, and
proposed [12]. Furthermore, based on URDGN, Yu $et\ al.$ [14] proposed a
multiscale network, namely MTDN. Nevertheless, MTDN cannot handle the
situation when the magnification factor is increased largely. The performance
degraded greatly and severe distortions appeared in the results. FSRGAN [11]
and FAN [15] introduced geometry prior knowledge and landmark heatmaps of face
images to the super resolution process. As aforementioned, these prior
knowledge-based methods have computational burdens in the training procedure.

Besides those drawbacks, the aforementioned methods also severely suffer from
the problem of model collapse and training instability, leading to notorious
oversmoothing artifacts [9, 10]. To break through these limitations and
produce photo-realistic super-resolved face images, we propose a novel GAN-
based SR method, namely Semantic Encoder guided Generative Adversarial Face
Ultra-Resolution Network (SEGA-FURN). The overview of the proposed SEGA-FURN
is demonstrated in Fig. 1, where it contains four components: Semantic
Encoder, Joint Discriminator, Generator, and Feature Extractor. The LR image
and HR image, both have two sets of paths. The set of paths shown in red
indicates a real tuple, and symmetrically the set of paths shown in blue
represents a fake tuple. As for the red paths (real tuple), Path 1 shows that
the HR image is sent to the joint discriminator directly and Path 2 shows the
HR image is passed to the semantic encoder, where the semantics extraction is
employed on the HR image to acquire its embedded semantics. And for the blue
paths (fake tuple), the two paths mean two different flows of LR images. For
Path 1, the LR image is passed to the semantic encoder to extract its embedded
semantics. And for Path 2, the LR image is fed into the generator, where the
residual in the internal dense block (RIDB) is put forward as the basic
building unit for the generator to obtain the image features at different
levels and the inside upsampling operation is able to super resolve the LR
image to be the realistic HR image with fine details. There are two sets of
paths entering into the joint discriminator. The red tuple includes the HR
image and its embedded semantics. Similarly, the blue tuple is composed of the
SR image produced by the generator and embedded semantics extracted from the
LR image. The joint discriminator aims to evaluate how an input real tuple is
more authentic compared to a given fake generated tuple. In addition, both SR
and HR images are put into a feature extractor to extract the intermediate
feature maps which are used to calculate the perceptual loss. Such a learning
strategy allows our proposed method to achieve superior SR performance
compared with other state-of-the-art methods. It is noted that the basic idea
of SEGA-FURN is recorded by the first author’s dissertation [16].

The main contributions of our proposed method can be summarized as follows:

1) Our proposed method is able to ultra-resolve an unaligned tiny face image
to a Super-Resolved (SR) face image with multiple upscaling factors (e.g., 4×
and 8×). In addition, our method does not need any prior information or facial
landmark points.

2) We design a semantic encoder to reverse the image information back to the
embedded semantics reflecting facial semantic attributes. The embedded
semantics combined with image data is fed into a joint discriminator. Such
innovation enables the semantic encoder to guide the discriminative process,
which enhances the discriminative ability of the discriminator.

3) We propose the Residual in Internal Dense Block (RIDB) as the basic
architecture for the generator. This innovation provides an effective way to
take advantage of hierarchical features, increasing the feature extraction
capability of the SEGA-FURN. This architecture was partially described in our
previous paper [17]

4) We propose a joint discriminator which is capable of learning the joint
probability constructed by embedded semantics and visual information (HR and
LR images), resulting in a powerful discriminative ability. Furthermore, in
order to remedy the problem of vanishing gradient and improve the model
stability, we make use of RaLS objective loss to optimize the training
process.

The remainder of this paper is organized as follows. In Section 2, we provide
some preliminary information on our proposed method to solve the problem. In
Section 3, we provide a detailed description of the proposed method and its
major components: generator, semantic encoder, joint discriminator, feature
extractor, and loss functions. The novelty of the proposed method is discussed
as well. In Section 4, a large amount of experimental results are presented
and compared with other state-of-the-art methods. Both qualitative and
quantitative analyses are conducted and performance superiority over other
methods are demonstrated. Section 5 concludes this paper.

Figure 2: Red dotted rectangle: The architecture of the Generator. Blue dotted
rectangle: The architecture of the Joint Discriminator. $F_{SF}$ denotes
shallow features, $F_{MDBM}$ denotes the outputs of MDBM, $F_{GF}$ represents
global features, and $F_{MHF}$ represents multiple hierarchical features. K,
n, and s are the kernel size, number of filters, and strides respectively. N
is the number of neurons in a dense layer.

##  II Preliminary

In this section, we first review the Standard Generative Adversarial Network
(SGAN) [18]. Secondly, we analyze the major problems associated with SGAN and
present improved GAN variants that can address current problems. Many GAN-
based SR methods utilize SGAN adversarial objective function as their
optimization strategy, but SGAN may cause unexpected issues resulting in
unstable training procedures and poor super resolution results [19, 20, 21].
In the following paragraphs, we analyze the underlying problems raised by
involving GAN in the SR methods and then explore effective solutions, so as to
improve super resolution performances.

###  II-A Overview of SGAN

SGAN consists of two networks, one of which is the Generator $G$, and the
other is the discriminator $D$. SGAN has been applied in many applications,
such as super-resolution [9], image translation [22] and face aging [23].
Through adversarial learning in the SGAN, the generator and discriminator
compete against each other. Both two networks try to optimize themselves to
solve the adversarial max-min problem. The objective function used in SGAN is:

$\begin{split}V(G,D)=&\max\limits_{d}\min\limits_{g}\mathbb{E}_{x\sim
p_{data}(x)}[logD_{d}(x)]\\\ &+\mathbb{E}_{z\sim
p_{z}(z)}[log(1-D_{d}(G_{g}(z)))]\end{split}$ (1)

where $V(G,D)$ is a binary cross entropy loss which is commonly used in GAN
applications, $G$ is supposed to map random noise $z$ from the prior
distribution $p_{z}(z)$ over real data $x$, and $D$ is the discriminator which
distinguishes whether its input comes from the $G$ or real data distribution
$p_{data}(x_{r})$. The ultimate goal of SGAN is that $D$ and $G$ are capable
of reaching the Nash equilibrium state, in which once SGAN attains Nash
equilibrium, the generator can generate realistic-looking images which fool
the discriminator.

###  II-B Problem Analysis

However, SGAN in [18, 12, 9] encounters the problem of gradient vanishing,
model collapse, and poor quality of the generated images. Several works [21,
19, 24] have proved that the objective function of the SGAN causes vanishing
gradients, resulting in the instability of GAN training. The discriminator of
the SGAN can be expressed as Eqn. 2:

$D(x)=\sigma(C(x))$ (2)

where $x$ denotes either $I^{HR}$ or $I^{SR}$ in this context, $\sigma$
represents the sigmoid function, $C(x)$ is the probability predicted by the
non-transformed discriminator. The restriction of SGAN is that they only
concentrate on increasing the probability that fake samples belong to real
rather than decreasing the probability that real samples belong to real
simultaneously. In SGAN, if the optimal discriminator is reached, it will stop
learning the real data but will only focus on the fake samples. As a result,
the generator cannot receive enough gradient information from real data to
make progress and the authenticity of fake samples will no longer be improved.
To address this problem, several improved GAN variants have been proposed to
find the objective function with smoother and non-vanishing gradients.

###  II-C Relativistic average GAN

RaGAN [19] was proposed as an improved SGAN by drawing up a Relativistic
average Discriminator (RaD). The RaD demonstrates that the SGAN ignores the
relative discriminant information between real samples and fake samples. This
key property is complemented in RaD, which RaD not only improves the
probability that the generated samples are real but also decreases the
possibility that the real samples are real. The RaD can be expressed as:

$D(x_{r},x_{f})=\sigma(C(x_{r})-E_{x_{f}}[C(x_{f})])$ (3)

where $E_{x_{f}}$ denotes the average of the fake samples in one batch
predicted by RaD. As Eqn. 3 shows, the RaD is capable of evaluating the
probability that the real image is more realistic than the fake image, which
addresses the issue of vanishing gradient and improves the stability of GAN.

###  II-D Least Squares GAN

The Least Squares GAN (LSGAN) [21] indicated that the vanishing gradient
problem is mainly caused by the discriminator of SGAN using the sigmoid cross
entropy loss. This work argued that the original discriminator penalizes a
small decision error to update the generator which makes the generated samples
stay on the correct side of the decision boundary, but are still far from
corresponding real samples, leading to a vanishing gradient problem during the
adversarial process. Motivated by this issue, LSGAN proposed the least squares
loss function to penalize large errors coming from fake samples that lie far
away from the decision boundary. The formulation can be expressed as:

$\displaystyle L_{D}^{LSGAN}=$ $\displaystyle\mathbb{E}_{x_{r}\sim
p_{x_{r}}}[(C_{d}(x_{real})-0)^{2}]$ (4) $\displaystyle+\mathbb{E}_{x_{f}\sim
p_{x_{f}}}[(C_{d}(x_{fake})-1)^{2}]$

Thus, the discriminator utilizing least squares loss is capable of providing
sufficient gradients when optimizing the generator, which is able to remedy
the vanishing gradient problem. In our proposed method, we further take
advantage of the Relativistic average Least Squares (RaLS) loss function which
incorporates RaD into least squares loss. Such an optimization strategy can
help our method stabilize the learning process and accelerate the convergence
speed, producing authentic SR images.

Figure 3: Top: Dense Nested Block (DNB) consists of multiple RIDBs. Bottom:
The proposed Residual in Internal Dense Block (RIDB).

##  III Proposed Method

In this section, we present the proposed method SEGA-FURN in detail. First, we
describe the novel architecture of SEGA-FURN through four main parts:
generator, semantic encoder, joint discriminator, and feature extractor. Next,
we introduce the objective loss function RaLS [19] for optimizing the
generator and discriminator respectively. Furthermore, we discuss the benefit
of the perceptual loss used in our method. Finally, we provide the overall
total loss used in SEGA-FURN. The overview of our method is shown in Fig. 1.
The architecture of the discriminator and generator can be seen in Fig. 2.
Moreover, the structures of our proposed DNB and RIDB [17] are presented in
Fig. 3.

###  III-A Generator

We use the Dense Nested Blocks (DNBs) and Residual in Internal Dense Blocks
(RIDBs) that we proposed in GAFH-RIDN [17] as basic units to construct $G$. As
shown at the top of Fig. 2, the proposed generator mainly consists of three
stages: Shallow Feature Module (SFM), Multi-level Dense Block Module (MDBM),
and Upsampling Module (UM). The LR face image $I^{LR}$ is fed into the SFM as
the initial input. In the end, SR face image $I^{SR}$ is obtained from the UM.
The overall super-resolution process can be formulated as:

$I^{SR}=G(I^{LR})$ (5)

Specifically, the MDBM is built up by multiple DNBs formed by several RIDBs.
As shown in Fig. 3, each DNB includes 3 RIDBs cascaded by residual connections
and one scaling layer. The RIDB is able to extract hierarchical features and
address the vanishing-gradient problem, which is the commonly encountered
issue in [9, 10, 25, 12, 26]. Owing to the designed architecture, the feature
maps of each layer are propagated into all succeeding layers, promoting an
effective way to extract hierarchical features and alleviating the gradient
vanishing problem. It is emphasized that Local Residual Learning (LRL) is
introduced to take effective use of the local residual features extracted by
RIDBs. In addition, in order to help the generator fully take advantage of
hierarchical features, we design the Global Residual Learning (GRL) to fuse
the shallow features $F_{SF}$ and global features $F_{GF}$. Overall,
benefiting from the proposed architecture [17], the generator is capable of
exploiting abundant hierarchical features and super-resolving from the LR
space to the HR space.

###  III-B Semantic Encoder

The proposed semantic encoder is designed to extract embedded semantics (as
shown in Fig. 1), which is used to project visual information (HR, LR) back to
the latent space. The GAN-based SR models [9, 10, 12] only exploit visual
information during the discriminative procedure, ignoring the semantic
information reflected by latent representation. The proposed semantic encoder
will complement the missing critical property. Previous GAN’s work [27, 28]
has proved that the semantic representation is beneficial to the
discriminator.

Based on this observation, the proposed semantic encoder is designed to
inversely map the image to the embedded semantics. The most significant
advantage of the new semantic encode is that it can guide the discriminative
process since the embedded semantics obtained from the semantic encoder can
reflect semantic attributes, such as the facial features (shape and gender)
and the spatial relationship between various components of the face (eyes,
mouth). According to the methodology of semantic encoder and ablation studies,
this advance is verified. It can be emphasized that the embedded semantics is
fed into the joint discriminator along with HR and LR images. Thanks to this
property, the semantic encoder can guide the discriminator to optimize,
thereby enhancing its discriminative ability.

In this context, we use two side-by-side pre-trained VGG19 [29] networks as
the semantic encoder to obtain the embedded semantics of the HR face image and
the LR face image from different convolutional layers respectively. These two
side-by-side VGG19 networks have the same structure except for different input
dimensions since it needs to satisfy the different dimensions of the HR and LR
face image respectively.

Figure 4: Qualitative comparison against state-of-the-art methods. The top two
rows: the results of 4× upscaling factor from 32×32 pixels to 256×256 pixels.
The bottom two rows: the results of 8× upsampling factor from 16×16 pixels to
256×256 pixels. From left to right: (a) HR images, (b) LR inputs, (c) Bicubic
interpolation, (d) Results of SRGAN [9], (e) Results of ESRGAN [10], and (f)
Our method.

###  III-C Joint Discriminator

As shown in Fig. 1, the proposed joint discriminator takes the tuple
incorporating both visual information and embedded semantics as the input,
where Embedded Semantics-Level Discriminative Sub-Net (ESLDSN) receives the
input embedded semantics while the image information is sent to Image-Level
Discriminative Sub-Net (ILDSN). Next, through the operation of the Fully
Connected Module (FCM) on a concatenated vector, the final probability is
predicted. Thus, the joint discriminator has the ability to learn the joint
probability distribution of image data ($I^{HR},I^{SR}$) and embedded
semantics ($E(I^{HR}),E(I^{LR})$). There are two sets of paths entering into
the joint discriminator. The set of paths shown in red indicates the real
tuple which consists of the real sample $I^{HR}$ from the dataset and its
embedded semantics $E(I^{HR})$. For the blue paths, a fake tuple is
constructed from SR image $I^{SR}$ generated from the generator and
$E(I^{LR})$ obtained from the LR image through a semantic encoder. As a
result, different from [9, 10, 12, 11], our joint discriminator has the
ability to evaluate the difference between real tuple $(I^{HR},E(I^{HR}))$ and
fake tuple $(I^{SR},E(I^{LR}))$.

Moreover, in order to alleviate the problem of gradient vanishing and enhance
the model stability, we adopt the Relativistic average Least Squares GAN
(RaLSGAN) objective loss for the joint discriminator by applying the RaD to
the least squares loss function [21]. Let’s denote the real tuple by
$X_{real}=(I^{HR},E(I^{HR}))$ and denote the fake tuple by
$X_{fake}=(I^{SR},E(I^{LR}))$. The process that makes joint discriminator
relativistic can be expressed as follows:

$\displaystyle\tilde{C}(X_{real})=(C(X_{real})-E_{x_{f}}[C(X_{fake})])$ (6)
$\displaystyle\tilde{C}(X_{fake})=(C(X_{fake})-E_{x_{r}}[C(X_{real})])$

where $\tilde{C}(\cdot)$ denotes the probability predicted by joint
discriminator, $E_{x_{f}}$ and $E_{x_{r}}$ describe the average of the SR
images (fake) and HR images (real) in a training batch. Moreover, the least
squares loss is used to measure the distance between HR and SR images.
According to Eqn. 7, we optimize the joint discriminator by adversarial loss
$L_{D}^{RaLS}$ and the generator is updated by $L_{G}^{RaLS}$, as in Eqn. 8.

$\displaystyle L_{D}^{RaLS}=$ $\displaystyle\mathbb{E}_{I^{HR}\sim
p_{(I^{HR})}}[(\tilde{C}(X_{real})-1)^{2}]$ (7)
$\displaystyle+\mathbb{E}_{I^{SR}\sim
p_{(I^{SR})}}[(\tilde{C}(X_{fake})+1)^{2}]$ $\displaystyle L_{G}^{RaLS}=$
$\displaystyle\mathbb{E}_{I^{SR}\sim
p_{(I^{SR})}}[(\tilde{C}(X_{fake})-1)^{2}]$ (8)
$\displaystyle+\mathbb{E}_{I^{HR}\sim
p_{(I^{HR})}}[(\tilde{C}(X_{real})+1)^{2}]$

where $I_{HR}\sim P_{I^{HR}}$ and $I_{SR}\sim P_{I^{SR}}$ indicate the HR
images and SR images distribution respectively. In addition, with the help of
least squares loss and relativism in RaLS, SEGA-FURN is remarkably more stable
and generates authentic and visually pleasant SR images.

In a joint discriminator, the ESLDSN takes the embedded semantics as the input
and downsamples it through 6 convolutional layers with 3×3 kernels and the
stride of 1 or 2 alternately, and then reshapes it to a 32-dimensional vector.
The ILDSN receives an image and performs feature extraction through 9 groups
of convolutional layers using the kernel of the size 3×3 followed by the
LeakyReLU [30] and the Batch Normalization (BN) layer to obtain the flattened
64-dimensional vector. Next, the resulting two vectors are concatenated by the
concatenation layer and then fed into the FCM. As for FCM, it contains six
dense layer blocks, where each dense layer block includes a dense layer, a
LeakyReLU activation layer, and a dropout layer except for the last single
dense layer. These six dense layers have 256, 128, 64, 32, 16, and 1 neurons
respectively. Finally, the output is the probability that how the given HR
face image is more realistic than the Super-Resolved (SR) face image.

###  III-D Feature Extractor

We further exploit pre-trained VGG19 [29] network as feature extractor $\phi$
in SEGA-FURN to obtain feature representations used to calculate the
perceptual loss $L_{perceptual}$, where $L_{perceptual}$ is utilized in SEGA-
FURN to eliminate the facial ambiguity and recover missing details of SR
images. Instead of using high-level features as in SRGAN, ESRGAN for
perceptual loss, we adopt low-level features before the activation layer
(i.e., feature representations from ‘Conv3_3’ layer in the feature extractor),
which contains complex edge texture.

###  III-E Loss Function

We involve perceptual loss $L_{perceptual}$ to constrain the intensity and
feature similarities between HR and SR images [31, 32]. Furthermore,
adversarial loss $L_{G}^{RaLS}$ is adopted to super-resolve SR images
containing visually appealing details and faithful to the HR.

Perceptual Loss: $L_{perceptual}$ is able to reduce the gap between the SR
image and the HR image. It is formulated as:

$L_{perceptual}=\frac{1}{WH}\sum_{q=1}^{W}\sum_{r=1}^{H}(\phi_{i,j}(I^{HR})_{q,r}-\phi_{i,j}(I^{SR})_{q,r})^{2}$
(9)

where $W$, $H$ describe the height and width of the feature maps,
$\phi(\cdot)$ denotes the output of feature extractor, $\phi_{i,j}$ indicates
the feature representations obtained from $j$-th convolution layer before
$i$-th maxpooling layer.

Total Loss: The total loss function $L_{perceptual}$ for the generator can be
represented as a weighted combination of two parts: perceptual loss
$L_{perceptual}$ and adversarial loss $L_{G}^{RaLS}$, the formula is described
as follows:

$L_{total}=\lambda_{con}L_{perceptual}+\lambda_{adv}L_{G}^{RaLS}$ (10)

where $\lambda_{con}$, $\lambda_{adv}$ are the trade-off weights for the
$L_{perceptual}$ and the $L_{G}^{RaLS}$. We set $\lambda_{con}$,
$\lambda_{adv}$ empirically to 1 and $10^{-3}$ respectively.

Figure 5: Qualitative comparison of ablation studies. The top two rows: the
results of upscaling factor 4×. The bottom two rows: the results of upscaling
factor 8×. From left to right: (a) HR images, (b) LR inputs, (c) Results of
RIDB-Net, (d) Results of RIDB-RaLS-Net (e) Results of RIDB-SE-Net, and (f)
Results of RIDB-SE-RaLS-Net (SEGA-FURN).

##  IV Experiments

In this section, we first present the details of the dataset and training
implementation. Then, we demonstrate the experiments and evaluation results.
We further compare our method with state-of-the-art methods. Moreover, in
order to prove the effectiveness of SEGA-FURN, we conduct ablation experiments
to verify the contributions of the proposed components.

Please note that we involved experimental results which were demonstrated in
the corresponding published papers. In addition, the quantitative results of
some state-of-the-art methods were missing when the upscaling factor is 4×,
because they did not conduct experiments for upscaling factor 4×, and some
methods did not exploit Structural Similarity (SSIM) as the evaluation
criterion. For the qualitative comparison, we adopted the traditional super-
resolution method, Bicubic interpolation, and the Generative Adversarial
Network (GAN) [18] -based super-resolution methods, SRGAN [9] and ESRGAN [10].
Note that, in order to compare the visual effects fairly, we used their
official published code of the above methods. As for SRGAN and ESRGAN, we
download their models and scripts from the official GitHub
repository1,211footnotetext:
https://github.com/tensorlayer/srgan22footnotetext:
https://github.com/xinntao/ESRGAN respectively. Moreover, the results of
ESRGAN have been confirmed by the author of ESRGAN.

###  IV-A Datasets

We conducted experiments on the public large-scale CelebFaces Attributes
dataset, CelebA [33]. It consists of 200K celebrity face images of 10,177
celebrities. We used a total of 202,599 images, where we randomly selected
162,048 HR face images as the training set, and all the rest 40,511 images
were chosen as the testing set.

###  IV-B Implementation Details

To verify the effectiveness of our method, we conducted experiments with
multiple upscaling factors 4× and 8× respectively. We resized and cropped the
images to 256x256 pixels as our HR face images do not have any alignment
operation. In order to obtain two groups of LR downsampled face images, we
used the bicubic interpolation method with downsampling factor $r$=4 to
produce 64×64 pixels, and factor $r$=8 to produce 32×32 pixels LR images.

We trained our network 30k epochs using the Adam optimizer [34] by setting
$\beta_{1}$=0.9, $\beta_{2}$=0.999 with the learning rate of $10^{-4}$ and
batch size of 8. We alternately updated the generator and discriminator until
the model converges. For the quantitative comparison, we adopted Peak Signal-
to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as the evaluation
metrics.

###  IV-C Comparisons with State-of-the-art Methods

We compare the proposed method with the state-of-the-art SR methods [35, 4,
36, 37, 38, 39, 40, 12, 41, 42, 43, 9, 10, 13, 44, 5, 45] quantitatively and
qualitatively.

TABLE I: Quantitative comparison on CelebA dataset for upscaling factor 4× and 8×, in terms of average PSNR(dB) and SSIM. Numbers in bold are the best evaluation results among state-of-the-art methods. Method |  | CelebA 4×
---
PSNR SSIM
|  | CelebA 8×
---
PSNR SSIM
Bicubic | 26.50 | 0.79 |  | 22.90 | 0.65
Yang $et\ al.$ [35] | - | - |  | 21.35 | 0.60
SRCNN [4] | 28.93 | 0.79 |  | 23.11 | 0.65
Ma $et\ al.$ [36] | - | - |  | 23.12 | 0.64
FAN [15] | - | - |  | 22.96 | 0.69
CSGM-BP [37] | 26.44 | - |  | 22.71 | -
DIP [38] | 27.35 | - |  | 23.45 | -
IAGAN [41] | 27.16 | - |  | 23.49 | -
Yang $et\ al.$ [39] | - | - |  | 23.07 | 0.65
FSRNet [11] |  |  |  | 21.19 | 0.60
FSRGAN [11] |  |  |  | 20.95 | 0.51
CBN [40] | 29.37 | 0.79 |  | 18.77 | 0.54
URDGN [12] | 29.10 | 0.79 |  | 24.82 | 0.70
TDN [42] | - | - |  | 22.66 | 0.66
FSRFCH [43] | - | - |  | 23.14 | 0.82
SRGAN [9] | 26.76 | 0.82 |  | 20.64 | 0.62
ESRGAN [10] | 23.82 | 0.71 |  | 20.32 | 0.57
TDAE [13] | 29.49 | 0.81 |  | 20.40 | 0.57
FaceAttr [44] | 29.78 | 0.82 |  | 21.82 | 0.62
WGAN-GP [20] | 23.28 | 0.69 |  | - | -
VDSR [5] | - | - |  | 19.58 | 0.57
Liu $et\ al.$ [45] | - | - |  | 21.60 | 0.66
Ours | 30.14 | 0.87 |  | 25.21 | 0.73

Qualitative Comparison: The 4× and 8× qualitative results are depicted in Fig.
4. The top two rows show the 4× visual results. As for Bicubic interpolation,
we observe that its results contain over-smooth visualization effects. SRGAN
[9] relatively enhances the SR results compared to Bicubic interpolation, but
it still fails to generate fine details, especially in facial components, such
as eyes, and mouth. It is obvious that ESRGAN [10] produces overly smoothed
visual results and misses specific textures. On the contrary, the 4× SR images
produced by our method retain facial-specific details and are faithful to HR
counterparts.

To reveal the powerful super-resolution ability of our proposed method, we
further conduct experiments with 8× ultra upscaling factor. As shown in the
bottom two rows in Fig. 4, the SR visual quality obtained by Bicubic
interpolation, SRGAN, and ESRGAN is decreased, since the magnification is
increased and the correspondence between HR and LR images becomes
incompatible. The outputs of Bicubic interpolation generate unpleasant noises.
SRGAN encounters a mode collapse problem during the super-resolution process
so it produces severe distortions in SR images. As for ESRGAN, it produces SR
images that show broken textures and noticeable artifacts around facial
components. The other methods all have their respective problems with the 8×
ultra scaling factor. In contrast, our method is capable of producing photo-
realistic SR images which preserve perceptually sharper edges and fine facial
textures.

Quantitative Comparison: The quantitative results with multiple ultra
upscaling factors 4× and 8× are shown in Table I. It is obvious that our
method attains the best in both PSNR and SSIM evaluations, 30.14dB/0.87 for 4×
and 25.21dB/0.73 for 8×, among all methods. FaceAttr [44] is the second best
method for 4×, 29.78dB/0.82, however, it degrades dramatically and performs
poorly when the upscaling factor increases to 8×, obtaining 21.82dB/0.62. In
contrast, our method ranks the first for both upscaling factors 4× and 8×,
which reflects the robustness of the proposed SEGA-FURN. Moreover, it is
notable that our proposed method not only boosts PSNR/SSIM by a large margin
of 1.04dB/0.08 over the classic method URDGN [12] with upscaling factor 4× but
also is higher than URDGN which is the second best for the 8× upscaling. This
result proves the stability of our method with multiple upscaling factors. In
addition, we compare with SRGAN [9] and ESRGAN [10] which also use generative
adversarial structure. It is obvious that our method not only improves SR
image quality from a perceptual aspect but also achieves impressively
numerical results.

TABLE II: Description of SEGA-FURN variants with different components in experiments. Variant | RIDB | SE | RaLS
---|---|---|---
RIDB-Net | $\surd$ |  |
RIDB-RaLS-Net | $\surd$ |  | $\surd$
RIDB-SE-Net | $\surd$ | $\surd$ |
RIDB-SE-RaLS-Net (SEGA-FURN) | $\surd$ | $\surd$ | $\surd$
TABLE III: Quantitative comparison of different variants on CelebA dataset for upscaling factor 4× and 8×. Ablation |  | CelebA 4×
---
PSNR/SSIM
| CelebA 8×
---
PSNR/SSIM
(A) RIDB-Net | 28.64/0.8514 | 24.25/0.7177
(B) RIDB-RaLS-Net | 28.71/0.8526 | 24.37/0.7218
(C) RIDB-SE-Net | 29.60/0.8607 | 24.44/0.7181
(D) RIDB-SE-RaLS-Net | 30.14/0.8682 | 25.21/0.7250

###  IV-D Ablation Study

We further implemented ablation studies to investigate the performance of the
proposed method. As shown in Table II, we list several variants based on
different proposed components. First, among them, RIDB-Net is used as the
baseline variant, which only contains a single component RIDB. Second, the
RIDB-RaLS-Net is constructed by removing the Semantic Encoder (SE) from the
SEGA-FURN. Next, RIDB-SE-Net means to remove RaLS loss of SEGA-FURN, and RIDB-
SE-RaLS-Net equals to SEGA-FURN including all the three components. We provide
the visual results of these variants in Fig. 5, and a quantitative comparison
in Table III.

Effect of RIDB We compare the proposed RIDB with other feature extraction
blocks, such as Residual Block (RB) from SRGAN [9] and Residual in Residual
Dense Block (RRDB) of ESRGAN [10]. As shown in Fig. 4 and Table I, the method
using our generator employing RIDB outperforms SRGAN using RB and ESRGAN using
RRDB in both qualitative and quantitative comparisons. The reason why our
method outperforms the others is analyzed below. Different from RB, our RIDB
introduces a densely connected structure that can combine different levels of
features. In addition, different from RRDB, the proposed RIDB designs multi-
level residual learning within each basic internal dense block, which is able
to boost the flow of features through the generator and provide hierarchical
features for the super-resolution process. Based on these observations and
analysis, the effectiveness of the new generator using the proposed RIDB[17]
is validated.

Effect of SE The Ablation (A) and (C) performed by RIDB-Net and RIDB-SE-Net
aim to illustrate the advantage of SE and also verify the effectiveness of the
joint discriminator. The RIDB-SE-Net can obtain embedded semantics extracted
by SE and further feed these semantics along with image data to the joint
discriminator. In the training process, the embedded semantics is capable of
providing useful semantic information for the joint discriminator. Such
innovation can enhance the discriminative ability of the joint discriminator.
Compared with RIDB-Net which does not employ SE and joint discriminator, the
RIDB-SE-Net achieves significant improvements in terms of quantitative
comparisons. Furthermore, as shown in Fig. 5, there is also a noticeable
refinement in detailed texture. The enhanced performance can verify that the
extracted embedded semantics has a superior impact on SR results and that the
SE along with the joint discriminator plays a critical role in the proposed
method.

Effect of RaLS The ablation (A) and (B) are conducted to demonstrate the
effect of RaLS loss. We replace the RaLS loss of RIDB-Net with the generic GAN
loss, Binary Cross Entropy (BCE), and keep all the other components the same.
As shown in Table III, it is obvious that once we remove the RaLS loss in
RIDB-Net, the quantitative results are lower than RIDB-RaLS-Net which has RaLS
loss. As expected, BCE used in RIDB-Net shows unrefined textures. In contrast,
when RaLS is utilized in variant, the visual results are perceptually pleasant
with more natural textures and edges. Thus, it can demonstrate that the RaLS
loss is capable of greatly improving the performance of super-resolution.

Final Effect From the comparison between ablation (D), and other ablation
variants, it is obvious that the large enhancement is noticeable by
integrating all these three components together (RIDB, Semantic Encoder, and
RaLS loss). Finally, we refer the RIDB-SE-RaLS-Net to SEGA-FURN which is the
ultimate proposed method.

##  V Conclusion

In this paper, we proposed a novel Semantic Encoder guided Generative
Adversarial Face Ultra-resolution Network (SEGAFURN) to super-resolve a tiny
LR unaligned face image to its HR version with multiple large ultra-upscaling
factors (e.g., 4× and 8×). In SEGA-FURN, a novel Semantic Encoder, a novel
Generator using Residual in Internal Dense Block, and a novel Joint
Discriminator adopting RaLS loss, are proposed to solve the existing issues
with other methods. The details of the three major components of the method
are described in this paper, and their performance advantages over other
state-of-the-art methods are analyzed. Extensive experiments demonstrate that
SEGA-FURN is superior to state-of-the-art methods. Photo-realistic SR face
images can be produced using the newly proposed SEGA-FURN method.

## References

  * [1] W. W. Zou and P. C. Yuen, “Very low resolution face recognition problem,” _IEEE Transactions on image processing_ , vol. 21, no. 1, pp. 327–340, 2011\.
  * [2] J. Yang, L. Luo, J. Qian, Y. Tai, F. Zhang, and Y. Xu, “Nuclear norm based matrix regression with applications to face recognition with occlusion and illumination changes,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 1, pp. 156–171, 2016.
  * [3] J. S. Isaac and R. Kulkarni, “Super resolution techniques for medical image processing,” in _2015 International Conference on Technologies for Sustainable Development (ICTSD)_. IEEE, 2015, pp. 1–6.
  * [4] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [5] J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * [6] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin, “Learning face hallucination in the wild,” in _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015, pp. 3871–3877.
  * [7] C. Ma, Z. Jiang, Y. Rao, J. Lu, and J. Zhou, “Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5569–5578.
  * [8] X. Hu, W. Ren, J. LaMaster, X. Cao, X. Li, Z. Li, B. Menze, and W. Liu, “Face super-resolution guided by 3d facial priors,” in _European Conference on Computer Vision_. Springer, 2020, pp. 763–780.
  * [9] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [10] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [11] Y. Chen, Y. Tai, X. Liu, C. Shen, and J. Yang, “Fsrnet: End-to-end learning face super-resolution with facial priors,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2492–2501.
  * [12] X. Yu and F. Porikli, “Ultra-resolving face images by discriminative generative networks,” in _European conference on computer vision_. Springer, 2016, pp. 318–333.
  * [13] ——, “Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3760–3768.
  * [14] X. Yu, F. Porikli, B. Fernando, and R. Hartley, “Hallucinating unaligned face images by multiscale transformative discriminative networks,” _International Journal of Computer Vision_ , vol. 128, no. 2, pp. 500–526, 2020.
  * [15] D. Kim, M. Kim, G. Kwon, and D.-S. Kim, “Progressive face super-resolution via attention to facial landmark,” _arXiv preprint arXiv:1908.08239_ , 2019.
  * [16] X. Wang, “Deep learning based image super resolution,” Ph.D. dissertation, 2021\.
  * [17] X. Wang, Y. Yang, Q. Pang, Q. Tang, and S. Du, “End-to-end generative adversarial face hallucination through residual in internal dense network,” in _29th European Signal Processing Conference (EUSIPCO)_ , 2021.
  * [18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [19] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.
  * [20] Z. Chen and Y. Tong, “Face super-resolution through wasserstein gans,” _arXiv preprint arXiv:1705.02438_ , 2017.
  * [21] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2794–2802.
  * [22] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [23] G. Antipov, M. Baccouche, and J.-L. Dugelay, “Face aging with conditional generative adversarial networks,” in _2017 IEEE international conference on image processing (ICIP)_. IEEE, 2017, pp. 2089–2093.
  * [24] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” _arXiv preprint arXiv:1701.07875_ , 2017.
  * [25] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4799–4807.
  * [26] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2472–2481.
  * [27] J. Donahue, P. Krähenbühl, and T. Darrell, “Adversarial feature learning,” _arXiv preprint arXiv:1605.09782_ , 2016.
  * [28] V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, and A. Courville, “Adversarially learned inference,” _arXiv preprint arXiv:1606.00704_ , 2016.
  * [29] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [30] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” _arXiv preprint arXiv:1505.00853_ , 2015.
  * [31] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [32] A. Dosovitskiy and T. Brox, “Generating images with perceptual similarity metrics based on deep networks,” in _Advances in neural information processing systems_ , 2016, pp. 658–666.
  * [33] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [35] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _IEEE transactions on image processing_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [36] X. Ma, J. Zhang, and C. Qi, “Hallucinating face by position-patch,” _Pattern Recognition_ , vol. 43, no. 6, pp. 2224–2236, 2010.
  * [37] A. Bora, A. Jalal, E. Price, and A. G. Dimakis, “Compressed sensing using generative models,” _arXiv preprint arXiv:1703.03208_ , 2017.
  * [38] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9446–9454.
  * [39] C.-Y. Yang, S. Liu, and M.-H. Yang, “Structured face hallucination,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 1099–1106.
  * [40] S. Zhu, S. Liu, C. C. Loy, and X. Tang, “Deep cascaded bi-network for face hallucination,” in _European conference on computer vision_. Springer, 2016, pp. 614–630.
  * [41] S. Abu Hussein, T. Tirer, and R. Giryes, “Image-adaptive gan based reconstruction,” _arXiv_ , pp. arXiv–1906, 2019.
  * [42] X. Yu and F. Porikli, “Face hallucination with tiny unaligned images by transformative discriminative neural networks,” in _Thirty-First AAAI conference on artificial intelligence_ , 2017.
  * [43] X. Yu, B. Fernando, B. Ghanem, F. Porikli, and R. Hartley, “Face super-resolution guided by facial component heatmaps,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 217–233.
  * [44] X. Yu, B. Fernando, R. Hartley, and F. Porikli, “Super-resolving very low-resolution face images with supplementary attributes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 908–917.
  * [45] C. Liu, H.-Y. Shum, and W. T. Freeman, “Face hallucination: Theory and practice,” _International Journal of Computer Vision_ , vol. 75, no. 1, pp. 115–134, 2007.

|  Xiang Wang received the M.Sc. degree in computer science at Lakehead
University, Thunder Bay, Canada. He worked as a research assistant at the
Computer Vision and Machine Learning Laboratory in the Lakehead University for
the research and design of computer vision and machine learning methods. His
research interests include computer vision, image enhancement, image
generation, and deep learning.
---|---
|  Yimin Yang (S’10-M’13-SM’19) received his Ph.D. degrees in Pattern
Recognition and Intelligent System from the College of Electrical and
Information Engineering, Hunan University, China, in 2013. He is currently an
Assistant Professor at Department of electrical and computer engineering,
Western University and he is also a faculty affiliate at the Vector Institute
for Artificial Intelligence (Canada). From 2014 to 2018, he was a PostDoctoral
Fellow with the Department of Electrical and Computer Engineering at the
University of Windsor (Canada). He has authored or coauthored more than 50
refereed papers. His research interests are artificial neural networks, hybrid
system approximation, and image feature selection. Dr. Yang was the recipient
of the Outstanding Ph.D. Thesis Award of Hunan Province, and the Outstanding
Ph.D. Thesis Award Nominations of Chinese Association of Automation, China, in
2014 and 2015, respectively. He is an Associate Editor for the IEEE
Transactions on Circuits and Systems for Video Technology, and the
Neurocomputing. He has been serving as a Reviewer for many international
journals of his research field and a Program Committee Member of some
international conferences.
---|---
|  Qixiang Pang is currently a professor at the Okanagan College, Canada.
Previously, he worked with Motorola, TELUS, General Dynamics, SED, the
University of British Columbia, and, the Chinese University of Hong Kong. He
received his Ph.D. degree from Beijing University of Posts and
Telecommunications, China. He did his postdoctoral research at the University
of British Columbia, Canada. His research interests lie in Cloud/Edge
Computing, IoT, 6G, and AI. He has been serving on the Technical Program
Committee of many international conferences.
---|---
|  Xiao Lu received the Ph.D. degree in pattern recognition and intelligent
system from the College of Electrical and Information Engineering, Hunan
University, Changsha, China, in 2015. She is currently an Assistant Professor
with the College of Engineering and Design, Hunan Normal University, Changsha,
China. Her current research interests include computer vision, machine
learning, and robotics.
---|---
|  Yu Liu received the B.Eng in electronic engineering from Tianjin
University, Tianjin, China in 1998. During 1998-2000, he was an electronic
engineer of Nantian Electronics Information Corp, Shenzhen, China. In 2000, he
backed to Tianjin University and received M.Eng in information and
communication engineering and the Ph.D. degree in signal and information
processing from the same University in 2002 and 2005 respectively. Currently,
he is a professor with School of Microelectronics, Tianjin University. During
2011-2012, Dr. Liu was a visiting fellow with the Department of Electrical
Engineering, Princeton University, Princeton, NJ, U.S. His research interests
include signal/video processing, medical signal processing, multimedia
systems, compressed sensing, indoor positioning system and machine learning.
---|---
|  Shan Du (S’05-M’09-SM’16) received the Ph.D. degree in Electrical and
Computer Engineering from the University of British Columbia, Vancouver, BC,
Canada in 2009. She is currently an assistant professor with the Department of
Computer Science, Mathematics, Physics & Statistics, the University of British
Columbia (Okanagan Campus). Before joining UBC, she was working as an
assistant professor with the Department of Computer Science, Lakehead
University, Canada and as a Research Scientist/Software Engineer with
IntelliView Technologies Inc., Canada. Shan has more than 15 years research
and development experience on image/video processing, image/video analytics,
pattern recognition, computer vision and machine learning. Shan was recipient
of many awards and grants, including NSERC-IRDF, NSERC-CGS D, AITF Industry
r&D Associates Grant, and ICASSP Best Paper Award. Shan is a senior member of
IEEE, IEEE Signal Processing Society and IEEE Circuits and Systems Society.
She is serving as an Associate Editor of IEEE Trans. on Circuits and Systems
for Video Technology and IEEE Canadian Journal of Electrical and Computer
Engineering, Area Chair of ICIP 2021, and served as Area Chair and Session
Chair of ICIP 2019, TPC member and reviewer for many international journals
and conferences.
---|---