11institutetext: Gilad Cohen 22institutetext: Tel Aviv University, Tel Aviv
6997801, 22email<EMAIL_ADDRESS>33institutetext: Raja Giryes
44institutetext: Tel Aviv University, Tel Aviv 6997801 44email:
<EMAIL_ADDRESS>
# Generative Adversarial Networks

Gilad Cohen and Raja Giryes

###### Abstract

Generative Adversarial Networks (GANs) are very popular frameworks for
generating high-quality data, and are immensely used in both the academia and
industry in many domains. Arguably, their most substantial impact has been in
the area of computer vision, where they achieve state-of-the-art image
generation. This chapter gives an introduction to GANs, by discussing their
principle mechanism and presenting some of their inherent problems during
training and evaluation. We focus on these three issues: (1) mode collapse,
(2) vanishing gradients, and (3) generation of low-quality images. We then
list some architecture-variant and loss-variant GANs that remedy the above
challenges. Lastly, we present two utilization examples of GANs for real-world
applications: Data augmentation and face images generation.

##  1 Introduction to GANs

Generative adversarial networks (GANs) are currently the leading method to
learn a distribution of a given dataset and generate new examples from it. To
begin to understand the concept of GANs, let us consider an interplay between
the police and money counterfeiters. A state just launched its new currency
and millions of bills and coins are widespread all over the country. Recently,
the police detected a flood of counterfeit money in circulation. Further
inspection reveals that the forged bills are lighter than the genuine bills,
and can be easily filtered out. The criminals then find out about the police’s
discovery and use new printers which control better the cash weight. In turn,
the police investigate and discover that the new counterfeit bills have a
different texture near the corners, and remove them from the system. After
many iterations of generating and discriminating forged money, the counterfeit
money becomes almost indistinguishable from the real money. Note that the
system has two agents: 1) The counterfeiters who create ”close to real” money
and 2) The police who detect counterfeit bills adequately.

Back to deep learning: The above two agents are called ”generator” and
”discriminator”, respectively. These two entities are trained jointly, where
the generator learns how to fool the discriminator with new adversarial
examples out of the dataset distribution; and the discriminator learns to
distinguish between real and fake data samples. The GAN architecture is used
in more and more applications since its introduction in 2014. It was proved
successful in many domains such as computer vision Dziugaite et al. (2015);
Karras et al. (2018); Hussein et al. (2020); Ledig et al. (2017), semantic
segmentation Luc et al. (2016); Isola et al. (2016); Wang et al. (2018);
Hoffman et al. (2018), time-series synthesis Brophy et al. (2019); Hartmann et
al. (2018), image editing Rott Shaham et al. (2019); Lample et al. (2017); Gal
et al. (2021b); Abdal et al. (2021); Xia et al. (2021), natural language
processing Fedus et al. (2018); Jetchev et al. (2016); Guo et al. (2018),
text-to-image generation Ramesh et al. (2021); Radford et al. (2021);
Patashnik et al. (2021), and many more. In the next section, we depict the
basic GAN architecture and loss. Later, we will present more sophisticated
architectures, losses, and common usages.

##  2 The basic GAN concept

The first GAN that was introduced by Goodfellow et al. (2014) is depicted in
Fig. 1.

Figure 1: Basic GAN structure. The discriminator and generator are two deep
neural networks trained jointly. The discriminator is trained for the task of
classifying whether an input image is natural (real) or generated (fake),
while the generator is optimized to fool the discriminator.

The architecture of GANs is comprised from two individual components: A
discriminator ($D$) and a generator ($G$). $D$ is trained to distinguish
between real images from the natural distribution and generated images, while
$G$ is trained to craft fake images which fool the discriminator. A random
distribution, $\textbf{z}\sim p_{\textbf{z}}$, is given as input to G. The
purpose of GANs is to learn the generated samples’ distribution,
$G(\textbf{z})\sim p_{g}$ that estimates the real world distribution $p_{r}$.
GANs are optimized by solving the following min-max optimization problem:

$\min_{G}\max_{D}\mathbb{E}_{\textbf{x}\sim
p_{r}}\log[D(\textbf{x})]+\mathbb{E}_{\textbf{z}\sim
p_{\textbf{z}}}\log[1-D(G(\textbf{z}))].$ (1)

On the one hand, $D$ aims at predicting $D(\textbf{x})=1$ for real data
samples and $D(G(\textbf{z}))=0$ for fake samples. On the other hand, the GAN
learns how to fool $D$ by finding $G$ which is optimized on hampering the
second term in Eq. (1).

On the first iteration, only the discriminator weights $\theta_{D}$ are
updated. We sample a minibatch of $m$ noise samples
{$\textbf{z}^{(1)},...,\textbf{z}^{(m)}$} from $p_{\textbf{z}}$ and a
minibatch of $m$ real data examples {$\textbf{x}^{(1)},...,\textbf{x}^{(m)}$}
from $p_{r}$. We then calculate the discriminator’s gradients

$\nabla_{\theta_{D}}\frac{1}{m}\sum_{i=1}^{m}\left[\log
D\left(\textbf{x}^{(i)}\right)+\log\left(1-D\left(G\left(\textbf{z}^{(i)}\right)\right)\right)\right],$

and update the discriminator weights, $\theta_{D}$, by ascending this term. On
the second iteration, only the generator’s weights $\theta_{G}$ are updated.
We sample a minibatch of $m$ noise samples
{$\textbf{z}^{(1)},...,\textbf{z}^{(m)}$} from $p_{\textbf{z}}$, and calculate
the generator’s gradients

$\nabla_{\theta_{G}}\frac{1}{m}\sum_{i=1}^{m}\log\left(1-D\left(G\left(\textbf{z}^{(i)}\right)\right)\right),$
(2)

and update the generator weights, $\theta_{G}$, by descending this term.
Goodfellow et al. (2014) showed that under certain conditions on $D$, $G$, and
the training procedure, the distribution $p_{\textbf{z}}$ converges to
$p_{r}$.

##  3 GAN Advantages and Problems

Since their first introduction in 2014, GANs have attracted a growing interest
all over the academia and industry, thanks to many advantages over other
generative models (mainly Variational Auto-encoders (VAEs) Kingma and Welling
(2013)):

  * •

Sharp images: GANs produce sharper images than other generative models. The
images at the output of the Generator look more natural and with better
quality than images generated using VAEs, which tend to be blurrier.

  * •

Configurable size: The latent random variable size is not restricted,
enriching the generator search space.

  * •

Versatile generator: The GAN framework can support many different generator
networks, unlike other generative models that may have architectural
constraints. VAEs, for example, enforce using a Gaussian at the generator’s
first layer.

The above advantages make GANs very attractive in the deep learning community,
achieving state-of-the-art results in a variety of domains, and generating
very natural images. Yet, the original GAN suffers from three major problems
that are described in detail below. We first present a short summary of them:

  * •

Mode collapse: During the synchronized training of the generator and
discriminator, the generator tends to learn to produce a specific pattern
(mode) which fools the discriminator. Although this pattern minimizes Eq. (1),
the generator does not cover the full distribution of the dataset.

  * •

Vanishing gradients: Very frequently the discriminator is trained ”too well”
to distinguish real images from adversarial images; in this scenario, the
training step of the generator back propagates very low gradients, which does
not help the generator to learn.

  * •

Instability: The model ($\theta_{D}$ or $\theta_{G}$) parameters fluctuate,
and generally not stable during the training. The generator seldom achieves a
point where it outputs very high-quality images.

###  3.1 Mode collapse

Data distributions are multi-modal, meaning that every sample is classified
(usually) to only one label. For example, in MNIST there are ten classes of
digits (or modes) labeled from ’0’ to ’9’. Mode collapse is the phenomenon
where the generator only yields a small subset of the possible modes. In Fig.
2 you can observe generated MNIST images by two different GANs. The top row
shows the training of a ”good” GAN, not suffering from mode collapse. It
generates every kind of mode (digit type) throughout the training. The bottom
row exhibits a GAN training with mode collapse, generating just the digit ’6’.

Figure 2: Example of the mode collapse problem in GANs. The top row shows a
training without mode collapse, where all MNIST modes (digits) are generated.
The bottom row shows a bad training with mode collapse, where the generator
outputs only the digit ’6’. Figure was taken from Metz et al. (2017).

###  3.2 Vanishing gradients

GANs often suffer from training instability, where $D$ performs very well and
$G$ does not get a chance to train a good distribution. We turn to provide
some mathematical observations and understanding of why training a generator
$G$ is extremely hard when the discriminator $D$ is close to optimal.

The global optimality, stated in Goodfellow et al. (2014), is defined when $D$
is optimized for any given $G$. The optimal $D$ is achieved when its
derivative for Eq. (1) equals 0:

$\begin{gathered}\frac{p_{r}(\textbf{x})}{D(\textbf{x})}-\frac{p_{g}(\textbf{x})}{1-D(\textbf{x})}=0,\\\
D^{*}(\textbf{x})=\frac{p_{r}(\textbf{x})}{p_{r}(\textbf{x})+p_{g}(\textbf{x})},\end{gathered}$
(3)

where x is the real input data, $D^{*}(\textbf{x})$ is the optimal
discriminator, and $p_{r}(\textbf{x})$/$p_{g}(\textbf{x})$ is the distribution
of the real/generated data, respectively, over the real data x. If we
substitute the optimal discriminator $D^{*}(\textbf{x})$ into Eq. (1), we can
visualize the loss for the generator $G$:

$\displaystyle\mathbb{L}_{G}=\mathbb{E}_{\textbf{x}\sim
p_{r}}\log\frac{p_{r}(\textbf{x})}{\frac{1}{2}[p_{r}(\textbf{x})+p_{g}(\textbf{x})]}+\mathbb{E}_{\textbf{x}\sim
p_{g}}\log\frac{p_{g}(\textbf{x})}{\frac{1}{2}[p_{r}(\textbf{x})+p_{g}(\textbf{x})]}-2\cdot\log
2.$ (4)

Before we continue any further, we mention two important metrics for
probability measurement. The first one is the Kullback-Leiblar (KL)
divergence:

$KL(p_{1}||p_{2})=\mathbb{E}_{\textbf{x}\sim p_{1}}\log\frac{p_{1}}{p_{2}},$
(5)

which measures how much the distribution $p_{2}$ differs from the distribution
$p_{1}$. Note that this metric is not symmetrical, i.e., $KL(p_{1}||p_{2})\neq
KL(p_{2}||p_{1})$. A symmetrical metric is Jensen-Shannon (JS) divergence,
defined as:

$JS(p_{1}||p_{2})=\frac{1}{2}KL(p_{1}||\frac{p_{1}+p_{2}}{2})+\frac{1}{2}KL(p_{2}||\frac{p_{1}+p_{2}}{2}).$
(6)

Back to our GAN with the optimal $D$, Eq. (4) shows that the GAN loss function
can be reformulated as:

$\mathbb{L}_{G}=2\cdot JS(p_{r}||p_{g})-2\cdot\log 2,$ (7)

which shows that for $D^{*}$, the generator loss turns to be a minimization of
the JS divergence between $p_{r}$ and $p_{g}$. The relation between GAN
training and the JS divergence may explain its instability. To understand
this, view Fig. 3, which shows an example for JS divergences of different
distributions. In Fig. 3(a) we see the real image distribution $p_{r}$, as a
Gaussian with zero mean, and we consider three examples for generated image
distributions: $p_{g1}$, $p_{g2}$, and $p_{g3}$. Fig. 3(b) plots the
$JS(p_{r}(\textbf{x}),p_{q}(\textbf{x}))$ measure between $p_{r}$ and some
$p_{q}$ distribution, where the mean of $p_{q}$ ranges from $0$ to $80$. As
shown in the red box, the gradient of the JS divergence vanishes after a mean
of 30. In other words, when the discriminator is close to optimal
($D^{*}(\textbf{x})$), and we try to train a ”poor” generator with $p_{g}$ far
from the real distribution $p_{r}$, training will not be feasible due to
extremely low gradients. This is prominent especially in the beginning of the
training where $G$ weights are randomized.

Figure 3: Vanishing gradients in GANs. (a) An example of a real image
distribution $p_{r}(\textbf{x})$, a Gaussian with zero mean, with three more
different Gaussian distributions: $p_{g1}$, $p_{g2}$, and $p_{g3}$. (b)
Calculating the JS divergence measure between $p_{r}(\textbf{x})$ and a
Gaussian distribution with mean from 0 to 80. When training with optimal
discriminator ($D^{*}(\textbf{x})$), the generator $G$ minimizes the loss in
Eq. (6), pushing $p_{g}(\textbf{x})$ left towards $p_{r}(\textbf{x})$, alas,
it could take a very long time due to diminished gradients when being far from
$p_{r}(\textbf{x})$.

Due to the vanishing gradient problem, Goodfellow et al. Goodfellow et al.
(2014) proposed a change to the original adversarial loss. Instead of
minimizing $\log(1-D(G(\textbf{z})))$ (as in Eq. (2)), they suggest to
maximize $\log(D(G(\textbf{z})))$. The latter cost function yields the same
fixed point of $D$ and $G$ dynamics but maintains higher gradients early in
the training when the distributions $p_{r}$ and $p_{g}$ are far from each
other. On the other hand, this training strategy promotes mode collapse, as we
show next.

With an optimal discriminator $D^{*}$, $KL(p_{g}||p_{r})$ can be reformulated
as:

$\begin{split}KL(p_{g}||p_{r})&=\mathbb{E}_{\textbf{x}\sim
p_{g}}\log\frac{p_{g}(x)/(p_{r}(x)+p_{g}(x))}{p_{r}(x)/(p_{r}(x)+p_{g}(x)))},\\\
&=\mathbb{E}_{\textbf{x}\sim p_{g}}\log\frac{1-D^{*}(x)}{D^{*}(x)},\\\
&=\mathbb{E}_{\textbf{x}\sim p_{g}}\log[1-D^{*}(x)]-\mathbb{E}_{\textbf{x}\sim
p_{g}}\log[D^{*}(x)].\end{split}$ (8)

If we switch the order of the two sides in Eq. (8), we get:

$\begin{split}-\mathbb{E}_{\textbf{x}\sim
p_{g}}\log[D^{*}(x)]&=KL(p_{g}||p_{r})-\mathbb{E}_{\textbf{x}\sim
p_{g}}\log[1-D^{*}(x)],\\\ &=KL(p_{g}||p_{r})-2\cdot
JS(p_{r}||p_{g})+2\cdot\log 2+\mathbb{E}_{\textbf{x}\sim
p_{r}}\log[D^{*}(x)].\end{split}$ (9)

The alternative loss for $G$ is thus only affected by the first two terms (the
last two terms are constant). Since $JS(p_{r}||p_{g})$ is bounded by $[0,\log
2]$ (see Fig. 3(b)), the loss function is dominated by $KL(p_{g}||p_{r})$,
which is also called the reverse KL divergence. since $KL(p_{g}||p_{r})$
usually does not equal $KL(p_{r}||p_{g})$, the optimized $p_{g}$ by the
reversed KL is totally different than $p_{g}$ optimized by the KL divergence.
Fig. 4 shows the difference of the two optimization where the distribution $p$
is a mixture of two Gaussians, and $q$ is a single Gaussian. When we optimize
for $KL(p_{r}||p_{g})$, $q$ averages all of $p$ modes to hit the mass center
(Fig. 4(a)). However, for the reverse KL divergence optimization, $q$
distribution chooses a single mode (Fig. 4(b)), which will cause a mode
collapse during training.

Figure 4: Optimized distribution $q$ for (a) minimizing KL divergence
$KL(p||q$) and for (b) minimizing reverse KL divergence $KL(q||p)$. Figure is
taken from Wang et al. (2019a).

In summary, using the original $G$ loss from Eq. (1) will result in vanishing
gradients for $G$, and using the other loss in Eq. (9) will result in a mode
collapse. These problems are inherent within the GAN loss and thus cannot be
solved using sophisticated architectures. In Section 5 we discuss other loss
functions for GANs, which solve these problems.

###  3.3 Instability and Image quality

Early GAN models which use the G losses described above
($\log\big{(}1-D(G(\textbf{z}))\big{)}$ and
$-\log\big{(}D(G(\textbf{z}))\big{)}$) exhibit great instability in their cost
values during training. Arjovsky et al. Arjovsky et al. (2017) experimented
with these two $G$ losses and claimed that in both cases the gradients cause
instability to the GAN training. The loss in Eq. (1) stays constant after the
first training steps (Fig. 5(a)), and the loss in Eq. (8) fluctuates during
the entire training (Fig. 5(b)). In both cases, they did not find a
significant correlation between the calculated loss and the generated image
quality. In other words, it is very difficult to predict when during the
training the generator actually produces good quality images, and the only way
to get a good generator is to stop the training and manually visualize many
generated images.

Using the above two G losses in Eq. (1) and Eq. (8) yield poor quality images
compared to modern GAN models. In the following sections we will cover more
sophisticated losses that enhance the generator’s resolution and image size.

Figure 5: Training instability in GANs. (a) $JS$ distance metric in GAN
training using the loss in Eq. (1). This quantity correlates poorly to the
generated image quality, saturating to $\log 2\approx 0.69$, the highest value
taken by the $JS$ distance. (b) Generator loss during training using a
different generator cost (maximizing $\log(D(G(\textbf{z})))$ instead of
minimizing $\log(1-D(G(\textbf{z})))$) showing increasing error without a
significant improvement in image quality. Plots are taken from Arjovsky et al.
(2017)

###  3.4 Problems: summary

The original GAN model and loss proposed by Goodfellow et al. Goodfellow et
al. (2014) suffer from three inherent challenges: (1) Mode collapse; (2)
Vanishing gradients; and (3) Image quality. Follow-up works improve the
performance of GANs on one or more of these problems by using different
architectures for $D$ or $G$, modifying the cost function, and more. A subset
of architecture-variant and loss-variant GANs are portrayed in Fig. 6(a),(b),
respectively. A sample of some recent prominent GAN models are presented in
the following sections.

Figure 6: Recent GAN models which solve the original GAN’s problems: Mode
diversity (collapse), vanishing gradients, and image quality. (a) A subset of
architecture-variant GANs, (b): A subset of loss-variant GANs. Larger axis
values indicate better performance. Red points indicate the model improves all
three challenges, blue points improve two, and black points improve only one.
Figures are taken from Wang et al. (2019b).

##  4 Improved GAN architectures

Many types of new GAN architectures have been proposed since 2014 (Berthelot
et al. (2017); Brock et al. (2019); Denton et al. (2015); Karras et al.
(2018); Radford et al. (2015); Zhang et al. (2019); Karras et al. (2019);
Feigin et al. (2020)). Different GAN architectures were proposed for different
tasks, such as image super-resolution Ledig et al. (2017) and image-to-image
transfer Zhu et al. (2017); Park et al. (2019). In this section we present
some of these models, which improved the performance on image quality,
vanishing gradients, and mode collapse, compared to the original GAN.

###  4.1 Semi-supervised GAN (SGAN)

Semi-supervised learning is a promising research field between supervised
learning and unsupervised learning. In supervised learning, each data is
labeled, and in unsupervised learning, no labels are provided; Semi-supervised
learning has labels only for a small subset of the training data, and no
annotations for the rest of the data, like many real-world problems.

SGAN Odena (2016) extended the original GAN learning to the semi-supervised
context by adding to the discriminator network an additional task of
classifying the image labels (Fig. 7). The generator’s architecture is the
same as before. In SGAN the discriminator utilizes two heads, a softmax and a
sigmoid. The sigmoid is used to distinguish between real and fake images, and
the softmax predicts the images’ labels (only for images predicted as real).
The results on MNIST showed that both $D$ and $G$ in SGAN are improved
compared to the original GAN.

Figure 7: SGAN architecture.

###  4.2 Conditional GAN (CGAN)

CGAN was originally proposed as an extension of the original GAN, where both
the discriminator and generator were fed by an additional class of the image
Mirza and Osindero (2014); Odena et al. (2017). An illustration of CGAN
architecture is shown in Fig. 8. In this setup the loss function in Eq. (1) is
slightly modified to condition both the real images $x$ and the latent
variable $z$ on $y$ (the label):

$\min_{G}\max_{D}\mathbb{E}_{\textbf{x}\sim
p_{r}}\log[D(\textbf{x}|\textbf{y})]+\mathbb{E}_{\textbf{z}\sim
p_{\textbf{z}}}\log[1-D(G(\textbf{z}|\textbf{y}))].$ (10)

All values ($x$, $y$, $z$) are first encoded by some neural layers prior to
their fusion in the discriminator and generator. This improves the ability of
the discriminator to classify real/fake images and enhances the generator’s
ability to control the modalities of the generated images. Mirza and Osindero
(2014) showed that their CGAN architecture used with a language model can
handle also multimodal datasets, such as Flicker that contains labeled image
data with their particular user tags. They demonstrated that their generator
is capable of producing an automatic tagging.

Figure 8: CGAN architecture. Both the generator and discriminator are fed with
a class label $y$ to condition both the images $x$ and latent variable $z$
(see Eq. (10)).

###  4.3 Deep Convolutional GAN (DCGAN)

Before describing DCGAN, we provide a brief reminder of what is a
convolutional neural network. Convolutional neural networks (CNNs) were
proposed by LeCun et al. LeCun et al. (1999); These networks consist of
trained spatial filters applied on hidden activations throughout their
architecture. These networks perform correlations using their trained kernels
with a sliding window over images (or hidden activations). This was shown to
improve the accuracy on many recognition tasks, especially in computer vision.
A very basic and popular CNN network called LeNet is shown in Fig. 9.

Figure 9: LeNet architecture. A popular Convolutional Neural Network (CNN)
used for image recognition. Image was taken from LeCun et al. (1999).

DCGAN proposed to create images using solely deconvolutional networks in their
generator Radford et al. (2015). The generator architecture is depicted in
Fig. 10. Deconvolutional networks can be conceived as CNNs that use the same
components but in reverse, projecting features into the image pixel space.
Zeiler and Fergus (2014) showed that deconvolutional layers achieve good
visualization for CNNs; this allowed the DCGAN generator to create high-
resolution images for the first time.

In addition to improved resolution, DCGAN showed better stability during the
training thanks to multiple modifications in the original GAN network:

  * •

All pooling layers were replaced. In the discriminator, they used convolution
kernels with stride ¿ 1, and the generator utilized fractional-strided
convolution to increase the spatial size.

  * •

Both the discriminator and generator were trained with batch normalization
which promotes similar statistics for real images and fake generated images.

  * •

The discriminator architecture replaces all normal ReLU activations with
Leaky-ReLU Maas (2013); this activation multiplies also the negative kernel
output by a small value ($0.2$), to prevent from ”dead” gradients to propagate
to the generator. The generator architecture used ReLU after every
deconvolution layer except the output, which uses Tanh activation.

The authors demonstrated empirically that the mere CNN architecture used in
DCGAN in not the key contributing factor for the GAN’s performance, and the
above modifications are crucial. To show that they measured GANs quality by
considering them as feature extractors on supervised datasets, and evaluating
the performance of linear models trained on these features (for more
information see Radford et al. (2015)). DCGAN yields a classification error of
22.48% on the StreetView House Numbers dataset (SVHN) Netzer et al. (2011),
whereas a purely supervised CNN with the same architecture achieved a
significantly higher 28.87% test error.

Figure 10: DCGAN generator architecture. This generator creates a complex
$64\times 64$ pixel image from a 100 dimensional uniform distribution $z$. No
fully connected or pooling layers are used. Figure is taken from Radford et
al. (2015)

###  4.4 Progressive GAN (PROGAN)

PROGAN described a novel training methodology for GANs, involving progressive
steps toward the development of the entire network architecture Karras et al.
(2018). This progressive architecture uses the idea of progressive neural
networks first proposed by Rusu et al. Rusu et al. (2016). These architectures
are immune to forgetting and can leverage prior knowledge via lateral
connections to previously learned features. The progressive training scheme is
shown in Fig. 11. First, they trained low resolution $4x4$ pixel images. Next,
both $D$ and $G$ grow to include a layer of $8x8$ spatial resolution. This
progressive training gradually adds more and more intermediate layers to
enhance the resolution, until reaching a high resolution of $1024x1024$ pixel
images with the CelebA dataset. All previous layers remain trainable in later
steps.

Many state-of-the-art GAN architectures utilize this type of progressive
training scheme, and it has resulted in very credible images Karras et al.
(2018, 2019); Brock et al. (2019); Karras et al. (2020b); Richardson et al.
(2021), and more stable learning for both $D$ and $G$.

Figure 11: Training process for the PROGAN progressive methodology. Training
starts with both $G$ and $D$ having a low spatial resolution of $4\times 4$
pixels. Every training step adds a new incremented intermediate layer to $D$
and $G$, enhancing the generated images’ resolution. All existing layers are
trainable throughout the process. The images on the right are examples of
generated images using PROGAN at $1024\times 1024$. Figure was taken from
Karras et al. (2018)

###  4.5 BigGAN

Attention in computer vision is a method that focuses the task on an
”interesting” region in the image. The self-attention is used to train the
network (usually CNN) in an unsupervised manner, teaching it to segment or
localize the most relevant pixels (or activations) for the vision task.

BigGAN Brock et al. (2019) has achieved state-of-the-art generation on the
ImageNet datasets. Its architecture is based on the Self-attention GAN (SAGAN)
Zhang et al. (2019), which employs a self-attention mechanism in both $D$ and
$G$, to capture a large receptive field without sacrificing computational
efficiency for CNNs Vaswani et al. (2017). The original SAGAN architecture can
learn global semantics and long-range dependencies for images, thus generating
excellent multi-label images based on the ImageNet datasets ($128\times 128$
pixels). BigGAN achieved improved performance by scaling up the GAN training:
Increasing the number of network parameters ($\times 4$) and increasing the
batch size ($\times 8$). They achieved better performance on ImageNet with
size $128\times 128$ and were also able to train BigGAN also on the
resolutions $256\times 256$ and $512\times 512$.

Unlike many previous GAN models, which randomized the latent variable from
either $z\sim\mathcal{N}(0,\,1)$ or $z\sim\mathcal{U}(-1,\,1)$, BigGAN uses a
simple truncation trick. During training $z$ is sampled from
$\mathcal{N}(0,\,1)$, but for generating images in inference $z$ is selected
from a truncated normal, where values that lie outside the range are re-
sampled until falling in the range. This truncation trick shows improvement in
individual sample quality at the cost of a reduction in overall sample
variety, as shown in Fig. 12(a). Notice though that using this different
inference sampling trick as is may not generate good images with some large
models, producing some saturation artifacts (Fig. 12(b)). To solve this issue,
the authors proposed to use Orthogonal Regularization to force $G$ to be more
amenable to truncation, making it smoother so that the full lateral space will
map to good generated images.

Figure 12: Using truncated sampling from the latent space. (a) Trade-off
between image quality and image variety. From left to right, the range
threshold is set to $2$, $1$, $0.5$, and $0.04$. (b) Saturation artifacts from
applying truncated normal distribution to a model training with
$z\sim\mathcal{N}(0,\,1)$. Images were taken from Brock et al. (2019).

###  4.6 StyleGAN

StyleGAN Karras et al. (2019) proposed an alternative generator architecture
for GANs. Unlike the traditional $G$ architecture that samples a random latent
variable at its input, their architecture starts from a learned constant input
and adjust the characteristics of the images in every convolutional layer
along $G$ using different, learned latent variables. Additionally, they inject
learned noise directly throughout $G$ (see Fig. 13).

Figure 13: Comparison between a traditional generator (left) to the StyleGAN
generator (right). Unlike the standard generator which selects a random latent
variable only in its input, in StyleGAN generator the latent input is mapped
to an intermediate latent space $\mathcal{W}$, which then controls the
generator through adaptive instance normalization (AdaIN). Also, Gaussian
noise is injected after every convolution layer. ”A” corresponds to a learned
affine transform, and ”B” applies learned scaling factors to the input noise.
Figure is taken from Karras et al. (2019).

This architectural change leads to automatic, unsupervised separation of high-
level attributes (e.g., pose and identity) from low-level variations (e.g.,
freckles, hair) in the generated images. StyleGAN did not modify the
discriminator or the loss function. They used the same $D$ architecture as in
Karras et al. (2018). In addition to state-of-the-art quality for face image
generation, StyleGAN demonstrates a higher degree of latent space
disentanglement, presenting more linear representations of different factors
of variation, turning the GAN synthesis to be much more controllable. Recently
a more advanced styleGAN architecture has been proposed Karras et al. (2020b);
Gal et al. (2021a); Karras et al. (2021). Some generated examples are
presented in Fig. 14. One may also use StyleGAN for image editing by
calculating the latent vector of a given input image in the styleGAN (this
operation is known as styleGAN inversion) and then manipulating this vector
for editing the image Xia et al. (2021); Richardson et al. (2021); Abdal et
al. (2019); Tov et al. (2021); Abdal et al. (2020); Shen et al. (2020); Shen
and Zhou (2021); Patashnik et al. (2021).

Figure 14: Selected face images generated using the StyleGAN2 architecture (an
improved version of the original StyleGAN), trained on the FFHQ dataset.
Figure from Karras et al. (2020b).

##  5 Improved GAN objectives

As described in Section 3.2, the original GAN’s min-max loss (Eq. (1))
promotes mode collapse and vanishing gradient phenomena. This section presents
selected loss functions and regularizations that remedy these problems and
also improves the image quality. This is just a partial list and other loss
functions and regularization methods exist such as the least square GANs Mao
et al. (2017) or optimal transport models Peyre and Cuturi (2019).

###  5.1 Wasserstein GAN (WGAN)

WGAN Arjovsky et al. (2017) has solved the vanishing gradient and mode
collapse problems of the original GAN by replacing the cost in Eq. (1) with
the Earth Mover (EM) distance, which is known also as the Wasserstein distance
and is defined as:

$W(p_{r},p_{g})=\inf_{\gamma\in\prod(p_{r},p_{g})}\mathbb{E}_{(x,y)\sim\gamma}[||x-y||],$
(11)

where $\prod(p_{r},p_{g})$ denotes the set of all joint distributions
$\gamma(x,y)$ whose marginals are $p_{r}$ and $p_{g}$, respectively. The EM
distance is therefore the minimum cost of transporting ”mass” in converting
distribution $p_{r}$ into the distribution $p_{g}$.

Figure 15: Differentiating two Gaussian distributions using optimal
discriminator $D^{*}$. The original GAN cost saturates (red line), resulting
in vanishing gradients, whereas WGAN cost objective (blue line) yields
measurable gradients. Plot was taken from Arjovsky et al. (2017).

Unlike KL and JS distances, EM is capable of indicating distance even when the
$p_{r}$ and $p_{g}$ distributions are far from each other; EM is also
continuous and thus provides useful gradients for training $G$. However, the
infimum in Eq. (11) is highly intractable, so the authors estimate the EM cost
with:

$\max_{w\sim\mathcal{W}}\mathbb{E}_{\textbf{x}\sim
p_{r}}[f_{w}(\textbf{x})]-\mathbb{E}_{\textbf{z}\sim
p_{z}}[f_{w}(G(\textbf{z}))],$ (12)

where $\\{f_{w}\\}_{w\in\mathcal{W}}$ is a parameterized family of all
functions that are $K$-Lipschitz for some $K$ ($||f||_{L}\leq K$). Readers are
referred to Arjovsky et al. (2017) for more details.

The authors proposed to find the best function $f_{w}$ that maximize Eq. (12)
by back-propagating $\mathbb{E}_{\textbf{z}\sim
p_{z}}\big{[}\nabla_{\theta}f\big{(}g_{\theta}(z)\big{)}\big{]}$, where
$g_{\theta}$ are the generator’s weights. $f_{w}$ can be realized by $D$ but
constrained to be K-Lipschitz and $z$ is the lateral input noise for $G$. $w$
in $f_{w}$ are the discriminator’s parameters, and the objective of $D$ is to
maximize Eq. (12), which approximates the EM distance. When $D$ is optimized,
Eq. (12) becomes the EM distance, and $G$ is optimized to minimize it:

$-\min_{G}\mathbb{E}_{\textbf{z}\sim p_{z}}[f_{w}(G(\textbf{z}))]$ (13)

Fig. 15 compares the gradient of WGAN to the original GAN from two non
overlapping Gaussian distributions. It can be observed that WGAN has a smooth
and measurable gradient everywhere (blue line), and learns better even when
$G$ is not producing good images.

###  5.2 Self Supervised GAN (SSGAN)

We showed in Section 4.2 that CGANs can generate natural images. However, they
require labeled images to do so, which is a major drawback. SSGANs Chen et al.
(2019) exploit two popular unsupervised learning techniques, adversarial
training, and self-supervision, bridging the gap between conditional and
unconditional GANs.

Neural networks have been shown to forget previous learned tasks French
(1999); Kirkpatrick et al. (2017), and catastrophic forgetting was previously
considered as a major cause for GAN training instability. Motivated by the
desire to counter the discriminator forgetting, SSGAN adds to the
discriminator an additional loss, which enables it to learn useful
representations, independently of the quality of the generator. In a self-
supervised manner, the authors train a model on a task of prediction a
rotation angle ($[$$,$$,$$,$$]$), as shown in Fig. 16. The objectives of $D$
and $G$ are updated to:

$\begin{split}L_{G}&=-V(G,D)-\alpha\mathbb{E}_{\textbf{x}\sim
P_{G}}\mathbb{E}_{r\sim\mathbb{R}}[\log Q_{D}(R=r|\textbf{x}^{r})]\\\
L_{D}&=V(G,D)-\beta\mathbb{E}_{\textbf{x}\sim
P_{data}}\mathbb{E}_{r\sim\mathbb{R}}[\log
Q_{D}(R=r|\textbf{x}^{r})],\end{split}$ (14)

where $V(G,D)$ is the original GAN objective in Eq. (1), $P_{data}$ and
$P_{G}$ are the real data and generated data distributions, respectively,
$r\in R$ is a rotation selected from a set of all allowed angles
($R=\\{$$,$$,$$,$$\\}$). An image x rotated by $r$ degrees is denoted as
$\textbf{x}^{r}$, and $Q(R|\textbf{x}^{r})$ is the discriminator’s predictive
distribution over the angles of rotation of the sample. These new losses
enforce $D$ to learn good representation via learning the rotation information
in a self-supervised approach.

Using the above scheme, SSGAN achieves good high-quality images, matching the
performance of conditional GANs without having access to labeled data.

Figure 16: Self-Supervised GAN (SSGAN) discriminator. The discriminator, $D$,
performs two tasks: Identifying real/fake images (as the original GAN) and
rotation classification. Both the real and fake images are rotated by
($[$$,$$,$$,$$]$), and sent to the rotation degree classifier (in blue). Only
the original $None$ images are sent to the real/fake classifier (in red). Plot
is taken from Chen et al. (2019).

###  5.3 Spectral Normalization GAN (SNGAN)

SNGAN Miyato et al. (2018) proposes to add weight normalization to stabilize
the training of the discriminator. Their technique is computationally
inexpensive and can be applied easily to existing GANs architectures. In
previous works that stabilized GAN training (Gulrajani et al. (2017); Arjovsky
et al. (2017); Qi (2019)) it was emphasized that $D$ should be a K-Lipshitz
continuous function, forcing it not to change rapidly. This characteristic of
$D$ stabilizes the training of GANs. SNGAN controls the Lipschitz constant of
$D$ by literally constraining the spectral norm of each layer, normalizing
each weight matrix $W$ so it satisfies the spectral constraint
$\sigma(\textbf{W})=1$ (i.e., the largest singular value of the weight matrix
of each layer is 1). This is performed by simply normalizing each layer:

$\bar{\textbf{W}}_{SN}(\textbf{W})=\frac{\textbf{W}}{\sigma({\textbf{W}})},$
(15)

where W are the weight parameters of each layer in $D$. This paper proves that
this will make the Lipschitz constant of the discriminator function to be
bound by $1$, which is important for the WGAN optimization.

SNGAN achieves an extraordinary advance on ImageNet, and better or equal
quality on CIFAR-10 and STL-10, compared to the previous training
stabilization techniques that include weight clipping Arjovsky et al. (2017),
gradient penalty Wu et al. (2019); Mescheder et al. (2018), batch
normalization Ioffe and Szegedy (2015), weight normalization Salimans and
Kingma (2016), layer normalization Ba et al. (2016), and orthonormal
regularization Brock et al. (2017).

###  5.4 SphereGAN

SphereGAN Park and Kwon (2019) is a novel integral probability metric
(IPM)-based GAN, which uses the hypersphere to bound IPMs in the objective
function, thus enhancing the stability of the training. By exploiting the
information of higher-order statistics of data using geometric moment
matching, they achieved more accurate results. The objective function of
SphereGAN is defined as

$\min_{G}\max_{D}\sum_{r}E_{x}[d^{r}_{s}(\textbf{N},D(x))]-\sum_{r}E_{z}[d^{r}_{s}(\textbf{N},D(G(z)))],$
(16)

for $r=1,\dots,R$ where the function $d^{r}_{s}$ measures the $r$-th moment
distance between each sample and the north pole of the hypersphere, N. Note
that the subscript $s$ indicates that $d^{r}_{s}$ is defined on
$\mathbb{S}^{n}$. Fig. 17 shows the pipeline of SphereGAN. By defining IPMs on
the hypersphere, SphereGAN can alleviate several constraints that should be
imposed on $D$ for stable training, such as the Lipschitz constraints required
from conventional discriminators based on the Wasserstein distance.

Figure 17: Pipeline of Sphere GAN. The generator creates fake data from noise
inputs. Real and fake data are fed to the discriminator, which maps the output
to an $n$-dimensional Euclidean feature space (yellow plane). Green and purple
points on the feature plane correspond to fake and real samples, respectively.
The key idea of SphereGAN is remapping the feature points into the
$n$-dimensional hypersphere by using geometric transformations. These mapped
points are then used to calculate the geometric moments centered at the north
pole of the hypersphere ($N$). While the discriminator tries to maximize the
moment differences of real and fake samples, the generator tries to interfere
with the discriminator by minimizing those moment differences. Figure was
taken from Park and Kwon (2019).

Unlike conventional approaches that use the Wasserstein distance and add
additional constraint terms (see Table 1 in Park and Kwon (2019)), SphereGAN
does not need any additional constraints to force $D$ in a desired function
space, due to the usage of geometric transformation in $D$.

##  6 Data augmentation with GAN

We turn now to describe how to use GAN for data augmentation. Data
augmentation is a crucial process for tasks lacking large training sets. There
are three circumstances which require data augmentation:

  * •

Limited annotations: Training a large DNN with a very few labeled data in the
training set.

  * •

Limited diversity: When the training data lacks variations. For example, it
may not cover diverse illuminations or a variety of appearances.

  * •

Restricted data: The database might contain sensitive information, and thus
accessing it directly is strictly restricted.

The first two scenarios can be solved via supervised learning but they will
cost a lot of human effort to enrich the labeled data or to use active
learning approaches Cohn et al. (1996). An alternative approach is to utilize
GANs to augment the data. Semi-supervised GAN (SGAN) (see Section 4.1) is a
simple example of such a GAN architecture that is able to generate new
annotated images and can automatically enrich training data with few labels.

The second case is more common in the real world. Data variability is
important for many psychology or neuroscience experiments. For example, a
human’s EEG is sensitive to different types of face images such as happy,
angry, and sad faces Mavratzakis et al. (2016). Preparing those stimuli in
traditional experiments is time-consuming and costly for researchers.
Architectures like StyleGAN Karras et al. (2019) (see Section 4.6) are
specialized to generate broad types of face images as a stimulus. More
important, these images can be controlled to exhibit specific attributes (e.g.
level of happiness or facial textures), and thus enhancing the stimuli
variation in the experiment Wang et al. (2020). Data augmentation can be also
helpful in the training of the GANs themselves Karras et al. (2020a); Zhao et
al. (2020), which allows to train them with only few examples and still get
high quality generated data (e.g., images).

The last case is related to unsupervised learning approaches. When a database
is restricted to preserve users’ privacy, researchers can use GANs to
synthesize this sensitive data themselves. For example, Delaney et al.
employed GANs to generate synthetic ECG signals that resemble real ECG data
Delaney et al. (2019).

##  7 Conclusion

This chapter has provided just a glimpse of GANs and their various usages.
This fast-growing technique has many variants and applications that have not
been presented here for the sake of brevity. Yet, we believe that the
description of GANs given here provides the reader with the tools to better
understand this tool and be able to navigate between the vast amount of recent
literature on this topic. It is worth mentioning also normalizing flows
Papamakarios et al. (2021) and score based generative models Song et al.
(2021b, a); Nichol and Dhariwal (2021), which become very popular recently and
show competitive performance with GANs.

###### Acknowledgements.

We would like to thank Yuval Alaluf, Yotam Nitzan and Ron Mokady for their
helpful comments.

## References

  * Abdal et al. [2019] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4431–4440, 2019.
  * Abdal et al. [2020] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Abdal et al. [2021] Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  _ACM Trans. Graph._ , 40(3), May 2021.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein Generative Adversarial Networks.  In _ICML_ , volume 70, pages 214–223, 2017.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer Normalization.  7 2016.
  * Berthelot et al. [2017] David Berthelot, Tom Schumm, and Luke Metz.  {BEGAN:} Boundary Equilibrium Generative Adversarial Networks.  _CoRR_ , abs/1703.1, 2017.
  * Brock et al. [2017] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Neural Photo Editing with Introspective Adversarial Networks.  _ArXiv_ , abs/1609.0, 2017.
  * Brock et al. [2019] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In _International Conference on Learning Representations_ , 2019.
  * Brophy et al. [2019] Eoin Brophy, Zhengwei Wang, and Tomas E. Ward.  Quick and easy time series generation with established image-based gans.  _ArXiv_ , abs/1902.05624, 2019.
  * Chen et al. [2019] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby.  Self-Supervised GANs via Auxiliary Rotation Loss.  _CVPR_ , pages 12146–12155, 2019.
  * Cohn et al. [1996] David A Cohn, Zoubin Ghahramani, and Michael I Jordan.  Active Learning with Statistical Models.  _J. Artif. Int. Res._ , 4(1):129–145, 3 1996\.  ISSN 1076-9757.
  * Delaney et al. [2019] Anne Marie Delaney, Eoin Brophy, and Tomas E Ward.  Synthesis of Realistic ECG using Generative Adversarial Networks.  _ArXiv_ , abs/1909.0, 2019.
  * Denton et al. [2015] Emily L Denton, Soumith Chintala, Arthur Szlam, and Robert Fergus.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks.  _CoRR_ , abs/1506.0, 2015.
  * Dziugaite et al. [2015] Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani.  Training generative neural networks via maximum mean discrepancy optimization.  In _Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence_ , page 258–267, 2015.
  * Fedus et al. [2018] William Fedus, Ian Goodfellow, and Andrew M. Dai.  MaskGAN: Better text generation via filling in the _.  In _International Conference on Learning Representations_ , 2018.
  * Feigin et al. [2020] Yuri Feigin, Hedva Spitzer, and Raja Giryes.  Generative adversarial encoder learning, 2020.
  * French [1999] Robert M French.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_ , 3(4):128–135, 1999.
  * Gal et al. [2021a] Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or.  Swagan: A style-based wavelet-driven generative model.  _ACM Trans. Graph._ , 40(4), July 2021a.
  * Gal et al. [2021b] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators, 2021b.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems 27_ , pages 2672–2680, 2014.
  * Gulrajani et al. [2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved Training of Wasserstein GANs.  In _Advances in Neural Information Processing Systems 30_ , pages 5767–5777, 2017.
  * Guo et al. [2018] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang.  Long text generation via adversarial training with leaked information.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_ , pages 5141–5148, 2018.
  * Hartmann et al. [2018] Kay Gregor Hartmann, Robin Tibor Schirrmeister, and Tonio Ball.  Eeg-gan: Generative adversarial networks for electroencephalograhic (eeg) brain signals.  _ArXiv_ , abs/1806.01875, 2018.
  * Hoffman et al. [2018] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In _International Conference on Machine Learning_ , volume 80, pages 1989–1998, 2018.
  * Hussein et al. [2020] Shady Abu Hussein, Tom Tirer, and Raja Giryes.  Image-adaptive gan based reconstruction.  In _AAAI_ , 2020.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In _ICML_ , 2015.
  * Isola et al. [2016] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5967–5976, 2016.
  * Jetchev et al. [2016] Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf.  Texture synthesis with spatial generative adversarial networks.  _CoRR_ , abs/1611.08207, 2016.
  * Karras et al. [2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In _International Conference on Learning Representations_ , 2018.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4396–4405, 2019.
  * Karras et al. [2020a] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In _Proc. NeurIPS_ , 2020a.
  * Karras et al. [2020b] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In _CVPR_ , 2020b.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In _Proc. NeurIPS_ , 2021.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * Kirkpatrick et al. [2017] James N Kirkpatrick, Razvan Pascanu, Neil C Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114:3521–3526, 2017.
  * Lample et al. [2017] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, et al.  Fader networks: Manipulating images by sliding attributes.  In _Advances in Neural Information Processing Systems_ , pages 5963–5972, 2017.
  * LeCun et al. [1999] Y LeCun, P Haffner, L Bottou, and Yoshua Bengio.  Object Recognition with Gradient-Based Learning.  In _Shape, Contour and Grouping in Computer Vision_ , 1999.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszár, José Antonio Caballero, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 105–114, 2017.
  * Luc et al. [2016] Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek.  Semantic segmentation using adversarial networks.  _ArXiv_ , abs/1611.08408, 2016.
  * Maas [2013] Andrew L Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models.  In _ICML_ , 2013.
  * Mao et al. [2017] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Mavratzakis et al. [2016] Aimee Mavratzakis, Cornelia Herbert, and Peter Walla.  Emotional facial expressions evoke faster orienting responses, but weaker emotional responses at neural and behavioural levels compared to scenes: A simultaneous EEG and facial EMG study.  _NeuroImage_ , 124:931–946, 2016.
  * Mescheder et al. [2018] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  Which training methods for gans do actually converge?  In _International Conference on Machine Learning (ICML)_ , 2018.
  * Metz et al. [2017] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein.  Unrolled generative adversarial networks.  In _ICLR_ , 2017.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional Generative Adversarial Nets.  _ArXiv_ , abs/1411.1, 2014.
  * Miyato et al. [2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral Normalization for Generative Adversarial Networks.  In _International Conference on Learning Representations_ , 2018.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading Digits in Natural Images with Unsupervised Feature Learning.  In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011_ , 2011.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , volume 139, pages 8162–8171, 2021.
  * Odena [2016] Augustus Odena.  Semi-Supervised Learning with Generative Adversarial Networks.  _ArXiv_ , abs/1606.0, 2016.
  * Odena et al. [2017] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional Image Synthesis with Auxiliary Classifier {GAN}s.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 2642–2651, 2017.
  * Papamakarios et al. [2021] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _Journal of Machine Learning Research_ , 22(57):1–64, 2021.
  * Park and Kwon [2019] Sung Woo Park and Junseok Kwon.  Sphere Generative Adversarial Network Based on Geometric Moment Matching.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4287–4296, 2019.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2085–2094, October 2021.
  * Peyre and Cuturi [2019] Gabriel Peyre and Marco Cuturi.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 11(5-6):355–607, 2019.
  * Qi [2019] Guo-Jun Qi.  Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities.  _International Journal of Computer Vision_ , 128:1118–1140, 2019.
  * Radford et al. [2015] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.  _CoRR_ , abs/1511.0, 2015.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * Richardson et al. [2021] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In _CVPR_ , 2021.
  * Rott Shaham et al. [2019] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019\.
  * Rusu et al. [2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive Neural Networks.  _ArXiv_ , abs/1606.0, 2016.
  * Salimans and Kingma [2016] Tim Salimans and Durk P Kingma.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.  In _Advances in Neural Information Processing Systems 29_ , pages 901–909. Curran Associates, Inc., 2016.
  * Shen and Zhou [2021] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In _CVPR_ , 2021.
  * Shen et al. [2020] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  _TPAMI_ , 2020.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021a.  URL https://openreview.net/forum?id=St1giarCHLP.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2021b.
  * Tov et al. [2021] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _ACM Trans. Graph._ , 40(4), 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008, 2017.
  * Wang et al. [2018] T. Wang, M. Liu, J. Zhu, A. Tao, J. Kautz, and B. Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8798–8807, June 2018.  doi: 10.1109/CVPR.2018.00917.
  * Wang et al. [2019a] Zhengwei Wang, Qi She, and Tomas E Ward.  Generative Adversarial Networks: {A} Survey and Taxonomy.  _CoRR_ , abs/1906.0, 2019a.
  * Wang et al. [2019b] Zhengwei Wang, Qi She, and Tomas E Ward.  Generative Adversarial Networks: {A} Survey and Taxonomy.  _CoRR_ , abs/1906.0, 2019b.
  * Wang et al. [2020] Zhengwei Wang, Qi She, Alan F Smeaton, Tomás E Ward, and Graham Healy.  Synthetic-Neuroscore: Using a neuro-AI interface for evaluating generative adversarial networks.  _Neurocomputing_ , 405:26–36, 2020.
  * Wu et al. [2019] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang.  GP-GAN: Towards Realistic High-Resolution Image Blending.  _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019.
  * Xia et al. [2021] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  Gan inversion: A survey, 2021.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and Understanding Convolutional Networks.  In _ECCV_ , 2014.
  * Zhang et al. [2019] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In _International Conference on Machine Learning_ , volume 97, pages 7354–7363, 2019.
  * Zhao et al. [2020] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2242–2251, 2017.


With infinite gratitude I dedicate this thesis to my parents.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleArXiv e-prints
titleA Generalized Blakers-Massey Theorem
Mathematics - Algebraic Topology,Mathematics - Category Theory,55P99,18F99

journaltitleArXiv e-prints
titleComputational Higher Type Theory I: Abstract Cubical Realizability
Computer Science - Logic in Computer Science

journaltitleArXiv e-prints
titleNatural models of homotopy type theory
Mathematics - Category Theory,Mathematics - Logic

Springer, Dordrecht

booktitleEpistemology versus ontology
seriesLog. Epistemol. Unity Sci.
titleType theory and homotopy

journaltitleMath. Proc. Cambridge Philos. Soc.
titleHomotopy theoretic models of identity types

Paris, France

booktitleProceedings of the 6th ACM SIGPLAN Conference on Certified Programs and Proofs
seriesCPP 2017
titleThe HoTT Library: A Formalization of Homotopy Type Theory in Coq
Coq,Higher inductive types,Homotopy type theory,Univalent foundations,Universe polymorphism

Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern

booktitle19th International Conference on Types for Proofs and Programs
seriesLIPIcs. Leibniz Int. Proc. Inform.
titleA model of type theory in cubical sets

journaltitleArXiv e-prints
titleOn the homotopy groups of spheres in homotopy type theory
Mathematics - Algebraic Topology,Computer Science - Logic in Computer Science,Mathematics - Logic

journaltitleArXiv e-prints
titleOn the homotopy groups of spheres in homotopy type theory
Mathematics - Algebraic Topology,Computer Science - Logic in Computer Science,Mathematics - Logic

journaltitleAdv. Math.
titleImplications of large-cardinal principles in homotopical localization

annotationLex-reflective subcategories = local factorization systems. See also Kelly, clubs and data-type constructors.
journaltitleJ. Austral. Math. Soc. Ser. A
titleReflective subcategories, localizations and factorization systems

journaltitleArXiv e-prints
titleLocalization in Homotopy Type Theory
Mathematics - Algebraic Topology,Mathematics - Category Theory,55P60 (Primary),18E35,03B15 (Secondary)
family=Coq Development Team,

titleThe Coq Proof Assistant Reference Manual
given=F. van,

noteIn preparation
titleIdentity types of sequential colimits

booktitleProceedings of the 5th ACM SIGPLAN Conference on Certified Programs and Proofs
titleConstructing the propositional truncation using non-recursive HITs

booktitleInteractive Theorem Proving (ITP 2017)
titleHomotopy Type Theory in Lean

am.ac.uk/ mhe/agda-new/UF-IdEmbedding.html

Springer-Verlag, Berlin

seriesLecture Notes in Mathematics
titleCellular spaces, null spaces and homotopy localization
family=LeFanu Lumsdaine,

journaltitleArXiv e-prints
titleThe Simplicial Model of Univalent Foundations (after Voevodsky)
Mathematics - Logic,Mathematics - Algebraic Topology,Mathematics - Category Theory,03B15,55U10 (Primary),18C50,55U35 (Secondary)

New York, NY, USA

booktitleProceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science
seriesLICS '16
titleConstructions with Non-Recursive Higher Inductive Types

titleHigher inductive types: a tour of the menagerie

titleReducing all HIT's to 1-HIT's

ACM, New York

booktitleProceedings of the Joint Meeting of the Twenty-Third EACSL Annual Conference on Computer Science Logic (CSL) and the Twenty-Ninth Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)
titleEilenberg-MacLane spaces in homotopy type theory
pagesArticle No. 6610

IEEE Computer Soc., Los Alamitos, CA

booktitle2013 28th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2013)
titleCalculating the fundamental group of the circle in homotopy type theory

Bibliopolis, Naples

noteNotes by Giovanni Sambin
seriesStudies in Proof Theory. Lecture Notes
titleIntuitionistic type theory

Chalmers University of Technology

titleTowards a practical programming language based on dependent type theory

journaltitleArXiv e-prints
titleThe join construction
Mathematics - Category Theory,Mathematics - Logic

journaltitleArXiv e-prints
titleModalities in homotopy type theory
Mathematics - Category Theory,Computer Science - Logic in Computer Science,Mathematics - Logic,03B15,03B35,03B45,03B70,03F65,03G30,18A32,18B25,18C35,18C50,18D30,F.3.1,F.4.1

journaltitleMath. Structures Comput. Sci.
titleSets in homotopy type theory

journaltitleMath. Structures Comput. Sci.
titleBrouwer's fixed-point theorem in real-cohesive homotopy type theory

titleHomotopy Type Theory, VI

journaltitleHomology Homotopy Appl.
titleThe univalence axiom for elegant Reedy presheaves
family=Univalent Foundations Program,

Institute for Advanced Study

titleHomotopy Type Theory: Univalent Foundations of Mathematics
family=Univalent Foundations Program,

titleHomotopy Type Theory: Univalent Foundations of Mathematics

journaltitleTheory Appl. Categ.
titleA C-system defined by a universe category
pagesNo. 3711811215

titleA very short note on homotopy $\lambda$-calculus
.ias.edu/ vladimir/Site3/Univalent_Foundations_files/Hlambda_short_current.pdf

titleUnivalent Foundations Project
.ias.edu/ vladimir/Site3/Univalent_Foundations_files/univalent_foundations_project.pdf

journaltitleArXiv e-prints
titleCartan Geometry in Modal Homotopy Type Theory
Mathematics - Differential Geometry,Mathematics - Algebraic Geometry,Mathematics - Category Theory,58A15