and caricatures of the same identities, and generates a caricature for an
input content image from its learned deformation model. To compare the
performance of WarpGAN and DST, we used content/style image pairs from [100]
and ran DST. In Figure 4.7, we show the outputs of DST and the outputs of
WarpGAN taken from [100]. Despite the lack of a _learning_ component, DST
results are competitive and sometimes more aesthetically pleasing than results
of FoA and WarpGAN.

####  4.5.2 Human Evaluation

Quantitatively evaluating and comparing style transfer is challenging, in part
because of the subjective nature of aesthetic properties defining style and
visual quality, and in part due to the inherent tradeoff between content
preservation and stylization [120, 63]. Following the intuition developed in
these papers, we conducted a human evaluation study using Amazon Mechanical
Turk on a set of 75 diverse style/content pairs. The goal was to study the
effect of DST on the stylization/content preservation tradeoff, in comparison
to the base style transfer methods. The evaluation was conducted separately
for STROTSS and Gatys-based methods. We considered three DST deformation
regimes: low ($\beta=0.3$,$\gamma=75$), medium ($\beta=0.5$,$\gamma=50$), and
high ($\beta=0.7$,$\gamma=10$) for STROTSS; low ($\beta=3$,$\gamma=750$),
medium ($\beta=7$,$\gamma=100$), and high ($\beta=15$,$\gamma=100$) for Gatys.
So for each base method, we compare four stylized output images. The effect of
varying $\beta$ and $\gamma$ is illustrated in Figure 4.4.

To measure content preservation, we asked MTurk users the question: “Does
image A represent the same scene as image B”, where A referred to the content
image and B to the output of style transfer. The users were forced to choose
one of four answers: “Yes”, “Yes, with minor errors”, “Yes, with major errors”
and “No”. Converting these answers to numerical scores (1 for “No”, 4 for
“Yes”) and averaging across content/style pairs and users, we get a _content
score_ between 1 and 4 for each of the four methods.

To evaluate the effect of the proposed deformable framework, we presented the
users with a pair of outputs, one from the base method (Gatys or STROTSS) and
the other from DST, along with the style image. The order of the first two is
randomized. We asked the users to choose which of the two output images better
matches the style. The fraction of time a method is preferred in all
comparisons (across methods compared to, users, content/style pairs) gives a
_style score_ between 0 and 1. 0.7 means that the method “wins” 70% of all
comparisons it was a part of. The evaluation interfaces are provided in the
supplementary material.

In total, there were 600 unique content comparisons: 4 questions$\times$75
images for Gatys and an equal number for STROTSS. 123 users participated in
the evaluation, and each comparison was evaluated by 9.55 users on average.
The standard deviation of the content choice agreement was 0.79 (over a range
of 1 to 4). For stylization, there were 450 unique comparisons in total: 3
comparisons between the base method and each of the 3 DST deformation
regimes$\times$75 images for Gatys and likewise for STROTSS. 103 users
participated in the stylization evaluation, and each comparison was evaluated
by 8.76 users on average. For each comparison, 6.47 users agreed in their
choice on average.

Results of this human evaluation are shown in Figure 4.9. Across the
deformation regimes (low, medium, high), for both STROTSS and Gatys, DST
significantly increases the perceived stylization quality, while only
minimally reducing the perceived content preservation. Note that some
reduction to the content score can be expected since we intentionally alter
the content more by deforming it, but our evaluation shows that this drop is
small.

Figure 4.9: Human evaluation results, comparing DST in different deformation
regimes with STROTSS (green) and Gatys (blue). DST provides a much higher
perceived degree of style capture without a significant sacrifice in content
preservation.

####  4.5.3 Limitations

In Figure 4.10, we show unsuccessful examples of DST where the output image
did not deform towards having a similar shape as the style image or deformed
only partially. We observed that bad deformations often stem from poorly
matching or too sparse set of keypoints. We expect finding better matching
keypoints between images and making the method more robust to poor matches
will improve results.

Figure 4.10: Examples of DST failures. We observed that stylization failures
are often due to correspondence errors or overly complex scene layout.

###  4.6 Conclusion

Style transfer research prior to DST largely ignored geometry and shape,
despite the important role these play in visual style. This chapter presented
deformable style transfer (DST), a novel approach that combines the
traditional texture and color transfer with spatial deformations. Our method
incorporates deformation targets, derived from domain-agnostic point matching
between content and style images, into the objective of an optimization-based
style transfer framework. This is to our knowledge the first effort to develop
a one-shot, domain-agnostic method for capturing and transferring geometric
aspects of style.

While this work represents an early step towards incorporating geometry into
style transfer, it is far from a satisfying solution. From the narrow
perspective of improving DST a better algorithm might develop more robust
keypoint matching algorithms for highly stylized images. However, it is
doubtful that modifying geometry using deformation fields defined by paired
keypoints is the most effective approach to this problem. Even in two-
dimensional art, the scenes represented are often three-dimensional, and an
artist’s stylization of shape and form often take this into account. It seems
likely that style transfer algorithms will ultimately have to explicitly model
the 3D geometry of the scene to be stylized (and potentially the scene
represented in the style exemplar as well). We hope that future work will
continue to explore how to more accurately encode, extract and stylize
artistic shape and form.

##  Chapter 5 Neural Neighbor Style Transfer

Figure 5.1: Examples produced by Neural Neighbor Style Transfer (NNST) at 1k
resolution. NNST synthesizes a stylized output by rearranging features
extracted from the target style by a pretrained CNN. Synthesis can be
implemented either as direct optimization of output pixels (NNST-Opt, pictured
above), or as inference of pixels from features by a learned decoder (NNST-D).
Figure 5.2: Overview of NNST. The fast and slow variants of out method, NNST-D
and NNST-Opt, only differ in step 4; mapping from the target features to image
pixels. This simplified diagram omits several details for clarity, namely: we
apply steps 1-4 at multiple scales, coarse to fine; we repeat steps 1-4
several times at the finest scale; and we only apply step 5 once (optionally)
at the very end.

As detailed in Sections 2.1 and 2.2, the earliest style transfer algorithms
were based on rearranging and blending patches of pixels taken directly from
the style image [18, 40]. Several recent style transfer algorithms [10, 69,
33] took inspiration from this approach, rearranging neural features extracted
from the style image, then recovering an RGB image from the resulting tensor.
However, based on our user study when evaluating STROTSS (see Figure 3.11),
these methods do not actually improve over the original style transfer
formulation proposed by Gatys et al. [29]. NNST is not based on any
fundamentally new ideas, at the end of the day it also rearranges neural
features and decodes an image from the resulting tensor; instead it outlines a
set of important design decisions that dramatically improve outputs’ visual
quality. The paper this chapter is based on is currently under review, and was
joint work with Michal Kučera, Daniel Sýkora, Eli Shechtman, Sylvain Paris,
and my advisor Greg Shakhnarovich.

###  5.1 Introduction

The resulting algorithm, Neural Neighbor Style Transfer (NNST), is a
straightforward new baseline for style transfer based on replacing content
features with their nearest neighbor style feature (in the feature space of a
neural network). It offers state-of-the-art visual quality and comes in two
variants. The fast variant NNST-D (NNST-Decoder) produces a 512x512 output in
$\approx 4.5$ seconds by using a learned decoder to efficiently recover pixels
from a rearranged tensor of style features. The slower variant NNST-Opt (NNST-
Optimization) trades off speed for quality, producing a 512x512 output in
$\approx 40$ seconds, by directly optimizes the pixels of the output image in
the same manner as Gatys [29], STROTSS [63], and others [69, 83].

Regardless of the mechanism for recovering pixels from features, the details
of exactly how the style features are rearranged are of vital importance.
Patches of images are in reality tensors, having both spatial extent (the
first two dimensions) and being a concatenation of the color channels (a third
dimension) in a particular region of the image. Similarly we can think of
patches of neural features, where the first two dimensions also correspond to
a spatial region, and the third corresponds to the concatenation of a set of
activation maps produced by a neural network. The original instinct of neural
style transfer algorithms inspired by patch-based synthesis was to follow
precisely in their footsteps and match patches of neural features, then
average in regions where the patches overlap, as had been done in pixel space.
Our first recommendation is that this is inappropriate for neural features, as
averaging them destroys distinctive features of the style. We find that a more
effective mechanism to recreate stylistic features with large spatial extent
is a coarse-to-fine stylization procedure, coupled with using features from
multiple neural network layers. Our second recommendation is to pre-process
feature maps by zero-centering (over spatial dimensions) before computing
matches. While using the cosine distance to match neural features is common,
we find zero-centering (which we have not seen proposed elsewhere)
dramatically increases the diversity of features used to synthesize the final
output. As a fortuitous side-effect, the visual contrast between diverse
features actually improves content preservation. Finally, we find that an
explicit content loss is unnecessary, the implicit bias provided by
initializing with the content image is enough to preserve the perceptual
contents of the input.

An additional component of NNST’s success is treating the luminance of the
final output separately from the hue and chroma. While the complex texture
synthesis afforded by neural style transfer is important for generating a
compelling luminance channel, a simpler procedure based on moment matching and
bilateral filtering often generates convincing hue and chroma (parameterized
as AB channels in the CIE-Lab colorspace in this work). This procedure is
extremely efficient, and due to it’s simplicity, is more robust than
generating colors using neural style transfer. While this prevents capturing
higher order terms of the style’s color distribution, and sometimes more
compelling results can be achieved by omitting this step, in the majority of
cases either higher order distribution terms are not an important part of the
style and preventing artifacts is preferable.

Qualitatively our results more accurately capture the texture of the target
media than prior work, particularly when seen at high resolution. In addition
our color processing dramatically increases the system’s reliability,
preventing the introduction of many jarring artifacts. We evaluate NNST-D and
NNST-Opt relative to prior feed-forward and optimization based methods Via a
large user study. NNST-D is judged to produce better stylizations than other
fast methods and is judged roughly equivalent to the much slower state-of-the-
art in optimization-based style transfer, STROTSS. NNST-Opt is judged to offer
even higher visual quality, setting a new standard for artistic style
transfer.

###  5.2 Neural Neighbor Style Transfer

####  5.2.1 Feature Extraction

Like other neural style transfer algorithms, NNST’s pipeline relies on a pre-
trained feature extractor $\Phi(x)$, where $x$ is an RGB image. Similar to
STROTSS, $\Phi(x)$ extracts the hypercolumns [85, 35] formed from the
activations produced for convolutional layers in the first four blocks of pre-
trained VGG16 [102] when $x$ is passed in. We use bilinear interpolation on
activations from all layers to give them spatial resolution equal to one
quarter of the original image. For an image with height H, and width W, this
yields an image representation
$\Phi(x)\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times 2688}$. Generally we
consider style to be rotation invariant, and to reflect this we extract
features from the style image rotated at $0^{\circ},90^{\circ},180^{\circ}$
and $270^{\circ}$ in all experiments.

####  5.2.2 Feature Matching

Content |  (a) |  (b) |  (c) |  (d) |  Style
---|---|---|---|---|---

Figure 5.3: Demonstration of the affect of zero-centering features before
nearest-neighbor matching. In (a) there is no zero-centering and no color
processing, resulting in lower quality feature pairings that lead to more
homogeneous colors and worse content preservation. (b) largely fixes the most
egregious errors of (a) by adding color processing, although the features of
the face are poorly defined. (c) is the default setting of NNST which uses
zero-centering and color-correction, producing nice results in both cases. (d)
is the same as (c) but with no color processing, this setting is less reliable
and more prone to introducing visual errors than (c), but when colors are
mapped correctly (as in the above examples) the results can be stunning. See
Figure 5.11 for more examples of the effect of zero-centering without color
post-processing.

The core steps of our pipeline are outlined in Figure 5.2. We extract features
from the style image and content image (1) zero-center the content features
and style features (2). Then use nearest-neighbors matching under cosine
distance (3) to replace each content feature (hypercolumn) with the closest
style feature. If the content image $C$ is of size $H_{c}\times W_{c}$, and
style image $S$ is of size $H_{s}\times W_{s}$, this yields a new target
representation for our stylized output
$T\in\mathbb{R}^{\frac{H_{c}}{4}\times\frac{W_{c}}{4}\times 2688}$ where the
feature vector $T_{i}\in\mathbb{R}^{2688}$ at each spatial location is derived
from the original style image, or a rotated copy (recall from Section 5.2.1
that we extract features from rotated copies of the style image as well). For
simplicity let $\Phi^{\prime}(x)$ be the function extracting features from $x$
and its rotations, where an individual feature vector (from any spatial
location in any rotation) can be indexed as $\Phi^{\prime}(x)_{j}$. Formally:

$\displaystyle T_{i}=$ $\displaystyle\Phi^{\prime}(S)_{j}$ (5.1)
$\displaystyle\operatorname*{argmin}_{j}\hskip 5.69046pt$ $\displaystyle
D\Big{(}\Phi(C)_{i}-\mu_{C},\hskip
5.69046pt\Phi^{\prime}(S)_{j}-\mu^{\prime}_{S}\Big{)}$ (5.2)

Where $D$ is the cosine distance, $\mu_{C}$ is the average feature extracted
from the content image, and $\mu^{\prime}_{S}$ is the average feature
extracted from the style image and its rotated copies. While mean subtraction
does not have a huge impact when using our color post-processing, it extremely
important without it, enabling some stunning results in cases where the
content and style are well matched (Figure 5.3) or a high degree of
stylization is desired (Figure 5.11).

Content |  (a) |  (b) |  (c) |  Style
---|---|---|---|---

Figure 5.4: Demonstration of the effect of our final feature splitting phase
(c). (a) is our result without feature splitting, content is well preserved,
but too many photographic details bleed into the output and the brushstrokes
in the 2nd row are poorly defined. (b) mimics our feature splitting phase,
recomputing feature matches after each update to the output image, but matches
complete hypercolumns instead of computing matches separately for each layer.
This leads to unnecessary loss of content details and muddier high frequencies
relative to (c).

In sections 5.2.3 and 5.2.4 we describe our feed-forward and optimization
based methods for recovering image pixels from $T$ (choosing between these
differentiates between NNST-D and NNST-Opt). In the main loop of our pipeline
we produce stylizations at each scale, coarse to fine, and each result is used
to initialize the next scale. Throughout this process we match hypercolumns
wholesale, and keep the $T$ unchanged throughout the synthesis process at a
particular scale. While this is efficient (since $T$ need only be computed
once per scale, and computing a single large distance matrix is well suited to
GPU parallelism), and the result roughly captures many aspects of the target
style, stopping at this point leads to images that fail to capture the high
frequencies of the target style (Figure 5.4).

We believe that this effect is due to incompatible hypercolumns, which are not
adjacent in the original style, being placed next to each other in $T$.
Because these features have overlapping receptive fields, the output is
optimized to produce the average of several features (each taken from a
different region of the style) at a single output location. This manifests
visually as a ’washed out’ quality, an issue noted by prior work in style
transfer [33], and other patch-based synthesis work [58, 48, 22].

We find that these issues can be largely resolved by a final phase where the
the feature matching process is less constrained, a similar solution to one
used by Luan et al. in the image compositing [78]. In this final phase, which
we call ’Feature Splitting’, matches are computed for each layer separately,
resulting in $T$ consisting of novel hypercolumns where features at different
layers are mixed and matched from different locations/rotations of the style
image. Unlike [78] we do not compute matches only once, we recompute them
after every update to the output image. In this phase features are matched
relative to the current output, rather than the initial content. When using
our learned decoder $\mathcal{G}$ to synthesize, this amounts to feeding the
output back into the same network as ’content’ five times (recomputing $T$
each time). When directly optimizing the output image, this amounts to
recomputing $T$ using the current output as the ’content’ after each Adam
update [62]. For more examples of the effect of feature splitting and
recomputing $T$ after each update, see Figure 5.14.

Figure 5.5: Overview of the architecture of the learned decoder used in
NNST-D, and the associated inference procedure at a particular scale (a
separate model is trained to produce final outputs that are $64,128,256$ and
$512$ pixels on the long side, note that a single model is pictured and each
model has 4 branches).

####  5.2.3 Neural Network Decoder (NNST-D)

##### Architecture

The decoder from feature tensor to pixels, $\mathcal{G}$, takes as input the
target representation
$T\in\mathbb{R}^{\frac{H_{c}}{4}\times\frac{W_{c}}{4}\times 2688}$. $T$ is
then fed into 4 independent branches, each responsible for producing one level
of a 4-level laplacian pyramid parameterizing the output image. Each branch
has virtually the same architecture (but separate parameters), consisting of
five 3x3 convolutional layers with leaky ReLU [79] activations (except the
last layer, which is linear), and a linear residual 3x3 convolution [37]
directly from $T$ to the branch’s output. All intermediate hidden states have
256 channels. The four branches differ only in number of output channels,
having $48$, $12$, $3$, and $3$ output channels respectively. Transposed
convolutions are applied to the first two branches to trade off channel depth
for resolution (resulting in one $H_{c}\times W_{c}\times 3$ output and one
$\frac{H_{c}}{2}\times\frac{W_{c}}{2}\times 3$ output). The third branch is
not altered (resulting in a $\frac{H_{c}}{4}\times\frac{W_{c}}{4}\times 3$
output), and the output of the fourth branch is bilinearly downsampled by a
factor of two (resulting in a $\frac{H_{c}}{8}\times\frac{W_{c}}{8}\times 3$
output). The final output image is synthesized by treating the output of the
four branches as levels of a laplacian pyramid and combining them
appropriately. In Figure 5.5 we give an overview of how we employ our
architecture at inference.

##### Training

We train our model using MS-COCO[76] as a source of content images, and
Wikiart [95] as a source of style images, matching the training regime of
[124, 3, 88]. These are the standard datasets for this task. MS-COCO is a
diverse dataset of roughly 300,000 photographs, and Wikiart is a dataset of
roughly 80,000 2D artworks from a variety of movements. Content/Style training
pairs are randomly sampled independently from each dataset. For each input
pair, two outputs are generated during training, a reconstruction of the style
image and a style transfer:

$\displaystyle\hat{S}$ $\displaystyle=\mathcal{G}(\Phi(S))$ (5.3)
$\displaystyle O$ $\displaystyle=\mathcal{G}(T)$ (5.4)

Recall that an intermediate output of $\mathcal{G}$, the decoder from feature
tensor to pixels, is a laplacian pyramid that is collapsed to form the final
output image (see Figure 5.5). Let the levels of this pyramid for the style
reconstruction be $\hat{S}_{0..3}$, let the levels of a 4-level laplacian
pyramid constructed from $S$ be denoted as $S_{0..3}$, let $P_{i}$ be the
number of pixels at level $i$. These are used to compute the reconstruction
loss:

$\mathcal{L}_{r}=\sum_{i=0}^{3}\frac{\|S_{i}-\hat{S}_{i}\|_{1}}{P_{i}}$ (5.5)

We do not know what the pixels of the style transferred result should be, so
we instead optimize this output using a cycle loss between $T$ and features
extracted from $O$ [127]:

$\mathcal{L}_{cycle}=\frac{16}{H_{C}W_{C}}\sum_{i=0}^{\frac{H_{C}W_{C}}{16}}D\Big{(}T_{i},\hskip
5.69046pt\Phi(O)_{i}\Big{)}$ (5.6)

Where $D$ computes the cosine distance, and $i$ indexes over the spatial
indexes of $T$ and $\Phi(O)$ (which are both a quarter of the original
resolution of $C$). To further improve the ’realism’ of our results and
encourage better capturing the target style we also employ the adversarial
patch co-occurrence loss proposed by Park et al. [90]:

$\mathcal{L}_{adv}=-\log\mathcal{D}(\Theta^{(4)}(S),\Theta^{(1)}(O))$ (5.7)

Where $\Theta^{(k)}(x)$ is a function that extracts $k$ random patches of size
$\frac{\max(H,W)}{8}$ from $x$, and $H,W$ are the height and width of $x$
respectively. $\mathcal{D}$ is a discriminator that evaluates whether a single
patch ($\Theta^{(1)}(O)$ or $\Theta^{(1)}(S)$) comes from the same image as 4
patches extracted from the style image ($\Theta^{(4)}(S)$). We use the same
discriminator architecture and discriminator training described in [90], where
further details can be found. We fit the parameters of our model,
$\theta_{\mathcal{G}}$ to minimize the full objective:

$\displaystyle\min_{\theta_{\mathcal{G}}}\mathbb{E}_{C\sim\mathbb{P}_{C},\,S\sim\mathbb{P}_{S}}\Big{[}\mathcal{L}_{r}+\mathcal{L}_{cycle}+\mathcal{L}_{adv}\Big{]}$
(5.8)

Where $\mathbb{P}_{C}$, $\mathbb{P}_{S}$ are the distributions of content and
style training images respectively. A separate decoder is trained for each
output scale (64, 128, 256, 512 pixels on the long side). Training converges
fairly quickly, and we use models trained for a three epochs on MS-COCO
(Wikiart images are sampled independently with replacement for each MS-COCO
example). We train using a batch size of 4, and the Adam optimizer [62] with
parameters $\eta=2e^{-3},\beta_{1}=0.0,\beta_{2}=0.99$. The same set of
decoder models (four total, one for each scale) are used in all experiments.

####  5.2.4 Image Optimization (NNST-Opt)

While performing style transfer using $\mathcal{G}$ is fast, there are many
cases where optimizing the output image directly produces sharper results with
fewer artifacts. Given our target features $T$ and feature extractor $\Phi$,
we find output image $O$ by minimizing following objective:

$\min_{O}\hskip
8.5359pt-\frac{1}{P}\sum_{i=0}^{P-1}\cos\left(\Phi(O)_{i},T_{i}\right)$ (5.9)

where $P=W_{c}H_{c}/16$, the number spatial locations in $\Phi(O)$ and $T$. As
in other optimization based methods note that we do not update the parameters
of $\Phi$, only the output image itself.

Equation 5.9 is minimized via 200 updates of $O$ using Adam [62] with
parameters $\eta=2e^{-3}$, $\beta_{1}=0.9$, and $\beta_{2}=0.999$. As in
STROTSS, we parameterize $O$ as a laplacian pyramid with 8 levels. Doing so
allows the average color of large regions to be controlled by a small number
of parameters (coefficients in coarse levels of the pyramid).

Figure 5.6: In the first two rows we give examples of NNST-Opt with color
post-processing (b) and without (a). Color post-processing helps fix common
content preservation errors due to features within a region of homogeneous
color in the original content being mapped to features of multiple colors in
the output. Below these examples we outline our color processing procedure.
First the stylized luminance (produced by NNST-D or NNST-Opt) is extracted.
Then it is used as a guide for bilateral filtering on the original content’s
AB channels, this aligns the boundaries of colored regions to the stylized L
channel. After combining the stylized L channel with the filtered AB channels
we use simple moment matching to align the output’s color distribution with
the style.

####  5.2.5 Color Post-Processing

We observed that a common source of perceptual errors in the outputs produced
by NNST-D, NNST-Opt, and other methods, is when a region with a single color
in the original content is mapped to multiple colors in the output (see the
first two rows of Figure 5.6). However, after converting our outputs to Lab
colorspace, we observed that the luminance channel generally matches the
target style well, and was free of artifacts. This motivated our post-
processing step, which is to take the luminance generated by NNST-D or NNST-
Opt, but discard the AB channels and replace them with slightly modified AB
channels from the original content. In order to match the content’s original
AB channels to the generated L channel we perform bilateral filtering [107,
87] on the AB channels guided by the L channel. Then we match the mean and
covariance of color distribution formed by the output’s L channel and filtered
AB channels to the style’s color distribution. (see the bottom section of
Figure 5.6). This has some similarity to the color control proposed in [30]
(which suggests disentangling style from color by inheriting hue and chroma
from the content image), but our addition of bilateral filtering helps
outputs’ final color to better conform to edges present in the luminance, and
moment matching enables stylizations which more closely match the palette of
the target style.

In the vast majority of cases this post-processing step improves results,
however we have observed a few scenarios where it does not.

First, for apparently monochrome styles (e.g., pencil or pen drawings),
visually imperceptible color variations in the style can distort the second-
order statistics used in moment matching, leading to results where desaturated
colors are visible. Fortunately in these cases the unprocessed results of NNST
are typically monochrome, and we detect this situation by examining the
maximimum value of the AB channels’ covariance matrix, and not apply the color
processing if the value is below a threshold (we find 4e-5 to work well).

Second, this post-processing step can prevent the output from matching
distinctive color features of the style. For example the local color
variations that defines the dots of pointillism, the abrupt color shifts
within objects of cubism, or the limited multi-modal palettes used in some
artwork (see Figures 5.3 and 5.7).

Third, colorful styles with a large white background can cause moment matching
to result in over-saturation. In these cases it can be better to not use this
post-processing step.

Fortunately, our post-processing step is simple and computationally efficient,
taking less than a millisecond in our PyTorch implementation. In a practical
setting it is essentially free for users to generate results both with and
without the post-processing, then choose the one best suited to their needs.

Content |  $\downarrow$ Stylization |  $\uparrow$ Stylization |  Style
---|---|---|---

Figure 5.7: NNST has two mechanisms for controlling the output’s level of
stylization. In the first row we demonstrate the effect of increasing
stylization by ommitting our color post-processing. This increases stylization
by allowing greater variation from the hue and chroma of the original content
image. In the second row we demonstrate the effect of varying $\alpha$, the
parameter controlling the weight of the stylization at the previous scale in
the initialization of the next scale. We show results with $\alpha=0.0$
(minimum stylization) and $\alpha=1.0$ (maximum stylization).

####  5.2.6 Control of stylization degree

Including or omitting our color processing is an important mechanism for
trading off between content preservation and stylization quality (row 1 of
Figure 5.7). However, we can also take advantage of NNST’s multi-scale
pipeline to control the stylization level of our final output (row 2 of Figure
5.7).

For both NNST-D and NNST-Opt we produce stylizations at eighth, quarter, half,
and full resolution. The upsampled output of the previous scale serves as
initialization for the next. We initialize the coarsest scale with a
downsampled version of the content image. Let $O_{s}$ be the output of our
algorithm at scale $s$. Let $C_{s+1},S_{s+1}$ be the content and style images
at finer scale $s+1$. Let $O_{s}^{\uparrow}$ be $O_{s}$ upsampled to be the
same resolution as $C_{s+1}$. Instead of constructing $T$ by finding matches
between $\Phi(C_{s+1})$ and $\Phi(S_{s+1})$, we instead find matches between
$\Phi(\alpha O_{s}^{\uparrow}+(1-\alpha)C_{s+1})$ and $\Phi(S_{s+1})$. The
parameter $\alpha$ controls stylization level, with $\alpha=0$ corresponding
to the lowest stylization level, and $\alpha=1$ the highest. By default, we
set $\alpha=0.25$, as this generally produces a visually pleasing balance
between stylization and content preservation.

####  5.2.7 Implementation and Timing Details

We implement our method using the Pytorch framework [91]. The feed-forward
variant of our method, NNST-D takes 4.5 seconds to process a pair of 512x512
content/style images. Our optimization based variant, NNST-Opt, takes 38
seconds to process the same input. Timing results are based on an NVIDIA
2080-TI GPU.

###  5.3 Design Decisions

A pithy description of NNST and non-parametric neural style transfer algorithm
proposed by Chen and Schmidt[10] much earlier in 2016 would reveal little
difference between the two. Both methods explicitly construct a tensor of
‘target features’ by replacing vectors of VGG-derived content features with
vectors of VGG-derived style features, then optimize the pixels of the output
image to produce the ’target features’ (or use a learned decoder). Yet, there
is a dramatic difference between the visual quality of the algorithms’
outputs. As is often the case, the devil is in the details, and this section
explores the important design decisions that can boost a style transfer
algorithm’s visual quality.

In Figures 5.8, 5.9, 5.10, 5.11, 5.12, 5.13, 5.14, we visually explore the
effects of the our design decisions relative to [10], and iteratively modify
their method until arriving at NNST. Where appropriate these figures also
demonstrate the effect of modifying individual design elements of NNST to
match [10].

Figure 5.8: Visual comparison between (a.) the outputs of Chen and Schmidt
[10] and (b.) a simplified variant of NNST which uses the feature splitting
regime across all scales and does not employ color correction. While both
algorithms share a similar high level framework, they differ in many details,
resulting in NNST much better recreating distinctive visual features of the
style image. Figure 5.9: Visual comparison between matching features
separately for each location ($1\times 1$ patches) and matching them as
$3\times 3$ patches: (a.) the outputs of Chen and Schmidt [10] (3x3 feature
patches matched and overlaps averaged), (b.) a variant of [10] where feature
patches are matched independently for each spatial location (1x1 patches, no
averaging), (c.) a variant of NNST where 3x3 patches are matched and averaged,
and (d.) the simplified NNST variant from Figure 5.8. Note that matching 1x1
rather than 3x3 patches (b. and d. relative to a. and c.) allows more high
frequency details of the style to appear in the output. Figure 5.10: Visual
comparison between single-scale and multi-scale stylization: (a.) [10] w/ 1x1
patches (row b of Figure 5.9), (b.) a. applied coarse-to-fine using the same
mechanisms as NNST ($\alpha=0.25$), (c) simplified NNST at only the finest
scale ($\alpha=1.0$), and (d.) the simplified NNST variant from Figure 5.8.
Stylizing coarse-to-fine increases stylization level and results in visual
features of the style with larger spatial extent appearing in the output (and
this effect increases with lower $\alpha$, see Figure 5.7). In addition,
stylizing coarse-to-fine allows stylistic details to be hallucinated in large
flat regions of the content image (compare the floor beneath the monk in c.
and d.). Figure 5.11: Visual comparison between using zero-centering or not
before matching features using the cosine distance: (a.) multi-scale [10] w/
1x1 patches (row b of Figure 5.10), (b.) a. using the centered cosine distance
for feature matching (instead of the standard cosine distance), (c) d. using
the standard cosine distance (instead of the centered cosine distance), and
(d.) the simplified NNST variant from Figure 5.8. Using the centered cosine
distance not only results in a more diverse set of style features appearing in
the output, the contrast between these features helps preserve the perceived
contents of the original input.

Figure 5.12: Visual comparison between using different individual
convolutional layers of pretrained VGG-16 as a source of features. All images
are produced using multi-scale [10] w/ 1x1 patches matched with the centered
cosine distance (row b. of Figure 5.11 corresponds to row conv3_1 of this
figure, the default style features used by [10]). Layers in the first two
conv. blocks capture color well, but not more complex visual elements. Layers
in the third and fourth conv block capture complex visual elements, but not
color. Layers in the fifth block do not seem closely tied to stylistic
features. No layer alone is sufficient to capture all desired stylistic
features. Figure 5.13: Visual comparison between using features from multiple
layers of pretrained VGG-16. The first 5 rows demonstrate the effect of using
all the layers from a given conv. block. Just as no single layer is
sufficient, no single conv. block contains a rich enough representation of
style to produce satisfactory outputs. The 5th-9th rows demonstrate the effect
of using all of the features up to a certain depth in the network. Most
important stylistic details can be captured using the first three conv.
blocks. Small improvements can be made using the 4th and 5th conv. blocks as
well, but it is probably not worth the computational cost (the 4th and 5th
blocks each contain 36% of the total feature channels). While NNST uses all
feature through conv. block 4, only using features through block 3 would be an
obvious means to increase efficiency. Figure 5.14: The design decisions so far
lead to an algorithm close to NNST, the only difference that remain to be
evaluated are the frequency of computing matches and whether or not to compute
matches separately for each layer: (a.) multi-scale [10] w/ 1x1 hypercolumns
using layers conv1_1-conv4_3, matched with the centered cosine distance (row 8
of Figure 5.13), (b.) a. with nearest neighbors recomputed after each update
of the output image, (c.) a. with nearest neighbors computed seperately for
each layer, and (d.) the simplified NNST variant from Figure 5.8 (i.e. a. w/
features matched separately for each layer and recomputing matches each
update). Nice results can already be achieved without feature splitting or
recomputing matches each iteration (a.), however slightly sharper high
frequencies can be achieved by recomputing matches (b.), and more diverse
stylistic features from each layer are matched seperately (c.). When both of
these modifications are applied, we essentially arrive at NNST (d.).

###  5.4 Evaluation

Content |  [73] |  [124] |  [3] |  NNST-D (Ours) |  Style
---|---|---|---|---|---

Figure 5.15: Qualitative comparision between NNST-D and the top three feed-
forward methods from our user study, using oil painting, ink, watercolor, and
pencil styles. Below each input and result is a zoomed-in portion of the
image. While no method to date reliably reproduces the long range correlations
and high frequency details of arbitrary styles, NNST dramatically improves
over prior work.

Content |  [75] |  [29] |  [63] |  NNST-Opt (Ours) |  Style
---|---|---|---|---|---

Figure 5.16: Qualitative comparision between NNST-Opt and the top three
optimization based methods from our user study, using oil painting, ink,
watercolor, and pencil styles. Below each input and result is a zoomed-in
portion of the image. While no method to date reliably reproduces the long
range correlations and high frequency details of arbitrary styles, NNST
dramatically improves over prior work.

####  5.4.1 Traditional Media Evaluation Set

In order to benchmark the performance of NNST and prior work we gathered a
dataset of 30 high-resolution content photographs from Flickr, chosen for
their diversity and under the constraint that they be available under a
creative commons license allowing modification and redistribution. We followed
the same procedure (also using Flickr) to gather ten ink drawings and ten
watercolor paintings. From the Rijksmuseum’s open-source collection we take
ten impressionist oil paintings created between 1800-1900. We supplement these
with ten pencil drawings taken from the dataset used in Im2Pencil [72]. In
total this gives us 40 high-resolution style images. We use this dataset in
the following user study, and will make it available to download.

Figure 5.17: Example of (a) our user study interface and (b) an example
attention verification question used in our study.

####  5.4.2 User Study

In order to assess the stylization quality of NNST-D and NNST-Opt relative
prior work we generated stylizations for all pairwise content/style
combinations in the traditional media evaluation set described above (A total
of 1200 outputs per method). We conduct a user study using Prolific
(https://www.prolific.co/) where users are shown the output of two algorithms
(randomly ordered) for the same content/style pair (randomly selected from the
1200 possible combinations), along with the target style, and asked ’Does
”Image A” or ”Image B” better match the ”Target Style”?’. Users are asked to
judge 9 such triplets in sequence, among which is mixed one attention
verification question (selecting the image that shows a cartoon whale in a
randomly ordered triplet). In total we collected 225 votes per method pair,
from a total of 400 unique participants. Examples of the study interface are
in Figure 5.17.

Optimization-based methods consistently produce outputs of higher visual
quality than fast feed-forward methods, therefore each family of techniques is
generally benchmarked separately. However, while we group these methods in
Table 5.1, we compare both variants of our method to both families of
technique. For optimization-based methods we benchmark against Gatys [29],
CNNMRF [69], Deep Image Analogies (DIA) [75], and STROTSS [63]. We were unable
to run the official code for CNNMRF and Deep Image Analogies, and re-
implemented their methods. For fast feed-forward methods we benchmark against
WCT [73], AvatarNet [99], MST [124], and ArtFlow [3].

The results of our study are summarized in Table 5.1, along with the p-values
of rejecting the null hypothesis that the preference rate for NNST-D/Opt is
less than 50%. We calculate these p-values under the assumption that the votes
are independent and the sum of votes received by a method is distributed as a
binomial. In summary there is a statistically significant preference for our
fast variant NNST-D over all benchmarked methods (fast and optimization-based)
except STROTSS (STROTSS is preferred but not by a statistically significant
margin). There is a statistically significant preference for NNST-Opt over all
benchmarked methods.

| Feed-Forward
---|---
| WCT | Avatar | MST | ArtFlow
NNST-D | 71% (1.0e-10) | 74% (1.0e-13) | 70% (2.5e-10) | 60% (1.6e-3)
NNST-Opt | 83% ($<$ 1e-15) | 85% ($<$ 1e-15) | 72% (5.9e-12) | 69% (1.4e-9)
| Optimization-Based
| DIA | CNNMRF | Gatys | STROTSS
NNST-D | 61% (4.1e-4) | 64% (1.6e-7) | 60% (1.0e-3) | 49% (0.66)
NNST-Opt | 61% (2.5e-4) | 82% ($<$1e-15) | 65% (1.3e-6) | 55% (1.2e-2)
Table 5.1: The percentage of votes received by NNST in our forced choice user
study when benchmarked against prior work. In parentheses is the p-value of
rejecting the null hypothesis that the preference rate for NNST is less than
50%.

####  5.4.3 Limitations

Our approach performs well in general but there is nonetheless areas where
there remains room for improvement. For instance, physical phenomena like the
drips of paint on the bear in Figure 5.18 are not reproduced. Also stylistic
features based on long range correlations such as the lines and hatching
patterns in Figure 5.15 (second and last rows) are not accurately reproduced.
These cases are challenging for all methods, and while NNST makes progress
relative to prior work, we believe that developing approaches which better
mimic the consistency of artwork (both imposed by an artist and by the
physical properties of media such as paint) remains an important unsolved
problem in artistic style transfer.

We also observe that there is still a marked gap between the visual quality of
outputs produced by NNST-D and NNST-Opt. Reducing this disparity will be key
to creating a high-quality practical stylization algorithm.

###  5.5 Extensions and Applications

Although the primary goal of our approach is to perform a generic style
transfer without requiring additional knowledge about the style and content
image, in the case when such information is available our technique can be
easily extended to incorporate it. In this scenario we follow the concept of
Image Analogies [40] and extend our objective (5.1) by adding a term that
incorporates further guidance on top of the cosine distance:

$T_{i}=\arg\min_{\Phi^{\prime}(S)_{j}}w_{\cos}D\Big{(}\Phi(C)_{i}-\mu_{C},\hskip
5.69046pt\Phi^{\prime}(S)_{j}-\mu^{\prime}_{S}\Big{)}+w_{guide}D^{g}(C^{g}_{i},S^{g}_{j}).$

Here $S^{g}_{i}$ and $C^{g}_{i}$ are downsampled versions of style and content
guiding channels (e.g., segmentation masks, see Figure 5.18), $D^{g}$ is a
metric which evaluates guide similarity at pixels $i$ and $j$ (in our
experiments we use sum of squared differences), and $w_{\cos}$ and $w_{guide}$
are weights that balance the influence of the cosine and guiding term (in our
experiments we set $w_{\cos}=0.5$ and $w_{guide}=0.5$). In Figure 5.18 we
demonstrate the effect of incorporating additional segmentation masks as a
guiding channels. Doing so forces features from the style to be transferred in
a more predictable and semantically meaningful way. In contrast to previous
neural approaches that also support guidance [30, 63] our technique better
preserves visual aspects of the original style exemplar.

Figure 5.18: Incorporating additional guidance (segmentation masks) into NNST-
Opt. In contrast to unguided version the style of the output sementically
matches the exemplar, i.e., background texture in the stylized image
corresponds to the background in the style exemplar, etc. When compared to the
current state-of-the-art in neural style transfer that also support guidance
[30, 63] our approach better preserves stylistic details.

Besides single image style transfer our approach is practical also in the
context of example-based video stylization [49, 106] where the aim is to
propagate the style from a sparse set of stylized keyframes to the rest of the
video sequence. In the original setting, the stylization of keyframes is
tedious as those need to be created by hand to stay perfectly aligned with the
content in the video. Using our approach, however, one can stylize the entire
sequence fully automatically without the need to preserve alignment. By
transferring the style from an arbitrary exemplar image one can stylize a
subset of frames and then run an existing keyframe-based video stylization
technique of Jamriška et al. [49] or Texler et al. [106] to propagate the
style to the rest of the sequence while maintaining temporal coherence (see
Figure 5.19 and online video).

Figure 5.19: Our approach used as a generator of stylized frames for example-
based video stylization—a few selected keyframes are stylized using NNST-Opt
and the rest of the sequence is stylized using the method of Jamriška et al.
[49]. Video results are available online.

One of the limiting factors of NNST is that the memory constraints of
currently available GPUs mean that it can only deliver outputs of moderate
resolution (up to 1k). To obtain higher resolution images NNST can be plugged
into the method of Texler et al. [105]. In this approach the result of neural
style transfer is used as a guide to drive patch-based synthesis algorithm of
Fišer et al. [22], producing a high-resolution counterpart of the stylized
image generated by the neural method (in Figure 5.20 nearest neighbor field is
upsampled to obtain a 4K output). However, a compromise here is that when
comparing middle scale features NNST performs better than patch-based
synthesis of [105] since it can adapt the style features to follow salient
structures visible in the content image.

Figure 5.20: NNST-Opt combined with the method of Texler et al. [105]. the
output of NNST-Opt is used as a guide for patch-based synthesis algorithm that
can operate at far higher resolution, in this case producing a 4K resolution
output. Producing outputs at very high-resolutions allows subtle but important
features of physical media (e.g. craquelure) to be reproduced. However, patch
synthesis cannot replicate the ability of NNST (and other neural methods) to
adapt the style features to salient contours of the content, resulting in
muddier middle frequencies (e.g. brushstrokes) in the 4k output.

###  5.6 Conclusion

We have demonstrated a conceptually simple approach to artistic stylization of
images, and explored several key design choices to motivate our algorithm. We
showed qualitatively and quantitatively that our approach is flexible enough
to support various scenarios and produce high-quality results in all these
cases. Put together, we believe that these characteristics make our approach
suitable for practical applications and a solid basis for future work.

##  Chapter 6 Conclusion

This thesis has detailed my efforts to discover better algorithms for artistic
style transfer, in particular by using non-parametric tools to better capture
important visual details of an artwork’s style. Chapter 3 describes a non-
parametric, but statistically well-motivated, style loss leveraging optimal
transport. Chapter 4 describes a non-parametric parameterization of geometry
and proportion based on keypoints which enabled stylization of these spatial
attributes, in contrast to prior work almost exclusively focused on the
stylization of texture. Chapter 5 outlines a framework which uses nearest
neighbors, one of the simplest non-parametric tools, to produce stylizations
with state-of-the-art visual quality, and explores the important design
decisions which lead to this success. The algorithms presented represent
progress in capturing the local textures of a particular style (STROTSS in
Chapter 3, and NNST in Chapter 5) and a first step towards stylizing the
geometry of arbitrary content (DST in Chapter 4); however, I believe the field
of style transfer has a long road to travel before it can truly model an
artist’s style.

All of the algorithms proposed in chapters 3 through 5 focus on the case where
only a single content and style example are available at inference. Ultimately
defining ‘style’ using a single image is an artificial constraint that does
not align well with peoples’ typical usage of the word. ‘Style’ is rarely
defined by just one artwork, instead it’s usually based on similarities within
a body of work created by one, or even multiple, artists. Defining style using
multiple images can help disentangle style from content [96, 64] by revealing
which visual features are consistently present, and which shift based on
content. While methods which currently take advantage of this are tuned to a
particular artist or a small number of similar artists, they produce
compelling results, making a strong empirical case that style should be
defined using multiple images. A challenge for future work is discovering
methods which can efficiently and effectively make use of a variable number of
novel style examples at inference. In addition, I believe that fully taking
advantage of the style images available will require methods which explicitly
leverage the semantics of artwork.

Figure 6.1: Example of the quality of style transfer created by human artists.
Achieving this seems impossible without semantic understanding of both the
content and style image. Semantic understanding would help explain how an
artist renders the content’s hair and shoes in style of the Simpsons despite
there being no close analogue in the style exemplar. Image credit Kevin
Mcshane.

One of the largest gaps between style transfer algorithms today and most
artist’s creative process is that artists will alter or generalize style based
on the semantics of the object to be represented. Figure 6.1 displays a
content image of artist Kevin McShane, a style image of Homer Simpson, and a
self-portrait drawn by McShane in the style of the Simpsons. What would be
required for an algorithm to produce an output of similar quality to McShane
given access to the single content image and style image? This feat is
probably super-human, McShane has obviously seen himself in numerous
poses/lighting conditions, and has almost certainly seen a large number of
images from the Simpsons. This allows him to render his hair and shoes in
style of the Simpsons despite there being no close analogue in the style
exemplar. It seems almost certain that stylization on the level of a human
artist will require semantic understanding of both the content and style
images. The semantics of objects in the content image must be understood so
they remain recognizable in the final output. The semantics of objects the
style image must be understood so that their appearance can be compared with
photorealistic analogues, giving some hope of inferring the underlying rules
the artist follows when depicting an object, and generalizing these rules to
objects not present in the original artwork.

Semantics are not the only aspect of scene understanding important to modeling
artistic style. As discussed in Chapter 4, shape and proportion play an
important role in art, and many artists’ styles are influenced by the
fundamentally three-dimensional nature of the scenes represented. This
manifests not only through artistic modifications of a scene’s underlying
geometry, but also through the perspective from which a viewer is shown the
rendered scene (close to the main subject, far from it, looking upwards,
etc.). Ultimate success in style transfer will likely require explicitly
representing the underlying geometry of content images, and the implied
geometry of artworks.

This implies that style transfer must become more closely tied to mainstream
computer vision tasks such as recognition, localization, and depth prediction.
Not only will this benefit style transfer, developing systems which can parse
the semantics and geometry of artwork can lead to models which better mimic
humans’ robust ability to understand scenes. Systems trained using large
datasets of photographs improve every year on benchmarks where the test data
matches the training distribution, but performance drops catastrophically when
the distribution of test data shifts [39]. This is in stark contrast to
humans, who can typically understand semantics, even in artistic styles they
have never seen before. When we go to the art museum, it is common to
encounter works by unfamiliar artists who produce images in a visually
distinctive style. Yet we are easily, often automatically, able to recognize
the semantic contents of these images. Representational art, unlike natural
imagery, is a product of the artist’s internal content representations. Not
only that, it is specially designed to be recognizable to other humans. These
properties make representational art an invaluable window into the invariances
of the human visual system. I hope that future work, my own and others’, will
continue to explore and leverage the connection between our capabilities in
recognition and creativity.

## References

  * [1] Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or.  Neural best-buddies.  ACM Transactions on Graphics, 37(4):1–14, Jul 2018.
  * [2] Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or.  Neural best-buddies: Sparse cross-domain correspondence.  ACM Transactions on Graphics (TOG), 37(4):69, 2018.
  * [3] Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jiebo Luo.  Artflow: Unbiased image style transfer via reversible neural flows.  2021\.
  * [4] Michael Ashikhmin.  Synthesizing natural textures.  In Symposium on Interactive 3D Graphics, pages 217–226, 2001.
  * [5] Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu.  PatchTable: Efficient patch queries for large datasets and applications.  ACM Transactions on Graphics, 34(4):97, 2015.
  * [6] James R Bergen and Edward H Adelson.  Early vision and texture perception.  Nature, 333(6171):363–364, 1988.
  * [7] Guillaume Berger and Roland Memisevic.  Incorporating long-range consistency in cnn-based texture generation.  arXiv preprint 1606.01286, 2016.
  * [8] Terry Caelli, Bela Julesz, and Edgar Gilbert.  On perceptual analyzers underlying visual texture discrimination: Part ii.  Biological Cybernetics, 29(4):201–214, 1978.
  * [9] Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua.  Stylebank: An explicit representation for neural image style transfer.  CVPR, pages 2770–2779, 2017.
  * [10] Tian Qi Chen and Mark Schmidt.  Fast patch-based style transfer of arbitrary style.  arXiv preprint 1612.04337, 2016.
  * [11] Li Cheng, SV N Vishwanathan, and Xinhua Zhang.  Consistent image analogies using semi-supervised learning.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [12] Tai-Yin Chiu.  Understanding generalized whitening and coloring transform for universal style transfer.  In ICCV, pages 4452–4460, 2019.
  * [13] Tai-Yin Chiu and Danna Gurari.  Iterative feature transformation for fast and versatile universal style transfer.  In ECCV, 2020.
  * [14] Forrester Cole, David Belanger, Dilip Krishnan, Aaron Sarna, Inbar Mosseri, and William T. Freeman.  Synthesizing normalized faces from facial identity features.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [15] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  * [16] Jeremy S. De Bonet.  Multiresolution sampling procedure for analysis and synthesis of texture images.  In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’97, page 361–368, USA, 1997\. ACM Press/Addison-Wesley Publishing Co.
  * [17] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur.  A learned representation for artistic style.  CoRR, abs/1610.07629, 2016.
  * [18] Alexei A Efros and William T Freeman.  Image quilting for texture synthesis and transfer.  In SIGGRAPH, pages 341–346. ACM, 2001.
  * [19] Alexei A Efros and Thomas K Leung.  Texture synthesis by non-parametric sampling.  In CVPR, volume 2, pages 1033–1038. IEEE, 1999.
  * [20] James Elkins.  Style.  The Dictionary of art, 29, 1996.
  * [21] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Visualizing higher-layer features of a deep network.  University of Montreal, 1341(3):1, 2009.
  * [22] Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sỳkora.  Stylit: illumination-guided example-based stylization of 3d renderings.  In TOG, volume 35, pages 1–11. ACM New York, NY, USA, 2016.
  * [23] Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sýkora.  Example-based synthesis of stylized facial animations.  ACM Transactions on Graphics, 36(4):155, 2017.
  * [24] Alain Fournier, Don Fussell, and Loren Carpenter.  Computer rendering of stochastic models.  Communications of the ACM, 25(6):371–384, 1982.
  * [25] Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier.  Split and match: Example-based adaptive patch sampling for unsupervised style transfer.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 553–561, 2016.
  * [26] Kunihiko Fukushima and Sei Miyake.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In Competition and cooperation in neural nets, pages 267–285. Springer, 1982.
  * [27] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Texture synthesis using convolutional neural networks.  arXiv preprint arXiv:1505.07376, 2015.
  * [28] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In CVPR, pages 2414–2423, 2016.
  * [29] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In CVPR, pages 2414–2423, 2016.
  * [30] Leon A Gatys, Alexander S Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman.  Controlling perceptual factors in neural style transfer.  In CVPR, 2017.
  * [31] Chris A Glasbey and Kantilal Vardichand Mardia.  A review of image-warping methods.  Journal of applied statistics, 25(2):155–171, 1998.
  * [32] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, pages 2672–2680, 2014.
  * [33] Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan.  Arbitrary style transfer with deep feature reshuffle.  In CVPR, pages 8222–8231, 2018.
  * [34] B. Guo, H. Shum, and Y. Xu.  Chaos mosaic: Fast and memory efficient texture synthesis.  2000\.
  * [35] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik.  Hypercolumns for object segmentation and fine-grained localization.  In CVPR, pages 447–456, 2015.
  * [36] Paul Harrison.  A non-hierarchical procedure for re-synthesis of complex textures.  2001\.
  * [37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [38] David J Heeger and James R Bergen.  Pyramid-based texture analysis/synthesis.  In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 229–238, 1995.
  * [39] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  arXiv preprint arXiv:2006.16241, 2020.
  * [40] Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin.  Image analogies.  In SIGGRAPH, pages 327–340. ACM, 2001.
  * [41] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin.  Image analogies.  In SIGGRAPH Conference Proceedings, pages 327–340, 2001.
  * [42] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
  * [43] D. Hofstadter.  Metamagical themas: Variations on a theme as the essence of imagination.  Scientific American, 247(4), 1983.
  * [44] Xun Huang and Serge J. Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  ICCV, pages 1510–1519, 2017.
  * [45] Xun Huang and Serge J Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  2017\.
  * [46] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European conference on computer vision (ECCV), pages 172–189, 2018.
  * [47] Philip TG Jackson, Amir Atapour Abarghouei, Stephen Bonner, Toby P Breckon, and Boguslaw Obara.  Style augmentation: data augmentation via style randomization.  In CVPR Workshops, pages 83–92, 2019.
  * [48] Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora.  LazyFluids: Appearance transfer for fluid animations.  In ACM Transactions on Graphics, volume 34, page 92, 2015.
  * [49] Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora.  Stylizing video by example.  ACM Transactions on Graphics, 38(4):107, 2019.
  * [50] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen.  Dynamic instance normalization for arbitrary style transfer.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4369–4376, 2020.
  * [51] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song.  Neural style transfer: A review.  IEEE transactions on visualization and computer graphics, 26(11):3365–3385, 2019.
  * [52] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, pages 694–711, 2016.
  * [53] Bela Julesz.  Texture and visual perception.  Scientific American, 212(2):38–49, 1965.
  * [54] Bela Julesz.  Experiments in the visual perception of texture.  Scientific American, 232(4):34–43, 1975.
  * [55] Bela Julesz.  Textons, the elements of texture perception, and their interactions.  Nature, 290(5802):91–97, 1981.
  * [56] Bela Julesz.  A theory of preattentive texture discrimination based on first-order statistics of textons.  Biological Cybernetics, 41(2):131–138, 1981.
  * [57] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [58] Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf.  Self tuning texture optimization.  In CGF, volume 34, pages 349–359. Wiley Online Library, 2015.
  * [59] Tsung-Wei Ke, Michael Maire, and Stella X Yu.  Multigrid neural architectures.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6665–6673, 2017.
  * [60] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn.  Fcss: Fully convolutional self-similarity for dense semantic correspondence.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6560–6569, 2017.
  * [61] Sunnie SY Kim, Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich.  Deformable style transfer.  ECCV, 2020.
  * [62] Diederik P Kingma and J Adam Ba.  Adam: A method for stochastic optimization.  ICLR, 2015.
  * [63] Nicholas Kolkin, Jason Salavon, and Greg Shakhnarovich.  Style transfer by relaxed optimal transport and self-similarity.  In CVPR, pages 10051–10060, 2019.
  * [64] Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjorn Ommer.  Content and style disentanglement for artistic style transfer.  In ICCV, pages 4422–4431, 2019.
  * [65] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [66] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger.  From word embeddings to document distances.  In International Conference on Machine Learning, pages 957–966, 2015.
  * [67] Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel.  Handwritten digit recognition with a back-propagation network.  In Advances in neural information processing systems, pages 396–404, 1990.
  * [68] John-Peter Lewis.  Algorithms for solid noise synthesis.  In Proceedings of the 16th annual conference on Computer graphics and interactive techniques, pages 263–270, 1989.
  * [69] Chuan Li and Michael Wand.  Combining markov random fields and convolutional neural networks for image synthesis.  In CVPR, pages 2479–2486, 2016.
  * [70] Wenbin Li, Wei Xiong, Haofu Liao, Jing Huo, Yang Gao, and Jiebo Luo.  Carigan: Caricature generation through weakly paired adversarial learning.  CoRR, abs/1811.00445, 2018.
  * [71] Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang.  Learning linear transformations for fast arbitrary style transfer.  arXiv preprint arXiv:1808.04537, 2018.
  * [72] Yijun Li, Chen Fang, Aaron Hertzmann, Eli Shechtman, and Ming-Hsuan Yang.  Im2pencil: Controllable pencil illustration from photographs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1525–1534, 2019.
  * [73] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Universal style transfer via feature transforms.  In NIPS, pages 385–395, 2017.
  * [74] Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung-Yeung Shum.  Real-time texture synthesis by patch-based sampling.  ACM Transactions on Graphics, 20(3):127–150, 2001.
  * [75] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang.  Visual attribute transfer through deep image analogy.  SIGGRAPH, 2017.
  * [76] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [77] Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang.  A closed-form solution to universal style transfer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5952–5961, 2019.
  * [78] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala.  Deep photo style transfer.  In CVPR, 2017.
  * [79] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.  Rectifier nonlinearities improve neural network acoustic models.  In Proc. icml, volume 30, page 3. Citeseer, 2013.
  * [80] Aravindh Mahendran and Andrea Vedaldi.  Understanding deep image representations by inverting them.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188–5196, 2015.
  * [81] Jitendra Malik and Pietro Perona.  Preattentive texture discrimination with early vision mechanisms.  JOSA A, 7(5):923–932, 1990.
  * [82] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proc. 8th Int’l Conf. Computer Vision, volume 2, pages 416–423, July 2001.
  * [83] Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor.  The contextual loss for image transformation with non-aligned data.  In ECCV, pages 768–783, 2018.
  * [84] Gaspard Monge.  Mémoire sur la théorie des déblais et des remblais.  Histoire de l’Académie Royale des Sciences de Paris, 1781.
  * [85] Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory Shakhnarovich.  Feedforward semantic segmentation with zoom-out features.  In CVPR, pages 3376–3385, 2015.
  * [86] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2017.  https://distill.pub/2017/feature-visualization.
  * [87] Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand.  Bilateral filtering: Theory and applications.  Now Publishers Inc, 2009.
  * [88] Dae Young Park and Kwang Hee Lee.  Arbitrary style transfer with style-attentional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5880–5888, 2019.
  * [89] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [90] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A Efros, and Richard Zhang.  Swapping autoencoder for deep image manipulation.  arXiv preprint arXiv:2007.00653, 2020.
  * [91] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [92] Javier Portilla and Eero P. Simoncelli.  A parametric texture model based on joint statistics of complex wavelet coefficients.  International Journal of Computer Vision, 40(1):49–70, 2000.
  * [93] Eric Risser, Pierre Wilmot, and Connelly Barnes.  Stable and controllable neural texture synthesis and style transfer using histogram losses.  arXiv preprint 1701.08893, 2017.
  * [94] Rómer Rosales, Kannan Achan, and Brendan J Frey.  Unsupervised image translation.  In iccv, pages 472–478, 2003.
  * [95] Babak Saleh and Ahmed Elgammal.  Large-scale classification of fine-art paintings: Learning the right metric on the right feature.  arXiv preprint arXiv:1505.00855, 2015.
  * [96] Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Bjorn Ommer.  A style-aware content loss for real-time hd style transfer.  In ECCV, pages 698–714, 2018.
  * [97] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [98] Eli Shechtman and Michal Irani.  Matching local self-similarities across images and videos.  In CVPR.
  * [99] Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang.  Avatar-net: Multi-scale zero-shot style transfer by feature decoration.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8242–8250, 2018.
  * [100] Yichun Shi, Debayan Deb, and Anil K. Jain.  Warpgan: Automatic caricature generation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [101] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [102] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  2014\.
  * [103] Peter-Pike J. Sloan, William Martin, Amy Gooch, and Bruce Gooch.  The Lit Sphere: A model for capturing NPR shading from art.  In Graphics Interface, pages 143–150, 2001.
  * [104] Jan Svoboda, Asha Anoosheh, Christian Osendorfer, and Jonathan Masci.  Two-stage peer-regularized feature recombination for arbitrary image style transfer.  In CVPR, pages 13816–13825, 2020.
  * [105] Ondřej Texler, David Futschik, Jakub Fišer, Michal Lukáč, Jingwan Lu, Eli Shechtman, and Daniel Sýkora.  Arbitrary style transfer using neurally-guided patch-based synthesis.  CAG, 2020.
  * [106] Ondřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora.  Interactive video stylization using few-shot patch-based training.  ACM Transactions on Graphics, 39(4):73, 2020.
  * [107] Carlo Tomasi and R. Manduchi.  Bilateral filtering for gray and color images.  International Conference on Computer Vision, pages 839–846, 1998\.
  * [108] Greg Turk.  Generating textures on arbitrary surfaces using reaction-diffusion.  Acm Siggraph Computer Graphics, 25(4):289–298, 1991.
  * [109] Mark R Turner.  Texture discrimination by gabor functions.  Biological cybernetics, 55(2):71–82, 1986.
  * [110] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky.  Texture networks: Feed-forward synthesis of textures and stylized images.  pages 1349–1357, 2016.
  * [111] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv preprint arXiv:1607.08022, 2016.
  * [112] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6924–6932, 2017.
  * [113] Cédric Villani.  Optimal transport: old and new, volume 338.  Springer Science & Business Media, 2008.
  * [114] Huan Wang, Yijun Li, Yuehai Wang, Haoji Hu, and Ming-Hsuan Yang.  Collaborative distillation for ultra-resolution universal style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1860–1869, 2020.
  * [115] Xin Wang, Geoffrey Oxholm, Da Zhang, and Yuan-Fang Wang.  Multimodal transfer: A hierarchical deep convolutional neural network for fast artistic style transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5239–5247, 2017.
  * [116] Li-Yi Wei and Marc Levoy.  Fast texture synthesis using tree-structured vector quantization.  In SIGGRAPH, pages 479–488, 2000.
  * [117] Andrew Witkin and Michael Kass.  Reaction-diffusion textures.  In Proceedings of the 18th annual conference on computer graphics and interactive techniques, pages 299–308, 1991.
  * [118] Jordan Yaniv, Yael Newman, and Ariel Shamir.  The face of art: Landmark detection and geometric style in portraits.  ACM Trans. Graph., 38(4):60:1–60:15, July 2019.
  * [119] Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang.  Attention-aware multi-stroke style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1467–1475, 2019.
  * [120] Mao-Chuang Yeh, Shuai Tang, Anand Bhattad, and David A Forsyth.  Quantitative evaluation of style transfer.  arXiv preprint 1804.00118, 2018.
  * [121] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In ECCV, pages 818–833. Springer, 2014.
  * [122] Richard Zhang.  Making convolutional networks shift-invariant again.  arXiv preprint 1904.11486, 2019.
  * [123] Wei Zhang, Chen Cao, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang.  Style transfer via image component analysis.  IEEE Transactions on multimedia, 15(7):1594–1601, 2013.
  * [124] Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang.  Multimodal style transfer via graph cuts.  In ICCV, pages 5943–5951, 2019.
  * [125] Xu Zheng, Tejo Chalasani, Koustav Ghosal, Sebastian Lutz, and Aljosa Smolic.  Stada: Style transfer as data augmentation.  arXiv preprint arXiv:1909.01056, 2019.
  * [126] Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal.  Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization.  ACM Transactions on mathematical software (TOMS), 23(4):550–560, 1997.
  * [127] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [128] Song Chun Zhu, Yingnian Wu, and David Mumford.  Frame: Filters, random field and maximum entropy: — towards a unified theory for texture modeling.  International Journal on Computer Vision, pages 1–20, 1998.