11institutetext: EleutherAI 22institutetext: Booz Allen Hamilton
33institutetext: AIDock 44institutetext: Georgia Institute of Technology

# VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language
Guidance

Katherine Crowson Co-first authors11 Stella Biderman∗ 1122 Daniel Kornis 33
Dashiell Stander 11 Eric Hallahan 11 Louis Castricato 1144 Edward Raff 22

###### Abstract

Generating and editing images from open domain text prompts is a challenging
task that heretofore has required expensive and specially trained models. We
demonstrate a novel methodology for both tasks which is capable of producing
images of high visual quality from text prompts of significant semantic
complexity without any training by using a multimodal encoder to guide image
generations. We demonstrate on a variety of tasks how using CLIP [40] to guide
VQGAN [12] produces higher visual quality outputs than prior, less flexible
approaches like minDALL-E [20], GLIDE [36] and Open-Edit [26], despite not
being trained for the tasks presented. Our code is available in a public
repository.

###### Keywords:

generative adversarial networks; grounded language; image manipulation

##  1 Introduction

Using free-form text to generate or manipulate high-quality images is a
challenging task, requiring a grounded learning between visual and textual
representations. Manipulating images in an open domain context was first
proposed by the seminal Open-Edit [26], which allowed text prompts to alter an
image’s content. This was done mostly with semantically simple transformations
(e.g., turn a red apple green), and does not allow generation of images. Soon
after DALL-E [41] and GLIDE [36] were developed, both of which can perform
generation (and inpainting) from arbitrary text prompts, but do not themselves
enable image manipulation.

In this work we propose the first a unified approach to semantic image
generation and editing, leveraging a pretrained joint image-text encoder [40]
to steer an image generative model [12]. Our methodology works by using the
multimodal encoder to define a loss function evaluating the similarity of a
(text, image) pair and backpropagating to the latent space of the image
generator. We iteratively update the candidate generation until it is
sufficiently similar to the target text. The difference between using our
technique for generation and editing is merely a matter of initializing the
generator with a particular image (for editing) or with random noise (for
generation).

A significant advantage of our methodology is the lack of additional training
required. Only a pretrained image generator and a joint image-text encoder are
necessary, while all three of [26, 41, 36] require training similar models
from scratch. Additionally [41, 36] train generators from scratch.

We demonstrate several significant contributions, including:

  1. 1.

High visual quality for both generation and manipulation of images.

  2. 2.

High semantic fidelity between text and generation, especially when
semantically unlikely content co-occurs.

  3. 3.

Efficiency in that our method requires no additional training beyond the pre-
trained models, using only a small amount of optimization per inference.

  4. 4.

The value of open development and research. This technique was developed in
public and open collaboration has been integral to its rapid real-world
success. Non-authors have already extended our approach to other modalities
(e.g., replacing text for audio) and commercial applications.

The rest of our manuscript is organized as follows. In Section 2 we discuss
how of how our methodology works, resulting in a simple and easy-to-apply
approach for combing multiple modalities for generation or manipulation. The
efficacy of vqgan-clip in generating high quality and semantically relevant
images is shown in Section 3, followed by superior manipulation ability in
Section 4. The design choices of vqgan-clip to obtain both high image quality
and fast generation are validated by ablations in Section 5, and Section 6
discusses resource usage and efficiency considerations. As our approach has
been public since April 2021, we are able to show further validation by
external groups in Section 7. This use includes extensions to other
modalities, showing the flexibility of our approach, as well as commercial use
of vqgan-clip that demonstrate its success at handling open-domain prompts and
images to a satisfying degree. Finally we conclude in Section 8.

##  2 Our Methodology

To demonstrate our method’s effectiveness we apply it using VQGAN [12] and
CLIP [40] as pre-trained models, and so refer to our approach as vqgan-clip.
We stress, however, that our approach is not specific to either model and that
subsequent work has already shown success that builds on our work using other
models [6, 13, 30, 50], and even in other modalities [19, 54].

We start with a text prompt and use a GAN to iteratively generate candidate
images, at each step using CLIP to improve the image. We optimize the image by
treating the squared spherical distance between the embedding of the candidate
and the embedding of the text prompt as a loss function, and differentiating
through CLIP with respect to the GAN’s latent vector representation of the
image, which we refer to as the “z-vector” following [38]. This process is
outlined in Fig. 1

Figure 1: Diagram showing how augmentations are added to stabilize and improve
the optimization. Multiple crops, each with different random augmentations,
are applied to produce an average loss over a single source generation. This
improves the results with respect to a single latent Z-vector.

To generate an image, the “initial image” contains random pixel values. The
optimization process is repeated to alter the image, until the output image
gradually improves such that it semantically matches the target text. We can
also edit existing images by starting with the image-to-edit as the “initial
image”. The text prompt used to describe how we want the image to change is
used identically to the text prompt for generating an image, and no changes to
the architecture exist between generation and manipulation besides how the
‘initial image” is selected.

We use Adam [21] to do the actual optimization, a learning rate of $0.15$,
$\beta=(0.9,0.999)$, and run for 400 iterations for the experiments in this
paper.

###  2.1 Discrete Latent Spaces for Images

Unlike the naturally discrete nature of text, the space of naturally occurring
images is inherently continuous and not trivially discretized. Prior work by
[38] borrows techniques from vector quantization (VQ) to represent a variety
of modalities with discrete latent representations by building a codebook
vocabulary with a finite set of learned embeddings. Given a codebook of
vocabulary size $K$ with embedding dimension $n_{k}$,
$\mathcal{Z}=\\{z_{i}\\}_{k}^{K}\in\mathbb{R}^{n_{k}}$.

This is applied to images by constructing a convolutional autoencoder with
encoder $\mathcal{E}$ and decoder $\mathcal{G}$. An input image $x\in I$ is
first embedded with the encoder $z=E(x)$. We can then compute the vector
quantized embedding $x$ as

$z_{q}=\operatorname*{argmin}_{z_{k}\in\mathcal{Z}}\|z_{i,j}-z_{k}\|$

which we can then multiply back through the vocabulary in order to perform
reconstruction. We can then use a straight-through estimator on the
quantization step in order to allow the CNN and codebook to be jointly trained
end-to-end. We use the popular VQGAN [12] model for the experiments in this
paper.

###  2.2 Contrastive Text-Image Models

To guide the generative model, we need a way to adjust the similarity of a
candidate generation with the guidance text. To achieve this, we use CLIP,
[40], a joint text-image encoder trained by using contrastive learning. We use
CLIP to embed text prompts and candidate generated images independently and
measure the cosine similarity between the embeddings. This similarity is then
reframed as a loss that we can use gradient descent to minimize.

###  2.3 Augmentations

One challenge of using vqgan-clip is that gradient updates from the CLIP loss
are quite noisy if calculated on a single image. To overcome this we take the
generated candidate image and modify it many times, producing a large number
of augmented images. We take random crops of the candidate image and then
apply further augmentations such as flipping, color jitter, noising, etc.
[42]. Most high level semantic features of an image are relatively invariant
to these changes, so averaging the CLIP loss with respect to all of the
augmented images reduces the variance of each update step. There is a risk
that a random crop might dramatically change the semantic content of an image
(e.g. by cropping out an important object), but we find that in practice this
does not cause any issues.

For the results presented in this paper we used an augmentation pipeline
consisting of: random horizontal flips, random affine projections, random
perspective projections, random color jitter, and adding random Gaussian
noise.

###  2.4 Regularizing the Latent Vector

When using an unconstrained VQGAN for image generation, we found that outputs
tended to be unstructured. Adding augmentations helps with general coherence,
but the final output will often still contain patches of unwanted textures. To
solve this problem we apply a weighted $L^{2}$ regularization to the the
z-vector.

This produces a regularized loss function given by the equation

$\mathit{Loss}=\mathit{L}_{\mathit{CLIP}}+\alpha\cdot\frac{1}{N}\sum_{i=0}^{N}Z_{i}^{2}$

where $\alpha$ is the regularization weight. This encourages parsimony in the
representation, sending low information codes in VQGAN’s codebook to zero. In
practice we note that regularization appears to improve the coherence of the
output and produces a better structured image. We decay the regularization
term by $0.005$ over the course of generation.

###  2.5 Additional Components

Our methodology is highly flexible and can be extended straightforwardly
depending on the use-case and context due to the ease of integrating
additional interventions on the intermediate steps of image generation.
Researchers using our framework have introduced a number of additional
components, ranging from using ensembles [7], to using Bézier curves for
latent representations [14, 6], to using perturbations to make the results
more robust to adversaries [27]. Although they aren’t used in the main
experiments of this paper, we wish to call attention to two in particular that
we use frequently: “prompt addition” and masked image editing. We give an
overview of both here, and provide additional experiments and information in
Appendix 0.G

##### Prompt Addition:

We have found that our users are often interested in applying multiple text
prompts at the same time. This can be achieved by computing the loss against
multiple target texts simultaneously and adding the results. In Section 0.G.1
we use this tool to explore the semantic cohesion of vqgan-clip’s generations.

##### Masking:

A common technique in image generation and editing is masking, where a portion
of an image is identified ahead of time as being where a model should
edit111In the context of image this is often referred to as “infilling,” but
we will use “masking” as a general term to refer to both. vqgan-clip is
compatible with masking by zeroing out the gradients in parts of the latent
vector that one wishes to not change. However vqgan-clip can also leverage the
semantic knowledge of CLIP to perform self-masking without any non-textual
human input.

##  3 Semantic Image Generation

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 2: Example vqgan-clip generations and their text prompts. Prompts
selected to demonstrate a range of visual styles that vqgan-clip is capable of
producing including classical art (g, i), modern art (l), drawings (e), oils
(a), and others not included due to space.

The primary application of our methodology is for generating images from text.
In contrast to previous work on this topic [41, 36, 53], we do not perceive
creating photo-realistic images or images that could convince a human that
they are real photographs as our primary goal. Our focus is on producing
images of high visual quality that are semantically meaningful in relation to
a natural language prompt, which we demonstrate in this section. This in fact
requires abandoning photo-realism when prompts may ask for artistic or
explicitly unrealistic generations and edits. A loosely curated set of example
generations is presented in Fig. 2.

As vqgan-clip has been publicly available for almost a year, we have had the
opportunity to observe people experimenting with and building off of vqgan-
clip in the wild. In Appendix 0.D we show a sample of artwork created by
people other than the authors of this paper are included to demonstrate the
power and range of vqgan-clip.

###  3.1 Artistic Impressions

We find that vqgan-clip is able to evoke the artistic style of famous artists
and major artistic styles from around the world. Fig. 2 features “an astronaut
in the style of van Gogh” whose background evokes Starry Night and “the Tower
of Babel by J. M. W. Turner” which draws on Turner’s color palate and use of
light. Another way this can be seen is by directly asking for “a painting by
[name]” or “art by [name].” In Fig. 3 we present six images created this way
drawing on artists from different regions, time periods, and artistic styles.
While the images often are missing cohesion (most likely due to the vagueness
of “a painting” as a prompt) they are each markedly reminiscent of the artist
in question. While CLIP would obviously find these images visually similar to
other works by the artist, we also find that non-CLIP-based image similarity
approaches reliably identify these images as visually similar to work by the
artists. To validate this we queried Google’s Reverse Image Search using each
generation in Fig. 3, and in every case a real painting by the target artist
was the most similar image.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Stylistic impressions of famous artists. Third party tools like
Google’s Reverse Image Search indicate that real paintings by the target
artists are the most visually similar images in every case.

###  3.2 Comparisons to Other Approaches

The closest prior work in open domain generation of images comes from DALL-E
[41] and GLIDE [36], which claim to train very large pretrained text-to-image
models. DALL-E and GLIDE are purported to be 12 billion and 5 billion
parameters, respectively, while vqgan-clip together is 227 million.
Unfortunately, we were denied permission to study the models purported in the
respective papers by their authors. We instead use the verifiable state-of-
the-art models using each methodology methodologies. This includes the
independent implementation minDALL-E [20] (1.3 B parameters) and two versions
of GLIDE (783 M parameters without CLIP and 941 M with) released by
OpenAI222It is important to stress that at time of writing there was no
verifiable evidence for the existence or performance of either of these models
other than the word of OpenAI employees. That such assertions are taken
seriously as science is a symptom of the pervasive power disparities across
machine learning research [5, 23, 29, 49].

To evaluate our model, we recruited humans and asked them to rate the
alignment of (text, image) pairs on a scale of 1 (low) to 5 (high). In
particular, they were directed to rate higher quality images that do not match
the prompt lower than lower quality images that do. Prompts were selected
based on principles learned from our experience working with these models but
without prior knowledge of how the models would behave on the particular
prompts in question. All prompts and generated images can be found in Appendix
0.D. To provide the maximal advantage to our competitors, minDALL-E and GLIDE
examples are cherry-picked best-of-five, while vqgan-clip examples are
uncherry-picked (best-of-one). Table 1 shows the mean score per prompt for
each model. We find that humans overwhelmingly view the generations using our
technique as more aligned with the input text.

| A | B | C | D | E | F | G | H | I | J | K | L | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
minDALL-E | 3.3 | 2.3 | 3.2 | 3.7 | 1.5 | 2.7 | 2.2 | 1.3 | 3.3 | 3.0 | 3.5 | 2.3 | 2.7
GLIDE (CF) | 3.0 | 4.0 | 2.7 | 3.2 | 1.3 | 2.5 | 1.3 | 1.2 | 2.0 | 2.3 | 2.0 | 2.8 | 2.3
GLIDE (CLIP) | 3.2 | 4.0 | 2.8 | 4.8 | 3.0 | 2.7 | 1.8 | 2.5 | 3.7 | 2.3 | 3.7 | 5.0 | 3.3
VQGAN-CLIP | 4.3 | 5.0 | 4.8 | 5.0 | 4.5 | 4.5 | 4.8 | 4.7 | 4.2 | 3.8 | 4.7 | 4.7 | 4.6
Table 1: Mean human ratings of generations by each model on a score of 1
(worst) to 5 (best).

###  3.3 Qualitative Analysis

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(a)

(b)

(c)

(d)

Figure 4: Text based generations of images. Top to bottom: minDALL-E, GLIDE
(CLIP-guided), GLIDE (CF-guided), and our vqgan-clip.

A sampling of representative results is shown in Fig. 4 for four different
prompts using minDALL-E, two variants of GLIDE, and our vqgan-clip. Further
comparisons, including the prompts in Fig. 2, can be found in Appendix 0.E. We
find that the minDALL-E and the GLIDE models are much more variable in the
quality of their generations. While they are able to produce images that are
clearly recognizable in response to the prompts “the universal library
trending on artstation” and “a charcoal drawing of a cathedral,” their
generations in response to “a child’s drawing of a baseball game” are largely
unrecognizable and their responses to “a forest rendered in low poly” ignore
the later half of the prompt. These latter cases demonstrate the low semantic
relevance of prior methods’ output given the prompt.

The “child’s drawing” case is of particular note here in that a child’s
drawing is expected to have lower visual clarity and lack of structure. That
vqgan-clip is able to correctly modulate its ability for fine details is thus
of note to show that vqgan-clip is not intrinsically biased toward producing
fine details when inappropriate, and correctly identifies the appropriate
context of multi-part prompts. Further evidence of this can be found in Figs.
11, 12, 13 and 14 where vqgan-clip is able to produce generations for the
prompts “A colored pencil drawing of a waterfall” and “sketch of a 3D printer
by Leonardo da Vinci” that showcase the properties of the medium (visible
strokes, the use of shading, the fact that the image is created on a piece of
paper) while still producing a compelling visual image.

##  4 Semantic Image Editing

As far as we are aware, our framework is the first in the literature to be
able to perform semantic image generation and semantic image editing. There
are other examples in the literature of generative models that can perform
style transfer [39], image inpainting [36, 41], and other types of image
manipulation [37], but we note that each of these represent distinct tasks
from open domain semantic image editing. By contrast, to adapt our generation
methodology to image editing all that is required is to replace the randomly
initialized starting image with the image we wish to edit.

###  4.1 Comparison to State-of-the-Art

For semantic image editing we compare to Open-Edit [26]. As far as we are
aware, Open-Edit is the only published research on open domain semantic image
editing other than our work. To avoid giving any accidental advantage to our
methodology, we focus primarily on the domains presented as examples in [26]
such as changing colors and textures. We use the default settings for their
model and the same prompting structure as in their paper.

####  4.1.1 Color editing

Here we prompt the model to change the dominant color palette without
degrading the image quality or any of the finer details. The results can be
seen in Fig. 5, where prior Open-Edit causes destructive transformations of
the content of the image. In the second case the “Red bus” also shows a single
desired target for manipulation that is respected by vqgan-clip, but Open-Edit
causes a change in coloration of the entire image.

Instruction Original VQGAN-CLIP Open-Edit “Green” “Red Bus”

Figure 5: Examples of editing the color in an image. Original on the left, our
vqgan-clip in the middle, and Open-Edit on the right. vqgan-clip better
maintains original structure of the content while limiting unintended
distortion.

####  4.1.2 Weather Modification

Another use case that [26] highlight as a success of their model is weather
modification, changing the overall weather conditions present in an image.
Results on this task are shown in Fig. 6, where Open-Edit’s reliance on edge
maps to maintain structure show a limitation in editing ability. The needed
alterations often change more of the image content that would violate the edge
maps, preventing Open-Edit from being as successful in achieving the desired
content change.

Instruction Original VQGAN-CLIP Open-Edit “Foggy Sky $\to$ Clear Sky” “Clear
Sky $\to$ Cloudy Sky” “Cloudy $\to$ Sunny”

Figure 6: Weather alteration can required greater alteration of scene
structure that Open-Edit is not able to perform, as shown in the “Cloudy $\to$
Sunny” example that needs to alter the sky in addition to brightness levels.

####  4.1.3 Misc

We include extra miscellaneous examples to emphasize that this is open domain
image editing and the performance is not limited to select types of
transformations. These are shown in Fig. 7, and we note the “wooden” and
“focused” examples demonstrate a task with less correlative semantics. This
further requires a more robust grounding between modalities for success and
the ability of our approach to better handle a breadth of possible inputs for
open-domain prompts and images.

Instruction Original VQGAN-CLIP Open-Edit “Wooden” “Withered Flowers”
“Focused”

Figure 7: More challenging modifications that required greater linguistic
grounding to visual content to achieve, again showing vqgan-clip is better
able to edit image content.

##  5 Ablations on Components

###  5.1 Latent Vector Regularization

Prior versions of this work used a methodology called Codebook Sampling, which
optimizes a categorical distribution over a grid superimposed on the latent
vectors. We found this approach was too slow for interactive use, and did
leave considerable room for visual improvement. In Fig. 8 we show the
improvement in quality of our current approach compared to the prior. By
adding a regularization term Section 2.4, we obtain a methodology that is both
faster and produces higher quality images than Codebook Sampling, giving the
results presented in this paper. Our approach to regularization is
incompatible with Codebook Sampling because of the tendency of Codebook
Sampling to commit to particular codes early in training before the effects of
regularization manifest.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

Figure 8: Comparisons between codebook sampling (top), z-quantize (middle),
and z-quantize with MSE regularization methods (bottom). Notice that
z-quantize with regularization is able to produce finer details than
alternative options, while non-regularized and codebook tend to yield muddled
images.

###  5.2 Number of Augmentations

In Fig. 9 we ablate the number of augmentations to show their importance in
consistent result quality.

no augmentation one augmentation all augmentations a child’s drawing of a
cathedral Random Affine Transformation the universal library, rendered in the
Unreal Engine Random Perspective a cityscape at night, rendered in the 8k
resolution. Random Noise

Figure 9: Examples of the impacts of various augmentations. Left the right:
unaugmented, only the named augmentation, all augmentations.

We find that the affine augmentation reduces clutter and unwanted duplicative
generations, prospective augmentation improves the consistency of the 3D
geometry, the noise augmentation improves the isolation of the foreground from
the background.

##  6 Resource Considerations

Our approach runs in $(935.2\pm 20.4)\,\mathrm{s}$ on an NVIDIA Tesla K80 and
$(229.5\pm 26.2)\,\mathrm{s}$ on an NVIDIA GeForce RTX 2080 Ti (10 runs in
each sample). This is approximately three times slower than minDALL-E and ten
times slower than GLIDE (filtered) in our testing. Although we would like to
analyze the trade-offs involved with the fact that both models required
extensive pretraining none of the papers we compare to report their training
requirements in enough detail to analyze the trade-offs brought about by this
difference. We encourage the authors to release more information about their
models so that more complete analysis can be done.

###  6.1 Efficiency as a Value

One of the goals of this research333The inclusion of this section is heavily
inspired by [4] is to increase the accessibility of AI image generative and
editing tools. We have deliberately limited our approach to something that
requires less than 11 GB of VRAM, so that it fits inside widely available
commercial GPUs such as K80s. This GPU is particularly important from an
accessibility standpoint, as it is the largest GPU that can be easily obtained
using a free account on Google Colaboratory. The full generative process takes
less than 3 minutes in a Google Colab notebook, making this a viable approach
for anyone with access to the internet and a Google account.

Researchers with significantly more resources can obtain higher quality images
using various augmentations left out of this paper, such as using an ensemble
or additional auxiliary models to regularize the generations. While pushing
the performance of our methodology to the maximum is a worthwhile endeavour,
the fact that we can outperform the current state-of-the-art while running on
freely available resources is something that we view as particularly worth
highlighting. We leave determining the optimal framework with unbound
resources to future work.

###  6.2 Runtime Analysis

DALL-E [41], GLIDE [36], and Open-Edit [26] all also incorporate image
generators and joint text-image encoders into their architecture. Unlike our
method however, they require computationally intensive training and
finetuning. This invites the question of trade-offs between training and
inference time. Unfortunately, none of the aforementioned papers report their
training requirements in enough detail to estimate their training
requirements. We are however able to estimate how long minDALL-E [20], the
current state-of-the-art DALL-E model, takes to train at $504$ V100-hours for
the base model plus an additional $288$ V100-hours to finetune on ImageNet
[9]. Through private communication with the authors, we were able to learn
that GLIDE (filtered) required $400$ A100-days to train, which we approximate
as $19,200$ V100-hours for ease of comparison.

Model | K80 | P100 | V100 | Training
---|---|---|---|---
minDALL-E | $216.0s\pm 07.6s$ | $60.0s\pm 5.5s$ | $016.3s\pm 2.7s$ |  $792$ V100-hours
GLIDE (filtered) | $096.2s\pm 00.1s$ | $19.2s\pm 0.3s$ | $009.7s\pm 1.1s$ |  $19,200$ V100-hours
vqgan-clip | $935.2s\pm 20.4s$ | $654.3s\pm 10.1s$ | $188.3s\pm 1.2s$ |  $0$ V100-hours
Table 2: Run-time of minDALL-E, GLIDE (filtered) and vqgan-clip on a variety
of GPUs. Each cell shows the mean and standard deviation of a 10-run sample.
We exclude Open-Edit due to a lack of information about its training costs.
minDALL-E becomes cheaper than vqgan-clip after $858$ V100-hours have been
expended while GLIDE (filtered) requires $20200$ V100-hours.

On all hardwares evaluated, our model is substantially slower than both
minDALL-E and GLIDE (filtered). However. In terms of trade-offs between
training and inference on V100 GPUs, minDALL-E’s total cost becomes cheaper
than vqgan-clip at $\approx 15,800$ generations, while GLIDE(filtered)
requires $\approx 384,000$. In terms of compute expended, minDALL-E becomes
cheaper than vqgan-clip after $858$ V100-hours while GLIDE (filtered) requires
$20200$ V100-hours. While cost and efficiency concerns depend significantly on
individual contexts, the fact that GLIDE (filtered) only becomes as efficient
as vqgan-clip efficient after tens of thousands of dollars of compute have
been expended substantially limits researchers’ ability to experiment with and
iterate on the methodology. The same applies to minDALL-E, albeit with a price
tag in the thousands rather than tens of thousands.

##  7 Adoption of VQGAN-CLIP

A unique aspect of vqgan-clip has been its public development over the past
year, which has resulted in an active community of users and real-world impact
within and beyond classical computer vision. [22, 14, 6, 27, 50] create
additional components (see Section 2.5) that they insert into our framework to
improve performance in particular target domains. [30, 13, 36] evaluate their
pretrained models by substituting them in for VQGAN or CLIP in our framework.
[2, 16] use our framework with diffusion models in place of GANs.

Beyond computer vision, [55] show that it is useful in the materials
engineering design processes. [54, 19] builds on our work by using the
framework to perform sound-guided image generation. In the domain of affective
computing and HCI, [15] has further found vqgan-clip able to elicit targeted
emotions from viewers.

This last example helps explain the widespread commercial adoption of vqgan-
clip, with over a dozen commercial apps built to provide it as a service and
over 500 NFTs produced using our method
sold444https://twitter.com/quasimondo/status/1438601395087753217. A sampling
of commercial websites using vqgan-clip include NightCafe, Wombo Art,
snowpixel.app, starryai.com, neuralblender.com, and hypnogram.xyz.
Collectively, across these sites, vqgan-clip has been used over 10 million
times, showing the veracity of our approach to handle unstructured and diverse
user content.

##  8 Conclusion

We have presented vqgan-clip, a method of generating and manipulating images
based on only human written text prompts. The quality of our model’s
generations have high visual fidelity and remain faithful to the textual
prompt, outperforming prior approaches like DALL-E and GLIDE. The fidelity has
been externally validated by commercial success and use by multiple companies.
Compared to the only comparable approach to text based image editing, vqgan-
clip continues to produce higher quality visual images — especially when the
textual prompt and image content have low semantic similarity. Our ablation
studies show the use of augmented images during the optimization process is a
key factor to our method’s success and that a code-book approach would
sacrifice both speed and quality in our approach.

### Acknowledgements

We would like to acknowledge Ryan Murdock, who developed a very similar
technique for combining VQGAN and CLIP simultaneously to us [33, 34] but did
not release his approach. Ryan was invited to coauthor this paper and
declined. We are also indebted to the thousands of people who have used and
built upon our methods and given us feedback that has allowed us to continue
to improve our techniques.

Many of the experiments in this paper were made possible by CoreWeave.
CoreWeave also provides compute for a free public demo in the EleutherAI
Discord server which enabled the methods described in this paper to be
iterated on rapidly by a great many people. The bot that runs the demo was
created by Discord user BoneAmputee.

## References

  * [1] Safinah Ali and Devi Parikh  “Telling Creative Stories Using Generative Visual Aids”, 2021  arXiv:2110.14810v1 [cs.HC]
  * [2] Omri Avrahami, Dani Lischinski and Ohad Fried  “Blended Diffusion for Text-driven Editing of Natural Images”, 2021  arXiv:2111.14818v1 [cs.CV]
  * [3] David Bau et al.  “Rewriting a deep generative model”  In _European Conference on Computer Vision_ , 2020, pp. 351–369  Springer
  * [4] Abeba Birhane et al.  “The values encoded in machine learning research”  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 173–184
  * [5] Sid Black et al.  “GPT-NeoX-20B: An Open-Source Autoregressive Language Model”  In _preprint_ , 2022
  * [6] Gerry Chen, Alice Dumay and Mengyi Tang  “diffvg+CLIP: Generating Painting Trajectories from Text”  In _preprint_ , 2021
  * [7] Guillaume Couairon et al.  “FlexIT: Towards Flexible Semantic Image Translation”, 2022  arXiv:2203.04705 [cs.CV]
  * [8] Nicola De Cao, Wilker Aziz and Ivan Titov  “Editing Factual Knowledge in Language Models”, 2021  arXiv:2104.08164v2 [cs.CL]
  * [9] Jia Deng et al.  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  IEEE
  * [10] Hao Dong, Simiao Yu, Chao Wu and Yike Guo  “Semantic Image Synthesis via Adversarial Learning”  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 5706–5714
  * [11] Constantin Eichenberg et al.  “MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning”, 2021  arXiv:2112.05253v1 [cs.CV]
  * [12] Patrick Esser, Robin Rombach and Björn Ommer  “Taming Transformers for High-Resolution Image Synthesis”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12873–12883
  * [13] Nanyi Fei et al.  “WenLan 2.0: Make AI Imagine via a Multimodal Foundation Model”, 2021  arXiv:2110.14378v1 [cs.AI]
  * [14] Kevin Frans, LB Soros and Olaf Witkowski  “CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders”, 2021  arXiv:2106.14843v1 [cs.CV]
  * [15] Theodoros Galanos, Antonios Liapis and Georgios N Yannakakis  “AffectGAN: Affect-Based Generative Art Driven by Semantics”  In _9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)_ , 2021
  * [16] Shuyang Gu et al.  “Vector Quantized Diffusion Model for Text-to-Image Synthesis”, 2021  arXiv:2111.14822v3 [cs.CV]
  * [17] Neil Houlsby et al.  “Parameter-efficient transfer learning for NLP”  In _International Conference on Machine Learning_ , 2019, pp. 2790–2799  PMLR
  * [18] Xiaodan Hu et al.  “MUSE: Textual Attributes Guided Portrait Painting Generation”  In _2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR)_ , 2021, pp. 386–392  IEEE
  * [19] Joel Jang, Sumin Shin and Yoonjeon Kim  “Music2Video: Automatic Generation of Music Video with fusion of audio and text”, 2022  arXiv:2201.03809v1 [cs.SD]
  * [20] Saehoon Kim et al.  “minDALL-E on Conceptual Captions”, 2021  URL: https://github.com/kakaobrain/minDALL-E
  * [21] Diederik P Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”, 2014  arXiv:1412.6980v9 [cs.LG]
  * [22] Gihyun Kwon and Jong Chul Ye  “CLIPstyler: Image Style Transfer with a Single Text Condition”, 2021  arXiv:2112.00374v2 [cs.CV]
  * [23] Connor Leahy and Stella Biderman  “The hard problem of aligning AI to human values”  In _The State of AI Ethics Report_ 4, 2021, pp. 180–183
  * [24] Brian Lester, Rami Al-Rfou and Noah Constant  “The Power of Scale for Parameter-Efficient Prompt Tuning”, 2021  arXiv:2104.08691v2 [cs.CL]
  * [25] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz and Philip HS Torr  “Manigan: Text-guided image manipulation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7880–7889
  * [26] Xihui Liu et al.  “Open-edit: Open-domain image manipulation with open-vocabulary instructions”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , 2020, pp. 89–106  Springer
  * [27] Xingchao Liu et al.  “FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization”, 2021  arXiv:2112.01573v1 [cs.CV]
  * [28] Michael Matena and Colin Raffel  “Merging Models with Fisher-Weighted Averaging”, 2021  arXiv:2111.09832v1 [cs.LG]
  * [29] Milagros Miceli, Julian Posada and Tianling Yang  “Studying up machine learning data: Why talk about bias when we mean power?”  In _Proceedings of the ACM on Human-Computer Interaction_ 6.GROUP ACM New York, NY, USA, 2022, pp. 1–14
  * [30] Oscar Michel et al.  “Text2Mesh: Text-Driven Neural Stylization for Meshes”, 2021  arXiv:2112.03221v1 [cs.CV]
  * [31] Eric Mitchell et al.  “Fast Model Editing at Scale”, 2021  arXiv:2110.11309v1 [cs.LG]
  * [32] Alexander Mordvintsev, Christopher Olah and Mike Tyka  “DeepDream - a code example for visualizing Neural Networks”, Google AI Blog, 2015  URL: https://ai.googleblog.com/2015/07/deepdream-code-example-for-visualizing.html
  * [33] Ryan Murdock  ““The Taming Transformers decoder really just goes! And this is with very little work.””  URL: https://twitter.com/advadnoun/status/1367556678896394240
  * [34] Ryan Murdock  ““Working on using the RN50x4 version of CLIP with the Taming Transformers VQGAN.””  URL: https://twitter.com/advadnoun/status/1368081153375105027
  * [35] Seonghyeon Nam, Yunji Kim and Seon Joo Kim  “Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language”  In _Advances in Neural Information Processing Systems_ 31 Curran Associates, Inc., 2018, pp. 42–51  URL: https://papers.neurips.cc/paper/2018/hash/d645920e395fedad7bbbed0eca3fe2e0-Abstract.html
  * [36] Alex Nichol et al.  “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”, 2021  arXiv:2112.10741v3 [cs.CV]
  * [37] Evangelos Ntavelis et al.  “SESAME: semantic editing of scenes by adding, manipulating or erasing objects”  In _European Conference on Computer Vision_ , 2020, pp. 394–411
  * [38] Aäron Oord, Oriol Vinyals and Koray Kavukcuoglu  “Neural Discrete Representation Learning”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017, pp. 6309–6318  URL: https://proceedings.neurips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstract.html
  * [39] Or Patashnik et al.  “StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2085–2094
  * [40] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”  In _Proceedings of the 38th International Conference on Machine Learning_ 139, Proceedings of Machine Learning Research  PMLR, 2021, pp. 8748–8763  URL: https://proceedings.mlr.press/v139/radford21a.html
  * [41] Aditya Ramesh et al.  “Zero-Shot Text-to-Image Generation”  In _Proceedings of the 38th International Conference on Machine Learning_ 139, Proceedings of Machine Learning Research  PMLR, 2021, pp. 8821–8831  URL: https://proceedings.mlr.press/v139/ramesh21a.html
  * [42] Edgar Riba et al.  “Kornia: an Open Source Differentiable Computer Vision Library for PyTorch”  In _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 3663–3672
  * [43] Dave Sayers et al.  “The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies.”, 2021
  * [44] Ramprasaath R Selvaraju et al.  “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 618–626
  * [45] Or Sharir, Barak Peleg and Yoav Shoham  “The Cost of Training NLP Models: A Concise Overview”, 2020  arXiv:2004.08900 [cs.CL]
  * [46] Assaf Shocher et al.  “Semantic Pyramid for Image Generation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7457–7466  DOI: 10.1109/CVPR42600.2020.00748
  * [47] Karen Simonyan, Andrea Vedaldi and Andrew Zisserman  “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps”, 2014  arXiv:1312.6034v2 [cs.CV]
  * [48] Charlie Snell  “Alien Dreams: An Emerging Art Scene”, Machine Learning @ Berkeley, 2020  URL: https://ml.berkeley.edu/blog/posts/clip-art/
  * [49] Yu Tao and Kush R Varshney  “Insiders and Outsiders in Research on Machine Learning and Society”  In _arXiv preprint arXiv:2102.02279_ , 2021
  * [50] Yingtao Tian and David Ha  “Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts”, 2021  arXiv:2109.08857v2 [cs.NE]
  * [51] Maria Tsimpoukelli et al.  “Multimodal Few-Shot Learning with Frozen Language Models”  In _Advances in Neural Information Processing Systems_ 34, 2021
  * [52] Ted Underwood  “Mapping the latent spaces of culture”, 2021  URL: https://tedunderwood.com/2021/10/21/latent-spaces-of-culture/
  * [53] Zihao Wang et al.  “CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP”, 2022  arXiv:2203.00386v1 [cs.CV]
  * [54] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar and Juan Pablo Bello  “Wav2CLIP: Learning Robust Audio Representations From CLIP”, 2021  arXiv:2110.11499v2 [cs.SD]
  * [55] Zhenze Yang and Markus J. Buehler  “Words to Matter: De novo Architected Materials Design Using Transformer Neural Networks”  In _Frontiers in Materials_ 8 Frontiers, 2021, pp. 417
  * [56] Jason Yosinski et al.  “Understanding Neural Networks Through Deep Visualization”, 2015  arXiv:1506.06579v1 [cs.CV]

##  Appendix 0.A Author Contributions

##### Katherine Crowson:

Originated the idea of combining VQGAN with CLIP and lead the development of
the approach.

##### Stella Biderman:

Lead the experimentation and writing of the paper.

##### Daniel Kornis:

developed latent vector regularization and carried out experiments on Open-
Edit.

##### Dashiell Stander:

Assisted with generative experiments and the writing of the paper.

##### Eric Hallahan:

Assisted with generative experiments and the writing of the paper.

##### Louis Castricato:

Developed semantic image editing and masking techniques.

##### Edward Raff:

Advised the project, proposed ablation experiments, and assisted in the
writing of the paper.

##  Appendix 0.B Additional Related Work

###  0.B.1 Image Generation and Editing

While there is a significant literature on semantic image generation and
editing beyond those cited in the main body of the paper, it is overwhelmingly
work that imposes strong constraints on and assumptions about the inputs it
work with. Closed domain work such as [10, 35, 25, 39] achieves much higher
quality images, but comes at the cost of being afunctional outside the domain
the model was trained on. Other authors simplify the problem by using non-
textual auxiliary inputs [46, 37] or filtering the inputs to a small, pre-
defined vocabulary [18].

###  0.B.2 CNN Visualisation and Interpretability

Our approach is surprisingly similar to older work on visualizing and
interpreting convolutional neural networks. Techniques for identifying what
region(s) of an image lead to it being assigned a label by a classifier such
as [47, 44, 56] operate on similar principles to our backpropagating of the
CLIP loss, and DeepDream [32] uses the same core idea of iteratively updating
an image to match a label.

###  0.B.3 Building Multimodal Models Cheaply

Due to the high cost of training large, state-of-the-art transformer models
[45, 5], methodologies that allow one to build off of previously trained
models cheaply are highly valuable. The bulk of the existing literature on
this topic focuses on editing or updating existing (typically unimodal) models
in both computer vision [3] and natural language processing [8, 31, 28].

Another thread of research aims to leverage prompting to efficiently produce
multimodal models from unimodal ones. [51] shows that an input-dependent
version of prompt tuning [24] enables them to convert a 7 billion parameter
language model into a multimodal model by only training 68 million of the
parameters. Building on their success, [11] replace the prompt tuning with
adapters [17] and incorporate CLIP embeddings, enabling them to train a state-
of-the-art visual question answering transformer at a fraction of the cost of
doing so from scratch. Our work continues in this theme by removing the
training entirely, at the cost of a modest increase in generation time.

##  Appendix 0.C Observations of Public Use

As our models have been public for nearly a year at time-of-writing, we have
had the opportunity to study how people use vqgan-clip in the wild. While a
systematic study is far outside the scope of this paper, we summarize our main
observations in the hopes of guiding future human-computer interaction
investigations. These observations are largely informed by casual
conversations and interviews with users of our methodology, and by in situ
observation of people interacting with the model. For further discussion of
the HCI and sociological influences on generative modeling, we direct an
interested reader to [48, 52, 1].

##### Human-AI Co-Creation

The overwhelming feedback we have gotten from users is that they view vqgan-
clip not as an AI working in a vacuum but as an AI and a human working
together to generate art. Users have also created new (and unintended) ways of
working with vqgan-clip that increase their direct agency over the art
created. The most prominent example of this is that many users halt generation
early to edit the partial generations directly, before restarting generation
from their newly edited image. In this way users can remove artifacts,
discourage undesirable components of an image, and try alternative paths
through the optimization landscape.

##### Notebook-based development

The overwhelming majority of people who have picked up and iterated on the
methodologies described in this paper use notebooks such as Jupyter Notebook
and Google Colab to do their work, instead of traditional repositories and
scripts. Based on our conversations with users, the cause of this appears to
be two-fold: widespread popularity among people who do not identify as
computer scientists or developers and a lack of the necessary computational
resources to run the model locally. Similar considerations drive the
popularity of websites that offer vqgan-clip-as-a-service and the bot in our
Discord server.

##### Iterated prompting

While the typical methodology in machine learning for improving results from
an algorithm focus on improvements to the training data or algorithm, our
users have exhibited a strong preference for modifying the prompts they
provide instead. We view this as exemplifying “natural language as an API,”
[43] wherein the actual mapping between the images and the text is less
important than the fact that text is a natural medium for human interaction.
The end-goal is to obtain desirable images using whatever text input is
necessary.

##  Appendix 0.D Additional Generations

The following is a selection of artwork generated using our technique by
people other than the authors of this paper, often with highly complex
prompting. We include them both to emphasize the diversity of artistic styles
vqgan-clip is capable of, as well as its widespread adoption. These images
have served as the covers of academic journals, sold for thousands of dollars,
formed the basis for physical paintings, and been displayed in art galleries.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

##  Appendix 0.E Additional Comparisons

The following images showcase the minDALL-E [20] and GLIDE (filtered) [36]
generations with the prompts found in Figure 3.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 11: Example minDALL-E generations and their text prompts.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 12: Example GLIDE (CLIP-Guided) generations and their text prompts.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 13: Example GLIDE (CF-Guided) generations and their text prompts.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 14: Example vqgan-clip generations and their text prompts. This is the
same figure as Fig. 2 repeated for ease of comparison with Figs. 11, 12 and
13.

##  Appendix 0.F Comparison of Artistic Impressions

We find that minDALL-E and both GLIDE (filtered) models have a much more
tenuous understanding of famous artists than vqgan-clip does. To showcase this
we invent six fictional but plausible painting titles by each artist in Figure
4 and prompt the models to generate, e.g., “Willow Trees by van Gogh.” While
vqgan-clip is able to consistently produce something that is recognizably in
the style of the artist and containing the subject, we find that the other
models frequently fail to produce artwork that matches the prompt, either in
subject or in style.

We note that the GLIDE (filtered) models will frequently produce
photorealistic images, despite the fact that none of the artists produce
photorealistic artwork. As mentioned in Section 4, our primary goal is to
produce high quality art rather than photorealistic images. Consequently,
GLIDE’s Hokusai generations - despite undoubtably being images of rice fields
- rate lowly in our estimation as they have nothing in common with Hokusai’s
artwork. GLIDE’s best results are in response to the van Gogh prompt in which
the models clearly generate impressionistic images of trees, though the art
resembles the work of Nicholas Verrall (in the case of CLIP-guided) and
William Langson Lathrop or Claude Monet (CF-guided) far more so than van Gogh.

minDALL-E appears to be noticeably better at producing art, it frequently
fails to agree with the prompt in both subject and in style. Perhaps most
jarringly, in response to “A Self-Portrait by Kahlo” minDALL-E generates what
is clearly a painting of a person, but the subject is a white man instead of a
Latin woman.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(a)

(b)

(c)

Figure 15: Attempts to generate novel works of art by famous artists. Top to
bottom: minDALL-E, GLIDE (CLIP-guided), GLIDE (CF-guided), and our vqgan-clip.
While VQGAN-CLIP’s generation quality varies, it is the only model that seems
to have grasped the desired task.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(a)

(b)

(c)

Figure 16: Attempts to generate novel works of art by famous artists. Top to
bottom: minDALL-E, GLIDE (CLIP-guided), GLIDE (CF-guided), and our vqgan-clip.
While VQGAN-CLIP’s generation quality varies, it is the only model that seems
to have grasped the desired task.

As before, all images are cherrypicked best-of-five except for vqgan-clip
which is uncherrypicked. We omit ablations over various wordings of the prompt
as our exploratory testing indicated that it would effect the overall quality
of the generation but not the extent to which they match the prompts.

##  Appendix 0.G Additional Components

###  0.G.1 Prompt Addition

One topic of immense interest for text-to-image models is compositionality:
the extent to which the model is able to take multiple discrete concepts and
combine them. While a detailed analysis of compositionality in vqgan-clip is
outside the scope of this paper, we have observed that vqgan-clip is able to
intelligently combine multiple prompts. By providing two separate text inputs
and averaging the loss, we can effectively “add” the two prompts together. In
Fig. 17 we show that adding a content and a style prompt results in an image
that is quite similar to one produced by combining the style and content into
a single prompt with natural language. We view this as a rich and exciting
area for future research.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 17: A comparison of natural semantic combination and addition in the
latent space. Left to right: the original prompt, a natural language prompt,
latent space addition of prompts, and the descriptive modifier.

###  0.G.2 Masked Image Editing

Given a source phrase (text) $s_{p}$ and a target phrase (text) $t_{p}$, we
aim to construct an optimization rule that replaces all instances of $s_{p}$
within a source image $s_{i}$ with $t_{p}$, resulting in a target image (which
is unknown at the beginning) $t_{i}$.

To generate $t_{i}$, we first need to mask the part component of our source
image that corresponds to $s_{p}$, for instance if $s_{p}$ is “Dog,” we need
to mask the part of the image containing a dog. We can utilize CLIP in a zero
shot setting to perform masking as follows:

  1. 1.

Crop $s_{i}$ into a number of smaller sub images,
$\tilde{S}=\\{S_{i}^{k}\\}_{k=1}^{N}$

  2. 2.

$\forall i\in\tilde{S}$, compute $L(i)=f(i)\cdot g(s_{p})$, where $f(\cdot)$
denotes our image encoder and $g(\cdot)$ denotes our text encoder.

  3. 3.

Record the value of L(i) at the center of the crop of $i$

  4. 4.

Normalize the resulting grey scale image, this is now our mask

(a)

(b)

(c)

(d)

(e)

(f)

Figure 18: Two examples of masked editing, showing Jacket $\to$ Fur and
Windmills $\to$ Mushrooms.

We can then threshold this mask in order to determine which components of the
mask actually contain our object of interest. In practice, we compute the
threshold as two standard deviations below the average weight.

During generation, we crop and augment our current generated image and for
every crop we compute the distance of the embedding of this crop to the
embedding of the original image. This is used to preserve some notion of
structure that was present in the original image. Similarly, we compute the
distance of this embedded crop against the target phrase.

By minimizing a weighted sum of these distances, we can perform in-image
object replacement and editing. See Figure 18.