# Multi-Concept Customization of Text-to-Image Diffusion

Nupur Kumari1 Bingliang Zhang2 Richard Zhang3 Eli Shechtman3 Jun-Yan Zhu1
1Carnegie Mellon University 2Tsinghua University 3Adobe Research

###### Abstract

While generative models produce high-quality images of concepts learned from a
large-scale database, a user often wishes to synthesize instantiations of
their own concepts (for example, their family, pets, or items). Can we teach a
model to quickly acquire a new concept, given a few examples? Furthermore, can
we compose multiple new concepts together? We propose Custom Diffusion, an
efficient method for augmenting existing text-to-image models. We find that
only optimizing a few parameters in the text-to-image conditioning mechanism
is sufficiently powerful to represent new concepts while enabling fast tuning
($\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}6$
minutes). Additionally, we can jointly train for multiple concepts or combine
multiple fine-tuned models into one via closed-form constrained optimization.
Our fine-tuned model generates variations of multiple new concepts and
seamlessly composes them with existing concepts in novel settings. Our method
outperforms or performs on par with several baselines and concurrent works in
both qualitative and quantitative evaluations while being memory and
computationally efficient.

Figure 1: Given a few images of a new concept, our method augments a pre-
trained text-to-image diffusion model, enabling new generations of the concept
in unseen contexts. Example concepts include personal objects, animals, e.g.,
a pet dog, and classes not well generated by the model, e.g., moongate (a
circular gate [83]). Furthermore, we propose a method for composing multiple
new concepts together, for example, V∗ dog wearing sunglasses in front of a
moongate. We denote personal categories with a new modifier token V∗.

##  1 Introduction

Recently released text-to-image models [59, 66, 63, 86] have represented a
watershed year in image generation. By simply querying a text prompt, users
are able to generate images of unprecedented quality. Such systems can
generate a wide variety of objects, styles, and scenes – seemingly “anything
and everything”.

However, despite the diverse, general capability of such models, users often
wish to synthesize specific concepts from their own personal lives. For
example, loved ones such as family, friends, pets, or personal objects and
places, such as a new sofa or a recently visited garden, make for intriguing
concepts. As these concepts are by nature personal, they are unseen during
large-scale model training. Describing these concepts after the fact, through
text, is unwieldy and unable to produce the personal concept with sufficient
fidelity.

This motivates a need for model customization. Given the few user-provided
images, can we augment existing text-to-image diffusion models with the new
concept (for example, their pet dog or a “moongate” as shown in Figure 1)? The
fine-tuned model should be able to generalize and compose them with existing
concepts to generate new variations. This poses a few challenges – first, the
model tends to forget [15, 58, 40] or change [39, 46] the meanings of existing
concepts: e.g., the meaning of “moon” being lost when adding the “moongate”
concept. Secondly, the model is prone to overfit the few training samples and
reduce sampling variations.

Moreover, we study a more challenging problem, _compositional fine-tuning_ –
the ability to extend beyond tuning for a single, individual concept and
compose multiple concepts together, e.g., pet dog in front of moongate (Figure
1). Improving compositional generation has been studied in recent works [45].
But composing multiple new concepts poses additional challenges, such as
mixing unseen concepts.

In this work, we propose a fine-tuning technique, Custom Diffusion for text-
to-image diffusion models. Our method is computationally and memory efficient.
To overcome the above-mentioned challenges, we identify a small subset of
model weights, namely the key and value mapping from text to latent features
in the cross-attention layers [76, 6]. Fine-tuning these is sufficient to
update the model with the new concept. To prevent model forgetting, we use a
small set of real images with similar captions as the target images. We also
introduce augmentation during fine-tuning, which leads to faster convergence
and improved results. To inject multiple concepts, our method supports
training on both simultaneously or training them separately and then merging.

We build our method on Stable Diffusion [2] and experiment on various datasets
with as few as four training images. For adding single concepts, our method
shows better text alignment and visual similarity to the target images than
concurrent works and baselines. More importantly, our method can compose
multiple new concepts efficiently, whereas concurrent methods struggle and
often omit one. Finally, our method only requires storing a small subset of
parameters ($3\%$ of the model weights) and reduces the fine-tuning time (6
minutes on 2 A100 GPUs, $2-4\times$ faster compared to concurrent works).
Please find the code and data at our website.

##  2 Related Work

Deep generative models Generative models aim to synthesize samples from a data
distribution, given a set of training examples. These include GANs [24, 34,
10], VAEs [36], auto-regressive [18, 61, 75], flow-based [16, 17], and
diffusion models [29, 71, 14]. To improve controllability, these models can be
conditioned on a class [10, 68], image [32, 80, 93, 48], or text prompt [94,
73, 50]. Our work mainly relates to text-conditioned synthesis [47]. While
earlier works [62, 87, 94, 73, 85, 31] were limited to a few classes [77, 43],
recent text-to-image models [50, 66, 63, 59, 14, 60, 86], trained on extremely
large-scale data, have demonstrated remarkable generalization ability.
However, such models are by nature generalists and struggle to generate
specific instances like personal toys or rare categories, e.g., “moongate”. We
aim to adapt these models to become specialists in new concepts.

Image and model editing. While generative models can sample random images, a
user often wishes to edit a single, specific image. Several works aim at
leveraging the capabilities of generative models, such as GANs [92, 55, 3, 4,
5] or diffusion models [12, 50, 35] towards editing. A challenge is
representing the specific image in the pre-trained model, and such methods
employ per-image or per-edit optimization. A closely related line of work
edits a generative model directly [7, 78, 22]. Whereas these methods aim to
customize GANs, our focus is on text-to-image models.

Transfer learning. A method of efficiently producing a whole distribution of
images is leveraging a pretrained model and then using transfer learning [82,
52, 81, 49, 89, 41, 53, 44, 25, 51, 22]. For example, one can adapt
photorealistic faces into cartoons [52, 49, 41, 53, 22]. To adapt with just a
few training images, efficient training techniques [33, 91, 74, 90, 67, 38]
are often useful. Different from these works, which focus on tuning whole
models to single domains, we wish to acquire multiple new concepts without
catastrophic forgetting [20, 37, 42, 58, 40]. By preserving the millions of
concepts captured in large-scale pretraining, we can synthesize the new
concepts in composition with these existing concepts. Relatedly, several
methods [56, 70, 23, 30] propose to train adapter modules or low rank updates
for large-scale models in the discriminative setting. In contrast, we adapt a
small number of existing parameters and do not require additional parameters.

Adapting text-to-image models. Similar to our goals, two concurrent works,
DreamBooth [65] and Textual Inversion [21], adopt transfer learning to text-
to-image diffusion models [66, 63] via either fine-tuning all the parameters
[65] or introducing and optimizing a word vector [21] for the new concept. Our
work differs in several aspects. First, our work tackles a challenging
setting: compositional fine-tuning of multiple concepts, where concurrent
works struggle. Second, we only fine-tune a subset of cross-attention layer
parameters, which significantly reduces the fine-tuning time. We find these
design choices lead to better results, validated by automatic metrics and
human preference studies.

Figure 2: Custom Diffusion. Given images of new concepts, we retrieve real
images with similar captions as the given concepts and create the training
dataset for fine-tuning, as shown on the left. To represent personal concepts
of a general category, we introduce a new modifier token V∗, used in front of
the category name. During training, we optimize key and value projection
matrices in the diffusion model cross-attention layers along with the modifier
token. The retrieved real images are used as a regularization dataset during
fine-tuning.

##  3 Method

Our proposed method for model fine-tuning, as shown in Figure 2, only updates
a small subset of weights in the cross-attention layers of the model. In
addition, we use a regularization set of real images to prevent overfitting on
the few training samples of the target concepts. In this section, we explain
our design choices and final algorithm in detail.

###  3.1 Single-Concept Fine-tuning

Given a pretrained text-to-image diffusion model, we aim to embed a new
concept in the model given as few as four images and the corresponding text
description. The fine-tuned model should retain its prior knowledge, allowing
for novel generations with the new concept, based on the text prompt. This can
be challenging as the updated text-to-image mapping might easily overfit the
few available images.

In our experiments, we use Stable Diffusion [2] as our backbone model, which
is built on the Latent Diffusion Model (LDM) [63]. LDM first encodes images
into a latent representation, using hybrid objectives of VAE [36], PatchGAN
[32], and LPIPS [88], such that running an encoder-decoder can recover an
input image. They then train a diffusion model [29] on the latent
representation with text condition injected in the model using cross-
attention.

Learning objective of diffusion models. Diffusion models [71, 29] are a class
of generative models that aim to approximate the original data distribution
$q({\mathbf{x}}_{0})$ with $p_{\theta}({\mathbf{x}}_{0})$:

$\displaystyle
p_{\theta}({\mathbf{x}}_{0})=\int\Bigr{[}p_{\theta}({\mathbf{x}}_{T})\prod
p_{\theta}^{t}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})\Bigr{]}d{\mathbf{x}}_{1:T},$
(1)

where ${\mathbf{x}}_{1}$ to ${\mathbf{x}}_{T}$ are latent variables of a
forward Markov chain s.t.
${\mathbf{x}}_{t}=\sqrt{\alpha_{t}}{\mathbf{x}}_{0}+\sqrt{1-\alpha_{t}}\epsilon$.
The model is trained to learn the reverse process of a fixed-length (usually
$1000$) Markov chain. Given noisy image ${\mathbf{x}}_{t}$ at timestep $t$,
the model learns to denoise the input image to obtain ${\mathbf{x}}_{t-1}$.
The training objective of the diffusion model can be simplified to:

$\displaystyle\mathbb{E}_{\epsilon,{\mathbf{x}},{\mathbf{c}},t}[w_{t}||\epsilon-\epsilon_{\theta}({\mathbf{x}}_{t},{\mathbf{c}},t)||],$
(2)

where $\epsilon_{\theta}$ is the model prediction and $w_{t}$ is a time-
dependent weight on the loss. The model is conditioned on timestep $t$ and can
be further conditioned on any other modality ${\mathbf{c}}$, e.g., text.
During inference, a random Gaussian image (or latent) ${\mathbf{x}}_{T}$ is
denoised for fixed timesteps using the model [72].

Figure 3: Analysis of change in weights on updating all network weights during
fine-tuning. The mean change in the cross-attention layers is significantly
higher than other layers even though they only make up $5\%$ of the total
parameter count.

A naïve baseline for the goal of fine-tuning is to update all layers to
minimize the loss in Eqn. 2 for the given text-image pairs. This can be
computationally inefficient for large-scale models and can easily lead to
overfitting when training on a few images. Therefore, we aim to identify a
minimal set of weights that is sufficient for the task of fine-tuning.

Rate of change of weights. Following Li  _et al_. [41], we analyze the change
in parameters for each layer in the fine-tuned model on the target dataset
with the loss in Eqn. 2, i.e.,
$\Delta_{l}=||\theta_{l}^{\prime}-\theta_{l}||/||\theta_{l}||,$ where
$\theta_{l}^{\prime}$ and $\theta_{l}$ are the updated and pretrained model
parameters of layer $l$. These parameters come from three types of layers –
(1) cross-attention (between the text and image), (2) self-attention (within
the image itself), and (3) the rest of the parameters, including convolutional
blocks and normalization layers in the diffusion model U-Net. Figure 3 shows
the mean $\Delta_{l}$ for the three categories when the model is fine-tuned on
“moongate” images. We observe similar plots for other datasets. As we see, the
cross-attention layer parameters have relatively higher $\Delta$ compared to
the rest of the parameters. Moreover, cross-attention layers are only $5\%$ of
the total parameter count in the model. This suggests it plays a significant
role during fine-tuning, and we leverage that in our method.

Model fine-tuning. Cross-attention block modifies the latent features of the
network according to the condition features, i.e., text features in the case
of text-to-image diffusion models. Given text features
${\mathbf{c}}\in\mathbb{R}^{s\times d}$ and latent image features
${\mathbf{f}}\in\mathbb{R}^{(h\times w)\times l}$, a single-head cross-
attention [76] operation consists of
$Q=W^{q}{\mathbf{f}},\;\;K=W^{k}{\mathbf{c}},\;\;V=W^{v}{\mathbf{c}}$ , and a
weighted sum over value features as:

$\displaystyle\text{Attention}(Q,K,V)=\text{Softmax}\Big{(}\frac{QK^{T}}{\sqrt{d^{\prime}}}\Big{)}V,$
(3)

where $W^{q}$, $W^{k}$, and $W^{v}$ map the inputs to a query, key, and value
feature, respectively, and $d^{\prime}$ is the output dimension of key and
query features. The latent feature is then updated with the attention block
output. The task of fine-tuning aims at updating the mapping from given text
to image distribution, and the text features are only input to $W^{k}$ and
$W^{v}$ projection matrix in the cross-attention block. Therefore, we propose
to only update $W^{k}$ and $W^{v}$ parameters of the diffusion model during
the fine-tuning process. As shown in our experiments, this is sufficient to
update the model with a new text-image paired concept. Figure 4 shows an
instance of the cross-attention layer and the trainable parameters.

Figure 4: Single-head Cross-Attention. Latent image feature ${\mathbf{f}}$ and
text feature ${\mathbf{c}}$ are projected into query $Q$, key $K$, and value
$V$. Output is a weighted sum of values, weighted by the similarity between
the query and key features. We highlight the updated parameters $W^{k}$ and
$W^{v}$ in our method.

Text encoding. Given target concept images, we require a text caption as well.
If there exists a text description, e.g., moongate, we use that as a text
caption. For personalization-related use-case where the target concept is a
unique instance of a general category, e.g., pet dog, we introduce a new
modifier token embedding, i.e., V∗ dog. During training, V∗ is initialized
with a rare occurring token embedding and optimized along with cross-attention
parameters. An example text caption used during training is, photo of a V∗
dog.

Regularization dataset. Fine-tuning on the target concept and text caption
pair can lead to the issue of language drift [39, 46]. For example, training
on “moongate” will lead to the model forgetting the association of “moon” and
“gate” with their previously trained visual concepts, as shown in Figure 5.
Similarly, training on a personalized concept of V∗ tortoise plushy can leak,
causing all examples with plushy to produce the specific target images. To
prevent this, we select a set of 200 regularization images from the LAION-400M
[69] dataset with corresponding captions that have a high similarity with the
target text prompt, above threshold $0.85$ in CLIP [57] text encoder feature
space.

Figure 5: Role of regularization data in mitigating overfitting behavior
during fine-tuning. $1^{\text{st}}$ row: samples from pre-trained models. In
$2^{\text{nd}}$ row, fine-tuning cross-attention key, value projection
matrices without any regularization dataset leads to moongate like images on
the text prompt photo of a moon. We largely mitigate this issue with the use
of regularization datasets as shown in $3^{\text{rd}}$ row. More results can
be found in Figure 24 (Appendix).

###  3.2 Multiple-Concept Compositional Fine-tuning

Joint training on multiple concepts. For fine-tuning with multiple concepts,
we combine the training datasets for each individual concept and train them
jointly with our method. To denote the target concepts, we use different
modifier tokens, V${}_{i}^{*}$, initialized with different rarely-occurring
tokens and optimize them along with cross-attention key and value matrices for
each layer. As shown in Figure 7, restricting the weight update to cross-
attention key and value parameters leads to significantly better results for
composing two concepts compared to methods like DreamBooth, which fine-tune
all the weights.

Constrained optimization to merge concepts. As our method only updates the key
and value projection matrices corresponding to the text features, we can
subsequently merge them to allow generation with multiple fine-tuned concepts.
Let set $\\{W^{k}_{0,l},W^{v}_{0,l}\\}_{l=1}^{L}$ represent the key and value
matrices for all $L$ cross-attention layers in the pretrained model and
$\\{W^{k}_{n,l},W^{v}_{n,l}\\}_{l=1}^{L}$ represent the corresponding updated
matrices for added concept $n\in\\{1\cdots N\\}$. As our subsequent
optimization applies to all layers and key-value matrices, we will omit
superscripts $\\{k,v\\}$ and layer $l$ for notational clarity. We formulate
the composition objective as the following constrained least squares problem:

$\begin{split}\hat{W}=\operatorname*{arg\,min}_{W}||WC_{\textnormal{reg}}^{\top}-&W_{0}C_{\textnormal{reg}}^{\top}||_{F}\\\
\textnormal{s.t. }\hskip 2.84526ptWC^{\top}=V,\textnormal{ whe}&\textnormal{re
}C=[{\mathbf{c}}_{1}\cdots{\mathbf{c}}_{N}]^{\top}\\\ \textnormal{and
}&V=[W_{1}{\mathbf{c}}_{1}^{\top}\cdots
W_{N}{\mathbf{c}}_{N}^{\top}]^{\top}.\end{split}$ (4)

Here, $C\in\mathbb{R}^{s\times d}$ is the text features of dimension $d$.
These are compiled of $s$ target words across all $N$ concepts, with all
captions for each concept flattened out and concatenated. Similarly,
$C_{\textnormal{reg}}\in\mathbb{R}^{s_{\textnormal{reg}}\times d}$ consists of
text features of
$\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}1000$
randomly sampled captions for regularization. Intuitively, the above
formulation aims to update the matrices in the original model, such that the
words in target captions in $C$ are mapped consistently to the values obtained
from fine-tuned concept matrices. The above objective can be solved in closed
form, assuming $C_{\textnormal{reg}}$ is non-degenerate and the solution
exists, by using the method of Lagrange multipliers [9]:

$\begin{split}\hat{W}=W_{0}+{\mathbf{v}}^{\top}{\mathbf{d}},\hskip
2.84526pt&\textnormal{where
}{\mathbf{d}}=C(C_{\textnormal{reg}}^{\top}C_{\textnormal{reg}})^{-1}\\\
\textnormal{and
}{\mathbf{v}}^{\top}&=(V-W_{0}C^{\top})({\mathbf{d}}C^{\top})^{-1}.\end{split}$
(5)

We show full derivation in the Appendix B. Compared to joint training, our
optimization-based method is faster
($\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}2$
seconds) if each individual fine-tuned model exists. Our proposed methods lead
to the coherent generation of two new concepts in a single scene, as shown in
Section 4.2.

Training details. We train with our method for $250$ steps in single-concept
and $500$ steps in two-concept joint training, on a batch size of $8$ and
learning rate $8\times 10^{-5}$. During training, we also randomly resize the
target images from $0.4-1.4\times$ and append the prompt “very small”, “far
away” or “zoomed in”, “close up” accordingly to the text prompt based on
resize ratio. We only backpropagate the loss on valid regions. This leads to
faster convergence and improved results. We provide more training details in
the Appendix E.

##  4 Experiments

In this section, we show the results of our method on multiple datasets in
both single concept fine-tuning and composition of two concepts on the Stable
Diffusion model [2].

Datasets. We perform experiments on ten target datasets spanning a variety of
categories and varying training samples. It consists of two scene categories,
two pets, and six objects, as shown in Figure 8. We also recently introduced a
new dataset, CustomConcept101, consisting of 101 concepts, and show our
results on it in Appendix A and the website.

Evaluation metrics. We evaluate our method on (1) _Image-alignment_ , i.e.,
the visual similarity of generated images with the target concept, using
similarity in CLIP image feature space [21], (2) _Text-alignment_ of the
generated images with given prompts, using text-image similarity in CLIP
feature space [27], and (3) _KID_ [8] on a validation set of $500$ real images
of a similar concept retrieved from LAION-400M to measure overfitting on the
target concept (e.g., V∗ dog)and forgetting of existing related concepts
(e.g., dog). (4) We also perform a _human preference_ study of our method with
baselines. Unless mentioned otherwise, we use $200$ steps of DDPM sampler with
a scale $6$. The prompts used for quantitative and human evaluation are shown
on our website.

Baselines. We compare our method with the two concurrent works, DreamBooth
[65] (third-party implementation [84]) and Textual Inversion [21]. DreamBooth
fine-tunes all the parameters in the diffusion model, keeping the text
transformer frozen, and uses generated images as the regularization dataset.
Each target concept is represented by a unique identifier, followed by its
category name, i.e., “[V] category”, where [V] is a rarely occurring token in
the text token space and not optimized during fine-tuning. Textual Inversion
optimizes a new V∗ token for each new concept. We also compare with the
competitive baseline of Custom Diffusion (w/ fine-tune all), where we fine-
tune all the parameters in the U-Net [64] diffusion model, along with the V∗
token embedding in our method. We provide implementation details for all
baselines in the supplement.

###  4.1 Single-Concept Fine-tuning Results

Qualitative evaluation. We test each fine-tuned model on a set of challenging
prompts. This includes generating the target concept in a new scene, in a
known art style, composing it with another known object, and changing certain
properties of the target concept: e.g., color, shape, or expression. Figure 6
shows the sample generations with our method, DreamBooth, and Textual
Inversion. Our method, Custom Diffusion, has higher text-image alignment while
capturing the visual details of the target object. It performs better than
Textual Inversion and is on par with DreamBooth while having a lower training
time and model storage
($\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}5\times$
faster and $75$MB vs $3$GB storage).

Quantitative evaluation. We evaluate on $20$ text prompts and $50$ samples per
prompt for each dataset, resulting in a total of $1000$ generated images. We
use DDPM sampling with $50$ steps and a classifier-free guidance scale of $6$
across all methods. As shown in Figure 8, our method outperforms the
concurrent methods [65, 21]. Also, Custom Diffusion works on par with our
proposed baseline of fine-tuning all the weights in the diffusion model, while
being more computationally and time efficient. Table 1 also shows the KID of
generated images by each fine-tuned model on a reference dataset, with
captions similar to the fine-tuned concept. As we observe, our method has
lower KID than most baselines, which suggests less overfitting to the target
concept. In Appendix C, we show that the updated matrices can be compressed to
further reduce model storage.

Computational requirements Training time of our method is
$\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}6$
minutes (2 A100 GPUs), compared to $20$ minutes for Ours (w/ fine-tune all) (4
A100s), $20$ minutes for Textual Inversion (2 A100s), and
$\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}1$
hour for DreamBooth (4 A100s). Also, since we update only $75$MB of weights,
our method has low memory requirements for storing each concept model. We keep
the batch size fixed at $8$ across all.

Figure 6: Single-concept fine-tuning results. Given images of a new concept
(target images shown on the left), our method can generate images with the
concept in unseen contexts and art styles. First row: representing the concept
in an artistic style of watercolor paintings. Our method can also generate the
mountains in the background, which DreamBooth and Textual Inversion omit.
Second row: changing the background scene. Our method and DreamBooth perform
similarly and better than Textual Inversion. Third row: adding another object,
e.g., an orange sofa with the target table. Our method successfully adds the
other object. Fourth row: changing object property, like color of petals.
Fifth row: accessorizing personal pet cat with sunglasses. Our method
preserves the visual similarity better than baselines while changing only the
petal colors or adding sunglasses to the cat. For Textual Inversion, the input
text prompt consists of only optimized V∗ instead of V∗ category. We show more
samples on our website.  Figure 7: Multi-concept fine-tuning results. First
row: our method has higher visual similarity with the personal cat and chair
images shown in the first column while following the text condition. Second
row: DreamBooth omits the cat in 3 out of 4 images, whereas our method
generates both cats and wooden pots. Third row: our method generates the
target flower in the wooden pot while maintaining the visual similarity to the
target images. Fourth row: generating the target table and chair together in a
garden. For all settings, our optimization-based approach and joint training
perform better than DreamBooth, and joint training performs better than the
optimization-based method.

###  4.2 Multiple-Concept Fine-tuning Results

We show the results of generating two new concepts in the same scene on the
following five pairs: (1) Moongate + Dog, (2) Cat + Chair, (3) Wooden Pot +
Cat, (4) Wooden Pot + flower, and (5) Table + Chair. We compare our method
with DreamBooth training on the two datasets together, using two different
$[V_{1}]$ and $[V_{2}]$ tokens for each concept. For Textual Inversion, we
perform inference using the individually fine-tuned tokens for each concept in
the same sentence. We compare our method with the baselines on a set of $400$
images generated with $8$ prompts for each composition setting in Figure 8 and
Table 1. We also compare with our baseline of sequentially training on the two
concepts or fine-tuning all weights in our method. Our method performs better
on all except the “Table+Chair” composition, where all methods perform
comparably except Textual Inversion, which doesn’t perform well at composition
as also shown in Figure 22 in the Appendix. This shows the importance of fine-
tuning only the cross-attention parameters for composition. In the case of
sequential training, we observe forgetting of the first concept. Figure 7
shows sample images of our proposed two methods and DreamBooth. As we can see,
our method is able to generate the two objects in the same scene in a coherent
manner while having high alignment with the input text prompt. We show more
samples on our website.

Figure 8: Text- and image-alignment for single-concept (left) and multi-
concept (right) fine-tuning. Compared to DreamBooth, Textual Inversion, and
our w/ fine-tune all baseline, our method lies further along the upper right
corner (with less variance). There exists a trade-off between high similarity
to target images vs. text-alignment on new prompts. Keeping in consideration
this trade-off, our method is on-par or better than the baselines. For multi-
concept both joint training and optimization based method are better than
other baselines.

| Method |  Text-alignment |  Image-alignment |  KID (validation)
---|---|---|---|---
Single- Concept | Textual Inversion | 0.670 | 0.827 | 22.27
DreamBooth | 0.781 | 0.776 | 32.53
2-5 | Ours (w/ fine-tune all) | 0.795 | 0.748 | 19.27
| Ours | 0.795 | 0.775 | 20.96
Multi- Concept | Textual Inversion | 0.544 | 0.630 | —
DreamBooth | 0.783 | 0.695
2-5 | Ours (w/ fine-tune all) | 0.787 | 0.691
| Ours (Sequential) | 0.797 | 0.700
| Ours (Optimization) | 0.800 | 0.695
| Ours (Joint) | 0.801 | 0.706

Table 1: Quantitative comparisons. Top row: single-concept fine-tuning
evaluation averaged across datasets. The last column shows the KID ($\times
10^{3}$) between real validation set images and generated images with the same
caption. Since our method uses a regularization set of real images, it
achieves lower KID and even improves slightly over the pretrained model.
Textual Inversion has the same KID as the pretrained model, as it does not
update the model. Bottom row: evaluation on multi-concept averaged across the
five composition pairs. We show individual scores for all in Table 7 and 8 in
the Appendix. We also evaluate single-concept fine-tuned models on FID [28]
(MS-COCO [43]) in Table 6 and show the trend of image-, text-alignment with
training steps in Figure 20.

Ours | Textual Inversion | DreamBooth | Ours (w/ fine-tune all)
---|---|---|---
|  Text Alignment |  Image Alignment |  Text Alignment |  Image Alignment |  Text Alignment |  Image Alignment
Single-concept |  72.62 $\pm$ 2.38$\%$ |  51.62 $\pm$ 2.62$\%$ |  53.50 $\pm$ 2.64$\%$ |  56.62 $\pm$ 2.44$\%$ |  55.17 $\pm$ 2.55$\%$ |  53.99 $\pm$ 2.44$\%$
Multi-concept (Joint) |  86.65 $\pm$ 2.25 $\%$ |  81.89  $\pm$ 2.09 $\%$ |  56.39 $\pm$ 2.46 $\%$ |  61.80 $\pm$ 2.59$\%$ |  59.00 $\pm$ 2.61 $\%$ |  59.12 $\pm$ 2.72$\%$
2-7  Multi-concept (Optimization) |  81.22 $\pm$ 2.72$\%$ |  83.11 $\pm$ 2.18$\%$ |  57.00 $\pm$ 2.62 $\%$ |  61.75 $\pm$ 2.68$\%$ |  57.60 $\pm$ 2.43 $\%$ |  53.49 $\pm$ 2.71$\%$

Table 2: Human preference study. For each paired comparison, our method is
preferred (over $\geq 50\%$) over the baseline in both image- and text-
alignment. Textual Inversion seems to overfit to target images and thus has a
similar image-alignment as ours but performs worse on text-alignment in the
single-concept setting.

###  4.3 Human Preference Study

We perform the human preference study using Amazon Mechanical Turk. We do a
paired test of our method with DreamBooth, Textual Inversion, and Ours (w/
fine-tune all). For text-alignment, we show the two generations from each
method (ours vs. baseline) on the same prompt with the question – “Which image
is more consistent with the text?”. In image-alignment, we show 2-4 training
samples of the relevant concept along with the generated images (same as in
text-alignment study) with the question – “Which image better represents the
objects as shown in target images?”. We collect $800$ responses for each
questionnaire. As shown in Table 2, our method is preferred over baselines in
both single-concept and multi-concept, even compared to Ours (w/ fine-tune
all) method, which shows the importance of only updating cross-attention
parameters.

Method | Text-alignment | Image-alignment | KID (validation)
---|---|---|---
Ours | 0.795 | 0.775 | 20.96
Ours w/o Aug | 0.800 | 0.736 | 20.67
Ours w/o Reg | 0.799 | 0.756 | 32.64
Ours w/ Gen | 0.791 | 0.768 | 34.70

Table 3: Ablation Study. No augmentation during training leads to lower image-
alignment. No regularization dataset or using generated images as
regularization produces much worse KID.  Figure 9: Custom Diffusion with
artistic styles. Our method can also be used to learn new styles. We finetune
our model on 13 and 15 images for the top and bottom rows, respectively. The
last column shows the composition of style with the new “wooden pot” concept
(target images shown in Figure 7) using the optimization-based method. Style
image credits: Mia Tang (top row), Aaron Hertzmann (bottom row).

###  4.4 Ablation and Applications

In this section, we ablate various components of our method to show its
contribution. We evaluate each experiment on the same setup as in Section 4.1.
We show sample generations for ablation experiments on our website.

Generated images as regularization (Ours w/ Gen). As detailed in Section 3.1,
we retrieve similar category real images and captions to use as regularization
during fine-tuning. Another way of creating the regularization dataset is to
generate images from the pretrained model [65]. We compare our method with
this setup, i.e., for the target concept of a “category” generate images using
the prompt, photo of a {category}, and show results in Table 3. Using
generated images results in a similar performance on the target concept.
However, this shows signs of overfitting, as measured by KID on a validation
set of similar category real images.

Figure 10: Image editing with our models using Prompt-to-Prompt [26]. Left:
replacement edit to only change the cap. Right: image stylization while
preserving the image content.  Figure 11: Failure cases on multi-concept fine-
tuning. Our method fails at difficult compositions like a cat and dog together
in a scene or similar category objects like teddybear and tortoise plushy as
shown on the right. Though as shown on the left, the pretrained model also
struggles with similar compositions.

Without regularization dataset (Ours w/o Reg). We show results when no
regularization dataset is used. We train the model for half the number of
iterations (the same number of target images seen during training). Table 3
shows that the model has slightly lower image-alignment and tends to forget
existing concepts, as evident from high KID on the validation set.

Without data augmentation (Ours w/o Aug). As mentioned in Section 3.2, we
augment by randomly resizing the target images during training and append
size-related prompts (e.g., “very small”) to the text. Here, we show the
effect of not using these augmentations. The model is trained for the same
number of steps. Table 3 shows that no augmentation leads to lower visual
similarity with target images.

Fine-tuning on a style. We show in Figure 9 that our method can also be used
for fine-tuning with specific styles. We change the text prompt for target
images to A painting in the style of V∗ art [21]. The regularization images
are retrieved with captions having high similarity with “art”.

Image editing with fine-tuned models Similar to the pretrained model, our
fine-tuned models can be used by existing image-editing methods. We show an
example of using the Prompt-to-prompt [26] method in Figure 10.

##  5 Discussion and Limitations

In conclusion, we have proposed a method for fine-tuning large-scale text-to-
image diffusion models on new concepts, categories, personal objects, or
artistic styles, using just a few image examples. Our computationally
efficient method can generate novel variations of the fine-tuned concept in
new contexts while preserving the visual similarity with the target images.
Moreover, we only need to save a small subset of model weights. Furthermore,
our method can coherently compose multiple new concepts in the same scene.

As shown in Figure 11, difficult compositions, e.g., a pet dog and a pet cat,
remain challenging. In this case, the pre-trained model also faces a similar
difficulty, and our model inherits these limitations. Additionally, composing
increasing three or more concepts together is also challenging. We show more
analysis and visualization in the Appendix C.

Acknowledgment. We are grateful to Nick Kolkin, David Bau, Sheng-Yu Wang,
Gaurav Parmar, John Nack, and Sylvain Paris for their helpful comments and
discussion and to Allie Chang, Chen Wu, Sumith Kulal, Minguk Kang, Yotam
Nitzan, and Taesung Park for proofreading the draft. We also thank Mia Tang
and Aaron Hertzmann for sharing their artwork. We also appreciate the help of
Sheng-Yu Wang, Songwei Ge, Daohan Lu, Ruihan Gao, Roni Shechtman, Avani Sethi,
Yijia Wang, Shagun Uppal, and Zhizhuo Zhou in collecting the dataset, and Nick
Kolkin for the feedback regarding it. This work was partly done by Nupur
Kumari during the Adobe internship. The work is partly supported by Adobe Inc.

## References

  * [1] Chatgpt.  https://chat.openai.com/chat, 2022.
  * [2] Stable diffusion.  https://huggingface.co/CompVis/stable-diffusion-v-1-4-original, 2022\.
  * [3] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [4] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [5] Rameen Abdal, Peihao Zhu, John Femiani, Niloy Mitra, and Peter Wonka.  Clip2stylegan: Unsupervised extraction of stylegan edit directions.  In ACM SIGGRAPH, 2022.
  * [6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations (ICLR), 2015\.
  * [7] David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba.  Rewriting a deep generative model.  In European Conference on Computer Vision (ECCV), 2020.
  * [8] Miko${\mathbf{l}}$aj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  In International Conference on Learning Representations (ICLR), 2018\.
  * [9] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe.  Convex optimization.  Cambridge university press, 2004.
  * [10] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In International Conference on Learning Representations (ICLR), 2019\.
  * [11] Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola.  What makes fake images detectable? understanding properties that generalize.  In European Conference on Computer Vision (ECCV), 2020.
  * [12] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon.  Ilvr: Conditioning method for denoising diffusion probabilistic models.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [13] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva.  On the detection of synthetic images generated by diffusion models.  arXiv preprint arXiv:2211.00680, 2022.
  * [14] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [15] Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding.  Don’t stop learning: Towards continual learning for the clip model.  arXiv preprint arXiv:2207.09248, 2022.
  * [16] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  In ICLR Workshop, 2015.
  * [17] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  In International Conference on Learning Representations (ICLR), 2017\.
  * [18] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [19] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In International Conference on Learning Representations (ICLR), 2019\.
  * [20] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [21] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [22] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  * [23] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  arXiv preprint arXiv:2110.04544, 2021.
  * [24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [25] Zheng Gu, Wenbin Li, Jing Huo, Lei Wang, and Yang Gao.  Lofgan: Fusing local representations for few-shot image generation.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [26] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [27] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In EMNLP, 2021.
  * [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Conference on Neural Information Processing Systems (NeurIPS), 2017.
  * [29] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [31] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu.  Multimodal conditional image synthesis with product-of-experts gans.  In European Conference on Computer Vision, pages 91–109. Springer, 2022.
  * [32] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [33] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [34] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [35] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [36] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In International Conference on Learning Representations (ICLR), 2014\.
  * [37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [38] Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Ensembling off-the-shelf models for gan training.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [39] Jason Lee, Kyun ghyun Cho, and Douwe Kiela.  Countering language drift via visual grounding.  In EMNLP, 2019.
  * [40] Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu.  Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation.  In NAACL, 2022.
  * [41] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman.  Few-shot image generation with elastic weight consolidation.  In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [42] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017.
  * [43] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision (ECCV), 2014.
  * [44] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.  Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.  In International Conference on Learning Representations (ICLR), 2021\.
  * [45] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In ECCV, 2022.
  * [46] Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin.  Countering language drift with seeded iterated learning.  In International Conference on Machine Learning (ICML), 2020.
  * [47] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov.  Generating images from captions with attention.  arXiv preprint arXiv:1511.02793, 2015.
  * [48] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations (ICLR), 2022\.
  * [49] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2020.
  * [50] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning (ICML), 2022.
  * [51] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or.  Mystyle: A personalized generative prior.  In SIGGRAPH ASIA, 2022.
  * [52] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [53] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [54] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On buggy resizing libraries and surprising subtleties in fid calculation.  arXiv preprint arXiv:2104.11222, 2021.
  * [55] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [56] Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder.  Mad-x: An adapter-based framework for multi-task cross-lingual transfer.  In EMNLP, 2020.
  * [57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning (ICML), 2021.
  * [58] Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer.  Effect of scale on catastrophic forgetting in neural networks.  In International Conference on Learning Representations (ICLR), 2021\.
  * [59] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [60] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning (ICML), 2021.
  * [61] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In NeurIPS, 2019.
  * [62] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In International Conference on Machine Learning (ICML), 2016.
  * [63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [65] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [66] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In NeurIPS, 2022.
  * [67] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger.  Projected gans converge faster.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [68] Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  In ACM SIGGRAPH, 2022.
  * [69] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [70] Gabriel Skantze and Bram Willemsen.  Collie: Continual learning of language grounding from language-image embeddings.  Journal of Artificial Intelligence Research, 74:1201–1223, 2022\.
  * [71] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning (ICML), 2015.
  * [72] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In International Conference on Learning Representations (ICLR), 2021\.
  * [73] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao.  Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [74] Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung.  Towards good practices for data augmentation in gan training.  arXiv preprint arXiv:2006.05338, 2, 2020.
  * [75] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In International Conference on Machine Learning (ICML), 2016.
  * [76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [77] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [78] Sheng-Yu Wang, David Bau, and Jun-Yan Zhu.  Rewriting geometric rules of a gan.  ACM SIGGRAPH, 2022.
  * [79] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.  Cnn-generated images are surprisingly easy to spot… for now.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [80] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [81] Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer.  Minegan: effective knowledge transfer from gans to target domains with few images.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [82] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu.  Transferring gans: generating images from limited data.  In European Conference on Computer Vision (ECCV), 2018.
  * [83] Wikipedia.  Moon gate.  https://en.wikipedia.org/wiki/Moon_gate, 2022.
  * [84] XavierXiao.  Dreambooth on stable diffusion.  https://github.com/XavierXiao/Dreambooth-Stable-Diffusion, 2022\.
  * [85] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [86] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [87] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [88] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [89] Miaoyun Zhao, Yulai Cong, and Lawrence Carin.  On leveraging pretrained gans for generation with limited data.  In International Conference on Machine Learning (ICML), 2020.
  * [90] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [91] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang.  Image augmentations for gan training.  arXiv preprint arXiv:2006.02595, 2020.
  * [92] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros.  Generative visual manipulation on the natural image manifold.  In European Conference on Computer Vision (ECCV), 2016.
  * [93] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [94] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Appendix

In Section A, we show the results of our method on the CustomConcept101
dataset, which consists of a significantly larger number of custom concepts
from a wide variety of categories. In Section B, we show the derivation of our
optimization-based method for merging multiple concepts into a single model.
In Section C, we show additional experiment results of our method. We then
provide more evaluation results, for example, the trend with training
iterations, in Section D and implementations details in Section E. Finally, we
discuss the societal implications of our method in Section F.

##  Appendix A CustomConcept101

We show the results on our newly released dataset of 101 concepts consisting
of a wide variety of categories, including toys, plushies, wearables, pets,
scenes, and human faces. Figure 12 shows one image of each concept from the
dataset. The images are collected from websites that allow redistribution,
e.g., Unsplash, or captured by ourselves. For multi-concept, we propose $101$
unique compositions among the concepts, e.g., V${}_{1}^{*}$ dog with
V${}_{2}^{*}$ sunglasses. We also create text prompts for evaluating the
customization, $20$ prompts each in the case of single-concepts, and $12$
prompts for multi-concept composition. We first used ChatGPT [1] to propose
$40$ prompts consisting of a particular concept or pair of concepts. We
instruct it to change the background scene, insert another object in the
scene, or create stylistic variations of the main subjects. We then manually
filtered and modified the proposals to create a final set of prompts.

Table 4 shows the comparison of our method with DreamBooth and Textual
Inversion. Both ours and DreamBooth models are trained with generated images
as regularization on this dataset. In the single-concept case, our method
performs on par with DreamBooth with marginally lower image-alignment and
higher text-alignment. Textual Inversion overfits the target images and has
low text-alignment. For multi-concept customization, both our optimization and
joint training method are better than DreamBooth on average. Sample images for
each method are shown in Figure 13. For more results and dataset details,
please refer to our code and website.

| Method |  Text-alignment |  Image-alignment
---|---|---|---
Single- Concept | Textual Inversion | 0.612 | 0.752
DreamBooth | 0.752 | 0.752
Ours | 0.760 | 0.744
Multi- Concept | DreamBooth | 0.738 | 0.662
Ours (Optimization) | 0.763 | 0.658
Ours (Joint) | 0.757 | 0.668

Table 4: Quantitative comparison on CustomConcept101. On single-concepts, our
method is better on text-alignment and slightly worse on image-ailgnment
compared to DreamBooth. For multi-concept, our method outperforms DreamBooth
on average. The metrics are calculated over $100$ images generated using $20$
prompts for single-concept and $120$ images generated using $12$ prompts for
multi-concept and then averaged across all models. We used DDPM sampler with
$200$ steps in each case. Figure 12: CustomConcept101 dataset. A example image
of each concept in the dataset. Out of $101$ categories, $31$ are collected
from Unsplash, $2$ concepts belonging to the flower category are collected
from other websites which allow redistribution, and the rest are captured by
ourselves. For more details regarding the dataset and license, please refer to
our webpage.  Figure 13: Qualitative comparison on CustomConcept101. Single-
concept top block (first three rows) shows sample comparisons of our method
with DreamBooth and Textual Inversion on three concepts from the dataset. Our
method shows higher text alignment, whereas DreamBooth retains more image
alignment with the target images. Multi-concept: bottom block (last two rows)
shows multi-concept composition using our method and DreamBooth. Our joint
training method outperforms it on image-alignment with both target images
while maintaining text-alignment with the given caption. The optimization
method fails to preserve the image similarity in some cases, e.g., the sofa in
the last row. We select the best three out of five images for single-concept
and the best three out of 10 images for multi-concept.

##  Appendix B Multi-Concept Optimization Based Method

To compose multiple concepts together, we solved a constrained least squares
problem (introduced in Section 3.2, Eqn. 4). We solved the problem in closed
form as shown in Eqn. 5 in the main paper. Here, we show the full derivation.
We first restate the objective below.

$\begin{split}\hat{W}=\operatorname*{arg\,min}_{W}||WC_{\textnormal{reg}}^{\top}-&W_{0}C_{\textnormal{reg}}^{\top}||\\\
\textnormal{s.t. }\hskip 2.84526ptWC^{\top}=V,\textnormal{ whe}&\textnormal{re
}C=[{\mathbf{c}}_{1}\cdots{\mathbf{c}}_{N}]^{\top}\\\ \textnormal{and
}&V=[W_{1}{\mathbf{c}}_{1}^{\top}\cdots
W_{N}{\mathbf{c}}_{N}^{\top}]^{\top}.\end{split}$ (6)

Here, the matrix norms are the Frobenius norm, $W_{0}\in\mathbb{R}^{o\times
d}$ is the matrix from the pretrained model, $C\in\mathbb{R}^{s\times d}$ is
the text features of dimension $d$. These are compiled of $s$ target words
across all $N$ concepts, with all captions for each concept flattened out and
concatenated. Similarly,
$C_{\textnormal{reg}}\in\mathbb{R}^{s_{\textnormal{reg}}\times d}$ consists of
text features of
$\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}1000$
randomly sampled captions for regularization.

By using the method of Lagrange multipliers [9], we need to minimize the
following objective:

$\begin{split}L=\frac{1}{2}||WC_{\textnormal{reg}}^{\top}-W_{0}C_{\textnormal{reg}}^{\top}||-\textnormal{trace}({\mathbf{v}}(WC^{\top}-V)),\end{split}$
(7)

here ${\mathbf{v}}\in\mathbb{R}^{s\times o}$ is the Lagrangian multiplier
corresponding to the constraints. Differentiating the above objective and
equating it to $0$, we obtain:

$\begin{split}WC_{\textnormal{reg}}^{\top}C_{\textnormal{reg}}-W_{0}C_{\textnormal{reg}}^{\top}C_{\textnormal{reg}}-{\mathbf{v}}^{\top}C=0\\\
\implies
W=W_{0}+{\mathbf{v}}^{\top}C(C_{\textnormal{reg}}^{\top}C_{\textnormal{reg}})^{-1}.\end{split}$
(8)

We assume $C_{\textnormal{reg}}$ is non-degenerate. Using the above solution
in Eqn. 6, $WC^{\top}=V$, we obtain:

$\begin{split}&(W_{0}+{\mathbf{v}}^{\top}C(C_{\textnormal{reg}}^{\top}C_{\textnormal{reg}})^{-1})C^{\top}=V\\\
&\textnormal{Let
}{\mathbf{d}}=C(C_{\textnormal{reg}}^{\top}C_{\textnormal{reg}})^{-1}\\\
&{\mathbf{v}}^{\top}=(V-W_{0}C^{\top})({\mathbf{d}}C^{\top})^{-1}.\end{split}$
(9)

Figure 14: Analysis of singular values of the difference of the key and value
projection matrices in the cross-attention layers between pretrained and fine-
tuned model. As shown in the plot, the singular values drop to 0, suggesting
that we can approximate the difference matrix with a low-rank matrix.  Figure
15: Quantitative results with compressed models. We save the low-rank
approximation of the difference between the pretrained and fine-tuned model
updated weights. Even with the top $60\%$ rank ($5\times$ compression), the
performance remains similar (overlapping blue and black points). As we
increase the compression, the image-alignment score decreases, and the model
approaches the pretrained model weights with high text-alignment, as
illustrated with qualitative samples in Figure 21.  Figure 16: Three-concept
fine-tuning results. We can also compose two objects with a style or three new
objects in the same scene for some combination of concepts.

##  Appendix C Experiments

Figure 17: Results of our method on DreamBooth [65] and Textual Inversion [21]
datasets. Our method works similarly or better in some instances, for example,
V∗ backpack with the night sky, and for a V∗ cat backpack. The samples are
generated with $200$ steps of the DDPM sampler, scale $6$.

In this section, we show more experiments, including ablation and analysis of
the limitations of our method.

Comparison with DreamBooth and Textual Inversion. We also show a comparison of
our method with DreamBooth and Textual Inversion on the target concept images
from the respective works. Figure 17 shows the sample generations for the same
input prompts. Our method performs better than baselines in text-alignment
while maintaining the visual similarity to target images.

Model compression. We analyze the singular values of the difference between
updated key and value projection matrices and the corresponding pretrained
matrices in all cross-attention layers of the model. As shown in Figure 14,
the singular values steeply drop. This suggests that the difference matrix can
be approximated well with a low-rank decomposition. Thus, we perform SVD and
only save the low-rank factorization of the difference matrices. This can
reduce the memory storage for each concept further. Figure 15 and 21 shows the
quantitative and qualitative results with decreasing compression ratio. Even
with an approximation using only the top $60\%$ of singular values ($5\times$
compression in model storage), the performance is similar. But as we increase
the compression to only include the top $1\%$ of singular values, the image-
alignment decreases. Top k$\%$ implies singular values till the rank where
cumulative sum is k$\%$ of total sum of singular values. We also attempted to
enforce a low-rank update to the key, value matrices during fine-tuning but
observed the results to be sub-optimal [19]. More sample generations are shown
on our website.

Figure 18: Qualitative analysis of Choice of V∗ (Ours w/ random init V∗) and
generated images as regularization (Ours w/ Gen). We show the generated
samples for the original category word, for e.g., images generated on the
prompt a dog for models fine-tuned on V∗ dog. We also show the sample
generated by pretrained model in column (d) for comparison. Column (b) shows
that initializing V∗ randomly from the normal distribution of existing token
embeddings and then optimizing leads to more shifts in original category words
getting mapped to target images compared to our method in column (a).
Similarly, with generated images as regularization, the quality of samples
gets worse for the original category as shown in column (c).

Choice of V∗. In all experiments, we initialized the unique modifier token
with the token-id $42170$. During joint training with two concepts, we
initialize the other with token-id $47629$ in the pretrained CLIP tokenizer
used in Stable Diffusion. We ablate our choice of V∗ with – (1) Random
initialization with the mean and standard deviation of existing token
embeddings and then optimizing the modifier token, (2) not optimizing the
modifier token, once initialized with the rarely occurring token. The
quantitative results are shown in Table 5. We observe that not optimizing V∗
leads to significantly lower image-alignment. Compared to random
initialization of V∗, our method has higher image-alignment and lower text-
alignment. But we observe that the generated samples with the prompt a
{category} shift more towards target image distribution of V∗ category,
compared to our final method, as shown in Figure 18.

Three concept fine-tuning. We test the limit of our multi-concept method and
show results on three concepts in Figure 16. Our method is able to compose two
new objects with a new style or synthesize three new objects.

Limitations of our multi-concept fine-tuning. As shown in Figure 11 in the
paper, our method fails at difficult compositions like generating personal cat
and dog in the same scene. We observe that the pretrained model also struggles
with such compositions and hypothesize that our model inherits these
limitations. Here, we analyze the attention map of each token on the latent
image features in Figure 19. The “dog” and “cat” token attention maps are
largely overlapping for both our and pretrained models, which might lead to
worse composition.

Generated images as regularization (Ours w/ Gen). We showed in Section 4.4
(Table 3) that using generated images as regularization leads to similar
performance on the target concept but higher KID [8] on the validation set. We
show here that it also leads to artifacts in the generations with category
word used to generate images which are used as the regularization set. For
example, fine-tuned model on V∗ dog generates images with saturation artifact
for the prompt a dog as shown in Figure 18. Since we use a high learning rate
compared to DreamBooth (which also uses generated images as regularization). A
lower learning rate might mitigate this issue but at the cost of increased
training time.

Figure 19: Attention map visualization of failed compositions. We show the
average attention across timestep and layers for each word (token). For both
our and pretrained models, the attention map of “cat” and “dog” overlap more
often. This can be one of the reasons for the failed compositions.

| Methods | Barn (7) |  Tortoise plushy (12) |  Teddy- Bear (7) |  Wooden Pot (4) | Dog (10) | Cat (5) | Flower (10) | Table (4) | Chair (4) | Mean
---|---|---|---|---|---|---|---|---|---|---|---
Text-alignment | Ours | 0.791 | 0.827 | 0.849 | 0.796 | 0.764 | 0.768 | 0.800 | 0.788 | 0.771 | 0.795
2-12 | Ours (w/ V∗ init normal dist.) | 0.817 | 0.809 | 0.853 | 0.807 | 0.781 | 0.805 | 0.820 | 0.793 | 0.766 | 0.805
| Ours (w/o V∗ opt) | 0.847 | 0.824 | 0.864 | 0.830 | 0.769 | 0.801 | 0.823 | 0.787 | 0.807 | 0.816
Image-alignment | Ours | 0.744 | 0.783 | 0.829 | 0.769 | 0.684 | 0.848 | 0.734 | 0.768 | 0.814 | 0.774
2-12 | Ours (w/ V∗ init normal dist.) | 0.747 | 0.780 | 0.829 | 0.787 | 0.670 | 0.785 | 0.709 | 0.772 | 0.811 | 0.765
| Ours (w/o V∗ opt) | 0.730 | 0.755 | 0.784 | 0.786 | 0.663 | 0.757 | 0.683 | 0.688 | 0.757 | 0.733
KID ($\times 10^{3}$) (Validation) | Ours | 09.00 | 26.82 | 40.33 | 08.77 | 19.46 | 27.39 | 36.47 | 15.77 | 17.94 | 22.43
2-12 | Ours (w/ V∗ init normal dist.) | 09.99 | 21.76 | 44.68 | 12.27 | 15.09 | 25.26 | 32.97 | 15.26 | 21.28 | 22.06
| Ours (w/o V∗ opt) | 10.22 | 23.75 | 41.64 | 11.97 | 18.19 | 26.41 | 28.83 | 16.56 | 19.20 | 21.86

Table 5: Quantitative results of different choices for modifier token V∗. We
ablate our method with two settings – (1) Ours (w/ V∗ init normal dist.),
where we initialize the modifier token randomly with mean and standard
deviation of the existing token embeddings, and then optimize during training.
(2) Ours (w/o V∗ opt), i.e., not optimizing the token once initialized with
the rare occurring token. We observe that not optimizing V∗ leads to worse
results on image-alignment, i.e., the model is not able to learn the target
concept. Similarly, random initialization with the normal distribution also
results in lower image-alignment and higher text-alignment, but as shown in
Figure 18, the category word mapping shifts to target images.

| Methods |  Moongate (135) [44] | Barn (7) |  Tortoise plushy (12) |  Teddy- Bear (7) |  Wooden Pot (4) | Dog (10) | Cat (5) | Flower (10) | Table (4) | Chair (4)
---|---|---|---|---|---|---|---|---|---|---|---
MS-COCO FID | Ours | 16.05 | 17.21 | 16.27 | 16.71 | 16.70 | 16.26 | 16.95 | 16.71 | 16.25 | 16.99
DreamBooth | 17.35 | 20.36 | 19.61 | 19.45 | 20.01 | 19.10 | 20.57 | 18.57 | 19.39 | 19.35

Table 6: MS-COCO FID evaluation with fine-tuned models is a standard
evaluation metric for text-to-image models. The pretrained stable diffusion
model has $16.35$ FID with the same setting of $50$ DDPM sampling steps, scale
$6$. Our method for most datasets has a similar FID, which shows that the
fine-tuned models are similar to the pretrained model on other unrelated
concepts. Since Textual Inversion does not update the model, it has the same
FID as the pretrained model. Figure 20: Text- and image-alignment scores with
training steps (mean across datasets). The text-alignment score gradually
decreases as we fine-tune the model on the target images. In the case of
Textual Inversion, we observe that the new token embedding introduced in the
method results in high image-alignment but the text-alignment remains
significantly lower across training iterations. Compared to DreamBooth, our
method at convergence has higher image text- and image-alignment.

| Methods |  Moongate (135) [44] | Barn (7) |  Tortoise plushy (12) |  Teddy- Bear (7) |  Wooden Pot (4) | Dog (10) | Cat (5) | Flower (10) | Table (4) | Chair (4) | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---
Text-alignment | Textual Inversion | 0.658 | 0.739 | 0.646 | 0.713 | 0.726 | 0.613 | 0.643 | 0.648 | 0.651 | 0.658 | 0.670
DreamBooth | 0.778 | 0.839 | 0.789 | 0.850 | 0.791 | 0.764 | 0.789 | 0.824 | 0.715 | 0.676 | 0.781
2-13 | Ours (w/ fine-tune all) | 0.762 | 0.782 | 0.822 | 0.854 | 0.820 | 0.752 | 0.792 | 0.803 | 0.763 | 0.767 | 0.795
| Ours | 0.772 | 0.791 | 0.828 | 0.848 | 0.798 | 0.764 | 0.768 | 0.800 | 0.787 | 0.771 | 0.795
Image-alignment | Textual Inversion | 0.740 | 0.753 | 0.755 | 0.885 | 0.802 | 0.760 | 0.909 | 0.815 | 0.891 | 0.872 | 0.827
DreamBooth | 0.608 | 0.679 | 0.847 | 0.822 | 0.788 | 0.674 | 0.786 | 0.688 | 0.846 | 0.863 | 0.776
2-13 | Ours (w/ fine-tune all) | 0.662 | 0.699 | 0.822 | 0.804 | 0.767 | 0.669 | 0.810 | 0.670 | 0.725 | 0.772 | 0.748
| Ours | 0.665 | 0.744 | 0.784 | 0.829 | 0.769 | 0.684 | 0.849 | 0.735 | 0.767 | 0.814 | 0.775
KID ($\times 10^{3}$) (Validation) | Pretrained Model | 11.33 | 11.18 | 33.51 | 39.82 | 12.88 | 25.40 | 35.01 | 30.96 | 13.35 | 9.26 | 22.27
2-13 | Textual Inversion | 11.33 | 11.18 | 33.51 | 39.82 | 12.88 | 25.40 | 35.01 | 30.96 | 13.35 | 9.26 | 22.27
| DreamBooth | 20.80 | 12.975 | 36.45 | 42.93 | 15.80 | 44.14 | 67.41 | 37.55 | 21.07 | 26.16 | 32.53
2-13 | Ours (w/ fine-tune all) | 9.04 | 9.35 | 28.55 | 42.78 | 10.60 | 19.58 | 14.79 | 22.95 | 18.69 | 16.34 | 19.27
| Ours | 7.65 | 9.00 | 26.82 | 40.33 | 8.77 | 19.46 | 27.39 | 36.47 | 15.77 | 17.94 | 20.96

Table 7: Quantitative evaluation on single-concept fine-tuning. First and
Second row: we show the text- and image-alignment in CLIP feature space
(higher is better for both). All metrics are calculated with $1$K generated
samples across $20$ prompts for each dataset. Our method performs better than
baselines when averaged across datasets. We show the trend with training steps
in Figure 20. Bottom row: KID between real validation set images ($500$) and
generated images ($1$K) with the same caption. Since our method uses a
regularization set of real images, it achieves lower KID than baselines and
even improves slightly over the pretrained model except on “Table” and
“Chair”. Textual Inversion has the same KID as the pretrained model since the
diffusion model is not updated in the method. We also evaluate our models with
FID [28] on MS-COCO [43] in Table 6 in the Appendix. We use the DDPM sampler
with $50$ steps and scale $6$ for both metrics. The number of training images
is shown in brackets.

##  Appendix D Evaluation

Text- and image-alignment scores with training iterations. There is usually a
trade-off between text-alignment and image-alignment. High image-alignment
leads to a decrease in text-alignment where sample generations have less
variance and are close to input target images. We show the trend of text- and
image-alignment in Figure 20, which shows that initially the model has high
text-alignment (as pretrained model) but low image-alignment and with training
the curves for text-/image-alignment gradually gets worse/better. Table 7 and
8 shows the individual text- and image-alignment scores for the single-concept
and multi-concept fine-tuning. To measure text-alignment, we remove the
modifier token V∗ from the text prompt for extracting CLIP text features.

MS-COCO FID evaluation for all models. We also evaluate MS-COCO FID for our
models and DreamBooth. As shown in Table 6, our method has lower FID and only
slightly worse occasionally compared to the $16.35$ FID of the pretrained
model. This suggests our method doesn’t change the generated image
distribution on unrelated concepts. We measure FID [28] using clean-fid
library [54].

##  Appendix E Implementation and Experiment Details

We describe additional training details for our method and baselines [21, 65].

Datasets. All datasets in the paper were captured manually or downloaded from
Unsplash except Moongate [44].

| Methods |  | Moongate + Dog
---
| Cat + Chair
---
| Wooden Pot + Cat
---
| Wooden Pot + Flower
---
| Table + Chair
---
Mean
Text-alignment | DreamBooth | 0.767 | 0.783 | 0.774 | 0.860 | 0.732 | 0.783
Textual Inversion | 0.538 | 0.445 | 0.542 | 0.552 | 0.644 | 0.544
2-9 | Ours | w/ fine-tune all | 0.797 | 0.742 | 0.800 | 0.856 | 0.745 | 0.787
| Sequential | 0.785 | 0.736 | 0.863 | 0.862 | 0.740 | 0.797
| Optimization | 0.786 | 0.774 | 0.806 | 0.870 | 0.766 | 0.800
| Joint | 0.793 | 0.798 | 0.833 | 0.845 | 0.737 | 0.801
Image-alignment, (target1) | DreamBooth | 0.492 | 0.715 | 0.719 | 0.697 | 0.817 | 0.688
Textual Inversion | 0.675 | 0.571 | 0.539 | 0.531 | 0.822 | 0.627
2-9 | Ours | w/ fine-tune all | 0.584 | 0.736 | 0.594 | 0.699 | 0.817 | 0.686
| Sequential | 0.534 | 0.696 | 0.636 | 0.671 | 0.833 | 0.674
| Optimization | 0.583 | 0.792 | 0.580 | 0.683 | 0.786 | 0.684
| Joint | 0.592 | 0.674 | 0.654 | 0.758 | 0.837 | 0.702
Image-alignment, (target2) | DreamBooth | 0.656 | 0.737 | 0.633 | 0.633 | 0.839 | 0.699
Textual Inversion | 0.473 | 0.614 | 0.673 | 0.580 | 0.831 | 0.634
2-9 | Ours | w/ fine-tune all | 0.592 | 0.646 | 0.815 | 0.644 | 0.783 | 0.695
| Sequential | 0.641 | 0.762 | 0.748 | 0.660 | 0.819 | 0.725
| Optimization | 0.598 | 0.639 | 0.819 | 0.675 | 0.803 | 0.706
| Joint | 0.582 | 0.767 | 0.757 | 0.640 | 0.807 | 0.710

Table 8: Text-alignment and image-alignment with the two target concepts in
CLIP feature space on multi-concept fine-tuning. We evaluate each composition
pair on $400$ images generated using $8$ prompts with $200$ steps of DDPM
sampler and scale=$6$. A high score on one metric at the cost of worse
performance on other metrics leads to overall worse results, as our
qualitative samples show as well. Our method performs better than concurrent
methods on the average metric in all settings except Table+Chair, where most
methods perform comparably.

Custom Diffusion (ours). As mentioned in Section 3.2, we train with a batch
size of $8$ and learning rate $10^{-5}$, which is scaled by batch size for an
effective learning rate of $8\times 10^{-5}$. We train for $250$ steps for
single-concept experiments and $500$ steps for multi-concept. During training,
we randomly resize the target images to $1.2-1.4\times$ every 1 out of 3 times
and append zoomed in or close up to the text prompt. The rest of the time the
target image is randomly resized to $0.4-1.0\times$ and if the resize ratio is
less than $0.6$ we append far away or very small to the text prompt, and only
propagate the loss in the valid image region. Since fine-tuning is done only
for a few iterations, we do not notice any augmentation leaking in the fine-
tuned model. We also detach the start token embedding during fine-tuning. For
selecting the rare token as the modifier token V∗, we count the occurrence of
the total $49408$ tokens in $200K$ captions sampled from the LAION-400M
dataset. We then select the token with
$\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}5-10$
occurrences, with alphabetic representation, and not a substring of another
token. During training, we oversample the target images to keep the ratio of
regularization samples ($200$) and target samples the same.

Ours (w/ fine-tune all). When fine-tuning all parameters, we reduce the
learning rate to $8\times 10^{-6}$, which works better than $8\times 10^{-5}$
and train for $500$ steps with a batch size of $8$. For multi-concept, we
train for $1000$ iterations. The rest of the settings are the same as our
final method.

Textual Inversion [21] We train with the recommended batch size of $8$, a
learning rate of $0.005$ (scaled by batch size for an effective learning rate
of $0.04$) for $5000$ steps. The new token is initialized with the category
word, e.g., “dog”. In cases when the category word is represented by multiple
tokens, e.g., “tortoise plushy”, we use a single word approximation like
“plush”, similarly “gate” for “moongate”, “pot” for “wooden pot”, and “bear”
for “teddybear”.

DreamBooth [65] We use the third-party implementation [84] of DreamBooth.
Training is done with the frozen text transformer and fine-tuning the U-net
diffusion model with a batch size of $8$ and learning rate $10^{-6}$ (without
scaling with batch size and $\\#$GPUs). The text prompt used for target images
is photo of [V] category where we initialize [V] with the same rare occurring
token-id $42170$ as ours. The regularization images are generated with $50$
steps of the DDPM sampler with the text prompt photo of a {category}. We train
for $2500$ steps for single-concept. For multi-concept, we train for $5000$
steps but pick the best checkpoint at $3000$ iterations. For results on
CustomConcept101 dataset which we updated later in Appendix A, we trained
DreamBooth with the learning rate of $5\times 10^{-6}$ as suggested in their
paper and batch-size $4$. The training was done for $1000$ iterations for
single-concept and $2000$ iterations for multi-concept.

Figure 21: Qualitative results on fine-tuned model’s compression for reduced
storage requirements. We save the low-rank approximation of the difference
between the pretrained model and fine-tuned model updated weights. The storage
requirements of models from left to right are 75MB, 15MB, 5MB, 1MB, 0.1MB, and
0.08MB (to save the optimized V∗). Even with $5\times$ compression with top
$60\%$ singular values, the performance remains similar. Top k$\%$ implies
singular values till the rank where cumulative sum is k$\%$ of total sum of
singular values. As we increase the compression, the image-alignment score
decreases, as evident from sample generations not being similar to target
images, especially in the case of tortoise plushy, teddybear, and cat.  Figure
22: Multi-concept composition using Textual Inversion. We observe that Textual
Inversion struggles with the composition of two fine-tuned objects as shown in
the above sample generations as well.  Figure 23: Generating target images
with long text prompts in the pretrained model. We show that even with long
text descriptions the pretrained model struggles to generate exact target
images. Thus, to generate the target images, we need model fine-tuning.
Figure 24: Overfitting on the training prompt template. Since during fine-
tuning, target images are trained with the text prompt, photo of a V∗
{category}, we show here random sample generations for the text prompt, photo
of a {category}, in both ours and ours (w/o reg) case. As shown, the
generations shift towards the target images and have less diversity compared
to the pretrained model. Between ours and ours (w/o reg), our method has less
shift and is more diverse on average.

##  Appendix F Societal Impact

While training massive-scale diffusion models is inaccessible to most people,
our method of fine-tuning pretrained models can help democratize such models
to everyday users. Users can customize these models according to their own
personal images, artworks, and objects of interest. Being compute and memory
efficient, it will increase the accessibility and usage of large-scale models
by individual users as well as enable easy collaboration for sharing millions
of fine-tuned concepts and their compositions. At the same time, the dangers
of generative technology accompany our method as well. Possible ways to
mitigate this is the reliable detection of fake generated data, which has been
studied in the context of GANs [79, 11] and, recently, diffusion models [13].

##  Appendix G Change log

v1: Original draft.

v2: CustomConcept101 dataset details and results in Appendix A, Figure 12, 13,
and Table 4. Updated citations and results with three concept compositions in
Figure 16.