#  ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image
Diffusion Models

Maitreya Patel Tejas Gokhale Chitta Baral Yezhou Yang
Arizona State University
https://conceptbed.github.io Corresponding Author<EMAIL_ADDRESS>
###### Abstract

The ability to understand visual concepts and replicate and compose these
concepts from images is a central goal for computer vision. Recent advances in
text-to-image (T2I) models have lead to high definition and realistic image
quality generation by learning from large databases of images and their
descriptions. However, the evaluation of T2I models has focused on
photorealism and limited qualitative measures of visual understanding. To
quantify the ability of T2I models in learning and synthesizing novel visual
concepts, we introduce ConceptBed, a large-scale dataset that consists of 284
unique visual concepts, 5K unique concept compositions, and 33K composite text
prompts. Along with the dataset, we propose an evaluation metric, Concept
Confidence Deviation (CCD), that uses the confidence of oracle concept
classifiers to measure the alignment between concepts generated by T2I
generators and concepts contained in ground truth images. We evaluate visual
concepts that are either objects, attributes, or styles, and also evaluate
four dimensions of compositionality: counting, attributes, relations, and
actions. Our human study shows that $\operatorname{\text{{CCD}{}}}$ is highly
correlated with human understanding of concepts. Our results point to a trade-
off between learning the concepts and preserving the compositionality which
existing approaches struggle to overcome.

##  1 Introduction

Humans reason about the visual world by aggregating entities that they see
into “visual concepts”: both cats and elephants are animals, and both palms
and pines are trees. When we describe images in natural language we often make
use of such concepts to describe things that we see. Although this type of
visual concept learning is well-defined in human psychology murphy2004big , it
remains elusive with respect to the development of data-driven techniques
capable of learning and reasoning from images and their natural language
descriptions.

Text-to-Image (T2I) generative models are trained to translate natural
language phrases into images that correspond to that input. High-quality T2I
models, therefore, serve as a link between human-level concepts (expressed in
natural language) and their visual representations and are one way to
reproduce visual concepts. On the other hand, this has also sparked interest
in visual concept learning through the procedure of “image inversion” – to
translate one or many images corresponding to a visual concept into a latent
representation of that visual concept. While early works primarily explored
image inversion using Generative Adversarial Networks (GANs) xia2022gan ,
methods such as Textual Inversion gal2022image  and Dreambooth
ruiz2022dreambooth  combine image inversion with T2I – this has led to an
effective way to quickly learn concepts from a few images and reproduce them
in novel combinations and compositions with other concepts, attributes,
styles, etc. These methods aim to learn concepts with minimal reference images
by fine-tuning pre-trained text-conditioned diffusion models (as shown in
Figure 1). Therefore this paradigm of T2I and image inversion is a powerful
new way of learning and reproducing concepts.

Within this paradigm of concept learning via image inversion, two primary
evaluation criteria have emerged: concept alignment, which assesses the
generated image’s correspondence to target concept images, and compositional
reasoning, which evaluates whether the generated images maintain
compositionality. Previous works on T2I concept learning have evaluated only a
small number of concepts and unique compositions, rendering their findings
non-generalizable to a wide range of potential concepts. Furthermore, the
established evaluation metrics such as DINO cosine similarity
ruiz2022dreambooth  (for measuring concept alignment), and CLIPScore
hessel2021clipscore  (for evaluating compositionality), have encountered
challenges in accurately capturing human preferences. Consequently, there is a
growing need for automated evaluations.

Therefore, we propose the ConceptBed benchmark: a comprehensive evaluation
strategy aligned with human preferences and accompanied by a concept dataset.
The ConceptBed dataset comprises 284 distinct concepts and over 33,000
composite text prompts, which can be further extended using the provided
automatic realistic dataset creation logic. The dataset focuses on four
diverse concept learning evaluation tasks: learning styles, learning objects,
learning attributes, and compositional reasoning. To gain a deeper
understanding of previous methodologies, we incorporate four composition
categories – Action, Attribution, Counting, and Relations.

Figure 1: The outline of visual concept learning via image inversion. Here, V*
denotes the target concept. First, image inversion is employed to learn a
mapping from concept images to the latent space. Then, this learned concept V*
is used in text prompts to generate images with novel compositions.

Armed with the dataset, we show how it can be used to evaluate concept
learners. To that end, we present a novel concept deviation-aware evaluation
framework that exhibits a strong correlation with human preferences. This
framework, combined with the ConceptBed dataset, offers an alternative to
existing evaluation strategies, facilitating more effective large-scale
evaluations that are aligned with human evaluations. For each of the four
tasks within ConceptBed, we train a supervised classifier (referred to as an
oracle) to detect the respective concepts. Subsequently, the confidence scores
outputted by these oracles are utilized to calculate the concept deviations of
the generated concept images in relation to the reference target ground truth
images using the proposed $\operatorname{\text{{CCD}{}}}$ metric. This
approach enables us to assess concept and compositional alignment accurately.

We conduct extensive experiments on four recently proposed concept learning
methodologies. In total, we fine-tune approximately 1100 models (one model per
concept) and generate over 200,000 concept-specific images. Our results reveal
a trade-off between concept alignment and compositional reasoning, wherein
methods excelling at concept alignment tend to fall short in preserving
compositions and vice versa. This suggests that previous concept learning
approaches are either highly overfitted or severely underfitted. Furthermore,
our experiments demonstrate that utilizing a pre-trained CLIP
radford2021learning  textual encoder aids in maintaining compositionality, but
it lacks the flexibility required to learn complex concepts, such as sketch.

In summary, we make the following key contributions:

  * •

We introduce ConceptBed, a comprehensive benchmark for grounded quantitative
evaluations of text-conditioned concept learners. It comprises 284 unique and
diverse concepts and over 33,000 composite text prompts across four
composition categories.

  * •

Our key contribution is the Concept Confidence Deviation (CCD) evaluation
metric, which measures the learners’ ability to preserve concepts and
compositions. On average, we demonstrate a strong negative correlation of
$-0.89$ (Pearson’s correlation) between $\operatorname{\text{{CCD}{}}}$ and
human preferences.

  * •

Through extensive experiments with 1,100+ models, we identify shortcomings in
prior works and suggest future research directions. ConceptBed sets a standard
for evaluating text-conditioned concept learners.

##  2 Concept Learning Preliminaries

In this section, we establish the formal definition of concept learning
(Section 2.1). Later, we delve into the concept learning under the paradigm of
text-conditioned diffusion models (Section 2.2).

###  2.1 Definition of Concept Learning

###### Definition 1

A concept ($c$) is characterized as a set of entities clustered together,
sharing commonly associated properties ($P_{c}$).

The concept represents an abstract notion, which can be identified by the
recurring properties observed among different entities. For instance, in an
image collection featuring various animals such as dogs, cats, and more, the
concept of animal emerges. Similarly, images of distinct golden retrievers and
bisons exemplify the concept of dog.

Figure 2:  A summary of the ConceptBed dataset for large-scale grounded
evaluations of concept learners. The collection of concepts is categorized
into three classes: (1) Domain, (2) Objects, and (3) Attributes. ConceptBed
has 284 unique concepts and four compositional categories. Here, V* is a
learned concept.

###### Definition 2

A Concept Learner ($CL$) refers to a model or modeling strategy capable of
acquiring the concept ($c$) and effectively reproducing and associating it
with other concepts.

An ideal concept learner should exhibit two crucial properties: 1) Reproduce
Concepts: It should be able to replicate the learned concepts faithfully. 2)
Compositional Reasoning: The concept learner must be adept at associating the
acquired concepts with prior knowledge, thereby retaining both the concept
itself and its composition.

To evaluate the set of concepts ($\mathcal{C}$), we assume the existence of a
relationship ($\mathcal{R}$) between two concepts ($a$, $b$) such that
$a\in\mathcal{C}$. This compositional relationship can be denoted by the
predicate $\mathcal{R}(a,b)$, representing the composition of the two
concepts. For example, when $\mathcal{R}(a,b)\implies$ with(bird, two legs)
and the composition can be expressed as "A bird with two legs." We further
assume that the combination of any $\mathcal{R}$ with $a$ and $b$ is realistic
(e.g., we do not evaluate a bird with five legs). Note that two concepts can
also be combined with each other, denoted as $\mathcal{R}_{1}(a,b)\text{
\text@underline{and} }\mathcal{R}_{2}(a,c)$. For instance, "A bird with two
legs and eating the fish" combines the concepts of a bird having two legs and
eating a fish.

###  2.2 Concept Learning Under the Text-to-Image Paradigm

Prior studies on concept learning have focused on text-conditioned diffusion
models, such as Textual Inversion, DreamBooth, and Custom Diffusion. These
models operate within the text-to-image (T2I) paradigm, where a text prompt
($y$) serves as input to generate the corresponding image ($x_{gen}$)
representing the given prompt $y$. A popular approach within T2I is the Latent
Diffusion Model (LDM), which incorporates two key modules: 1) Textual Encoder
($C_{\theta}$): This module generates embeddings corresponding to the input
text prompt; 2) Generator ($\mathcal{E}_{\phi}$): The generator estimates the
noise iteratively from the input randomly sampled matrix at timestamp $t$
($z_{t}$), conditioned on the text.

Since T2I models solely consider text input, it becomes necessary to represent
the target concept $c$ in terms of text tokens. These tokens can subsequently
be employed to generate images associated with concept $c$. Therefore, the
concept learning task is approached as an image inversion problem, aiming to
map the target concept back to the text-embedding space. Let V* denote the
text tokens corresponding to the learned concept $c$. Once the optimal mapping
from V* to the target concept is determined, we can generate concept-specific
images using the LDM by providing V* in the text prompt. Suppose we are
provided with $m$ images ($X_{1:m}$) of the target concept $c$. Now, in order
to learn the text tokens V* corresponding to the concept $c$ from the set of
images $X_{1:m}$, the textual inversion technique aims to optimize V* by
reconstructing $X_{1:m}$ using the objective function of the LDM with frozen
parameters $\theta$ and $\phi$:

$V^{*}=\underset{v}{\mathrm{argmin}}\mathbb{E}_{z\sim\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t,y,x\in
X_{1:m}}\Big{[}||\epsilon-\epsilon_{\phi}(z_{t},t,x,C_{\theta}(y))||_{2}^{2}\Big{]},$
(1)

In the case of DreamBooth, instead of finding the optimal V*, it optimizes the
model parameter $\phi$ associated with the noise estimator. This optimization
process enables the model to learn the mapping between randomly initialized V*
and the target concept $c$. Once $\phi^{*}$ is obtained, it can be used to
generate images related to the target concept whenever the V* token is
provided as an input text token. This optimization strategy follows the same
LDM objective function111DreamBooth uses the additional regularizer term to
maintain the compositionality.:

$\mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t,y,x\in
X_{1:m}}\Big{[}||\epsilon-\epsilon_{\theta}(z_{t},t,x,C_{\phi}(V^{*}))||_{2}^{2}\Big{]},$
(2)

Once the mapping from V* to concept $c$ is learned, we can generate concept
$c$ images. However, in order to evaluate these generated images, it is
essential to verify whether they align with the learned concepts while
maintaining compositionality. Suppose we have a set of reference concept
images $X_{1:m}$ that depicts a bird eating, a bird standing, a bird with two
legs, and so on. In this case, the common concept across these images is bird,
represented as $c\in\mathcal{C}$. Once we determine that the target $c$ is a
bird or a specific species of bird, it becomes straightforward to verify the
generated images. To assess the quality of the generated images, we can
utilize a concept dataset $\mathcal{D}=\\{(x,c)\\}_{1}^{n}$ to train the
Oracle which can be used as the concept identifier/verifier.

##  3 ConceptBed

To improve the reliability of evaluations, we propose a novel evaluation
framework called ConceptBed, designed to accurately estimate Concept and
Composition Alignment by quantifying deviations in the generated images. In
this section, we first introduce the ConceptBed dataset (Section 3.1) and
next, we put forward the concept deviation-aware evaluation framework (Section
3.2). Please refer to the appendix for insights on the proposed dataset and
evaluation framework.

###  3.1 ConceptBed: Dataset

We introduce the ConceptBed dataset, a comprehensive collection of concepts to
enhance quantitative evaluations of concept learning. Our dataset incorporates
popular existing datasets such as ImageNet deng2009imagenet , PACS
li2017deeper , CUB wah2011caltech , and Visual Genome krishna2017visual ,
enabling us to create a labeled dataset that improves evaluation accuracy.
Figure 2 provides an overview of the ConceptBed dataset.

Learning Styles. To learn the various styles, we utilize the PACS dataset,
which encompasses four distinct domains: Art Painting, Cartoon, Photo, and
Sketch. Each style contains images corresponding to seven entities. In this
task, the concept learner aims to use examples from one style as a reference
and generate this style-specific images for all seven entities.

Learning Objects. Extracting object-level concepts is accomplished through the
utilization of the ImageNet dataset. It comprises 1000 low-level concepts from
the WordNet fellbaum2010wordnet  hierarchy. However, due to the presence of
noise in ImageNet images and the lack of relevance to daily life for many
concepts, we employ an automated filtering pipeline to ensure the usefulness
and quality of the reference concept images. The pipeline involves extracting
a list of low-level concepts and their parent concepts from ImageNet, followed
by extracting text phrases from Visual Genome containing the concept as a
subject in the caption. If an insufficient number of such captions exists
(less than 10 in Visual Genome) or they cannot be found, the concepts are
discarded. This filtering process results in 80 concepts such as (brambling,
squirrel monkey, dutch oven, etc.). To obtain high-quality images for concept
learners, an object detector is deployed to identify the percentage of pixels
belonging to a given concept, ensuring the realistic requirement of the
compositions. Finally, we select the top 100 images for which the concept
pixel area is the largest and greater than 0.4.

Learning Attributes. Since ImageNet dataset images are not labeled based on
the attributes present in the image, it is necessary to rely on datasets that
provide attribute-level grounded labels. Therefore, we employ the CUB dataset,
which offers attribute-level labels (such as orange wing, blue forehead,
etc.), enabling the ConceptBed to perform evaluations and measure the
attribute-level accuracy of concept learners.

Compositional Reasoning. In addition to learning new concepts, it is crucial
to maintain prior knowledge and associate the acquired concepts with it. To
conduct comprehensive evaluations, we use Visual Genome to extract captions in
which the concept appears as the subject of the sentence. These captions are
categorized into four composition categories (actions, attributes, counting,
and relation) through few-shot classification using GPT3 brown2020language .
This categorization allows us to measure the performance of the baselines on
each category, providing an in-depth understanding of the varying difficulty
levels of different compositions.

Dataset Statistics: The ConceptBed dataset consists of 284 unique concepts,
comprising 80 concepts from ImageNet, 200 concepts from CUB, and 4 concepts
from PACS. The dataset encompasses over 5000 unique composite prompts
distributed across the 80 ImageNet concepts, with each composite prompt having
up to two associations. Specifically, we have 3028, 2891, 1267, and 203
prompts contributing to the attribute, relation, action, and counting
categories, respectively. In total, the dataset contains approximately 33,000
composite prompts (counting the total number of prompts over the entire set of
concepts) for the evaluation of all processed concepts from ImageNet.

We would like to emphasize that our dataset curation pipeline can be readily
extended to more extensive datasets, such as OpenImages-v7 kuznetsova2020open
and LAION-5B schuhmannlaion . However, it is important to note that this
extension would significantly increase the resource requirements. With the
introduction of this dataset, our primary objective is to provide a
standardized and benchmarked evaluation framework for concept learners,
enhancing research in the field.

###  3.2 ConceptBed: Evaluation Framework

Figure 3: The outline of ConceptBed evaluation framework.

In this section, we present our Concept Confidence Deviation (CCD) metric,
which measures the alignment of generated images with respect to a reference
concept. Furthermore, in Section 3.2.2, we outline the necessary evaluation
settings to accurately assess the performance of concept learners across
various tasks.

####  3.2.1 Concept Confidence Deviation (CCD)

For simplicity, let us consider a pre-trained text-conditioned diffusion
model, denoted as $g(\cdot)$, which can be further fine-tuned on a specific
concept $c$ such that $c\in\mathcal{C}_{\textbf{{ConceptBed}}{}}$. We assume
the availability of concept-specific training images from the ConceptBed
dataset, denoted as
$\mathcal{D}_{c}^{real}\in\mathcal{D}_{\textbf{{ConceptBed}}{}}^{test}$. The
concept learner $g(\cdot)$ fine-tuned on concept $c$ using
$\mathcal{D}_{c}^{real}$ is referred to as $g_{c}(\cdot)$. To begin, we
generate a collection of $N$ images, each representing the learned concept,
denoted as
$\mathcal{D}_{c}^{gen}={x_{i}^{gen}=g_{c}(p_{c}^{i},rs^{i});\quad\forall
i\in[0,N]}$, where $p_{c}^{i}$ denotes the concept-specific prompt and $rs$
represents the random seed. The existing quantitative evaluation strategies
can be formulated as: 1) Concept Alignment:
$score=DINO(\mathcal{D}_{c}^{gen},\mathcal{D}_{c}^{real})$, and 2)
Compositional Reasoning: $score=CLIPScore(\mathcal{D}_{c}^{gen},p_{c})$. Here,
DINO measures the pair-wise cosine similarities between generated and target
real concept images. While CLIPScore measures the image-text alignment on
generated images with respect to the input text prompt.

However, these methods fail to capture the inherent concept deviations within
the generated images, rendering them ineffective in measuring performance
accurately. To address this limitation, the next step involves training an
oracle classifier, denoted as $f(\cdot)$, specifically for the concept
detection task using the ConceptBed training dataset,
$\mathcal{D}_{\textbf{{ConceptBed}}{}}^{train}$. By leveraging the output
probabilities of the oracle (with respect to the concept label $y_{c}$) for
each generated image $x_{gen}$ with respect to the output probabilities of
real target images $x_{real}$, we can estimate the deviations associated with
the generated images. The Concept Confidence Deviation (CCD) evaluation metric
is defined as follows:

$\operatorname{\text{{CCD}{}}}=-\underset{c}{\operatorname{\mathbb{E}}}\bigg{[}\underset{x_{gen}}{\operatorname{\mathbb{E}}}\big{[}f(y_{c}|x_{gen})-\underset{x_{real}}{\operatorname{\mathbb{E}}}f(y_{c}|x_{real})\big{]}\bigg{]},$
(3)

$\operatorname{\text{{CCD}{}}}$ first calculates the mean target probability
on the ground truth images and then measures the difference in probability of
the generated images. A $\operatorname{\text{{CCD}{}}}$ value of $0$ indicates
that the generated images closely follow the distribution of the ground truth
concept images. A positive $\operatorname{\text{{CCD}{}}}$ value suggests that
the generated images deviate from the original distribution, while a negative
$\operatorname{\text{{CCD}{}}}$ value implies that the generated images adhere
to the exact distribution as the training concept images.

Figure 4: We present qualitative examples showcasing the effectiveness of
concept learners on the ConceptBed dataset. The leftmost column displays four
instances of ground truth target concept images. Subsequent columns exhibit
target concept-specific images generated by all four baseline methods. The
first row demonstrates an example of the style concept (sketch). The second
row illustrates the bird concept. Furthermore, the last two rows exhibit
generated images based on distinct composite text prompts, leveraging the same
learned concept from the second row.

####  3.2.2 Task Specific Evaluation Settings

To efficiently leverage the ConceptBed evaluation pipeline, we trained
separate oracles on the corresponding ConceptBed datasets. Two different types
of evaluations are conducted, each with its respective set of oracles: 1)
Concept alignment, measured by a concept classifiers, and 2) Compositional
reasoning, measured by a Visual Question Answering model.

Concept Alignment: Concept alignment evaluation was performed on all tasks,
including the generated concept images with different composite text prompts.
To evaluate the style, a ResNet18 he2015deep  model is trained to distinguish
the images between four style concepts. To evaluate the objects, a ConvNeXt
liu2022convnet  model is fine-tuned on 80 classes from the ConceptBed using
the ImageNet training subset. The Concept Embedding Model (CEM)
zarlenga2022concept  was trained on CUB to detect the concepts and respective
attributes. Images corresponding to the concepts were generated for each task
by following the prompts: "A photo of V*" for objects and "A photo of a
<entity-name> in the style of V*" for styles. Here, <entity-name> belongs to
the seven classes from the PACS dataset. The remaining task, composition,
utilized the same pre-trained ConvNeXt model for concept-alignment, as
ConceptBed compositions are for 80 ImageNet concepts.

Compositional Reasoning: To measure the image-text alignment with respect to
the input prompts, the concept-specific token (i.e., dogs, cats, etc.) was
removed and replaced with the corresponding ground truth label making
$\mathcal{R}(a,b)$ to $\mathcal{R}(\mathrm{V\textsuperscript{*}},b)$. The
image-text similarity was then measured. Unlike previous works, CLIP was not
used due to its inability to capture compositions thrush2022winoground .
Instead, taking after cho2022dall , boolean questions are generated for each
prompt, which could be answered by a pre-trained VQA model such as ViLT
kim2021vilt . As ViLT is essentially a classifier, the
$\operatorname{\text{{CCD}{}}}$ can be calculated with respect to the
confidence of the model associated with a "yes" answer.

##  4 Experiments & Results

In this section, we extensively study text-conditioned diffusion models’
ability to learn new concepts while preserving compositionality. To achieve
that, we benchmark four state-of-the-art concept learning methodologies. In
Section 4.1, we explain the setup of the experiments, and in Section 4.2 we
report the evaluation results using ConceptBed evaluation framework: Concept
Alignment and Compositional Reasoning. At last, we compare our findings with
Human Evaluations. Please refer to the appendix for an in-depth discussion
about the experimental setup, results, and human evaluations.

###  4.1 Experimental Setup

In this work, we study four text-conditioned diffusion modeling-based concept
learning strategies: 1) Textual Inversion (LDM) gal2022image , 2) Textual
Inversion (SD), 3) DreamBooth ruiz2022dreambooth , and 4) Custom Diffusion
kumari2022multi . We generate $N=100$ images for all concepts to measure the
concept alignment. While $N=1$ images for ImageNet concepts on about $33,000$
composite text prompts. For a total of 284 concepts, we train all four
different baselines. This leads to $1,100$\+ concept-specific fine-tuned
models and we generate a total of $240,000$\+ images for evaluations.

Model | Concepts |  | Fine-grainedCUB |  | Composition
---|---|---|---|---|---
DomainPACS | ObjectsImageNet |  | Object-level | Attribute-level |
Textual Inversion (LDM) | 0.0549 | 0.1196 |  | 0.2252 | 0.1244 |  | 0.1868
Textual Inversion (SD) | 0.2853 | 0.0662 |  | 0.0903 | 0.0350 |  | 0.1163
DreamBooth | 0.6849 | 0.0880 |  | 0.1055 | 0.0532 |  | 0.3551
Custom Diffusion | 0.6319 | 0.2309 |  | 0.3969 | 0.1877 |  | 0.4882
Original | 0.0000 | 0.0000 |  | 0.0000 | 0.000 |  | 0.0000

Table 1:  Concept Alignment Evaluation Results. The table shows the
performance of concept learners on the $\operatorname{\text{{CCD}{}}}$
($\downarrow$) metric. The table includes results in three parts: Concepts
(DomainPACS, and ObjectImageNet), Fine-grainedCUB (Object-level, and
Attribute-level), and Composition. The best and worst performing models are
indicated by bold and  bold numbers, respectively.  Figure 5: The sensitivity
distribution of concept learners on $\operatorname{\text{{CCD}{}}}$.

Models | Relation |  | Action |  | Attribute |  | Counting
---|---|---|---|---|---|---|---
CLIP | Acc. | $\operatorname{\text{{CCD}{}}}$ |  | CLIP | Acc. | $\operatorname{\text{{CCD}{}}}$ |  | CLIP | Acc. | $\operatorname{\text{{CCD}{}}}$ |  | CLIP | Acc. | $\operatorname{\text{{CCD}{}}}$
Textual Inversion (LDM) | 0.6587 | 80.47% | 0.2085 |  | 0.6518 | 83.18% | 0.2120 |  |  0.6680 | 81.55% | 0.1329 |  | 0.6538 | 76.67% | 0.1312
Textual Inversion (SD) | 0.6294 | 81.01% | 0.1729 |  | 0.6274 | 83.71% | 0.1885 |  |  0.6362 | 82.13% | 0.1089 |  | 0.6299 | 77.08% | 0.1041
DreamBooth |  0.7051 | 81.24% | 0.0545 |  |  0.6992 | 83.87% | 0.0480 |  |  0.6860 | 82.16% | 0.0347 |  |  0.6934 | 77.48% | 0.0047
Custom Diffusion |  0.7065 | 84.63% | 0.0460 |  |  0.7056 | 87.99% | 0.0320 |  |  0.6943 | 84.59% | 0.0168 |  |  0.6909 |  80.05% | 0.0131
Stable Diffusion |  0.7222 | 85.41% | 0.0412 |  |  0.7183 | 88.65% | 0.0213 |  |  0.7056 | 84.79% | 0.0192 |  |  0.7095 |  80.07% | 0.0263
Original | 0.6626 | 87.45% | 0.0000 |  | 0.6831 | 89.80% | 0.0000 |  | 0.6306 | 85.79% | 0.0000 |  | 0.6553 | 78.31% | 0.0000

Table 2:  Compositional Reasoning Evaluation Results. The table shows the
performance of the prior works for Composition Alignment. CLIP $(\uparrow)$ is
the traditional image-text alignment metric. Accuracy $(\uparrow)$ is the
accuracy of the ViLT VQA classifier on generated boolean questions. And
$\operatorname{\text{{CCD}{}}}$ ($\downarrow$) is the composition deviations
reported from the ViLT model with respect to its performance on original
images. The best-performing model is indicated by bold numbers, while the
performance that is higher than the original data is reported with underline.

Models |  DomainPACS |  |  ObjectsImageNet |  |  Compositional Reasoning
---|---|---|---|---|---
DINO ($\uparrow$) | $\operatorname{\text{{CCD}{}}}$ ($\downarrow$) | Human Score ($\uparrow$) |  | DINO ($\uparrow$) | $\operatorname{\text{{CCD}{}}}$ ($\downarrow$) | Human Score ($\uparrow$) |  | CLIP ($\uparrow$) | $\operatorname{\text{{CCD}{}}}$ ($\downarrow$) | Human Score ($\uparrow$)
Textual Inversion (LDM) | 0.5073 | 0.0549 | 4.028 |  | 0.4708 | 0.1196 | 4.069 |  | 0.6611 | 0.1711 | 2.851
Textual Inversion (SD) | 0.4104 | 0.2853 | 4.084 |  | 0.4457 | 0.0662 | 4.159 |  | 0.6309 | 0.1436 | 3.694
DreamBooth | 0.3925 | 0.6849 | 3.083 |  | 0.4525 | 0.0880 | 4.075 |  | 0.6919 | 0.0360 | 3.556
Custom Diffusion | 0.3956 | 0.6319 | 3.164 |  | 0.4450 | 0.2309 | 3.803 |  | 0.6968 | 0.0204 | 4.178
Correlation (w.r.t. Human) | 0.6557 | -0.9345 | 1.0 |  | 0.2787 | -0.9868 | 1.0 |  | 0.3486 | -0.7488 | 1.0

Table 3:  Human Evaluations. Comparison of prior quantitative metrics and
$\operatorname{\text{{CCD}{}}}$ metric with Human evaluations. DINO
caron2021emerging  based pairwise cosine similarity is the prior evaluation
metric ruiz2022dreambooth . CLIP (CLIPScore) hessel2021clipscore  is the
traditional reference-free image-text similarity metric.
$\operatorname{\text{{CCD}{}}}$ is our presented concept deviation-aware
evaluation metric. Here, DomainPACS and ObjectImageNet evaluations are for
concept alignment and Compositional reasoning is for image-text similarity.
The correlation between $\operatorname{\text{{CCD}{}}}$ and the human
evaluations is negative as a lower score on $\operatorname{\text{{CCD}{}}}$
and a higher score on human evaluations mean better performance.

###  4.2 Evaluation Results

Concept Alignment. Table 1 shows the overall performance of the baselines in
terms of $\operatorname{\text{{CCD}{}}}$, where lower score indicates better
performance. First, we can observe that CCD for concept alignment is low for
the original images; suggesting that the oracle is certain about its
predictions. Second, it can be inferred that Custom Diffusion performs poorly,
while Textual Inversion (SD) outperforms the other methodologies except for
the case of the learning styles. We attribute this behavior to differences in
textual encoders. LDM trains the BERT-style textual encoder from scratch while
SD uses pre-trained CLIP to condition the diffusion model. CLIP contains vast
image-text knowledge leading to better performance on learning objects but
less flexibility to learn different styles as a concept. Surprisingly, if we
compare the concept alignment performance with and without composite prompts,
we observe that the performance further drops significantly for all baseline
methodologies when composite prompts are used. This shows that existing
concept learning methodologies find it difficult to maintain the concepts
whenever the additional context is provided via composite prompts.

Compositional Reasoning. Previously, we discussed concept alignment on
composite prompts. Table 2 summarizes the evaluations on composition tasks.
Here, we observe the complete opposite trend in results. Custom Diffusion
outperforms the other approaches across the composition categories. This
result shows the trade-off between learning concepts and at the same time
maintaining compositionality in recent concept learning methodologies.
Moreover, it can be observed that neither the CLIP score nor VQA accuracy is a
reliable metric to evaluate the compositions.

Sensitivity. Figure 5 illustrates the sensitivity distribution observed by the
CCD from generated concept-specific images from different methods from Table
1. Here, we can observe that the sensitivity distribution of original ground
truth images is very concentrated and it is very high for generated images.

Qualitative Results. Figure 4 provides the qualitative examples of the concept
learning. It can be inferred that Textual Inversion (LDM) learns the sketch
concept very well (the first row). While DreamBooth and Custom Diffusion
struggle to learn it. All of the baselines perform comparatively well in
reproducing the learned concept (the second row). But interestingly, in the
case of compositions, DreamBooth and Custom Diffusion performs well with the
cost of losing the concept alignment (last two rows). At the same time,
textual inversion approaches cannot reproduce the compositions (like, Two V*)
but they maintain the concept alignment. Overall, these qualitative examples
align with our quantitative results and strengthen our evaluation framework.

Human Evaluations. We perform Human Evaluations using Amazon Mechanical Turk
for both types of evaluations: 1) Concept Alignment – to measure the alignment
between generated images and ground truth reference images on DomainPACS and
ObjectImageNet, and 2) Compositional Reasoning – to measure the image-text
alignment. In the case of Concept Alignment, we ask human annotators to rate
the likelihood of the target image the same as three reference images. While
for Compositional Reasoning we simply ask the annotators to rate the
likelihood alignment of the image and the corresponding caption. Table 3
summarizes the performance of prior and proposed
($\operatorname{\text{{CCD}{}}}$) quantitative metrics with respect to the
Human Evaluations. It can be inferred that the CCD is strongly correlated with
human preferences and outperforms the prior evaluation metrics by a large
amount. This further strengthens the importance of the introduced ConceptBed
framework.

##  5 Related Work

Concept Learning. Concept learning encompasses various problem statements and
approaches, depending on the perspective adopted. Concept Bottleneck Models
(CBMs) koh2020concept  and Concept Embedding Models (CEMs) zarlenga2022concept
treat object attributes as concepts and propose classification strategies to
identify these concepts. Neuro Symbolic Concept Learner (NS-CL) mao2018the
aims to learn visual concepts by associating them with language semantics,
enabling the model to perform visual question answering. ComPhy chen2022comphy
and CRIPP-VQA patel-etal-2022-cripp  proposes the challenge of learning
concepts that are visually concealed or not easily discernible. Image
Inversion Style Concept Learning, as exemplified by Xia et. al. xia2022gan ,
takes a different approach. Its objective is to invert a given concept image
back into the latent space of a pre-trained model. By faithfully
reconstructing the concept from the inverted code using the generator, this
method explores the latent representations of concepts.

Text-to-Image Generative Models. With advances in vector quantization
van2017neural  and diffusion modeling rombach2022high , text-to-image
generation has improved its performance. Notable works such as DALL-E
ramesh2021zero  train transformer models on quantized latent spaces. Current
State-of-the-Art, diffusion-based text-to-image models such as GLIDE
nichol2022glide , LDM rombach2022high , and Imagen saharia2022photorealistic ,
have surpassed prior approaches (such as StackGAN zhang2017stackgan ,
StackGAN++ zhang2018stackgan++ , TReCS koh2021text , and DALL-E ramesh2021zero
) and achieved superior performance. Although diffusion-based generative
models are state-of-the-art methods, they can be prone to perturbations in
input space. Hence, measuring this sensitivity to perturbations is essential.

Text-to-Image Concept Learning. Text-conditioned diffusion models, such as LDM
and Imagen, have demonstrated their potential for learning new concepts with
only a few reference images. Textual Inversion gal2022image  proposes learning
the embedding corresponding to the placeholder (V*) through optimization.
DreamBooth ruiz2022dreambooth  suggests optimizing the UNet parameters instead
of optimizing the placeholder embedding. Custom Diffusion kumari2022multi
combines both approaches by optimizing the placeholder embedding and Key/Value
weights from the Cross-Attentions layers of the UNet model for faster and
better concept learning. These concept learners are essentially text-
conditioned diffusion models and inherit the same limitations of diffusion
models. One limitation is the overfitting of concepts and language drift. By
optimizing model parameters on a handful of reference images, it is highly
likely that the model might overfit the given concept and cannot maintain
compositionality. Therefore, in this paper, we propose ConceptBed to evaluate
these systems more systematically.

Text-to-Image Generative Model Evaluations. Evaluating generative models is
not widely explored. The FID heusel2017gans  score is commonly used to measure
generated image quality. CLIPScore hessel2021clipscore  is another popular
evaluation metric for reference-free image-text alignment. Another study
focuses on compositional evaluations of text-to-image models on small subsets
(CU-Birds and Oxford-Flowers)park2021benchmark . DALL-Eval cho2022dall
evaluates reasoning skills on synthetic datasets and social biases of text-to-
image generative models. DALL-Evalcho2022dall , VISORgokhale2022benchmarking ,
LAYOUTBENCHcho2023diagnostic  evaluates spatial reasoning abilities. Although
text-to-image model evaluations are well-explored, they lack concept-specific
assessments and cannot be used for evaluating concept learning. Ruiz et. al.
preferred to use DINO (an object-level visual feature extractor) for pair-wise
cosine similarity ruiz2022dreambooth . However, DINO cannot distinguish two
similar concepts from the concept hierarchy (e.g., two different breeds of
dogs). These metrics do not capture the sensitivity of image generations
observed in diffusion models. This sensitivity can be measured by calculating
the confidence of the generated images being the same as ground truth
reference images. Therefore, as part of ConceptBed, we propose a concept
distortion-aware evaluation framework for better and more accurate
evaluations.

##  6 Conclusion

In this paper, we introduce a novel benchmark called ConceptBed designed to
assess the efficacy of text-conditioned diffusion models in learning new
concepts. The ConceptBed benchmark encompasses an end-to-end evaluation
pipeline, a comprehensive Concept Library, and a novel Concept Confidence
Deviation (CCD) evaluation metric. We conduct evaluations based on two key
criteria: Concept Alignment and Compositional Reasoning. Through extensive
experiments, we demonstrate that existing text-conditioned diffusion model-
based concept learners exhibit significant limitations in their performance.
Furthermore, we conduct human evaluations to validate the effectiveness of our
proposed evaluation metric ($\operatorname{\text{{CCD}{}}}$), which showcases
a strong correlation with human preferences. This finding positions
$\operatorname{\text{{CCD}{}}}$ as a viable alternative to human judgments,
enabling large-scale and comprehensive evaluations. To the best of our
knowledge, the ConceptBed dataset represents the first large-scale concept-
learning dataset that facilitates precise and accurate evaluations of concept-
learning abilities.

## Acknowledgement

This work was supported by NSF RI grants #1750082 and #2132724, and a grant
from Meta AI Learning Alliance. The views and opinions of the authors
expressed herein do not necessarily state or reflect those of the funding
agencies and employers.

## References

  * [1] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  arXiv preprint arXiv:2304.08466, 2023.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [4] Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, and Chuang Gan.  Comphy: Compositional physical reasoning of objects and events from videos.  In International Conference on Learning Representations, 2022.
  * [5] Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, and Mohit Bansal.  Diagnostic benchmark and iterative inpainting for layout-guided image generation.  arXiv preprint arXiv:2304.06671, 2023.
  * [6] Jaemin Cho, Abhay Zala, and Mohit Bansal.  Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers.  arXiv preprint arXiv:2202.04053, 2022.
  * [7] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  arXiv preprint arXiv:2210.11427, 2022.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [9] Christiane Fellbaum.  Wordnet.  In Theory and applications of ontology: computer applications, pages 231–243. Springer, 2010.
  * [10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [11] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang.  Benchmarking spatial relationships in text-to-image generation.  arXiv preprint arXiv:2212.10015, 2022.
  * [12] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition. arxiv 2015.  arXiv preprint arXiv:1512.03385, 14, 2015.
  * [14] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich.  Deep anomaly detection with outlier exposure.  In International Conference on Learning Representations, 2019.
  * [15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, 2021.
  * [17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [18] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021.
  * [19] Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Text-to-image generation grounded by fine-grained user attention.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 237–246, 2021.
  * [20] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang.  Concept bottleneck models.  In International Conference on Machine Learning, pages 5338–5348. PMLR, 2020.
  * [21] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123:32–73, 2017.
  * [22] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  arXiv preprint arXiv:2212.04488, 2022.
  * [23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  International Journal of Computer Vision, 128(7):1956–1981, 2020\.
  * [24] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  * [25] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [26] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  In International Conference on Learning Representations, 2019.
  * [27] Gregory Murphy.  The big book of concepts.  MIT press, 2004.
  * [28] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
  * [29] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning, pages 16784–16804. PMLR, 2022.
  * [30] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach.  Benchmark for compositional text-to-image synthesis.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  * [31] Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang.  CRIPP-VQA: Counterfactual reasoning about implicit physical properties via video question answering.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9856–9870, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [33] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [35] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [38] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022.
  * [39] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov.  Effective data augmentation with diffusion models.  arXiv preprint arXiv:2302.07944, 2023.
  * [40] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [41] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [42] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  Gan inversion: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [43] Mateo Espinosa Zarlenga, Barbiero Pietro, Ciravegna Gabriele, Marra Giuseppe, Francesco Giannini, Michelangelo Diligenti, Shams Zohreh, Precioso Frederic, Stefano Melacci, Weller Adrian, et al.  Concept embedding models: Beyond the accuracy-explainability trade-off.  In Advances in Neural Information Processing Systems, volume 35, pages 21400–21413. Curran Associates, Inc., 2022.
  * [44] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  * [45] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  IEEE transactions on pattern analysis and machine intelligence, 41(8):1947–1962, 2018.

## Appendix

##  Appendix A Social Impact

In this paper, we introduce ConceptBed, a novel benchmark and evaluation
framework designed for conducting comprehensive studies on few-shot Concept
Learning using T2I diffusion models. Previous evaluations of recent works in
this field have been limited to a small number of test concepts, thus
hindering our understanding of their practical applicability. Through our
benchmark, we demonstrate that while current concept learners exhibit
impressive performance, a substantial gap remains that must be addressed. As
pioneers in constructing this extensive evaluation set, we anticipate that
future research will incorporate a broader range of potential concepts.
Additionally, we propose a novel evaluation metric and framework that can be
applied to any concept learning setting, extending its efficacy beyond the
confines of ConceptBed dataset. Ultimately, this research directly contributes
to the advancement of Human-Level Artificial Intelligence (HLAI) objectives,
fostering the development of more robust and capable systems.

##  Appendix B ConceptBed Resources

Name | Link
---|---
Project Page: | https://conceptbed.github.io
Code: | https://github.com/ConceptBed/evaluations
Data: | https://conceptbed.github.io/data.html
Results Explorer: | https://conceptbed.github.io/explorer.html

We release the code (a collection of concept learners), data, and results
(randomly selected from over 200,000 images) for readers to refer to or browse
through.

##  Appendix C Extended Related Work

Evaluations of T2I Concept Learners. Previous studies on concept learning have
conducted evaluations and model comparisons using their own test sets. For
instance, Textual Inversion [10] employed approximately 20 concepts with
around 27 unique compositions, while DreamBooth [35] utilized 30 concepts with
50 unique compositions. Custom Diffusion [22], on the other hand, employed 10
concepts with 24 unique compositions. Notably, these works were evaluated on a
relatively small subset of concepts and a limited list of compositions. In
order to address the limitations associated with a centralized evaluation set,
we introduce the ConceptBed dataset, which consists of 284 concepts and over
5000 unique compositions. Additionally, we present an automated procedure for
concept and composition collection, enabling the creation of large-scale
datasets.

Downstream Applications of Diffusion Models. In addition to concept learning,
diffusion models have demonstrated potential for various downstream
applications. For example, approaches such as prompt-to-prompt [15] and
DiffEdit [7] have been proposed for image editing tasks. In another case,
diffusion-generated images have shown improvements in ImageNet accuracy [1].
Furthermore, methods similar to textual inversion have been found to enhance
few-shot classification performance [39].

Out-of-Distribution Detection and Domain Adaptation/Generalization. While the
research directions of out-of-distribution detection and domain
adaptation/generalization have been explored independently to a significant
extent, they share a common focus on measuring and controlling model
confidence. Prior works have employed various confidence quantification
methods, including: 1) Expected Calibration Error (ECE), which is a popular
metric for assessing classifier calibration by measuring the difference
between model accuracy and its probability [28], and 2) Expected Uncertainty
Calibration Error (UCE), a recently proposed metric that quantifies the
miscalibration of uncertainty by calculating the difference between model
error and its uncertainty [12]. Given the high variance observed in diffusion
models with respect to hyperparameters, we introduce a novel method,
leveraging the ConceptBed dataset, to quantify generation variances and
measure deviations using $\operatorname{\text{{CCD}{}}}$. ECE and UCE can
serve as alternative metrics for quantifying deviations and evaluating concept
learners. Our experimental results in Appendix H.1 demonstrate that ECE
performs equally well as $\operatorname{\text{{CCD}{}}}$ in assessing concept
alignment. In the context of concept alignment, ECE and UCE can be computed
based on generated concept-specific images, without considering the
performance on the ground truth target images. Lower values of these metrics
indicate better performance, albeit at the cost of explainability regarding
the source of errors (e.g., overconfidence or lack of confidence in the
model). To address these potential ambiguities, we propose
$\operatorname{\text{{CCD}{}}}$, which measures the discrepancy in
probabilities between ground truth and generated concept-specific images,
thereby facilitating a more nuanced understanding of the limitations of
concept learners.

##  Appendix D Preliminaries on text-conditioned diffusion models

Diffusion Models: The training procedure of Stable Diffusion can be described
as follows: given a training pair $(\mathcal{I},y)$, the input image
$\mathcal{I}$ is first mapped to a latent vector $z$ and get a variably-noised
vector $z^{t}:=\alpha^{t}z^{t-1}+\sigma^{t}\epsilon$, where
$\epsilon\sim\mathcal{N}(0,1)$ is a noise term and $\alpha^{t},\sigma^{t}$ are
terms that control the noise schedule and sample quality. At training time,
the time-conditioned UNet is optimized to predict the noise $\epsilon$ and
recover the initial $z$, via conditioning on the text prompt $y$, the model is
trained with a squared error loss on the predicted noise term as follows:

$\displaystyle\mathcal{L}_{\text{diffusion}}=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,1),t,y}\Big{[}||\epsilon-\epsilon_{\theta}(z^{t},t,y)||_{2}^{2}\Big{]}$
(4)

where $t$ is uniformly sampled from $\\{1,\dots,T\\}$.

At inference time, Stable Diffusion is sampled by iteratively denoising
$z^{T}\sim\mathcal{N}(0,I)$ conditioned on the text prompt $y$. Specifically,
at each denoising step $t=1,\dots,T$, $z^{t-1}$ is obtained from both $z^{t}$
and the predicted noise term of UNet whose input is $z^{t}$ and text prompt
$y$. After the final denoising step, $z^{0}$ will be mapped back to yield the
generated image $\mathcal{I}$.

Textual-Inversion (TI): TI uses the pre-trained Stable Diffusion and fine-
tunes it to learn the specific concepts using a few images. Given a small set
of images depicting the target concept
$\mathcal{X}_{c}=\\{x_{c}^{i};i\in\\{0,...,m\\}\\}$, and with the rare-token
$y_{k}$ (i.e., V*), we want to learn the embedding corresponding to $y_{k}$.
This input-conditioned text can be represented as “A photo of a V*”.

TI follows the exact same process of Stable Diffusion. Unlike Stable
Diffusion, TI optimizes the text conditional encoder ($C_{\phi}$) with respect
to the rarely occurring token $y_{k}$ using the Latent Diffusion Model (LDM)
objective function:

$\displaystyle\mathcal{L}_{\text{TI}}=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,1),t,y}\Big{[}||\epsilon-\epsilon_{\theta}(z^{t},t,C_{\phi}(y))||_{2}^{2}\Big{]}$

Note that $z^{t}$ is the noised $x$ where $x\in X_{c}$. Intuitively, the
objective is to correctly remove the added noise (while training) and optimize
$C_{\phi}$ with respect to $y_{k}$. At inference time, a random noise tensor
is sampled and a text prompt (containing the rare-token $y_{k}$) is used to
generate the image it using fine-tuned $C_{\phi}$.

DreamBooth: While textual-inversion can be used to learn various concepts
depending on the training images and corresponding set of text prompts,
DreamBooth is proposed to learn the specific properties of the target subject:
“A photo of a V* dog”. In the case of DreamBooth, we do not optimize
$C_{\phi}$ and instead, it optimizes $\epsilon_{\theta}$.

To overcome the challenges (overfitting and language drift) of fine-tuning the
full model, DreamBooth contains the class-specific prior-preserving loss.
Essentially, this method uses the pre-trained diffusion model generated
samples ($X_{pr}=\\{\hat{x_{i}};\hat{x_{i}}=f(\epsilon,c_{pr})\\}$) to
supervise the training. Here, $\epsilon\sim\mathcal{N}(0,1)$ and conditioning
vector $c_{pr}=C_{\phi}(``<\mathrm{concept-name}>^{\prime\prime})$. Therefore,
the proposed loss becomes:

$\displaystyle\mathcal{L}_{\text{DB}}$
$\displaystyle=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,1),t,y,x\in
X_{c}}\Big{[}||\epsilon-\epsilon_{\theta}(z^{t},t,C_{\phi}(y))||_{2}^{2}\Big{]}$
$\displaystyle+\lambda*\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,1),t,y_{pr},\hat{x}\in
X_{pr}}\Big{[}||\epsilon-\epsilon_{\theta}(z^{t},t,C_{\phi}(y_{pr}))||_{2}^{2}\Big{]}$

Custom-Diffusion: For single-concept learning, Custom Diffusion is essentially
the combination of Textual Inversion and DreamBooth. The objective function of
Custom Diffusion is the same as DreamBooth but instead of optimizing whole
UNet (i.e., $\epsilon_{\theta}$), Custom Diffusion optimizes the embedding
corresponding to V* from $C_{\phi}$ and key/value weights from Cross Attention
Layers of the UNet model.

##  Appendix E ConceptBed Dataset Analysis

###  E.1 Concept Statistics

Our dataset, ConceptBed, comprises a total of 284 concepts. Among these
concepts, 200 are sourced from the CUB dataset, 80 from ImageNet, and 4 from
PACS. The concepts and their respective categories are presented in Table 9.
We use the CUB dataset for attribute-level analysis, while the ImageNet
concepts are included to ensure a diversity of concepts.

###  E.2 Composition Categorization

Figure 6: Left-most figure shows the world-cloud of the ConceptBed
compositions. Middle figure shows the statistics of composition categories.
Right-most shows the statistics of multiple composition categories combined
together to generate composite text prompts.

We leverage the Visual Genome dataset to create composite prompts for each of
the 80 ImageNet concepts. This process yields over 5000 unique compositions,
resulting in a rich variety of composite prompts. Table 10 provides detailed
statistics on the compositions for each concept. Furthermore, Figure 6
illustrates the distribution of composition categories within the ConceptBed
dataset. For the sake of simplicity, ConceptBed contains composite prompts
that combine up to two different compositions. To determine the composition
type, we employ the GPT3 (text-davinci-003) model for few-shot classification.
Figure 7 showcases the instruction and in-context examples used to categorize
each text phrase.

Figure 7: Screenshot depicting few-shot classification using GPT3. We keep the
instruction and few-shot examples constant and change the target phrase to get
the corresponding categories.

###  E.3 Question Generations

Caption | Generated Questions
---|---
two birds standing on branches | are there two birds ?
are there branches ?
tongue hanging out a dog’s mouth | is there tongue ?
is there a dog’s mouth ?
cat is licking herself | is there cat ?
is the cat licking herself ?
the monkey is holding onto a red bar | is there the monkey ?
is there a red bar ?
Propeller blade on an aircraft | is there propeller blade ?
is there an aircraft ?
the vine is around clock | is there the vine ?
is there clock ?
man driving the boat | is there man ?
is there the boat ?
Table 4: Question Generation. Examples of generated existence-related
questions from captions.

Rather than relying solely on image-text similarity, we evaluate compositions
through VQA performance using synthetically generated boolean questions (i.e.,
yes or no) based on the composite text phrases. To create these questions, we
manually filter out salient words such as nouns, attributes, and verbs, and
formulate questions corresponding to each of these words. This process enables
the creation of existence-related questions where the ground truth answer is
always yes. Table 4 provides examples of the questions generated for different
composite text prompts.

##  Appendix F Experimental Setup

Hyper-Parameter | Textual Inversion (LDM) | Textual Inversion (SD) | DreamBooth | Custom Diffusion
---|---|---|---|---
Base Model | LDM | Stable Diffusion - v1.5 | Stable Diffusion - v1.5 | Stable Diffusion - v1.5
Optimized | V* | V* | UNet | V* \+ CrossAtten(k,v)
Optimization Steps | 3000 | 3000 | 400 | 250
Learning Rate | 5e-4 | 5e-4 | 5e-6 | 1e-5
Place-Holder Token | * | <object> | sks | <new1>
Regularizer | - | - | ✔ | ✔
Regularization Images | - | - | 200 | 200
# if inference steps | 50 | 50 | 50 | 50
Guidance Scale | 7.5 | 7.5 | 7.5 | 7.5
Noise Scheduler | - | PNDMScheduler | PNDMScheduler | PNDMScheduler

Table 5: Hyper-parameters. The table summarizes the different hyper-parameter
settings for all baselines considered in this work to help the reproducibility
of results.

Hyper-parameters | DomainPACS | ObjectImageNet | Fine-GrainedCUB | Compositions
---|---|---|---|---
Model Architecture | ResNet18 | ConceNeXt-base | - | ViLT
Pre-trained Weights | timm | timm | - | dandelin/vilt-b32-finetuned-vqa
Pre-training Dataset | ImageNet | ImageNet | - | MSCOCO, GCC, SBU, VG
# of target concepts | 4 | 80 | 200/112 | 1
Objective Function | NLL | NLL (w outlier exposure) [14] | NLL | NLL

Table 6: Oracle Hyper-parameters. This table summarizes the different hyper-
parameters used for Oracles. We first take the pre-trained model weights and
then fine-tune them on target concepts from the ConceptBed dataset. Here, NLL
refers to the Negative Log-Likelihood.

Table 5 presents the hyperparameter details for the baseline methodologies
employed in our benchmarking process. To ensure fair comparisons, we generate
images for all concepts using the same number of inference steps and guidance
scale. Specifically, Textual Inversion (SD), DreamBooth, and Custom Diffusion
utilize the Stable Diffusion V1.5 pre-trained
model222https://huggingface.co/runwayml/stable-diffusion-v1-5. Regarding
Textual Inversion, we explore two variants: 1) Latent Diffusion Model-based,
and 2) Stable Diffusion-based. By incorporating different pre-trained models,
we aim to investigate their impact on learning novel concepts, and our
findings reveal significant differences. For instance, Textual Inversion (LDM)
outperforms Textual Inversion (SD) when learning style as a concept, while the
SD version excels in adapting object-level concepts.

Table 6 outlines the hyperparameter settings for each oracle. It is important
to note that we can employ any type of classifier as an oracle, as elaborated
in the subsequent section. In our approach, we initially take pre-trained
models and further, fine-tune them on concepts using the negative log-
likelihood objective. However, it is well-established that classifiers may
exhibit misclassification tendencies with high confidence. To address this,
for ObjectsImageNet, we additionally incorporate an outlier-exposure objective
function.

##  Appendix G Human Annotations

To evaluate the effectiveness of our ConceptBed evaluation framework, we
conducted human evaluations consisting of three distinct studies: object-based
concept similarity, style-based concept similarity, and traditional image-text
similarity for evaluating compositions. For the object and style-based concept
similarity, we asked participants to rate the likelihood of the target image
being the same as three reference images on a scale of 1 to 5. A rating of 1
indicated the least similarity, while a rating of 5 indicated an exact match
in terms of concept. We ensured that human annotators did not compare
generated images from different concept learning strategies; instead, they
rated each image independently. Regarding the composition evaluation, we
simply asked annotators to rate the image-text similarities on the same 1-5
scale, with 1 representing the least similarity and 5 representing an exact
match. Figures 8, 9, and 10 present screenshots of the MTurk interface used
for each type of human evaluation.

To ensure comprehensive coverage, we randomly selected 100 generated images
and obtained evaluations from three unique workers for each image. This
resulted in a total of 900 evaluations from human annotators. To assess the
relationship between human evaluations and various baseline evaluation
metrics, as well as our $\operatorname{\text{{CCD}{}}}$ method, we computed
Pearson’s correlation. Our findings indicate a strong correlation between the
human evaluations and our CCD evaluation metric.

##  Appendix H Ablations

###  H.1 Different Confidence Measures

Table 7 presents a comprehensive comparison of various confidence
quantification metrics employed in Out-Of-Distribution (OOD) detection.
Notably, all these metrics outperform the baseline metric DINO, as evidenced
by their consistently high correlation scores, reaching at least $-0.88$.
However, two metrics, namely $\operatorname{\text{{CCD}{}}}$, and ECE, exhibit
a significantly higher correlation with human scores, with values of $-0.98$.
This remarkable correlation can be attributed to the comprehensive nature of
these metrics, which consider both the probability and the change in class
prediction. In contrast, MSP solely focuses on the probability aspect, while
Accuracy only takes into account the change in class prediction. Furthermore,
both $\operatorname{\text{{CCD}{}}}$ and ECE exhibit strong correlations with
human scores, with values of $-0.9868$ and $-0.9860$, respectively. Although
$\operatorname{\text{{CCD}{}}}$ slightly outperforms ECE, both metrics can be
used interchangeably. Nonetheless, for the sake of simplicity and continuous
relative performance measurements, we introduce the
$\operatorname{\text{{CCD}{}}}$ metric, which offers better explainability. It
is worth noting that the negative sign in the correlation coefficients stems
from the inherent differences between the nature of these metrics.
Specifically, lower values of $\operatorname{\text{{CCD}{}}}$ indicate better
performance, while higher scores in human evaluations indicate superior
performance.

Models | Accuracy | MSP | ECE | $\operatorname{\text{{CCD}{}}}$
---|---|---|---|---
Textual Inversion (LDM) | 78.99% | 0.8601 | 0.0799 | 0.1196
Textual Inversion (SD) | 78.15% | 0.8963 | 0.0620 | 0.0662
DreamBooth | 79.25% | 0.8882 | 0.0686 | 0.0880
Custom Diffusion | 74.63% | 0.8370 | 0.1650 | 0.2309
original | 89.37% | 0.9412 | 0.0505 | 0.0000
Table 7: Possible Confidence Quantification Metrics. This table summarizes the results using different existing confidence quantification metrics on ConceptBed dataset on 80 concepts from ImageNet. Here, MSP refers to the Maximum Softmax Probability and ECE refers to Expected Calibration Error. Models | ResNet18 | Inception-V4 | ViT-Large | ConvNeXt
---|---|---|---|---
Textual Inversion (LDM) | 0.0107 | 0.0773 | 0.1165 | 0.1196
Textual Inversion (SD) | -0.0100 | 0.0201 | 0.0599 | 0.0662
DreamBooth | 0.0214 | 0.0485 | 0.0786 | 0.0880
Custom Diffusion | 0.1538 | 0.1845 | 0.2286 | 0.2309
original | 0.0000 | 0.0000 | 0.0000 | 0.0000
Table 8: Effects of different classifiers as Oracles. This table summarizes
the $\operatorname{\text{{CCD}{}}}$ performance based on the different types
of classifiers across the parameters range.

###  H.2 Choice of Classifiers for Oracles

In Table 8, we explore the impact of utilizing different types of classifiers
as oracles. Our analysis encompasses four distinct classifiers, each
characterized by an increasing number of parameters. Intriguingly, the choice
of classifier appears to have a negligible effect, as
$\operatorname{\text{{CCD}{}}}$ consistently demonstrates strong correlations
with human scores, surpassing a Pearson’s correlation of at least $-0.98$.

##  Appendix I Qualitative Results

Figure 11 presents qualitative examples showcasing the performance of various
baseline methods across different style concepts. Notably, the textual
inversion methods demonstrate limitations in preserving object-specific
features and accurately learning the desired style. Furthermore, both
DreamBooth and Custom Diffusion exhibit challenges in effectively capturing
and reproducing the intended styles. In Figure 12, we delve into the object-
specific learned concepts obtained through the baseline methodologies.
Notably, Custom Diffusion struggles in acquiring and comprehending new
concepts, thus offering an explanation for its relatively lower performance in
terms of concept alignment. To gain further insights, Figure 13 offers a
comparison of the generated images using Custom Diffusion at different random
seeds. The results indicate that Custom Diffusion successfully generates the
learned concepts in three out of four instances. However, when tasked with
generating concept-specific images based on composite text prompts, Custom
Diffusion struggles to maintain fidelity to the learned concept.

To facilitate a more comprehensive understanding of the ConceptBed benchmark
and its results, we have developed an online results explorer, which provides
readers with a user-friendly interface for exploring and analyzing the
benchmark outcomes. This results explorer can be found at:
https://conceptbed.github.io/explorer.html

Concept Source |  Concepts
---|---
PACS |  Art-Painting Cartoon Photo Sketch
ImageNet |  langur hand-held_computer guenon brambling desktop_computer speedboat titi airship tiger_cat organ squirrel_monkey bluetick siamang yawl lifeboat ambulance beagle digital_clock fire_engine Walker_hound gondola pill_bottle fireboat proboscis_monkey moving_van rotisserie slide_rule Irish_wolfhound junco cab magpie robin jeep colobus airliner gibbon letter_opener garbage_truck limousine English_foxhound borzoi baboon basset capuchin convertible analog_clock redbone canoe spider_monkey bulbul Afghan_hound goldfinch patas tabby web_site grand_piano laptop chickadee Dutch_oven black-and-tan_coonhound marmoset chimpanzee macaque police_van tow_truck cleaver howler_monkey bloodhound pickup house_finch beer_bottle notebook water_ouzel orangutan Madagascar_cat gorilla indri beach_wagon jay indigo_bunting
CUB |  Black_footed_Albatross Laysan_Albatross Sooty_Albatross Groove_billed_Ani Crested_Auklet Least_Auklet Parakeet_Auklet Rhinoceros_Auklet Brewer_Blackbird Red_winged_Blackbird Rusty_Blackbird Yellow_headed_Blackbird Bobolink Indigo_Bunting Lazuli_Bunting Painted_Bunting Cardinal Spotted_Catbird Gray_Catbird Yellow_breasted_Chat Eastern_Towhee Chuck_will_Widow Brandt_Cormorant Red_faced_Cormorant Pelagic_Cormorant Bronzed_Cowbird Shiny_Cowbird Brown_Creeper American_Crow Fish_Crow Black_billed_Cuckoo Mangrove_Cuckoo Yellow_billed_Cuckoo Gray_crowned_Rosy_Finch Purple_Finch Northern_Flicker Acadian_Flycatcher Great_Crested_Flycatcher Least_Flycatcher Olive_sided_Flycatcher Scissor_tailed_Flycatcher Vermilion_Flycatcher Yellow_bellied_Flycatcher Frigatebird Northern_Fulmar Gadwall American_Goldfinch European_Goldfinch Boat_tailed_Grackle Eared_Grebe Horned_Grebe Pied_billed_Grebe Western_Grebe Blue_Grosbeak Evening_Grosbeak Pine_Grosbeak Rose_breasted_Grosbeak Pigeon_Guillemot California_Gull Glaucous_winged_Gull Heermann_Gull Herring_Gull Ivory_Gull Ring_billed_Gull Slaty_backed_Gull Western_Gull Anna_Hummingbird Ruby_throated_Hummingbird Rufous_Hummingbird Green_Violetear Long_tailed_Jaeger Pomarine_Jaeger Blue_Jay Florida_Jay Green_Jay Dark_eyed_Junco Tropical_Kingbird Gray_Kingbird Belted_Kingfisher Green_Kingfisher Pied_Kingfisher Ringed_Kingfisher White_breasted_Kingfisher Red_legged_Kittiwake Horned_Lark Pacific_Loon Mallard Western_Meadowlark Hooded_Merganser Red_breasted_Merganser Mockingbird Nighthawk Clark_Nutcracker White_breasted_Nuthatch Baltimore_Oriole Hooded_Oriole Orchard_Oriole Scott_Oriole Ovenbird Brown_Pelican White_Pelican Western_Wood_Pewee Sayornis American_Pipit Whip_poor_Will Horned_Puffin Common_Raven White_necked_Raven American_Redstart Geococcyx Loggerhead_Shrike Great_Grey_Shrike Baird_Sparrow Black_throated_Sparrow Brewer_Sparrow Chipping_Sparrow Clay_colored_Sparrow House_Sparrow Field_Sparrow Fox_Sparrow Grasshopper_Sparrow Harris_Sparrow Henslow_Sparrow Le_Conte_Sparrow Lincoln_Sparrow Nelson_Sharp_tailed_Sparrow Savannah_Sparrow Seaside_Sparrow Song_Sparrow Tree_Sparrow Vesper_Sparrow White_crowned_Sparrow White_throated_Sparrow Cape_Glossy_Starling Bank_Swallow Barn_Swallow Cliff_Swallow Tree_Swallow Scarlet_Tanager Summer_Tanager Artic_Tern Black_Tern Caspian_Tern Common_Tern Elegant_Tern Forsters_Tern Least_Tern Green_tailed_Towhee Brown_Thrasher Sage_Thrasher Black_capped_Vireo Blue_headed_Vireo Philadelphia_Vireo Red_eyed_Vireo Warbling_Vireo White_eyed_Vireo Yellow_throated_Vireo Bay_breasted_Warbler Black_and_white_Warbler Black_throated_Blue_Warbler Blue_winged_Warbler Canada_Warbler Cape_May_Warbler Cerulean_Warbler Chestnut_sided_Warbler Golden_winged_Warbler Hooded_Warbler Kentucky_Warbler Magnolia_Warbler Mourning_Warbler Myrtle_Warbler Nashville_Warbler Orange_crowned_Warbler Palm_Warbler Pine_Warbler Prairie_Warbler Prothonotary_Warbler Swainson_Warbler Tennessee_Warbler Wilson_Warbler Worm_eating_Warbler Yellow_Warbler Northern_Waterthrush Louisiana_Waterthrush Bohemian_Waxwing Cedar_Waxwing American_Three_toed_Woodpecker Pileated_Woodpecker Red_bellied_Woodpecker Red_cockaded_Woodpecker Red_headed_Woodpecker Downy_Woodpecker Bewick_Wren Cactus_Wren Carolina_Wren House_Wren Marsh_Wren Rock_Wren Winter_Wren Common_Yellowthroat

Table 9: List of all concepts from ConceptBed library based on their data source. Concept | Action | Attribute | Counting | Relation | Overall
---|---|---|---|---|---
laptop | 17 | 18 | 2 | 40 | 52
tow_truck | 97 | 348 | 35 | 409 | 645
hand-held_computer | 17 | 18 | 2 | 40 | 52
gorilla | 9 | 11 | 0 | 11 | 19
chimpanzee | 9 | 11 | 0 | 11 | 19
pickup | 97 | 348 | 35 | 409 | 645
yawl | 116 | 178 | 36 | 466 | 567
beagle | 380 | 807 | 24 | 496 | 1222
bulbul | 62 | 270 | 11 | 142 | 374
spider_monkey | 9 | 11 | 0 | 11 | 19
borzoi | 380 | 807 | 24 | 496 | 1222
analog_clock | 1 | 61 | 10 | 71 | 109
letter_opener | 11 | 10 | 0 | 21 | 31
water_ouzel | 62 | 270 | 11 | 142 | 374
web_site | 17 | 18 | 2 | 40 | 52
garbage_truck | 97 | 348 | 35 | 409 | 645
bloodhound | 380 | 807 | 24 | 496 | 1222
basset | 380 | 807 | 24 | 496 | 1222
proboscis_monkey | 9 | 11 | 0 | 11 | 19
Dutch_oven | 58 | 85 | 13 | 111 | 194
fireboat | 116 | 178 | 36 | 466 | 567
black-and-tan_coonhound | 380 | 807 | 24 | 496 | 1222
speedboat | 116 | 178 | 36 | 466 | 567
beach_wagon | 98 | 213 | 17 | 363 | 497
airliner | 4 | 8 | 2 | 11 | 20
titi | 9 | 11 | 0 | 11 | 19
marmoset | 9 | 11 | 0 | 11 | 19
beer_bottle | 1 | 9 | 0 | 14 | 20
magpie | 62 | 270 | 11 | 142 | 374
Irish_wolfhound | 380 | 807 | 24 | 496 | 1222
lifeboat | 116 | 178 | 36 | 466 | 567
brambling | 62 | 270 | 11 | 142 | 374
rotisserie | 58 | 85 | 13 | 111 | 194
junco | 62 | 270 | 11 | 142 | 374
ambulance | 98 | 213 | 17 | 363 | 497
gondola | 116 | 178 | 36 | 466 | 567
tabby | 424 | 992 | 42 | 727 | 1592
cleaver | 11 | 10 | 0 | 21 | 31
limousine | 98 | 213 | 17 | 363 | 497
desktop_computer | 17 | 18 | 2 | 40 | 52
colobus | 9 | 11 | 0 | 11 | 19
house_finch | 62 | 270 | 11 | 142 | 374
chickadee | 62 | 270 | 11 | 142 | 374
cab | 98 | 213 | 17 | 363 | 497
notebook | 17 | 18 | 2 | 40 | 52
squirrel_monkey | 9 | 11 | 0 | 11 | 19
digital_clock | 1 | 61 | 10 | 71 | 109
canoe | 116 | 178 | 36 | 466 | 567
indri | 9 | 11 | 0 | 11 | 19
English_foxhound | 380 | 807 | 24 | 496 | 1222
airship | 4 | 8 | 2 | 11 | 20
capuchin | 9 | 11 | 0 | 11 | 19
tiger_cat | 424 | 992 | 42 | 727 | 1592
bluetick | 380 | 807 | 24 | 496 | 1222
Afghan_hound | 380 | 807 | 24 | 496 | 1222
moving_van | 97 | 348 | 35 | 409 | 645
jay | 62 | 270 | 11 | 142 | 374
police_van | 97 | 348 | 35 | 409 | 645
howler_monkey | 9 | 11 | 0 | 11 | 19
langur | 9 | 11 | 0 | 11 | 19
gibbon | 9 | 11 | 0 | 11 | 19
redbone | 380 | 807 | 24 | 496 | 1222
organ | 3 | 24 | 12 | 41 | 68
slide_rule | 17 | 18 | 2 | 40 | 52
goldfinch | 62 | 270 | 11 | 142 | 374
pill_bottle | 1 | 9 | 0 | 14 | 20
siamang | 9 | 11 | 0 | 11 | 19
convertible | 98 | 213 | 17 | 363 | 497
baboon | 9 | 11 | 0 | 11 | 19
Walker_hound | 380 | 807 | 24 | 496 | 1222
guenon | 9 | 11 | 0 | 11 | 19
indigo_bunting | 62 | 270 | 11 | 142 | 374
grand_piano | 3 | 24 | 12 | 41 | 68
fire_engine | 97 | 348 | 35 | 409 | 645
robin | 62 | 270 | 11 | 142 | 374
macaque | 9 | 11 | 0 | 11 | 19
orangutan | 9 | 11 | 0 | 11 | 19
jeep | 98 | 213 | 17 | 363 | 497
patas | 9 | 11 | 0 | 11 | 19
Madagascar_cat | 9 | 11 | 0 | 11 | 19
Table 10: This table shows the composition statistics by categories. Here,
overall means the unique compositions per concept and less than or equal to
the sum of all four compositions as one composite prompt can belong up to two
composition categories.

Figure 8: An example of human annotation for determining the concept alignment
for object-specific concepts. Figure 9: An exmaple of human annotationfor
determining the concept alignment for  style-specific concepts. Figure 10: The
example of Human Annotation for determining the image-text alignment.

Figure 11: Qualitative examples of the style specific four concepts. Figure
12: Qualitative examples of the object-specific four concepts. Figure 13:
Qualitative examples from Custom Diffusions at different random seeds. Left
most four figures are the target concept images. Top-Right four images are
object-specific generated images. While Bottom-Right four generated images are
on different composite text prompts.

##  Appendix J Limitations

We introduce the first comprehensive benchmark for large-scale concept
learning, encompassing 284 distinct concepts and a vast collection of 33,000
composite prompts. However, there are infinitely many concepts, and evaluating
all of them is next to impossible. Therefore, we recommend that future works
benchmark the novel methodologies with the combination of both ConceptBed and
selective qualitative examples. While training and evaluating numerous models
on an expanded subset of concepts can be resource-intensive, our approach,
ConceptBed, employs an automated strategy that effortlessly scales to
incorporate an extensive range of concepts. Our benchmark primarily evaluates
concept learning strategies derived from Stable Diffusion models. However, the
dataset and evaluation framework we present in ConceptBed can serve as a good
foundation for assessing any text-conditioned concept learners, including
inversion methodologies. It is important to note that the limitations inherent
to Stable Diffusion models, which form the core of our experiments, extend to
other concept learners, such as spatial relationships. Hence, while ConceptBed
utilizes composite text prompts pre-trained on text-to-image models, future
work will explore strategies to enable concept learners to adapt rapidly to
novel concepts and achieve state-of-the-art performance on our benchmark. In
addition to the above, concept learning holds promise for enhancing
performance in various application domains, such as refining existing concepts
to mitigate potential biases present in Stable Diffusion models and
incorporating spatial relations like left/right. These areas offer fertile
ground for further exploration and can contribute to the advancement of
concept learning techniques. By addressing these limitations and exploring
potential application areas, we aim to propel the development of concept
learning methods that consistently push the boundaries of performance on the
ConceptBed benchmark.