[

# Understand, Improve and Iterate: Towards Developing Explainable Metrics for
Evaluating and Improving Fine-Grain Text-to-Image Alignment

Jaskirat Singh1 Liang Zheng1,2
1The Australian National University 2Australian Centre for Robotic Vision
https://1jsingh.github.io/divide-evaluate-and-refine

# Divide, Evaluate, and Conquer: Developing Explainable Metrics for Evaluating
and Improving Fine-Grain Text-to-Image Alignment

Jaskirat Singh1 Liang Zheng1,2
1The Australian National University 2Australian Centre for Robotic Vision
https://1jsingh.github.io/divide-evaluate-and-refine

# Divide, Evaluate, and Refine: A Simple Decompositional Perspective on
Evaluating and Improving Text-to-Image Alignment

Jaskirat Singh1 Liang Zheng1,2
1The Australian National University 2Australian Centre for Robotic Vision
https://1jsingh.github.io/divide-evaluate-and-refine

# Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image
Alignment with
Iterative VQA Feedback

Jaskirat Singh1 Liang Zheng1,2
1The Australian National University 2Australian Centre for Robotic Vision
https://1jsingh.github.io/divide-evaluate-and-refine

###### Abstract

The field of text-conditioned image generation has made unparalleled progress
with the recent advent of latent diffusion models. While remarkable, as the
complexity of given text input increases, the state-of-the-art diffusion
models may still fail in generating images which accurately convey the
semantics of the given prompt. Furthermore, it has been observed that such
misalignments are often left undetected by pretrained multi-modal models such
as _CLIP_. To address these problems, in this paper we explore a simple yet
effective decompositional approach towards both evaluation and improvement of
text-to-image alignment. In particular, we first introduce a _D
ecompositional-Alignment-Score_ which given a complex prompt decomposes it
into a set of disjoint assertions. The alignment of each assertion with
generated images is then measured using a VQA model. Finally, alignment scores
for different assertions are combined aposteriori to give the final text-to-
image alignment score. Experimental analysis reveals that the proposed
alignment metric shows significantly higher correlation with human ratings as
opposed to traditional _CLIP, BLIP_ scores. Furthermore, we also find that the
assertion level alignment scores provide a useful feedback which can then be
used in a simple iterative procedure to gradually increase the expression of
different assertions in the final image outputs. Human user studies indicate
that the proposed approach surpasses previous state-of-the-art by 8.7% in
overall text-to-image alignment accuracy.

Figure 1:  We propose a training-free decompositional framework which helps
both better evaluate (Sec. 3.1) and gradually improve (Sec. 3.2) text-to-image
alignment using iterative VQA feedback.

##  1 Introduction

The field of text-to-image generation has made significant advancements with
the recent advent of large-scale language-image (LLI) models [1, 2, 3, 4, 5].
In particular, text-conditioned latent diffusion models have shown
unparalleled success in generating creative imagery corresponding to a diverse
range of free-form textual descriptions. However, while remarkable, it has
been observed [6, 7, 8] that as the complexity of the input text increases,
the generated images do not always accurately align with the semantic meaning
of the textual prompt.

To facilitate the reliable use of current text-to-image generation models for
practical applications, it is essential to answer two key questions: 1) Can we
detect such fine-grain misalignments between the input text and the generated
output in a robust manner? and 2) Once detected, can we improve the text-to-
image alignment for failure cases? While several metrics for evaluating text-
to-image alignment (_e.g_., CLIP [9], BLIP [10], BLIP2 [11]) exist, it has
been observed [7, 12] that a high score with these metrics can be achieved
even if the image does not fully correspond with input prompt. For instance,
in Fig. 2, an output image (containing only pink trees) shows high CLIP/BLIP
scores with the text “pink trees and yellow car” even if yellow car is not
present. Evaluating text-to-image matching using the image-text-matching (ITM)
head of BLIP models has also been recently explored [10, 11]. However, the
generated scores also show a similar tendency to favor the main subject of
input prompt. Furthermore, even if such misalignments are detected, it is not
clear how such information can be used for improving the quality of generated
image outputs in a reliable manner.

Figure 2: _Overview._ _Top:_ Traditional methods for evaluating text-to-image
alignment _e.g_., CLIP [9], BLIP-2 [10] and BLIP2-ITM (which provides a binary
image-text matching score between 0 and 1) often fail to distinguish between
good (_right_) and bad (_left_) image outputs and can give high scores even if
the generated image is not an accurate match for input prompt (missing yellow
car). In contrast, by breaking down the prompt into a set of disjoint
assertions and then evaluating their alignment with the generated image using
a VQA model [10], the proposed Decompositional-Alignment Score (DA-score)
shows much better correlation with human ratings (refer Sec. 4.1). _Bottom:_
Furthermore, we show that the assertion-level alignment scores can be used
along with a simple iterative refinement strategy to reliably improve the
alignment of generated image outputs (refer Sec. 4.2).

To address these problems, in this paper we explore a simple yet effective
decompositional approach towards both evaluation and improvement of fine-grain
text-to-image alignment. In particular, we propose a _D ecompositional-
Alignment-Score_ (DA-Score) which given a complex text prompt, first
decomposes it into a set of disjoint assertions about the content of the
prompt. The alignment of each of these assertions with the generated image is
then measured using a VQA model [13, 10]. Finally, the alignment scores for
diffferent assertions are combined to give an overall text-to-image alignment
score. Our experiments reveal that the proposed evaluation score shows
significantly higher correlation with human ratings over prior evaluation
metrics (_e.g_., CLIP, BLIP, BLIP2) (Sec. 4.1).

Furthermore, we also find that the assertion-level alignment scores provide a
useful and explainable feedback for determining which parts of the input
prompt are not being accurately described in the output image. We show that
this feedback can then be used to gradually improve the alignment of the
generated images with the input text prompt. To this end, we propose a simple
iterative refinement procedure (Fig. 3), wherein at each iteration the
expressivity of the least-aligned assertion is improved by increasing the
weightage/cross-attention strength (refer Sec. 3.2) of corresponding prompt
tokens during the reverse diffusion process. Through both qualitative and
quantitative analysis, we find that the proposed iterative refinement process
allows for generation of better aligned image outputs over prior works [7, 6,
8] while on average showing comparable inference times (Sec. 4.2).

##  2 Related Work

Figure 3: _Iterative refinement (Col:1-3;4-6) for improving text-to-image
alignment._ We propose a simple iterative refinement approach which uses the
decompositional alignment scores (refer Sec. 3.1) as feedback to gradually
improve the alignment of the generated images with the input text-prompt.

Text to Image Generation Models. Text conditional image synthesis is a topic
of keen interest in the vision community. For instance, [14, 15, 16, 17, 18]
use GANs to perform text guided image generation. Similarly, [19, 5] explore
the use of autoregressive models for zero-shot text to image generation.
Recently, diffusion-based-models [1, 2, 3, 4, 5, 20, 21] have emerged as a
powerful class of methods for performing text-conditional image synthesis over
diverse range of target domains.

While remarkable, generating images which align perfectly with the input text-
prompt remains a challenging problem [6, 8, 7, 22]. To enforce, heavier
reliance of generated outputs on the provided text, classifier-free guidance
methods [23, 2, 3] have been proposed. Similarly, use of an additional
guidance input to improve controllability of text-to-image generation have
recently been extensively explored [24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35]. However, even with their application, the generated images are often
observed to exhibit fine-grain misalignments such as missing secondary objects
[7, 6] with the input text prompt.

Evaluating Image-Text Alignment. Various protocols for evaluating text-image
alignment in a reference-free manner have been proposed [9, 10, 11]. Most
prior works [9, 3, 2, 5] typically use the cosine similarity between the text
and image embedding from large-scale multi-modal models [9, 36, 37, 38] such
as CLIP [9], BLIP [10], BLIP-2 [11] for evaluating the alignment scores.
Recently, [10, 11] also show the application of BLIP/BLIP-2 models for image-
text matching using image retrieval. However, as shown in Fig. 2, these scores
can give very high scores even if the generated images do not full align with
the input text prompt. Furthermore, unlike our approach image-text alignment
is often represented through a single scalar value which does not provide an
explainable measure which can be used to identify/improve weaknesses of the
image generation process.

Recent contemporary work [39] also explores the idea of VQA for evaluating T2I
alignment. However, the use of VQA is limited to evaluation alone. In
contrast, we further demonstrate how the interpretability of VQA scores can
then be used to reliably improve the performance of T2I models.

Improving Image-Text Alignment. Recently several works [6, 8, 7] have been
proposed to explore the problem of improving image-text alignment in a
training free manner. Liu _et al_. [6] propose to modify the reverse diffusion
process by composing denoising vectors for different image components.
However, it has been observed [7] that it struggles while generating
photorealistic compositions of diverse objects. Feng _et al_. [8] use scene
graphs to split the input sentence into several noun phrases and then assign a
designed attention map to the output of the cross-attention operation. In
another recent work, Chefer _et al_. [7] extend the idea of cross-attention
map modification to minimize missing objects but instead do so by modifying
the noise latents during the reverse diffusion process. While effective at
reducing missing objects, we find that the performance / quality of output
images can suffer as the number of subjects in the input prompt increases
(refer Sec. 4.2).

Besides training-free methods, recent contemporary work [40, 41] has also
explored the possibility of improving image-text alignment using human
feedback to finetune existing latent diffusion models. However this often
requires the collection of large-scale human evaluation scores and finetuning
the diffusion model across a range of diverse data modalities which can be
expensive. In contrast, we explore a training free approach for improvement of
fine-grain text-to-image alignment.

##  3 Our Method

Figure 4: _Method Overview._ Given a text prompt $\mathcal{P}$ and an
initially generated output $\mathcal{I}_{0}$, we first generate a set of
disjoint assertions $a_{i}$ regarding the content of the caption. The
alignment of the output image $\mathcal{I}_{0}$ with each of these assertions
is then calculated using a VQA model. Finally, we use the assertion-based-
alignment scores $u_{i}(\mathcal{I}_{0},\mathcal{P})$ as feedback to increase
the weightage $w_{i}$ (of the assertion with least alignment score) in a
parameterized diffusion model formulation $\mathcal{D}$ (Sec. 3.2). This
process can then be performed in an iterative manner to gradually improve the
quality of the generated outputs until a desirable threshold for the overall
alignment score $\Omega(\mathcal{I}_{k},\mathcal{P})$ is reached.

Given the image generation output $\mathcal{I}$ corresponding to a text prompt
$\mathcal{P}$, we wish to develop a mechanism for evaluation and improvement
of fine-grain text-to-image alignment. The core idea of our approach is to
take a decompositional strategy for both these tasks. To this end, we first
generate a set of disjoint assertions regarding the content of the input
prompt. The alignment of the output image $\mathcal{I}$ with each of these
assertions is then calculated using a VQA model. Finally, we use the
assertion-based-alignment scores as feedback to improve the expressiveness of
the assertion with the least alignment score. This process can then be
performed in an iterative manner to gradually improve the quality of generated
outputs until a desired value for the overall alignment score is attained.

In the next sections, we discuss each of these steps in detail. In Sec. 3.1 we
first discuss the process for evaluating decompositional-alignment scores. We
then discuss the iterative refinement process for improving text-to-image
alignment in Sec. 3.2. Fig. 4 provides an overview for the overall approach.

###  3.1 Evaluating Text-to-Image Alignment

Prompt Decomposition Model. Given an input prompt $\mathcal{P}$, we first
decompose its textual information into a set of disjoint assertions (and
corresponding questions) which exhaustively cover the contents of the input
prompt. Instead of relying on human-inputs as in [6, 7]111Prior works on
improving image-text alignment often rely on human-user inputs for expressing
contents of the input prompt into its simpler constituents. For instance, Feng
_et al_. [6] require the user to describe the prompt as a
conjunction/disjunction of simpler statements. Similarly, Chefer _et al_. [7]
require the user to provide a set of entities / subjects in the prompt, over
which their optimization should be performed., we leverage the in-context
learning capability [42] of large-language models [43, 44] for predicting such
decompositions in an autonomous manner. In particular, given an input prompt
$\mathcal{P}$ and large-language model $\mathcal{M}$, the prompt decomposition
is performed using in-context learning as,

$\displaystyle\mathbf{x}=\\{x_{0},x_{1},\dots
x_{n}\\}=\mathcal{M}(\mathbf{x}\mid\mathcal{P},D_{exempler},\mathcal{T}),$ (1)

where $\mathbf{x}$ is the model output, $n$ is the number of decompositions,
$D_{exemplar}$ is the in-context learning dataset consisting 4-5 human
generated examples for prompt decomposition, and $\mathcal{T}$ is task
description. Please refer supp. material for further details on exemplar-
dataset and task-description design.

The model output $\mathbf{x}$ is predicted to contain tuples
${x}_{i}=\\{a_{i},p_{i}\\}$, where each tuple is formatted to contain
assertions $a_{i}$ and the sub-part $p_{i}$ of the original prompt
$\mathcal{P}$ corresponding to the generated assertion. For instance, given
$\mathcal{P}:\textit{`a cat and a dog'}$ the prompt decomposition can be
written as,

$\displaystyle\mathcal{M}(\mathbf{x}\mid\mathcal{P}:\textit{`a cat and a
dog'},D_{exempler},\mathcal{T})=\left[\\{\textit{`there is a cat'},\textit{`a
cat'}\\},\\{\textit{`there is a dog',`a dog'}\\}\right].$

Computing Assertion-based Alignment Scores. We next compute the alignment of
the generated image $\mathcal{I}$ with each of the disjoint assertions using a
Visual-Question-Answering (VQA) model [10]. In particular, given image
$\mathcal{I}$, assertions $a_{i},i=1,\dots n$, their rephrasing in question
format $a^{q}_{i}$ and VQA-model $\mathcal{V}$, the assertion-level alignment
scores $u_{i}(\mathcal{I},a_{i})$ are computed as,

$\displaystyle
u_{i}(\mathcal{I},a_{i})=\frac{\exp{(\alpha_{i}/\tau)}}{\exp{(\alpha_{i}/\tau)}+\exp{(\beta_{i}/\tau)}},\
\textit{where}\quad\alpha_{i}=\mathcal{V}(\textit{`yes'}\mid\mathcal{I},a^{q}_{i}),\quad\beta_{i}=\mathcal{V}(\textit{`no'}\mid\mathcal{I},a^{q}_{i}),$

where $\alpha_{i},\beta_{i}$ refer to the logit-scores of VQA-model
$\mathcal{V}$ for input tuple (image $\mathcal{I}$, question $a^{q}_{i}$)
corresponding to output tokens _‘yes’,‘no’_ respectively. Hyperparameter
$\tau$ controls the temperature of the softmax operation and controls the
confidence of the alignment predictions.

Figure 5: _Visualizing the prompt decomposition process._ By dividing a
complex prompt $\mathcal{P}$ into a set of disjoint assertions $a_{i}$, we are
able to identify the sub-prompts $p_{i}$ (circled) which are not expressed in
the image output using VQA, and thereby address them using iterative
refinement (Sec. 3.2).

Combining Alignment Scores. Finally, the assertion level alignment-scores
$u_{i}(\mathcal{I},a_{i})$ are combined to give the overall text-to-image
alignment score $\Omega(\mathcal{I},\mathcal{P})$ between image $\mathcal{I}$
and prompt $\mathcal{P}$ as,

$\displaystyle\Omega(\mathcal{I},\mathcal{P})=\frac{\sum_{i}\lambda_{i}(\mathcal{P},a_{i})\
u_{i}(\mathcal{I}_{k},a_{i})}{\sum_{i}\lambda_{i}(\mathcal{P},a_{i})},$ (2)

where weights $\lambda_{i}(\mathcal{P},a_{i})$ refer to the importance of
assertion $a_{i}$ in capturing the overall content of the input prompt
$\mathcal{P}$, and allows the user to control the relative importance of
different assertions in generating the final image output222For simplicity
reasons, we mainly use $\lambda_{i}=1\forall i$ in the main paper. Further
analysis on variable $\lambda_{i}$ to account for variable information content
or visual verifiability of an assertion are provided in supp. material..
Please refer Fig. 4 for the overall implementation.

###  3.2 Improving Text to Image Alignment

In addition to predicting overall text-to-image alignment score, we find that
assertion-level alignment scores $u_{i}(\mathcal{I},a_{i})$ also provide a
useful and explainable way for determining which parts of the input prompt
$\mathcal{P}$ are not being accurately described in the output image
$\mathcal{I}$. This feedback can then be used in an iterative manner to
improve the expressivity of the assertion with least alignment score
$u_{i}(\mathcal{I},q_{i})$, until a desired threshold for the overall text-
image alignment score $\Omega(\mathcal{I},\mathcal{P})$ is obtained.

Parameterized Diffusion Model. We first modify the image generation process of
standard diffusion models in order to control the expressiveness of different
assertions $a_{i}$ in parametric manner. In particular, we modify the reverse
diffusion process to also receive inputs weights $w_{i}$, where each $w_{i}$
controls the relative importance of assertion $a_{i}$ during the image
generation process. In this paper, we mainly consider the following two
methods for obtaining such parametric control.

Prompt Weighting. Instead of computing the CLIP [36] features from original
prompt $\mathcal{P}$ we use prompt-weighting [45] to modify the input CLIP
embeddings to the diffusion model as,

$\displaystyle\text{CLIP}(\mathcal{P})=\mathcal{W}(\mathcal{P},\\{\text{CLIP}(p_{i}),w_{i}\\}_{i=1}^{n}))$
(3)

where $\mathcal{W}$ refers to the prompt-weighting function from [1, 45],
$p_{i}$ refers to the sub-prompt (Sec. 3.1) corresponding to assertion
$a_{i}$, and weights $w_{i}$ control the relative weight of different sub-
prompts $p_{i}$ in computing the overall CLIP embedding for prompt
$\mathcal{P}$.

Cross-Attention Control. Similar to [7], we also explore the idea of modifying
the noise latents $z_{t}$ during the reverse diffusion process, to increase
the cross-attention strength of the main noun-subject for each sub-assertion
$a_{i}$. However, instead of only applying the gradient update for the least
dominant subject [7], we modify the loss for the latent update in parametric
form as,

$\displaystyle
z_{t}=z_{t}-\alpha\nabla_{z_{t}}\mathcal{L}(z_{t},\\{w_{i}\\}_{i=1}^{n})),\
\mbox{where}\
\quad\mathcal{L}(z_{t},\\{w_{i}\\}_{i=1}^{n})=\sum_{i}w_{i}(1-\text{max}\
G(\mathcal{A}^{t}_{i})),$ (4)

where $\alpha$ is the step-size, $\mathcal{A}^{t}_{i}$ refer to the attention
map corresponding to the main noun-subject in assertion $a_{i}$, $G$ is a
smoothing function and weights $w_{i}$ control the extent to which the
expression of different noun-subjects in the prompt (for each assertion) will
be increased in the next iteration.

Iterative Refinement. Given the above parametric formulation for controlling
expression of different assertions, we next propose a simple yet effective
iterative refinement approach towards improving text-to-image alignment. In
particular, at any iteration $k\in[1,5]$ during the refinement process, we
first compute both overall text-image similarity score
$\Omega(\mathcal{I}_{k},\mathcal{P})$ and assertion-level alignment scores
$u_{i}(\mathcal{I}_{k},\mathcal{P})$. The image generation output
$\mathcal{I}_{k+1}$ for the next iteration is then computed as,

$\displaystyle\mathcal{I}_{k+1}=\mathcal{D}(\mathcal{P},\\{w^{k+1}_{i}\\}_{i=1}^{n}));\
\mbox{where}\quad w_{i}^{k+1}=\begin{cases}w_{i}^{k}+\Delta,\ \text{if}\quad
i=\text{argmin}_{l}\ u_{l}(\mathcal{I},\mathcal{P})\\\
w_{i}^{k}\quad\text{otherwise}\end{cases},$ (5)

where $\mathcal{D}$ refers to the parametrized diffusion model and $\Delta$ is
a hyper-parameter. This iterative process is then performed until a desirable
threshold for the overall alignment score
$\Omega(\mathcal{I}_{k},\mathcal{P})$ is reached. The image generation output
$\mathcal{I}^{\star}$ at the end of the refinement process is then computed
as,

$\displaystyle\mathcal{I}^{\star}=\text{argmax}_{\mathcal{I}_{k}}\Omega(\mathcal{I}_{k},\mathcal{P}).$
(6)

##  4 Experiments

Figure 6: _Method comparisons w.r.t correlation with human ratings._ We
compare the correlation of different text-to-image alignment scores with those
obtained from human subjects, as the number of subjects in the input prompt
(refer Sec. 4) is varied. We observe that the proposed alignment score (DA-
score) provides a better match for human-ratings over traditional text-to-
image alignment scores.

Figure 7: _Qualitative comparison w.r.t text-to-image alignment._ We compare
the outputs of our iterative refinement approach with prior works [1, 6, 8, 7]
on improving quality of generated images with changing number of subjects
(underlined) from 2 to 5. Please zoom-in for best comparisons.

(a) _Object Relationship_ : Eval-and-Refine helps better capture both presence
and relationship between the objects.

(b) _Overlapping entities_ : Proposed approach can better handle cases with
overlapping entities (spacesuit, person).

(c) _Prompt Complexity_ : Eval-and-Refine shows better alignment as number of
subjects in input prompt increase.

Figure 8: _Additional comparisons with Attend-and-Excite._ We analyse three
main ways in which the proposed iterative-refinement improves over Attend-and-
Excite [7] (refer Sec. 4.2 for details).

Dataset. Since there are no openly available datasets addressing semantic
challenges in text-based image generation with human annotations, we introduce
a new benchmark dataset Decomposable-Captions-4k for method comparison. The
dataset consists an overall of 24960 human annotations on images generated
using all methods [1, 6, 7] (including ours) across a diverse set of 4160
input prompts. Each image is a given rating between 1 and 5 (where 1
represents that ‘image is irrelevant to the prompt’ and 5 represents that
‘image is an accurate match for the prompt’).

Furthermore, unlike prior works [7] which predominantly analyse the
performance on relatively simple prompts with two subjects (_e.g_. object a
and object b), we construct a systematically diverse pool of input prompts for
better understanding text-to-image alignment across varying complexities in
the text prompt. In particular, the prompts for the dataset are designed to
encapsulate two axis of complexity: _number of subjects_ and _realism_. The
number of subjects refers to the number of main objects described in the input
prompt and varies from 2 (_e.g_., _a cat with a ball_) to 5 (_e.g_., _a woman
walking her dog on a leash by the beach during sunset_). Similarly, the
_realism_ of a prompt is defined as the degree to which different concepts
naturally co-occur together and varies as _easy_ , _medium_ , _hard_ and _very
hard_. _easy_ typically refers to prompts where concepts are naturally co-
occurring together (_e.g_., _a dog in a park_) while _very hard_ refers to
prompts where concept combination is very rare (_e.g_., _a dog playing a
piano_). Further details regarding the dataset are provided in supplementary
material.

###  4.1 Evaluating Text-to-Image Alignment

Baselines. We compare the performance of the _Decompositional-Alignment Score_
with prior works on evaluating text-to-image alignment in a reference-free
manner. In particular, we show comparisons with CLIP [9], BLIP [10] and BLIP2
[11] scores where the text-to-image alignment score is computed using the
cosine similarity between the corresponding image and text embeddings. We also
include comparisons with BLIP-ITM and BLIP2-ITM which directly predict a
binary image-text matching score (between 0 and 1) for input prompt and output
image. Finally, we report results on the recently proposed text-to-text (T2T)
similarity metric [7] which computes image-text similarity as the average
cosine similarity between input prompt and captions generated (using BLIP)
from the input image.

Quantitative Results. Fig. 6 shows the correlation between human annotations
and predicted text-to-image alignment scores across different metrics on the
_Decomposable-Captions_ dataset. We observe that the _DA-Score_ shows a
significantly higher correlation with human evaluation ratings as opposed to
prior works across varying number of subjects $N\in[2,5]$ in the input prompt.
We also note that while the recently proposed T2T similarity score [7] shows
comparable correlation with ours for $N=2$, its performance significantly
drops as the number of subjects in the input prompt increases.

###  4.2 Improving Text-to-Image Alignment

Figure 9: _Variation of alignment accuracy with prompt difficulty._ We observe
that while the accuracy of all methods decreases with increasing difficulty in
prompt _realism_ (refer Sec. 4), the proposed iterative refinement approach
consistently performs better than prior works.

In this section, we compare the performance of our iterative refinement
approach with prior works on improving text-to-image alignment in a training-
free manner. In particular, we show comparisons with 1) Stable Diffusion [1],
2) Composable Diffusion [6] 3) StructureDiffusion [8] and 4) Attend-and-Excite
[7]. All images are generated using the same seed across all methods.

Qualitative Results. Results are shown in Fig. 7. As shown, we observe that
Composable Diffusion [6] struggles to generate photorealistic combinations of
objects especially as number of subjects in the prompt increase.
StructureDiffusion [8] helps in addressing some missing objects _e.g_.,
telescope in example-1, but the generated images tend to be semantically
similar to those produced by the original Stable Diffusion model, and thus
does not significantly improve text-to-image alignment.

Attend-and-Excite [7] shows much better performance in addressing missing
objects (_e.g_., telescope in example-1 and umbrella in example-4). However,
as sumamrized in Fig. 8 we observe that it suffers from 3 main challenges: 1)
_Object Relationship_ (Fig. 8a): we observe that despite having desired
objects, generated images may sometimes fail to convey relationship between
them. For _e.g_., in row-1 Fig. 8 while output images show both a _lion and
guitar_, the lion does not seem to be playing the guitar. In contrast, Eval-
and-Refine is able to describe both presence and relation between objects in a
better manner. 2) _Overlapping Entities_ (Fig. 8b): For images with
overlapping entities (_e.g_., _person and spacesuit_), we observe that Attend-
and-Excite [7] typically spends most of gradient updates balancing between the
overlapping entities, as both entities (_person and spacesuit_) occupy the
same cross-attention region. This can lead to outputs where a) other important
aspects (_e.g_., _lake_ in Col-3) or b) one of the two entities (_e.g_.,
_spacesuit_) are ignored. 3) _Prompt Complexity_ (Fig. 8c): Finally, we note
that since Attend-and-Excite [7] is limited to applying the cross-attention
update _w.r.t_ the least dominant subject, as the complexity of input prompt
$\mathcal{P}$ increases, it may miss some objects (_e.g_., _umbrella, beach,
sunny day_) during the generation process. In contrast, the iterative nature
of our approach allows it to keep refining the output image $\mathcal{I}$
until a desirable threshold for the overall image-text alignment score
$\Omega(\mathcal{I},\mathcal{P})$ is reached.

Quantitative Results. In addition to qualitative experiments, we also evaluate
the efficacy of our approach using human evaluations. In this regard, we
report three metrics: 1) _normalized human score_ : which refers to the
average human rating (normalized between 0-1) for images generated on the
Decomposable-Captions-4k dataset. 2) _accuracy_ : indicating the percentage of
generated images which are considered as an accurate match (rating: 5) for the
input text prompt by a human subject. 3) _pairwise-preference_ : where human
subjects are shown pair of images generated using our method and prior work,
and are supposed to classify each image-pair as a win, loss or tie (win
meaning our method is preferred). For our approach we consider two variants 1)
_Ours (PW)_ which performs iterative refinement using only prompt-weighting,
and 2) _Ours (PW + CA)_ where iterative refinement is performed using both
prompt weighting and introducing cross-attention updates (Sec. 3.2). Pairwise
preference scores are reported while using _Ours (PW + CA)_ while comparing
with prior works.

Results are shown in Fig. 9 and Tab. 1. We observe that while the text-to-
image alignment accuracy for all methods decreases with an increased
difficulty in input text prompts (Fig. 9), we find that the our approach with
only prompt-weighting is able to consistently perform on-par or better than
Attend-and-Excite [7]. Further introduction of cross-attention updates (Sec.
3.2), allows our approach to exhibit even better performance, which
outperforms Attend-and-Excite [7] by 8.67 % in terms of overall alignment
accuracy of the generated images. These improvements are also reflected in the
pairwise comparisons where human subjects tend to prefer our approach over
prior works [7, 6, 8].

Method | Norm. Human | Alignment | Pairwise Comparison % | Inference
---|---|---|---|---
Score (%) | Accuracy (%) |  Win $\uparrow$ | Tie |  Lose $\downarrow$ | Time (s)
Stable-Diffusion [1] | 72.98 | 43.66 | 41.7 | 50.1 | 8.2 | 3.54 s
Composable-Diffusion [6] | 70.28 | 37.72 | 57.1 | 38.5 | 4.4 | 10.89 s
Structure-Diffusion [8] | 74.93 | 45.23 | 37.5 | 54.6 | 7.9 | 11.51 s
Attend-and-Excite [7] | 85.94 | 65.50 | 23.6 | 62.3 | 14.1 | 8.59 s
Ours (PW) | 89.53 | 70.28 | N/A | N/A | N/A | 10.32 s
Ours (PW + CA) | 90.25 | 74.16 | N/A | N/A | N/A | 12.24 s
Table 1: _Quantitative Results_. We report text-to-image alignment comparisons
_w.r.t_ normalized human rating score (Col:2), average alignment accuracy
evaluated by human subjects (Col:3) and pairwise user-preference scores (ours
vs prior work) (Col:4-6). Finally, we also report average inference time per
image for different methods in Col:7. We observe that our approach shows
better text-to-image alignment performance while on average using marginally
higher inference time.

Inference time comparison. Tab. 1 shows comparison for the average inference
time (per image) for our approach with prior works [7, 8, 6]. We observe that
despite the use of an iterative process for our approach, the overall
inference time is comparable with prior works. This occurs because prior works
themselves often include additional steps. For instance, Composable-Diffusion
[6] requires the computation of separate denoising latents for each statement
in the confunction/disjunction operation, thereby increasing the overall
inference time almost linearly with number of subjects. Similarly, Attend-and-
Excite [7] includes additional gradient descent steps for modifying cross-
attention maps. Moreover, such an increase is accumulated even if the baseline
Stable-Diffusion [1] model already generates accurate images. In contrast, the
proposed iterative refinement approach is able to adaptively adjust the number
of iterations required for the generation process by monitoring the proposed
_DA- Score_ for evaluating whether the generation outputs are already good
enough.

##  5 Conclusion

In this paper, we explore a simple yet effective decompositional approach for
both evaluation and improvement of text-to-image alignment with latent
diffusion models. To this end, we first propose a _Decompositional-Alignment
Score_ which given a complex prompt breaks it down into a set of disjoint
assertions. The alignment of each of these assertions with the generated image
is then measured using a VQA model. The assertion-based alignment scores are
finally combined to a give an overall text-to-image alignment score.
Experimental results show that proposed metric shows significantly higher
correlation with human subject ratings over traditional CLIP, BLIP based
image-text matching scores. Finally, we propose a simple iterative refinement
approach which uses the decompositional-alignment scores as feedback to
gradually improve the quality of the generated images. Despite its simplicity,
we find that the proposed approach is able to surpass previous state-of-the-
art on text-to-image alignment accuracy while on average using only marginally
higher inference times. We hope that our research can open new avenues for
robust deployment of text-to-image models for practical applications.

## References

  * [1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [2] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [5] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [6] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII, pages 423–439. Springer, 2022.
  * [7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  arXiv preprint arXiv:2301.13826, 2023.
  * [8] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  arXiv preprint arXiv:2212.05032, 2022.
  * [9] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [10] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [11] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [12] Roni Paiss, Hila Chefer, and Lior Wolf.  No token left behind: Explainability-aided image classification and generation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII, pages 334–350. Springer, 2022.
  * [13] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision, 2021.
  * [14] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [15] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022.
  * [16] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji.  Improving text-to-image synthesis using contrastive learning.  arXiv preprint arXiv:2107.02423, 2021.
  * [17] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 833–842, 2021.
  * [18] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5802–5810, 2019.
  * [19] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [20] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [21] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion model in generative ai: A survey.  arXiv preprint arXiv:2303.07909, 2023.
  * [22] Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant.  Character-aware models improve visual text rendering.  arXiv preprint arXiv:2212.10562, 2022.
  * [23] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [24] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2022.
  * [25] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon.  Ilvr: Conditioning method for denoising diffusion probabilistic models.  arXiv preprint arXiv:2108.02938, 2021.
  * [26] Jaskirat Singh, Stephen Gould, and Liang Zheng.  High-fidelity guided image synthesis with latent diffusion models.  arXiv preprint arXiv:2211.17084, 2022.
  * [27] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin.  Spatext: Spatio-textual representation for controllable image generation.  arXiv preprint arXiv:2211.14305, 2022.
  * [28] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: Fusing diffusion paths for controlled image generation.  arXiv preprint arXiv:2302.08113, 2, 2023.
  * [29] Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian.  Collage diffusion.  arXiv preprint arXiv:2303.00262, 2023.
  * [30] Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Vishal M Patel.  Scenecomposer: Any-level semantic image synthesis.  arXiv preprint arXiv:2211.11742, 2022.
  * [31] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al.  Reco: Region-controlled text-to-image generation.  arXiv preprint arXiv:2211.15518, 2022.
  * [32] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [33] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [35] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [37] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [38] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [39] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  arXiv preprint arXiv:2303.11897, 2023.
  * [40] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu.  Aligning text-to-image models using human feedback.  arXiv preprint arXiv:2302.12192, 2023.
  * [41] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Better aligning text-to-image models with human preference, 2023.
  * [42] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [44] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  arXiv preprint arXiv:2304.01852, 2023.
  * [45] AUTOMATIC1111.  Stable-diffusion-webui.  https://github.com/AUTOMATIC1111/stable-diffusion-webui, 2022.
  * [46] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [47] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [48] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment
with
Iterative VQA Feedback

##  Appendix A Additional Results

In this section, we include additional results for our approach which could
not be included in the main paper due to space constraints. In particular, we
report additional results for visualizing the iterative refinement process in
Sec. A.1. We also provide additional results comparing our method performance
with Attend-and-Excite [7] in Sec. A.2. Finally, in Sec. A.3, we compare our
approach with recent contemporary work on using human feedback for improving
text-to-image alignment.

###  A.1 Visualizing the Iterative Refinement Process

Figure 10: _Visualizing iterative refinement process for improving text-to-
image alignment._

Figure 11: _Visualizing iterative refinement process for improving text-to-
image alignment._

###  A.2 Additional Comparisons with Attend-and-Excite

Figure 12: _Additional Comparisons with Attend-and-Exite: Missing
Relationship._ Due to the pure object focused nature of Attend-and-Excite [7],
it may result in images where all objects are present but the relationship
between them is not accurately described. In contrast, we observe that the
iterative refinement approach is able to better describe both presence and
relationship between the objects.
Figure 13: _Additional Comparisons with Attend-and-Exite: Overlapping
Entities._ For images with overlapping entities (_e.g_., _person and
spacesuit_), we observe that Attend-and-Excite [7] typically spends most of
gradient updates balancing between the overlapping entities, as both entities
(_person and spacesuit_) occupy the same cross-attention region. This can lead
to outputs where a) other important aspects (_e.g_., _lake_ in Col-3) or b)
one of the two entities (_e.g_., _spacesuit_) are ignored.  Figure 14:
_Additional Comparisons with Attend-and-Exite: Missing Objects._ Since Attend-
and-Excite [7] is limited to applying the cross-attention update _w.r.t_ the
least dominant subject, as the complexity of input prompt increases, it may
miss some objects (_e.g_., umbrella, beach, sunny day) during the generation
process. In contrast, the iterative nature of our approach allows it to keep
refining the output image until a desirable threhold for the overall image-
text alignment score is reached.

###  A.3 Comparisons with Human-Feedback based Methods

Figure 15: _Method comparisons w.r.t correlation with human ratings._ We
compare the correlation of HPS score (CLIP finetuned with human feedback) from
Wu _et al_. [41]. We observe that while the HPS score shows improved
performance over CLIP [36], the proposed DA-score shows better correlation
with human ratings across varying number of subjects in input prompt.

Figure 16: _Comparing image alignment performance with human-feedback based
models._ We find that while human-feedback based finetuned diffusion model
from [41] improves the aesthetics (_e.g_. lighting) of the output, it does not
visually improve alignment with the input text for complex prompts.

Besides training-free methods, recent contemporary work [40, 41] has also
explored the possibility of improving image-text alignment using human
feedback to finetune existing latent diffusion models. For instance, Wu _et
al_. [41] recently release new versions of CLIP [36] and Stable Diffusion [1]
models which have been finetuned to better align with user preferences using
human-feedback data. In this section, we compare the performance of our simple
training-free approach with models released by [41] in terms of both
evaluation and improvement of fine-grain text-image matching.

Results. Fig. 15 compares the correlation of 1) Original CLIP [36] model, 2)
HPS scores (CLIP model finetuned to better align with human preference scores)
by [41], and 3) DA-Scores predicted by our method on the Decomposable-
Captions-4k dataset. We observe that while the HPS scores with the finetuned
CLIP model, show significantly higher correlation with human annotations as
opposed to original CLIP model, it still performs worse than the proposed DA-
scores.

Similarly, Fig. 16 compares the image outputs for 1) Original Stable Diffusion
[1] model, 2) HPS Adapted Stable Diffusion model from [41] and 3) DA-score
based iterative-refinement approach. We observe that while the HPS Adapted
model improves the aesthetics of the generated model (_e.g_., improved
lighting in Col-1,5 in example-1 Fig. 16), it does not improve the semantic
alignment with content of the input prompt. In contrast, while our approach
does not improve the aesthetics of the generated image, the output images show
significantly higher alignment with the input prompt.

Reason. As shown above while human preference finetuned CLIP and Stable
Diffusion models from [41] show better performance in terms of aesthetics,
they do not significantly improve visual alignment between generated images
and input text as the complexity of the prompts increases. _We believe that a
major reason behind the same is the heavy data-driven nature of human-feedback
based methods. That is, the generalization performance of the final finetuned
model often relies heavily on the diversity and nature of the human-feedback
dataset._ In current works [40, 41], human-feedback is typically collected by
showing users $4-10$ images (for the same input prompt) generated by a
pretrained Stable-Diffusion [1] model, and then asking the users to select the
best match. However as shown in the main paper, as the complexity of the input
prompt increases, the original Stable-Diffusion [1] model shows a very low
text-to-image aligment accuracy. As a result, the collected human-feedback
data is often biased towards predicting more aesthically pleasing outputs, as
opposed to outputs which improve fine-grain alignment with the content of the
input prompt.

In contrast, we propose a simple training-free approach which is able to
generalize well to both simple and hard prompts, for both evaluation and
improvement of text-to-image alignment. We also note that training-free
methods such as ours or prior works [7, 6, 8], can in-turn help improve the
performance of human-feedback based methods by providing a better quality
dataset for determining both aesthetics and content alignment of the generated
images.

##  Appendix B Implementation Details

In this section, we provide further details for the implementation of our
approach as well as other baselines [1, 7, 6, 8] used while reporting results
in the main paper. The full detailed implementation for both evaluation and
improvement of text-to-image alignment is provided in Alg. 1, 2.

Model Details. Similar to [7, 8], we use the official Stable Diffusion v1.4
model as the underlying pretrained text-to-image generation model while
reporting results with all methods [7, 8, 6] (including the iterative
refinement approach proposed in the main paper). All results are reported at
$512\times 512$ resolution while using 50 inference steps during the reverse
diffusion process. Unless otherwise specified, a fixed classifier-free
guidance scale [23] $\alpha_{cfg}=7.5$ is used for all experiments. By
default, we use the pretrained BLIP-VQA [10] model (BLIP model finetuned for
visual question answering) for predicting the assertion-alignment scores while
reporting all results.

Figure 17: _Visualizing the outputs of prompt decomposition._ By dividing a
complex prompt $\mathcal{P}$ into a set of disjoint assertions $a_{i}$, we are
able to identify the sub-prompts $p_{i}$ (circled) which are not being
expressed in the image output using VQA, and thereby address them using
iterative refinement.

Prompt Decomposition. Similar to our approach, prior works on improving image-
text alignment often rely on human-user inputs for expressing contents of the
input prompt into its simpler constituents. For instance, Feng _et al_. [6]
require the user to describe the prompt as a conjunction / disjunction of
simpler statements. Chefer _et al_. [7] require the user to provide a set of
entities / subjects in the prompt, over which the cross-attention optimization
should be performed. Similarly, in order to evaluate the _Decompositional
Alignement Scores_ , our approach relies on decomposing the input caption into
a set of disjoint assertions (with their rephrasing as a question).

Instead of relying of human inputs as in prior works, we leverage the in-
context learning capability of large-language models (LLMs) [44, 43] for
obtaining such decompositions in an autonomous manner. This allows us to
perform large-scale quantitative evaluations across different methods in a
robust manner. In particular, for all methods [7, 6] and ours, we first
collect a set of 4-5 human generated examples describing the desired outputs
(_e.g_., main subjects for [7]) for the input prompts. This example-set is
then used as an in-context dataset which is then fed to the _‘gpt-4’_ [44]
model to generate the desired prompt decompositions on prompts across the
_Decomposable-Captions-4k_ dataset. Fig. 17 provides an overview of the
different prompt decomposition outputs for our approach.

Note that while it is possible to explore relatively simpler methods _e.g_.,
extracting noun phrases for subject extraction in [7], it can lead to errors
as the complexity of the input prompt increases. For instance, for the input
_prompt: “a penguin wearing a bowtie with a bright sun in the background”_,
the extracted noun phrases will also include the word _“background”_. The use
of a LLM-based in-context framework allows us to avoid such errors. In future,
the proposed approach can be extended to allow for much lightweight
decomposition by using a low-rank finetuning [46] for adaptation of recently
released instruction-following models [47]. However, since the same is not the
main focus of our work, we leave it as a directive for future research.

Hyperparameters and Overall Algorithm. The proposed iterative refinement
approach uses a maximum of $K=5$ iterations for the refinement process. The
iterative process is terminated early if a threshold of $0.8$ for the overall
image alignment score $\Omega(\mathcal{I}_{k},\mathcal{P})$ is obtained. The
iterative refinement weights are initialized as $w_{i}=1\forall i$ for prompt
weighting (PW), and, $\gamma_{i}=0\forall i$ for cross-attention (CA) updates.
An increment $\Delta$ of $0.1$ and $1.0$ is used for updating the assertion
weights for prompt-weighting (PW) and cross-attention (CA) update methods
respectively. Furthermore, to reduce the inference time for each iteration as
compared to [7], cross-attention updates are only applied for first 20 steps
of the reverse diffusion process. Furthermore, the use of iterative cross
attention updates is also discarded. The image generation output
$\mathcal{I}^{\star}$ at the end of the refinement process is computed as,

$\displaystyle\mathcal{I}^{\star}=\text{argmax}_{\mathcal{I}_{k}}\Omega(\mathcal{I}_{k},\mathcal{P}).$
(7)

Please refer Alg. 1, 2 for the full detailed implementation (with
hyperparameters) for our approach.

Algorithm 1 DA-Score: Evaluating Text-to-Image Alignment

Input: Text prompt $\mathcal{P}$, generated image $\mathcal{I}$.
Output: Text-to-Image Alignment Score between $\mathcal{P},\mathcal{I}$
Require: Large-language model $\mathcal{M}$, VQA-model $\mathcal{V}$, exempler
dataset $\mathcal{D}$, task description $\mathcal{T}$, softmax-temperature
$\tau=0.9$

1:$\triangleright$ Prompt Decomposition

2:$\mathbf{x}=\\{x_{0},x_{1},\dots
x_{n}\\}=\mathcal{M}(\mathbf{x}\mid\mathcal{P},\mathcal{D}_{exempler},\mathcal{T}),\quad\textit{where}\
\ x_{i}=\\{a_{i},p_{i},a^{q}_{i}\\}$;

3:

4:$\triangleright$ Compute Assertion Alignment Scores using VQA

5:$u_{i}(\mathcal{I},a_{i})=\frac{\exp{(\alpha_{i}/\tau)}}{\exp{(\alpha_{i}/\tau)}+\exp{(\beta_{i}/\tau)}},\
\textit{where}\quad\alpha_{i}=\mathcal{V}(\textit{`yes'}\mid\mathcal{I},a^{q}_{i}),\quad\beta_{i}=\mathcal{V}(\textit{`no'}\mid\mathcal{I},a^{q}_{i})$

6:

7:$\triangleright$ Overall Image-Text Alignment Score

8:$\Omega(\mathcal{I},\mathcal{P})={\sum_{i}\lambda_{i}(\mathcal{P},a_{i})\
u_{i}(\mathcal{I},a_{i})}/{\sum_{i}\lambda_{i}(\mathcal{P},a_{i})},$

9:

10:return $\Omega(\mathcal{I},\mathcal{P})$.

Algorithm 2 Iterative Refinement: Improving Text-to-Image Alignment

Input: Text prompt $\mathcal{P}$, subprompts $p_{i}$, disjoint assertions in
question format $a^{q}_{i}$.
Output: Image generation output $\mathcal{I}^{\star}$ conditioned on
$\mathcal{P}$
Require: Pretrained diffusion model $\mathcal{D}$, VQA-model $\mathcal{V}$,
prompt-weighting function $\mathcal{W}$
Hyperparameters: max number of iterations $K=5$, alignment threshold
$\Omega_{max}=0.8$, weight increments $\Delta_{w}=0.1$, $\Delta_{\gamma}=1.0$,
step-size $\alpha=10$, number of cross-attention update steps $t_{0}=20$.

1:$\triangleright$ Initialize Assertion Weights

2:Initialize $w^{0}_{i}=1\ \forall i$; $\triangleright$ for prompt-weighting

3:Initialize $\gamma^{0}_{i}=0\ \forall i$; $\triangleright$ for cross-
attention updates

4:

5:$\triangleright$ Iterative Refinement

6:for $0\leq k<K$ do

7:

8: $\triangleright$ prompt weighting

9:
$y_{prompt}=\mathcal{W}(\mathcal{P},\\{\text{CLIP}(p_{i}),w^{k}_{i}\\}_{i=1}^{n}))$;

10:

11: $\triangleright$ Parameterized Reverse Diffusion

12: Sample $z_{T}\sim\mathcal{N}(0,\bm{I})$;

13: for $0<t\leq T$ do

14: $\\_,\mathcal{A}_{i}^{t}=\mathcal{D}(z_{t},y_{prompt},t)$;$\triangleright$
compute cross-attention maps

15: if $t\geq T-t_{0}$ then

16: $\triangleright$ Weighted Cross-Attention Updates

17:
$\mathcal{L}(z_{t},\\{\gamma^{k}_{i}\\}_{i=1}^{n})=\sum_{i}\gamma^{k}_{i}(1-\text{max}\
G(\mathcal{A}^{t}_{i}))$;

18:
$z_{t}=z_{t}-\alpha\nabla_{z_{t}}\mathcal{L}(z_{t},\\{\gamma^{k}_{i}\\}_{i=1}^{n}))$;

19: end if

20: $z_{t-1}=\textsc{ReverseDiff}(z_{t},t\rightarrow t-1\mid y_{prompt})$;

21: end for

22:

23: $\triangleright$ Get Decompositional-Alignment Scores

24: $\mathcal{I}_{k}=x_{0}=\textsc{Decoder}(z_{0})$;

25:
$\Omega(\mathcal{I}_{k},\mathcal{P}),\\{u_{i}(\mathcal{I}_{k},\mathcal{P})\\}_{i}^{n}=\textsc{DA-
Score}(\mathcal{I}_{k},\\{a^{q}_{i}\\}_{i})$;

26:

27: $\triangleright$ Finish If Output is Good Enough

28: if $\Omega(\mathcal{I}_{k},\mathcal{P})\geq\Omega_{max}$ then

29: return $\mathcal{I}^{\star}=\mathcal{I}_{k}$.

30: end if

31:

32: $\triangleright$ Update Assertion Weights

33: $w_{i}^{k+1}=\begin{cases}w_{i}^{k}+\Delta_{w},\ \text{if}\quad
i=\text{argmin}_{l}\ u_{l}(\mathcal{I}_{k},\mathcal{P})\\\
w_{i}^{k}\quad\text{otherwise}\end{cases}$ $\triangleright$ for prompt-
weighting

34: $\gamma_{i}^{k+1}=\begin{cases}\gamma_{i}^{k}+\Delta_{\gamma},\
\text{if}\quad i=\text{argmin}_{l}\ u_{l}(\mathcal{I}_{k},\mathcal{P})\\\
\gamma_{i}^{k}\quad\text{otherwise}\end{cases}$ $\triangleright$ for cross-
attention updates

35:end for

36:

37:$\triangleright$ Return Best Output

38:return
$\mathcal{I}^{\star}=\text{argmax}_{\mathcal{I}_{k}}\Omega(\mathcal{I}_{k},\mathcal{P})$.

##  Appendix C Decomposable Captions 4K Dataset

Overview. Since there are no openly available datasets addressing semantic
challenges in text-based image generation with human annotations, we introduce
a new benchmark dataset Decomposable-Captions-4k for method comparison. The
dataset consists an overall of 24960 human annotations on images generated
using all methods [1, 6, 7] (including ours) across a diverse set of 4160
input prompts. Each image is a given rating between 1 and 5 (where 1
represents that ‘image is semantically irrelevant to the prompt’ and 5
represents that ‘image is an accurate match for the prompt’). Fig. 18 provides
an overview of some user annotations for image-prompt pairs from the curated
dataset.

(a) _Example Samples with User Rating: 5 (Image is an accurate match for the
prompt)_

(b) _Example Samples with User Rating: 4 (Image is a good match for prompt
with minor mistakes)_

(c) _Example Samples with User Rating: 3 (Image seems like a 50-50 match for
the prompt)_

(d) _Example Samples with User Rating: 2 (Image captures minor aspects about
the input prompt)_

(e) _Example Samples with User Rating: 1 (Image seems semantically irrelevant
to the prompt)_

Figure 18: _Visualizing samples with human annotations from the Decomposable-
Captions-4k dataset._

Collecting Diverse Prompts of Varying Complexity. Furthermore, unlike prior
works [7] which predominantly analyse the performance on relatively simple
prompts with two subjects (_e.g_., object a and object b), we construct a
systematically diverse pool of input prompts for better understanding text-to-
image alignment across varying complexities in the text prompt. In particular,
the prompts for the dataset are designed to encapsulate two axis of
complexity: _number of subjects_ and _realism_. The number of subjects refers
to the number of main objects described in the input prompt and varies from 2
(_e.g_., _a cat with a ball_) to 5 (_e.g_., _a woman walking her dog on a
leash by the beach during sunset_). Similarly, the _realism_ of a prompt is
defined as the degree to which different concepts naturally co-occur together
and varies as _easy_ , _medium_ , _hard_ and _very hard_. _easy_ typically
refers to prompts where concepts are naturally co-occurring together (_e.g_.,
_a dog in a park_) while _very hard_ refers to prompts where concept
combination is very rare (_e.g_., _a dog playing a piano_).

(a) _Example sample outputs with our method for number of subjects: 2_

(b) _Example sample outputs with our method for number of subjects: 3_

(c) _Example sample outputs with our method for number of subjects: 4_

(d) _Example sample outputs with our method for number of subjects: 5_

Figure 19: _Visualizing variation in number of subjects in prompts from the
Decomposable-Captions-4k dataset. All images are generated using the proposed
iterative refinement approach._

(a) _Example sample outputs with our method for realism difficulty: easy_

(b) _Example sample outputs with our method for realism difficulty: medium_

(c) _Example sample outputs with our method for realism difficulty: hard_

(d) _Example sample outputs with our method for realism difficulty: very hard_

Figure 20: _Visualizing variation in realism difficulty from the Decomposable-
Captions-4k dataset. All prompts have been sampled using number of subjects =
3 subset of the overall dataset._

Prompt Generation. A key part of the _Decomposable-Captions-4k_ dataset is to
collect a set of diverse input prompts of varying complexity which would allow
for a much more comprehensive evaluation across different methods. Moreover
unlike prompts found in typical large-scale image-text datasets [48, 49], the
generated prompts should be imaginative and be able to describe novel and
often non-realistic combinations of different concepts (_e.g_. _a lion playing
a piano_).

To this end, we leverage the diverse language modelling capabilities of large-
scale large language models (LLM) [44, 43] in order to generate novel prompts
of varying complexity and realism. In particular, given a desired number of
subjects $N$ (between 2 and 5), we first use the _GPT-4_ model [44] API with
8K context length to come up with an initial random subject _e.g_., a dog. The
model is then instructed to conditionally generate a second subject (_e.g_.,
sunglasses) which is then combined with first subject to generate the sub-
prompt _“a dog wearing sunglasses”_. This process is continued until a
complete prompt with a desired number of subjects is obtained (_e.g_., _a dog
wearing sunglasses on a skateboard in a park_). At the end of generation
process, prompts which are grammatically inaccurate are filtered and removed.
An overview of example prompts with variable number of subjects (along with
corresponding image generation outputs with our approach) is provided in Fig.
19.

Furthermore, in order to generate prompts of varying level of _realism
difficulty_ , we generate prompts in batches of 4. In particular, during the
sequential generation process (described above) the model is prompted to
generate prompts of increasing level of _realism difficulty_ by asking it
generate additional subjects whose combination in a sentence is increasingly
more rare. For instance, for _realism difficulty: easy_ , the model is tasked
to generate additional subjects which typically co-occur together in natural
captions, which then leads to natural realistic prompts (_e.g_., _a group of
friends having picnic under a tree_). As the _realism difficulty_ is increased
the model generates input prompts where the co-occurence of different subjects
is more and more rare, thus allowing it to generate more imaginative and
challenging prompts (_e.g_. _a lion baking cookies in a kitchen_). Fig. 20
provides an overview of some example prompts with varying levels of realism
difficulty.

Figure 21: _Setup for pairwise user study comparing our method with prior
works._

Pairwise Human User Study. In addition to obtaining human annotations rating
(between 1 to 5) for each image-prompt pair (refer Fig. 18), we also perform a
pairwise user study comparing our method with prior works. In particular,
given an input text prompt $\mathcal{P}$, the participants were shown a pair
of image generation outputs comparing our method with prior works. For each
pair, the human subject is then asked to select the output image which better
aligns with the input prompt description. The human subjects are given three
options _left_ , _right_ and _tie_ , where _tie_ indicates that both images
are equally good or bad. All comparison images are generated using the same
seed at $512\times 512$ resolution with 50 inference steps for the reverse
diffusion process. Fig. 21 provides a screenshot of the user interface for
collecting the above human annotation data with pairwise comparisons.

##  Appendix D Discussion and Limitations

While the proposed iterative refinement approach shows better performance than
previous works [7, 8, 6], it still has some limitations. _First_ , the
proposed decompositional approach relies on a pretrained BLIP-VQA model [10],
for determining the alignment of the generated image with each of the disjoint
assertions. Thus, weaknesses of the pretrained BLIP-VQA model become our
weaknesses. Recall that the VQA scores help identify the areas in which the
current image generation output is lacking, which can then be addressed in the
next refinement step. However, if the VQA output is not correct, then the
model might focus on assertions which are already well expressed, which can
reduce the efficiency of the proposed iterative refinement strategy. In
future, the use of more accurate VQA models _e.g_., BLIP2-VQA can help
alleviate this problem. Furthermore, as noted through extensive quantitative
experiments across a diverse range of input prompts (refer main paper and App.
C), we find that the use of the BLIP-VQA model still yields quite competitive
results.

_Second_ , without additional information from the user, the iterative
refinement approach considers all assertions to be equally important in
determining the overall content of the input prompt. However, as the
complexity of the input prompt increases, we may wish to give more weight to
certain parts of the prompt over the others. Furthermore, it is possible that
the prompt may contain assertions which are not visually verifiable from the
output image. For instance, for the prompt _“a penguin in a shopping mall on a
weekend”_, the assertion about whether it is a weekend or not is not visually
verifiable. Similarly certain actions _e.g_. searching, singing are also not
verifiable from a single image. In future, we would like to explore a more
autonomous mechanism for including additional information like 1) user ranking
for different assertions (_i.e_., what the user considers as important in a
prompt) as well as 2) visual verifiability of a given assertion while
computing the decompositional alignment scores.

_Finally,_ as noted in Fig. 7 of the main paper, we note that while the
proposed iterative refinement approach leads to consistent improvements in
alignment accuracy over prior works, the accuracy of the alignment process
decreases as the complexity of input prompt is increased. In particular, for
prompts with _very hard_ realism difficulty, the overall alignment accuracy is
only 62.9% (Attend-and-Excite has 49.5%). This leaves much room for
improvement of text-to-image generation models. As discussed in App. A.3 one
potential solution in this direction would be to combine recent works on
human-feedback based diffusion model finetuning with the proposed training-
free approach for data collection. In particular, by generating training data
(on which human feedback is obtained) using the proposed iterative refinement
strategy instead of previously used pretrained Stable Diffusion [1] models, we
can increase the quality of the finetuning process. Using the proposed
decompositional alignment scores as pseudo-labels for learning the human-
feedback based reward model [40] is another interesting direction for future
work. However, the same is out of scope of this paper, and we leave it as a
direction for future research.