ContraFusion: Contrastively Improving Compositional Understanding in Diffusion Models via Fine-Grained Negative Images
Under review as a conference paper at ICLR 2025

CONTRAFUSION: CONTRASTIVELY IMPROVING COM-
POSITIONAL UNDERSTANDING IN DIFFUSION MODELS
VIA FINE-GRAINED NEGATIVE IMAGES

Anonymous authors
Paper under double-blind review

ABSTRACT

Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion mod-
els, they often struggle to understand compositional relationships between objects
and attributes, especially in complex settings. Existing solutions have tackled
these challenges through optimizing the cross-attention mechanism or learning
from the caption pairs with minimal semantic changes. However, can we gen-
erate high-quality complex contrastive images that diffusion models can directly
discriminate based on visual representations? In this work, we leverage large-
language models (LLMs) to compose realistic, complex scenarios and harness
Visual-Question Answering (VQA) systems alongside diffusion models to auto-
matically curate a contrastive dataset, COM-DIFF, consisting of 15k pairs of high-
quality contrastive images. These pairs feature minimal visual discrepancies and
cover a wide range of attribute categories, especially complex and natural scenar-
ios. To learn effectively from these error cases, i.e., hard negative images, we
propose CONTRAFUSION, a new multi-stage curriculum for contrastive learning
of diffusion models. Through extensive experiments across a wide range of com-
positional scenarios, we showcase the effectiveness of our proposed framework on
compositional T2I benchmarks. We will release our contrastive dataset to support
the development of generative models.

1

INTRODUCTION

The rapid advancement of text-to-image generative models (Saharia et al., 2022; Ramesh et al.,
2022) has revolutionized the field of image synthesis, driving significant progress in various appli-
cations such as image editing (Brooks et al., 2023; Zhang et al., 2024), video generation (Brooks
et al., 2024) and medical imaging (Han et al., 2024a). Despite their remarkable capabilities, state-
of-the-art models such as Stable Diffusion (Rombach et al., 2022) and DALL-E 3 (Betker et al.,
2023) still face challenges with composing multiple objects into a coherent scene (Huang et al.,
2023; Liang et al., 2024; Majumdar et al., 2024). Common issues include incorrect attribute bind-
ing, miscounting, and flawed object relationships as shown in Figure 1. For example, when given
the prompt “a red motorcycle and a yellow door”, the model might incorrectly bind the colors to the
objects, resulting in a yellow motorcycle.

Recent progress focuses on optimizing the attention mechanism within diffusion models to better
capture the semantic information conveyed by input text prompts (Agarwal et al., 2023; Chefer et al.,
2023; Pandey et al., 2023). For example, Meral et al. (2023) proposes manipulating the attention on
objects and attributes as contrastive samples during test-time to optimize model performance. While
more focused, the practical application of these methods still falls short of fully addressing attribute
binding and object relationships. Other works advocate to develop compositional generative models
to improve compositional performance as each constituent model captures distributions of an in-
dependent domain (Du & Kaelbling, 2024). However, such approach assumes a fixed prespecified
structure to compose models, limiting generalization to new distributions.

In this paper, we argue that curriculum training is crucial to equip diffusion models with a fun-
damental understanding of compositionality. Given that existing models often struggle with even
basic tasks (e.g., generating three cats when prompted with “Two cats are playing”) (Wang et al.,

1

000

001

002

003

004

005

006

007

008

009

010

011

012

013

014

015

016

017

018

019

020

021

022

023

024

025

026

027

028

029

030

031

032

033

034

035

036

037

038

039

040

041

042

043

044

045

046

047

048

049

050

051

052

053

Under review as a conference paper at ICLR 2025

A woman standing in front of 
three paintings on the wall. 
(Counting)

A white cat walks towards a 
brown dog. 
(Action, Spatial)

A yellow bowl, a blue mug 
and a pink plate on the table. 
(Obj., Counting, Color) 

Two cats are playing under a 
tree in a windy day. 
(Scene, Action, Counting)

A dog is feeding a girl. 
(Logic, Action)

!! SDXL

!! SD3

!! SDXL

!! DALLE

!! SDXL

"""

"""

"""

"""

"""

Figure 1: Limited Compositionality Understanding in Diffusion Models. Existing SOTA models
such as SDXL, DALL-E 3 often fail to correctly compose objects and attributes. The bottom are
images generated by our CONTRAFUSION.

2024), we progressively introduce more complex compositional scenarios during fine-tuning. This
staged training strategy helps models build a solid foundation before tackling intricate cases before
improving their performance on a wide range of compositional tasks.

Although many datasets exist for compositional generation (Wang et al., 2023; Feng et al., 2023a),
there remains a significant gap in datasets that offer a clear progression from simple to complex
samples within natural and reasonable contexts. Moreover, creating high-quality contrastive image
datasets is both costly and labor-intensive, especially given the current limitations of generative mod-
els in handling compositional tasks. To address this, we propose an automatic pipeline to generate
faithful contrastive image pairs, which we find crucial for guiding models to focus on compositional
discrepancies. In summary, our work can be summarized as follows:

Contrastive compositional dataset. We introduce COM-DIFF, a meticulously crafted compo-
sitional dataset consisting of high-quality contrastive images with minimal visual representation
differences, covering a wide range of attribute categories. By leveraging LLMs, we scale up the
complexity of compositional prompts while maintaining a natural context design. Our dataset fea-
tures faithful images generated by diffusion models, assisted by VQA systems to ensure accurate
alignment with the text prompts.

CONTRAFUSION: Curriculum contrastive learning. We are the first work to incorporate cur-
riculum contrastive learning into a diffusion model to improve compositional understanding. The
process is broken into three streamlined sub-tasks: (1) learning single object-attribute composi-
tion, (2) mastering attribute binding between two objects, and (3) handling complex scenes with
multiple objects. We conduct extensive experiments using the latest benchmarks and demonstrate
that CONTRAFUSION significantly boosts the model’s compositional understanding, outperforming
most baseline generative methods.

2 PRELIMINARY BACKGROUND

2.1 DIFFUSION MODELS

We implement our method on top of the state-of-the-art text-to-image (T2I) model, Stable Diffu-
sion (SD) (Rombach et al., 2022). In this framework, an encoder
into a spatial latent code z =

∈ X
reconstructs the original image, ensuring

maps a given image x

(x), while a decoder

E

(

(x))

x.

E

D

E

≈

D
A pre-trained denoising diffusion probabilistic model (DDPM) (Sohl-Dickstein et al., 2015; Ho
et al., 2020) for noise estimation and a pre-trained CLIP text encoder (Radford et al., 2021) to

2

054

055

056

057

058

059

060

061

062

063

064

065

066

067

068

069

070

071

072

073

074

075

076

077

078

079

080

081

082

083

084

085

086

087

088

089

090

091

092

093

094

095

096

097

098

099

100

101

102

103

104

105

106

107

Under review as a conference paper at ICLR 2025

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

Dataset

# Samples

DRAWBENCH (Saharia et al., 2022)
CC-500 (Feng et al., 2023a)
ATTN-AND-EXCT (Chefer et al., 2023)
T2I-COMPBENCH (Huang et al., 2023)

GEN-AI (Li et al., 2024a)

ABC-6K (Feng et al., 2023a)

WINOGROUNDT2I (Zhu et al., 2023)

COMP. SPLITS (Park et al., 2021)
WINOGROUND (Thrush et al., 2022)
EQBEN (Wang et al., 2023)
ARO (Yuksekgonul et al., 2023)
COM-DIFF (ours)

200
500
210
6000

1600

6000

22k

31k
400
250k
50k
15k

Contra.
text
✗
✗
✗
✗

Contra.
Image
✗
✗
✗
✗

✗

✓

✓

✓
✓
✓
✓
✓

✗

✗

✗

✓
✓
✓
✓
✓

Categories

Complex

3 (color, spatial, action)
1 (color)
2 (color, animal obj.)
6 (color, counting, texture, shape, (non-)spatial, complex)
8 (scene, attribute, relation, counting,
comparison, differentiation, logic)

1 (color)
20 (action, spatial, direction, color, number, size, texture,
shape, age, weight, manner, sentiment, procedure, speed, etc.)

2 (color, shape)
5 (object, relation, symbolic, series, pragmatics)
4 (attribute, location, object, count)
(relations, attributes)
8 (color, counting, shape, texture, (non-)spatial relations, scene, complex)

✓
✗
✗
✓

✓

✗

✗

✗
✗
✗
✗
✓

Table 1: The comparison of compositional T2I datasets. Contra. is the abbreviation of Contrastive.
Complex refers the samples that have multiple objects and complicated attributes and relationships.

process text prompts into conditioning vectors c(y). The DDPM model ϵ(θ) is trained to minimize
the difference between the added noise ϵ and the model’s estimate at each timestep t,

L = Ez∼E(x),y,ε∼N (0,1),t

(cid:2)||ε − εθ(zt, t, c(y))||2

2

(cid:3) .

(1)

During inference, a latent zT is sampled from N (0, 1) and is iteratively denoised to produce a latent
z0. The denoised latent z0 is then passed to the decoder to obtain the image x′ = D(z0).

2.2 COMPOSITIONAL DATASETS AND BENCHMARKS

The most commonly used data sets for object-attribute binding, including DRAWBENCH (Saharia
et al., 2022), CC-500 (Feng et al., 2023a) and ATTEND-AND-EXCITE (Chefer et al., 2023) con-
struct text prompts by conjunctions of objects and a few of common attributes like color and shape.
To more carefully examine how generative models work on each compositional category, recent
work explores the disentanglement of different aspects of text-to-image compositionality. Huang
et al. (2023) introduces T2I-COMPBENCH that constructing prompts by LLMs which covers six
categories including color, shape, textual, (non-)spatial relationships and complex compositions;
Recently, GEN-AI (Li et al., 2024a) collects prompts from professional designers which captures
more enhanced reasoning aspects such as differentiation, logic and comparison.

Another line of work proposes contrastive textual benchmarks to evaluate the compositional capa-
bility of generative models. ABC-6K (Feng et al., 2023a) contains contrast pairs by either swap-
ping the order objects or attributes while they focus on negative text prompts with minimal changes.
WINOGROUNDT2I (Zhu et al., 2023) contains 11K complex, high-quality contrastive sentence pairs
spanning 20 categories. However, such benchmarks focus on text perturbations but do not have im-
ages, which have become realistic with the advancement of generative models.

Several benchmarks featuring contrastive image pairs have also been introduced. COMPOSITIONAL
SPLITS C-CUB AND C-FLOWERS (Park et al., 2021) mainly focused on the color and shape at-
tributes of birds and flowers, sourcing from Caltech-UCSD Birds (Wah et al., 2011), Oxford-102
(Flowers) (Nilsback & Zisserman, 2008). Thrush et al. (2022) curated WINOGROUND consists of
400 high-quality contrastive text-image examples. EQBEN (Wang et al., 2023) is an early effort
to use Stable Diffusion to synthesize images to evaluate the equivariance of VLMs similarity, but
it lacks more complex scenarios. Yuksekgonul et al. (2023) emphasizes the importance of hard
negative samples and constructs negative text prompts in ARO by swapping different linguistic el-
ements in the captions sourced from COCO and sampling negative images by the nearest-neighbor
algorithm. However, it is not guaranteed the negative images found in the datasets truly match the
semantic meaning of the prompts.

3 DATA CONSTRUCTION: COM-DIFF

To address attribute binding and compositional generation, we propose a new high-quality con-
trastive dataset, COM-DIFF. Next, we introduce our design principle for constructing COM-DIFF.
Each sample in COM-DIFF consists of a pair of images (x+, x−) associated with a positive caption
t+.

3

Under review as a conference paper at ICLR 2025

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

Category

Stage-I

Stage-II

Shape

Color

Counting

Texture

Spatial

An american football. (l)
A volleyball. ((cid:204))

Ablue backpack.
A red backpack

Three birds.
Two birds.

A plastic toy.
A fluffy toy.

–

An american football and a volleyball.
A badminton ball and Frisbee.

A blue backpack and a yellow purse.
A yellow purse and a blue backpack.

Two cats and one dog.
Two dogs and one cat.

A rubber tire and a glass mirror.
A rubber mirror and a glass tire

A plate on the right of a bee.
A bee on the right of a place.

Non-spatial

A basketball player is eating dinner.
A basketball player is dancing.

A woman is passing a tennis ball to a man.
A man is passing a tennis ball to a woman.

Scene

Complex

Complex

A snowy night.
A rainy night.

Two round clock.
Three square clock.

In a serene lake during a thunderstorm.
In a serene lake on a sunny day.

Two fluffy dogs are eating apples to the right of a brown cat.
A brown dog are eating pears to the left of two fluffy cats.

Stage-III

Two green birds standing next to two orange birds on a willow tree.
An orange bird standing next to three green birds on the grass.

A man wearing a blue hat is throwing an american football from the left to the right
to a woman wearing a green pants on the playground during a snowy day.

A woman wearing a green hat is throwing a tennis ball from the right to the left
to a woman wearing a blue hat on the playground during a rainy night.

Table 2: Examples of text prompts. Each sample has a positive (top) and a negative prompt (bottom).

3.1 GENERATING TEXT PROMPTS

Our text prompts cover eight categories of compositionality: color, shape, texture, counting, spa-
tial relationship, non-spatial relationship, scene, and complex. To obtain prompts, we utilize the
in-context learning capability of LLMs. We provide hand-crafted seed prompts as examples and
predefined templates (e.g., “A {color} {object} and a {color} {object}.”) and then ask GPT-4 to
generate similar textual prompts. We include additional instructions that specify the prompt length,
no repetition, etc. In total, we generate 15400 positive text prompts. More information on the text
prompt generation is provided in the appendix A.
To generate a negative text prompt t−, we use GPT-4 to perturb the specified attributes or rela-
tionships of the objects for Stage-I data. In Stage-II, we either swap the objects or the attributes,
depending on which option makes more sense in the given context. For complex sentences, we
prompt GPT-4 to construct contrastive samples by altering the attributes or relationships within the
sentences. Table 2 presents our example contrastive text prompts.

3.2 GENERATING CONTRASTIVE IMAGES

Minimal Visual Differences. Our key idea is to generate contrastive images that are mini-
mally different in visual representations. By ”minimal,” we mean that, aside from the altered at-
tribute/relation, other elements in the images remain consistent or similar. In practice, we source
negative image samples in two ways: 1) generate negative images by prompting negative prompts
to diffusion models; 2) edit the positive image by providing instructions (e.g., change motorcycle
color to red) using MagicBrush (Zhang et al., 2024), as shown at the left of Figure 2.

Text-Image Alignment. The high-level objective of COM-DIFF is to generate positive images that
faithfully adhere to the positive text guidance, while the corresponding negative images do not align
with the positive text, despite having minimal visual differences from the positive images. As the
quality of images generated by diffusion-based T2I generative models varies significantly (Karthik
et al., 2023), we first generate 10-20 candidate images per prompt. However, how to select the most
faithful image is difficult. Existing automatic metrics like CLIPScore are not always effective at
comparing the faithfulness of images when they are visually similar. To address this, we propose

4

Under review as a conference paper at ICLR 2025

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

Data Generation

Curriculum Contrastive Learning

Neg Prompt

Instruction

Image Generation 
(SD3)

Image Editing 
(MagicBrush)

Select

GPT-4

Pos Prompt  

Text-Image Similarity

GPT-4

LLaVA
LLaVA
LLaVA

SD3

Minimal Visual

Diff.

z+
T

t+

z−
T

Obj. Attr. Rel. ↑

Dataset

Stage-I
Stage-I

Stage-II
Stage-II

Stage-III

Multi-Stage Fine-tuning

Stable Diﬀusion 3

LLaVA

1. Yes 
2. Red 
3. Two 
4. …

Select Best
CLIPScore

High

Alignment

Better
describe

Prompt:  
“A blue bowl and a pink mug “

GPT-4

Questions 
1. Is there a bowl?  
2. What color is the bowl? 
3. How many bowls are there? 
4. Is there a mug? …. 

Answers 
1. Yes 
2. Red 
3. One 
4. …

Low

LLaVA

Revised 
Caption

Figure 2: CONTRAFUSION Framework. Data generation pipeline (left) and curricumlum con-
trastive learning (right). Quality control of image geneartion (bottom): Given a prompt, SD3
generates multiple candidate images, which are evaluated by LLaVA. We select the best image by
alignment and CLIPScore. If the alignment score is low, we prompt LLaVA to describe the image
as a new revised caption based on the generated image.

decomposing each text prompt into a set of questions using an LLM and leverage the capabilities of
VQA models to rank candidate images by their alignment score, as illustrated in Figure 2 (bottom) 1.
Note the correct answers can be directly extracted from the prompts. Intuitively, we consider an
image a success if all the answers are correct or if the alignment is greater than θalign for certain
categories, such as Complex. After getting aligned images, we select the best image by automatic
metric (e.g., CLIPScore).

Empirically, we find this procedure fails to generate faithful images particularly when the prompts
become complex, as limited by the compositionality understanding of existing generative models,
which aligns with the observations of Sun et al. (2023). In response to such cases–i.e., the align-
ment scores for all candidate images are low–we introduce an innovative reverse-alignment strategy.
Instead of simply discarding low-alignment images, we leverage a VLM to dynamically revise the
text prompts based on the content of the generated images. By doing so, we generate new captions
that correct the previous inaccuracies while preserving the original descriptions, thereby improving
the alignment between the text and image.

Image-Image Similarity. Given each positive sample, we generate 20 negative images and select
the one with the highest similarity to the corresponding positive image, ensuring that the changes
between the positive and negative image pairs are minimal. In case of color and texture, we use
image editing rather than generation, as it delivers better performance for these attributes. Han et al.
(2024b) proposes that human feedback plays a vital role in enhancing model performance. For
quality assurance, 3 annotators randomly manually reviewed the pairs in the dataset and filtered 647
pairs that were obviously invalid.

4 CONTRAFUSION: CURRICULUM CONTRASTIVE FINE-TUNING

A common challenge in training models with data of mixed difficulty is that it can overwhelm the
model and lead to suboptimal learning (Bengio et al., 2009). Therefore, we divide the dataset into
three stages and introduce a simple but effective multi-stage fine-tuning paradigm, allowing the
model to gradually progress from simpler compositional tasks to more complex ones.

1Examples of decomposed questions are provided in the Appendix A.3

5

Under review as a conference paper at ICLR 2025

Figure 3: Contrastive dataset examples. Each pair includes a positive image generated from the
given prompt (left) and a negative image that is semantically inconsistent with the prompt (right),
differing only minimally from the positive image.

Stage-I: Single object.
In the first stage, the samples consist of a single object with either a specific
attribute (e.g., shape, color, quantity, or texture), a specific action, or within a simple static scene.
The differences between the corresponding negative and positive images are designed to be clear
and noticeable. For instance, “A man is walking” vs. “A man is eating”, where the actions differ
significantly, allowing the model to easily learn to distinguish between them.

Stage-II: Object compositions. We compose two objects with specified interactions and spatial
relationships. An example of non-spatial relationship is “A woman chases a dog” vs. “A yellow dog
chases a woman.” This setup helps the models learn to differentiate the relationships between two
objects.

Stage-III: Complex compositions. To further complicate the scenarios, we propose prompts with
complex compositions of attributes, objects, and scenes. Data in this stage can be: 1) contain more
than two objects; 2) assign more than two attributes to each object, or 3) involve intricate relation-
ships between objects.

Ultimately, our goal is to equip the model with the capability to inherently tackle challenges in
compositional generation. Next, we discuss how to design the contrastive loss during fine-tuning
at each stage. Given a positive text prompt t, a generated positive image x+, and corresponding
negative image x−, the framework comprises the following three major components:

Diffusion Model. The autoencoder converts the positive image and negative image to latent space
as z+
t . The encoder of the noise
estimator ϵθ is used to extract feature maps z+

0 . The noisy latent at timestep t is represented as z+

0 and z−

t and z−

et respectively.

et and z−

6

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

Counting, Missing Object Two cats, one dog, and one rabbit  are on the grass.Color A red motorcycle in front of  a yellow door Attribute BindingSpatial A black dog is in the left of a pigObject RelationshipsAction A man in yellow T-shirt is cryingComplexThree differently colored apples (yellow, green, red from left to right) with a transparent water bottle placed behind the middle apple.Action A white cat is chasing a little girl in a yellow floral dress on the grass In the British Museum, a dinosaur fossil is fighting with four caveman specimens on a circular platform Color Two blood moons hang in the night sky, and a flock of bats flies over a medieval-style castle A fully armored knight wearing a blue cape and a small golden dragon perched on their shoulder, is staring at a red evil dragon. Under review as a conference paper at ICLR 2025

Model

STABLE V1.4 (Rombach et al., 2022)
STABLE V2 (Rombach et al., 2022)
DALL-E 2 (Ramesh et al., 2022)
SDXL (Podell et al., 2023)
COMPOSABLE V2 (Liu et al., 2023)
STRUCTURED V2 (Feng et al., 2023a)
ATTN-EXCT V2 Chefer et al. (2023)
GORS (Huang et al., 2023)
PIXART-α (Chen et al., 2023)
MARS (He et al., 2024)

Attribute Binding

Object Relationship

Complex

Color

37.65
50.65
57.00
64.00
40.63
49.90
64.00
66.03
68.86
69.13

Shape

Texture

Spatial

Non-Spatial

35.76
42.21
55.00
54.00
32.99
42.18
45.17
47.85
55.82
54.31

41.56
49.22
63.74
36.45
36.45
49.00
59.63
62.87
70.44
71.23

12.46
13.42
13.00
20.00
8.00
13.86
14.55
18.15
20.82
19.24

30.79
30.96
30.00
31.00
29.80
31.11
31.09
31.93
31.79
32.10

30.80
33.86
37.00
41.00
28.98
33.55
34.01
33.28
41.17
40.49

CONTRAFUSION (Ours)

71.040.13

54.570.25

72.340.26

21.760.18

33.080.35

42.520.38

Table 3: Alignment evaluation on T2I-CompBench. We report average and standard deviations
across three runs. The best results are in bold.

Projection head. We apply a small neural network projection head g(·) that maps image repre-
sentations to the space where contrastive loss is applied. We use a MLP with one hidden layer to
obtain ht = g(zet) = W (2)σ(W (1)(zet)).

Contrastive loss. For the contrastive objective, we utilize a variant of the InfoNCE loss (van den
Oord et al., 2019), which is widely used in contrastive learning frameworks. This loss function is
designed to maximize the similarity between the positive image and its corresponding text prompt,
while minimizing the similarity between the negative image and the same text prompt. The loss for
a positive-negative image pair is expressed as follows:

L = − log

exp(sim(h+

t , f (t))/τ )

exp(sim(h+

t , f (t))/τ ) + exp(sim(h−

t , f (t))/τ )

(2)

where τ is a temperature parameter, f (·) is CLIP text encoder, sim function represents cosine simi-
larity:

sim(u, v) =

uT · v
∥u∥∥v∥

(3)

This encourages the model to distinguish between positive and negative image-text pairs.

5 EXPERIMENTS AND DISCUSSIONS

5.1

IMPLEMENTATION DETAILS

Experimental Setup In an attempt to evaluate the faithfulness of generated images, we use GPT-4
to decompose a text prompt into a pair of questions and answers, which serve as the input of our
VQA model, LLaVA v1.5 (Liu et al., 2024). Following previous work (Huang et al., 2023; Feng
et al., 2023a), we evaluate CONTRAFUSION on Stable Diffusion v2 (Rombach et al., 2022).

Baselines We compare our results with several state-ofthe-art methods, including trending open-
sourced T2I models that trained on large training data, Stable Diffusion v1.4 and Stable Diffusion
v2 (Rombach et al., 2022), DALL-E 2 (Ramesh et al., 2022) and SDXL (Podell et al., 2023). Com-
posableDiffusion v2 (Liu et al., 2023) is designed for conjunction and negation of concepts for
pretrained diffusion models. StructureDiffusion v2 (Feng et al., 2023a), Divide-Bind (Li et al.,
2024b) and Attn-Exct v2 (Chefer et al., 2023) are designed for attribute binding for pretrained dif-
fusion models. GORs (Huang et al., 2023) finetunes Stable Diffusion v2 with selected samples and
rewards. PixArt-α (Chen et al., 2023) incorporates cross-attention modules into the Diffusion Trans-
former. MARS (He et al., 2024) adapts from auto-regressive pre-trained LLMs for T2I generation
tasks.

Evaluation Metrics To quantitatively assess the efficacy of our approach, we comprehensively
evaluate our method via two primary metrics: 1) compositionality on T2I-CompBench (Huang et al.,

7

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

Under review as a conference paper at ICLR 2025

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Animal-Animal

Animal-Object

Object-Object

0.36

0.27

0.18

0.09

0

0.38

0.285

0.19

0.095

0

0.38

0.285

0.19

0.095

0

Full Prompt Minimum Object

Full Prompt Minimum Object

Full Prompt Minimum Object

Stable Diﬀusion v1.4

Attn & Exct

Composable

Structured

COMFORM

ContraFusion

Figure 5: Average CLIP image-text similarities between the text prompts and the images generated
by different models. The Full Prompt Similarity considers full text prompt. Minimum Object rep-
resents the minimum of the similarities between the generated image and each of the two object
prompts. Example of this benchmark is in subsection C.3.

2023) 2 and 2) color-object compositionality prompts (Chefer et al., 2023). We quantify the perfor-
mance using CLIPScore (Hessel et al., 2022) by evaluating the alignment of textual conditions and
corresponding generated images.

5.2 PERFORMANCE COMPARISON AND ANALYSIS

Alignment Assessment. To examine the
quality of COM-DIFF, we measure the align-
ment of the positive image and texts using CLIP
similarity. Figure 4 compares directly select-
ing the best image based on CLIPScore with
our pipeline, which leverages a VQA model to
guide image generation. These results confirm
that our approach consistently improves image
faithfulness across all categories with VQA as-
sistance during image generation and demon-
strate COM-DIFF contains high-quality image-
text pairs.

CLIPScore

VQA+CLIPScore

0.36

0.343

0.325

0.308

0.29

Color

Shape Counting Texture Spatial Non-spatial Scene Complex

Figure 4: Average CLIP similarity of image-text
pairs in COM-DIFF.

Benchmark Results Beyond the above evaluation, we also assess the alignment between the gen-
erated images using CONTRAFUSION and text condition on T2I-Compbench. As depicted in Ta-
ble 3, we evaluate several crucial aspects, including attribute binding, object relationships, and
complex compositions. CONTRAFUSION exhibits outstanding performance across 5/6 evaluation
metrics. The remarkable improvement of Complex performance is primarily attributed to Stage-III
training, where high-quality contrastive samples with complicated compositional components are
leveraged to achieve superior alignment capabilities.

Figure 5 presents the average image-text similarity on the benchmark proposed by Chefer et al.
(2023), which evaluates the composition of objects, animals, and color attributes. Compared to other
diffusion-based models, our method consistently outperforms in both full and minimum similarities
across three categories, except for the minimum similarity on Object-Object prompts. These results
demonstrate the effectiveness of our approach.

Ablation Study We conduct ablation studies on T2I-CompBench by exploring three key design
choices. First, we assess the effectiveness of our constructed dataset, COM-DIFF, by fine-tuning
Stable Diffusion v2 directly using COM-DIFF. As shown in Table 4, our results consistently outper-
form the baseline evaluation on Stable Diffusion v2 across all categories, demonstrating that our data
generation pipeline is effective. Next, we validate the impact of our contrastive loss by comparing it
with fine-tuning without this loss. The contrastive loss improves performance in the attribute binding

2More details about specific metrics used in T2I-CompBench are in Appendix.

8

Under review as a conference paper at ICLR 2025

Model

Attribute Binding

Object Relationship Complex

Color

Shape Texture

Spatial Non-Spatial

STABLE V2 (Rombach et al., 2022)
50.65
COM-DIFF
63.63
69.45
COM-DIFF + Contra. Loss
COM-DIFF + Contra. Loss + Multi-stage FT 71.04

42.21
47.64
54.39
54.57

49.22
61.64
67.72
72.34

13.42
17.77
20.21
21.76

30.96
31.21
32.09
33.08

33.86
35.02
38.14
42.52

Table 4: Ablation on T2I-CompBench. COM-DIFF refers to directly finetune SDv2 on COM-DIFF.

Figure 6: Qualitative comparison between CONTRAFUSION and other SOTA T2I modes with dif-
ferent prompts.

category, though it has less impact on object relationships and complex scenes. We hypothesize this
is because attribute discrepancies are easier for the model to detect, while relationship differences
are more complex. Finally, applying the multi-stage fine-tuning strategy leads to further improve-
ments, particularly in the Complex category, suggesting that building a foundational understanding
of simpler cases better equips the model to handle more intricate scenarios.

Qualitative Evaluation Figure 6 presents a side-by-side comparison between CONTRAFUSION
and other state-of-the-art T2I models, including SDXL, DALL-E 3, SD v3 and PixArt-α. CON-
TRAFUSION consistently outperforms the other models in generating accurate images based on the
given prompts. SDXL frequently generates incorrect actions and binds attributes to the wrong ob-
jects. DALL-E 3 fails to correctly count objects in two examples and misses attributes in the first
case. SD v3 struggles with counting and attribute binding but performs well in generating actions.
PixArt-α is unable to handle attributes, spatial relationships, and fails to count objects accurately in
the second prompt.

9

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

SD3.0DALL-E 3 OursSDXLTo the left of a stone statue, two men are running and talking.
PixArt-AlphaA brown bear and a white cat, both wearing spacesuits, are playing frisbee on Mars
Three spaceships are sailing through space, and on the top spaceship, an octagonal clock is hanging.
Two little bears swimming in the pool on the terrace through the window of a skyscraper on the rainy day
Under review as a conference paper at ICLR 2025

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

A bear and a skateboard

On the street, a bear wearing a blue 
baseball cap is standing on a 
skateboard

On the street, a bear wearing a blue 
baseball cap is holding a skateboard 
and standing to the right of a red fire 
hydrant

On a rainy day, a bear wearing a blue baseball 
cap is riding a colorful skateboard away from a 
gray trash can on the street, while a flock of 
birds flies past him

Two gray cats

Three gray cats on the grass. 

Three cats are playing on the grass, 
with one gray cat on the right of the two 
white cats.

Under the blue sky, three cats are 
playing on the grass, with the two 
white cats walking to the gray cat.

Easier, less compositionally

More complex, more compositionally

Figure 7: Examples of CONTRAFUSION for complex compositionality.

Stable Diﬀusion v3

SDXL

DALL-E 2

PixArt-Alpha

Ours

70

52.5

35

17.5

0

Aesthetic

Alignment

Aesthetic

Alignment

Aesthetic

Alignment

Aesthetic

Alignment

Figure 8: User study on 100 randomly selected prompts from Feng et al. (2023a). The ratio values
indicate the percentages of participants preferring the corresponding model.

Next, we evaluate how our approach handles complex compositionality, as shown in Figure 7. Using
the same object, “bear” and “cat,” we gradually increase the complexity by introducing variations
in attributes, counting, scene settings, interactions between objects, and spatial relationships. The
generated results indicate that our model effectively mitigates the attribute binding issues present
in existing models, demonstrating a significant improvement in maintaining accurate compositional
relationships.

User Study We conducted a user study to complement our evaluation and provide a more intuitive
assessment of CONTRAFUSION’s performance. Due to the time-intensive nature of user studies
involving human evaluators, we selected top-performing comparable models—DALLE-2, SD v3,
SDXL, and PixArt-α—all accessible through APIs and capable of generating images. As shown in
Figure 8, the results demonstrate CONTRAFUSION’s superior performance in alignment, though the
aesthetic quality may be slightly lower compared to other models.

6 CONCLUSION

In this work, we present CONTRAFUSION, a curriculum contrastive framework to overcome the
limitations of diffusion models in compositional text-to-image generation, such as incorrect attribute
binding and object relationships. By leveraging a curated dataset of positive-negative image pairs
and a multi-stage fine-tuning process, CONTRAFUSION progressively improves model performance,
particularly in complex scenarios. Our experiments demonstrate the effectiveness of this method,
paving the way for more robust and accurate generative models.

10

Under review as a conference paper at ICLR 2025

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

7 LIMITATION

Despite the effectiveness of our current approach, there are a few limitations that can be addressed
in future work. First, our dataset, while comprehensive, could be further expanded to cover an even
broader range of compositional scenarios and object-attribute relationships. This would enhance
the model’s generalization capabilities. Additionally, although we employ a VQA-guided image
generation process, there is still room for improvement in ensuring the faithfulness of the generated
images to their corresponding prompts, particularly in more complex settings. Refining this process
and incorporating more advanced techniques could further boost the alignment between the text and
image.

8 REPRODUCIBILITY

We have made efforts to ensure that our method is reproducible. Appendix A provides a description
of how we construct our dataset. Especially, Appndix A.1 and A.2 presents how we prompt GPT-
4 and use predefined template to generate text prompts of our dataset. Appendix A.3 provides an
example how we utilize VQA system to decompose a prompt into a set of questions, and answers.
Appendix B provides the details of implementation, to make sure the fine-tuning is reproducible.

REFERENCES

Aishwarya Agarwal, Srikrishna Karanam, K. J. Joseph, Apoorv Saxena, Koustava Goswami, and
Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image
synthesis. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2283–2293,
2023. URL https://api.semanticscholar.org/CorpusID:259252450.

Yoshua Bengio, J´erˆome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
ICML ’09, pp. 41–48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN
9781605585161. doi: 10.1145/1553374.1553380. URL https://doi.org/10.1145/
1553374.1553380.

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang
Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer
Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image

editing instructions, 2023. URL https://arxiv.org/abs/2211.09800.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe
Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video
generation models as world simulators. 2024. URL https://openai.com/research/
video-generation-models-as-world-simulators.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite:
Attention-based semantic guidance for text-to-image diffusion models, 2023. URL https:
//arxiv.org/abs/2301.13826.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James
Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion transformer
for photorealistic text-to-image synthesis, 2023. URL https://arxiv.org/abs/2310.
00426.

Andrew Drozdov, Nathanael Sch¨arli, Ekin Aky¨urek, Nathan Scales, Xinying Song, Xinyun Chen,
Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models,
2022. URL https://arxiv.org/abs/2209.15003.

Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model is not all you

need, 2024. URL https://arxiv.org/abs/2402.01103.

11

Under review as a conference paper at ICLR 2025

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sug-
ato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance
for compositional text-to-image synthesis, 2023a. URL https://arxiv.org/abs/2212.
05032.

Weixi Feng, Wanrong Zhu, Tsu jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu,
Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and genera-
tion with large language models, 2023b. URL https://arxiv.org/abs/2305.15393.

Xu Han, Fangfang Fan, Jingzhao Rong, and Xiaofeng Liu. Fair text to medical image diffusion
model with subgroup distribution aligned tuning, 2024a. URL https://arxiv.org/abs/
2406.14847.

Xu Han, Felix Yu, Joao Sedoc, and Benjamin Van Durme. Baby bear: Seeking a just right rating

scale for scalar annotations, 2024b. URL https://arxiv.org/abs/2408.09765.

Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei
Zhang, Zhelun Yu, Haoyuan Li, Ziwei Huang, LeiLei Gan, and Hao Jiang. Mars: Mixture of
auto-regressive models for fine-grained text-to-image synthesis, 2024. URL https://arxiv.
org/abs/2407.07614.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A
reference-free evaluation metric for image captioning, 2022. URL https://arxiv.org/
abs/2104.08718.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in

Neural Information Processing Systems, 33:6840–6851, 2020.

Hexiang Hu, Ishan Misra, and Laurens van der Maaten. Evaluating text-to-image matching using
binary image selection (bison). In 2019 IEEE/CVF International Conference on Computer Vision
Workshop (ICCVW), pp. 1887–1890, 2019. doi: 10.1109/ICCVW.2019.00237.

Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A com-
prehensive benchmark for open-world compositional text-to-image generation, 2023. URL
https://arxiv.org/abs/2307.06350.

Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don’t
succeed, try, try again: Faithful diffusion-based text-to-image generation by selection, 2023. URL
https://arxiv.org/abs/2305.13308.

Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia,
Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Genai-bench: Evaluating and improving
compositional text-to-visual generation, 2024a. URL https://arxiv.org/abs/2406.
13743.

Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide bind your attention for
improved generative semantic nursing, 2024b. URL https://arxiv.org/abs/2307.
10864.

Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal
machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10):
1–42, 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning, 2024. URL https://arxiv.org/abs/2310.03744.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional vi-
sual generation with composable diffusion models, 2023. URL https://arxiv.org/abs/
2206.01714.

12

Under review as a conference paper at ICLR 2025

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff,
Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li,
Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk,
Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind
Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. 2024
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16488–16498,
2024. URL https://api.semanticscholar.org/CorpusID:268066655.

Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is
all you need for high-fidelity text-to-image diffusion models, 2023. URL https://arxiv.
org/abs/2312.06059.

Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-
thought prompting for large multimodal models, 2024. URL https://arxiv.org/abs/
2311.17076.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number
of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp.
722–729, 2008. URL https://api.semanticscholar.org/CorpusID:15193013.

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency.
Cross-modal attention congruence regularization for vision-language relation alignment, 2023.
URL https://arxiv.org/abs/2212.10549.

Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark
for compositional text-to-image synthesis. In NeurIPS Datasets and Benchmarks, 2021. URL
https://api.semanticscholar.org/CorpusID:244906179.

Eric Pasewark, Kyle Montgomery, Kefei Duan, Dawn Song, and Chenguang Wang. Re-tuning:
Overcoming the compositionality limits of large language models with recursive tuning, 2024.
URL https://arxiv.org/abs/2407.04787.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe
Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image
synthesis, 2023. URL https://arxiv.org/abs/2307.01952.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International Conference on Machine Learning,
pp. 8748–8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-
conditional image generation with clip latents, 2022. URL https://arxiv.org/abs/
2204.06125.

Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate
Saenko. Cola: A benchmark for compositional text-to-image retrieval, 2023. URL https:
//arxiv.org/abs/2305.03689.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-
resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con-
ference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar
Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic
text-to-image diffusion models with deep language understanding. Advances in Neural Informa-
tion Processing Systems, 35:36479–36494, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised
learning using nonequilibrium thermodynamics. In International Conference on Machine Learn-
ing, pp. 2256–2265. PMLR, 2015.

13

Under review as a conference paper at ICLR 2025

Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Her-
rmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dreamsync: Aligning
text-to-image generation with image understanding feedback, 2023. URL https://arxiv.
org/abs/2311.17946.

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and
Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi-
tionality, 2022. URL https://arxiv.org/abs/2204.03162.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic-

tive coding, 2019. URL https://arxiv.org/abs/1807.03748.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011

dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng
Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models, 2023. URL
https://arxiv.org/abs/2303.14465.

Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and
conquer: Language models can plan and self-correct for compositional text-to-image generation,
2024. URL https://arxiv.org/abs/2401.15688.

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and
why vision-language models behave like bags-of-words, and what to do about it?, 2023. URL
https://arxiv.org/abs/2210.01936.

Arman Zarei, Keivan Rezaei, Samyadeep Basu, Mehrdad Saberi, Mazda Moayeri, Priyatham Kat-
takinda, and Soheil Feizi. Understanding and mitigating compositional issues in text-to-image
generative models, 2024. URL https://arxiv.org/abs/2406.07844.

Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated
dataset for instruction-guided image editing, 2024. URL https://arxiv.org/abs/2306.
10012.

Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang.
A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to-
image fidelity metrics, 2023. URL https://arxiv.org/abs/2312.02338.

14

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

Under review as a conference paper at ICLR 2025

A COM-DIFF DATA CONSTRUCTION

A.1 TEXT PROMPTS GENERATION

Here, we design the template and rules to generate text prompts by GPT-4 as follows:

• Color: Current state-of-the-art text-to-image models often confuse the colors of objects
when there are multiple objects. Color prompts in Stage-I follow fixed sentence template
“A {color} {object}.” and “A {color} {object} and a {color} {object}.” for Stage-II.

• Texture: Following Huang et al. (2023), we emphasize in the GPT-4 instructions to require
valid combinations of an object and a textural attribute. The texture prompts follows the
template “A {texture} {object}.” for Stage-I and “A {texture} {object} and a {texture}
{object}.” for Stage-II.

• Shape: We first generate objects with common geometric shapes using fixed template “A
{shape} {object}.” for Stage-I and “A {shape} {object} and a {shape} {object}.” for
Stage-II. Moreover, we ask GPT-4 to generate objects in the same category but with differ-
ent shapes, e.g., American football vs. Volleyball, as contrastive samples.

• Counting: Counting prompts in Stage-I follows fixed sentence template “{count}

{object}.” and “{count} {object} and {count} {object}.” for Stage-II.

• Spatial Relationship: Given predefined spatial relationship such as next to, on the left,
etc, we prompt GPT-4 to generate a sentence in a fixed template as “{object} {spatial}
{object}.” for Stage-II.

• Non-spatial Relationship: Non-spatial relationships usually describe the interactions be-
tween two objects. We prompt GPT-4 to generate text prompts with non-spatial relation-
ships (e.g., actions) and arbitrary nouns. We guarantee there is only one object in the
sentence for Stage-I, and two objects in Stage-II. We also find generative models fails to
understand texts like “A woman is passing a ball to a man”. It’s hard for the model to
correctly generate the directions of actions. We specially design prompts like this.

• Scene: We ask GPT-4 to generate scenes such as weather, place and background. For
Stage-I, the scene is simple, less than 5 words (e.g., on a rainy night.); For Stage-II, scenes
combine weather and background or location (e.g., in a serene lake during a thunderstorm.).

• Complex: Here, we refer to prompts that either contain more than two objects or assign
more than two attributes to each object, or involve intricate relationships between objects.
We first manually curate 10 such complex prompts, each involving multiple objects bound
to various attributes. These manually generated prompts serve as a context for GPT-4 to
generate additional natural prompts that emphasize compositionality. The complex cases
in Stage-II will be two objects with more attributes; Stage-III involves more objects.

Note that when constructing our prompts, we consciously avoided using the same ones as those
in T2I-Compbench, especially considering some prompts from T2I-CompBench are empirically
difficult to generate aligned image (e.g., “a pentagonal warning sign and a pyramidal bookend” as
shown in Figure 9), which are not well-suited for our dataset. We have filtered out similar prompts
from our dataset using LLMs to identify uncommon combinations of objects and attributes.

Figure 9: Example image that is hard to generate to align the prompt from T2I-CompBench.

15

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

Under review as a conference paper at ICLR 2025

A.2 NEGATIVE TEXT PROMPTS GENERATION

We apply in-context learning, everytime we prompt GPT-4 to generate negative cases, we give 5-
10 example test prompts each time, and make sure the generation is not repetitive, within certain
lengths.

• In Stage-I, we prompt GPT-4 to change the attribute of the object such as color, shape,
texture, counting, action, or scene, with instruction the differences should be noticeable.
• In Stage-II, we either swap the objects or attributes and let GPT-4 to validate the swapped
text prompts. For complex cases, we generate negative text by asking GPT-4 to change the
attributes/relationship/scenes.

• In Stage-III, we carefully curate complicated examples with 3-6 objects, each object have
1-3 attributes, with negative prompts change attributes, actions and spatial relationships,
scenes. And we prompt GPT-4 with such examples.

A.3 VQA ASSISTANCE

Instruction for QA Generation. Given an image description, generate one or two multiple-choice
questions that verify if the image description is correct. Table 5 shows an example of a generated
prompt and QA.

Prompt

A brown bear and a white cat, both wearing spacesuits,
are playing frisbee on Mars

Question

Answer

Is there a bear?
Is there a cat?
What color is the bear?
What color is the cat?
Does the bear wear a spacesuit?
Does the cat wear a spacesuit?
Is the bear playing the frisbee?
Is the cat playing the frisbee?
Where are they playing?

Yes
Yes
Brown
White
Yes
Yes
Yes
Yes
Mars

Table 5: VQA generated questions from a prompt.

Modifying Caption to Align Image. Next, we illustrate how we prompt VQA to revise the caption
when alignment scores of all candidate images are low. Given a generated image and a original text
prompt, we prompt VQA model with the following instruction:

Instruction: “Given the original text prompt describing the image, identify any parts that inac-
curately reflect the image. Then,generate a revised text prompt with correct descriptions, making
minimal semantic changes. Focusing on the counting, color, shape, texture, scene, spatial relation-
ship, non-spatial relationship. ”. At the same time, we will provide examples of revised caption for
in-context learning.

For example, given the following image (Figure 10) and the original text prompt, the modified
prompt generated by VQA model is as following:

Original text prompt: Three puppies are
playing on the sandy field on a
sunny day, with two black ones
walking toward a brown one.

Four puppies are

Modified prompt:
standing on a sandy field on
a sunny day, with three black
puppies and one brown puppy
facing forward.

the instruction ”Focusing on the
Note that
counting, color, shape, texture, scene, spatial

Figure 10: Image applies reverse-alignment.

16

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

Under review as a conference paper at ICLR 2025

relationship, non-spatial relationship” plays a crucial role in guiding the VQA model to provide
answers that accurately correspond to the specific attributes and categories we are interested in.
Without this directive, the model may occasionally fail to generate precise captions that correctly
describe the image.

A.4 DATA STATISTICS

Stage-I

Stage-II

In Stage-I,

the dataset

Stage-III Total

The dataset
is organized into three stages,
each progressively increasing in complex-
ity.
includes sim-
pler tasks such as Shape (500 samples), Color
(800), Counting (800), Texture (800), Non-
spatial relationships (800), and Scene (800),
totaling 4,500 samples.
Stage-II introduces
more complex compositions, with each cate-
gory—including Shape, Color, Counting, Tex-
ture, Spatial relationships, Non-spatial relation-
ships, and Scene—containing 1,000 samples,
for a total of 7,500 samples. Stage-III repre-
sents the most complex scenarios, with fewer but more intricate samples. We also include some
simple cases like Stage-I and II, each contain 200 samples, while the Complex category includes
2,000 samples, totaling 3,400 samples. Across all stages, the dataset contains 15,400 samples, pro-
viding a wide range of compositional tasks for model training and evaluation. Figure 11 show more
examples of images in our dataset.

Shape
Color
Counting
Texture
Spatial
Non-spatial
Scene
Complex

1700
2000
2000
2000
1200
2000
2000
2500

1000
1000
1000
1000
1000
1000
1000
500

200
200
200
200
200
200
200
2000

Table 6: Corpus Statistics.

500
800
800
800
-
800
800
-

A.5 COMPARISON WITH REAL CONTRASTIVE DATASET

To evaluate how our model would fare with a real hard-negative dataset, we include the results of
fine-tuning our model with COLA (Ray et al., 2023), BISON (Hu et al., 2019) evaluated by T2I-
CompBench in Table 7 (randomly sampled consistent number of samples across datasets).

Although COLA and BISON try to construct semantically hard-negative queries, the majority
of the retrieved image pairs are quite different in practice, often introducing a lot of noisy ob-
jects/background elements in the real images, due to the nature of retrieval from existing dataset.
We hypothesize this makes it hard for the model to focus on specific attributes/relationships in
compositionality. In addition, they don’t have complex prompts with multiple attributes and don’t
involve action, or scene.

In contrast, our dataset ensures the generated image pairs are contrastive with minimal visual
changes, enforcing the model to learn subtle differences in the pair, focusing on a certain cate-
gory. To the best of our knowledge, no real contrastive image dataset only differs on minimal visual
characteristics.

Dataset Color

Shape Texture

Spatial Non-Spatial Complex

COLA
BISON
Ours

62.20
59.49
71.04

48.98
49.36
54.57

53.73
48.77
72.34

15.21
14.64
21.76

30.87
31.25
33.08

33.15
32.91
42.52

Table 7: Performance of fine-tuning CONTRAFUSION on T2I-CompBench across different dataset.

A.6 QUALITY CONTROL

Coverage of LLM-generated QA Pairs We conducted human evaluations on Amazon Mechan-
ical Turk (AMT). We sampled 1500 prompt-image pairs (about 10% of the dataset, proportionally
across 3 stages) to perform the following user-study experiments. Each sample is annotated by 5
human annotators.

17

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

Under review as a conference paper at ICLR 2025

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

Figure 11: Example contrastive Image pairs in COM-DIFF

To analyze if the generated question-answer pairs by GPT-4 cover all the elements in the prompt, we
performed a user-study wherein for each question-prompt pair, the human subject is asked to answer
if the question-set cover all the objects in the prompt. The interface is presented in Figure 13.

Empirically, we find about 96% of the questions generated by GPT-4 cover all the objects, 94%
cover all the attributes/relationships.

Accuracy of Question-Answering of VQA Models To analyze the accuracy of the VQA model’s
answering results, we performed an additional user-study wherein for each question-image pair,
the human subject is asked to answer the same question. The accuracy of the VQA model is then
predicted using the human labels as ground truths. Results are displayed in Table 8.

Image Stage VQA Accuracy % Annotation Time / Image (s)

Stage-I
Stage-II
Stage-III

93.1%
91.4%
88.9%

8.7s
15.3s
22.6s

Table 8: VQA accuracy and annotation time for sampled images across different stages.

18

A green apple and a yellow pearA square clock hanging on the wallA yellow dog running with a womanA golden vase and a clear glassA cat is chasing a womanA man is walking on the streetA pyramid in the desert under the skyThree wolves in the foggy weather and a bright moon in the sky. Four hobbits are preparing to cross a hexagonal time tunnel in an underground volcanic caveA garden with various flowers, five garden gnomes—three wearing red hats and two wearing green hats—a birdbath in the middle, and a wooden bench on the left, with two butterflies flying aboveA green car is parked between two blue motorcycles, and a person wearing a red jacket is standing to the rear left of the car, holding a briefcase. Cupid is playing with a pink rabbit on white clouds with a cotton candy texture Under review as a conference paper at ICLR 2025

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

Figure 12: Comparison with Real Contrastive Dataset: COLA and BISON.

We observe that the VQA model is effective at measuring image-text alignment for the majority of
questions even as the complexity of the text prompt increases, attesting the effectiveness of pipeline.

Alignment of Revised Caption with Images To further validate the effectiveness of revising cap-
tions by VQA, we randomly sampled 500 images that are obtained by revising caption and per-
formed an additional user-study for those samples that obtain low alignment score from VQA an-
swering, but use the reverse-alignment strategy. Specifically, for each revised caption-image pair,
the human subject is asked to answer how accurately the caption describes the image. The interface
is presented in Figure 14. Note we have 5 annotators, each is assigned 100 caption-image pairs.

Empirically, we found that 4% of the samples show that the revised caption similarly describes the
image as the original caption. 94.6% of the samples show the revised caption better describes the
image. Overall,with the following settings, the average rating of the alignment between revised
caption and image is 4.66, attesting that revised caption accurately describes the image.

19

Under review as a conference paper at ICLR 2025

Figure 13: Interface for User Study: Coverage of LLM-generated QA Pairs

Figure 14: Interface for User Study: Alignment of Revised Caption with Images

Similarity of Contrastive Image Pairs We have 3 annotators in total, each annotator is assigned
2550 images (about 50% samples) to check if the positive and negative image pairs aligns with its
text prompt and are similar with small visual changes on specific attributes/relationships. We filtered
647 images from the randomly selected 7650 images, which is 8.45%, attesting the quality of most
images in the dataset.

20

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

Under review as a conference paper at ICLR 2025

Method

SD V2.1
SD-XL TURBO
DEEPFLOYD-IF
SD-XL
MIDJOURNEY V6
SD3-MEDIUM
DALL-E 3

CONTRAFUSION- SD3-MEDIUM (OURS)

Attribute

Scene

Basic

Relation

Spatial Action Part

Advanced

Avg Count Differ Compare

Logical

Negate Universal

0.75
0.79
0.82
0.82
0.85
0.86
0.91

0.89

0.77
0.82
0.83
0.84
0.88
0.86
0.91

0.88

0.72
0.77
0.80
0.80
0.86
0.87
0.89

0.90

0.72
0.78
0.81
0.81
0.86
0.86
0.89

0.91

0.69
0.76
0.80
0.81
0.85
0.88
0.91

0.88

0.74
0.79
0.81
0.82
0.85
0.86
0.90

0.89

0.66
0.69
0.69
0.71
0.75
0.74
0.78

0.80

0.63
0.65
0.66
0.67
0.73
0.77
0.76

0.79

0.61
0.64
0.65
0.64
0.70
0.72
0.70

0.73

0.50
0.51
0.48
0.49
0.49
0.50
0.46

0.51

0.57
0.57
0.57
0.57
0.64
0.73
0.65

0.73

Avg

0.58
0.60
0.60
0.60
0.65
0.68
0.65

0.72

Table 9: Gen-AI Benchmark Results.

B TRAINING IMPLEMENTATION DETAILS

We implement our apprpoach upon stable Diffuion v2.1 and we employ the pre-trained text encoder
from the CLIP ViT-L/14 model. The VAE encoder is frozen during training. The resolution is 768,
the batch size is 16, and the learning rate is 3e-5 with linear decay.

C QUANTITATIVE RESULTS

C.1 T2I-COMPBENCH EVALUATION METRICS

Following T2I-CompBench, we use DisentangledBLIP-VQA for color, shape, texture, UniDet for
spatial, CLIP for non-spatial and 3-in-1 for complex categories.

C.2 GEN-AI BENCHMARK

We further evaluate CONTRAFUSION on the Gen-AI (Li et al., 2024a) benchmark. For a fair com-
parison with DALL-E 3, we finetune our model on Stable Diffusion v3 medium. As indicated in
Table 9, CONTRAFUSION performs best on all the Advanced prompts, although it exhibits relatively
weaker performance in some of the basic categories compared to DALL-E 3.

C.3 ATTN & EXCT BENCHMARK PROMPT EXAMPLES

Model

The benchmark protocol we follow comprises
structured prompts ‘a [animalA] and a [an-
imalB]’, ‘a [animal] and a [color][object]’,
‘a [colorA][objectA] and a [colorB][objectB]’
. Table 10 demonstrate the results of aver-
age CLIP similarities between text prompts
and captions generated by BLIP for Stable
Diffusion-based methods on this benchmark.
CONTRAFUSION outperform those models on three categories.

Ours

STABLE v1.4 (Rombach et al., 2022)
COMPOSABLE V2 (Liu et al., 2023)
STRUCTURED V2 (Feng et al., 2023a)
ATTN-EXCT V2 (Chefer et al., 2023)
CONFORM (Meral et al., 2023)

Animal-Animal Animal-Obj Obj-Obj

0.76
0.69
0.76
0.80
0.82

0.84

0.78
0.77
0.78
0.83
0.85

0.86

0.77
0.76
0.76
0.81
0.82

0.85

Table 10: Attn-Exct benchmark Results.

D QUALITATIVE RESULTS

Figure 15 presents more comparison between CONTRAFUSION and other state-of-the-art T2I mod-
els, including SDXL, DALL-E 3, SD v3 and PixArt-α.

E RELATED WORK

Understanding and addressing compositional challenges in text-to-image generative models has
been a growing focus in the field. (Thrush et al., 2022; Huang et al., 2023; Chefer et al., 2023)
In particular, Zarei et al. (2024) identifies key compositional challenges in text-to-image diffusion
models and proposes strategies to enhance attribute binding and object relationships. Leveraging
the power of large-language models (LLMs) for compositional generation is another area of active
research. (Drozdov et al., 2022; Mitra et al., 2024; Pasewark et al., 2024) For instance, Feng et al.

21

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

Under review as a conference paper at ICLR 2025

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

Figure 15: Qualitative Results.

(2023b) leverages large language models (LLMs) to generate visually coherent layouts and improve
compositional reasoning in visual generation tasks. Futhermore, Du & Kaelbling (2024) argues that
constructing complex generative models compositionally from simpler models can improve compo-
sitional performance.

22

Three westies are chasing two birds on the grass, one of which is yellow and the other is green.On the table in the living room, there is a hexagonal clock and three glass cups.
SD3.0DALL-E 3 OursSDXLPixArt-AlphaOn the snow, a man in red is throwing a ball to a woman wearing a blue hat.A wolf and three yellow puppies.
There are two chairs to the right of a house.
On a rainy day, a detective wants to go out but doesn't want to draw any attention.
