Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment
Under review as a conference paper at ICLR 2025

TOWARD GUIDANCE-FREE AR VISUAL GENERATION
VIA CONDITION CONTRASTIVE ALIGNMENT

Anonymous authors
Paper under double-blind review

ABSTRACT

Classifier-Free Guidance (CFG) is a critical technique for enhancing the sample
quality of visual generative models. However, in autoregressive (AR) multi-modal
generation, CFG introduces design inconsistencies between language and visual
content, contradicting the design philosophy of unifying different modalities for vi-
sual AR. Motivated by language model alignment methods, we propose Condition
Contrastive Alignment (CCA) to facilitate guidance-free AR visual generation with
high performance and analyzes its theoretical connection with guided sampling
methods. Unlike guidance methods that alter the sampling process to achieve the
ideal sampling distribution, CCA directly fine-tunes pretrained models to fit the
same distribution target. Experimental results show that CCA can significantly
enhance the guidance-free performance of all tested models with just one epoch
of fine-tuning (∼1% of pretraining epochs) on the pretraining dataset, on par with
guided sampling methods. This largely removes the need for guided sampling in
AR visual generation and cuts the sampling cost by half. Moreover, by adjusting
training parameters, CCA can achieve trade-offs between sample diversity and
fidelity similar to CFG. This experimentally confirms the strong theoretical connec-
tion between language-targeted alignment and visual-targeted guidance methods,
unifying two previously independent research fields.

(a) LlamaGen

(b) VAR

Figure 1: CCA significantly improves guidance-free sample quality for AR visual generative models
with just one epoch of fine-tuning on the pretraining dataset.

1

INTRODUCTION

Witnessing the scalability and generalizability of autoregressive (AR) models in language domains,
recent works have been striving to replicate similar success for visual generation (Esser et al., 2021;
Lee et al., 2022). By quantizing images into discrete tokens, AR visual models can process images
using the same next-token prediction approach as Large Language Models (LLMs). This approach is
attractive because it provides a potentially unified framework for vision and language, promoting
consistency in reasoning and generation across modalities (Team, 2024; Xie et al., 2024).

Despite the design philosophy of maximally aligning visual modeling with language modeling
methods, AR visual generation still differs from language generation in a notable aspect. AR visual
generation relies heavily on Classifier-Free Guidance (CFG) (Ho & Salimans, 2022), a sampling
technique unnecessary for language generation, which has caused design inconsistencies between

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

000%%),'&&$RXUVZRJXLGDQFHZJXLGDQFH000%%,6&&$RXUVZRJXLGDQFHZJXLGDQFH00%%),'&&$RXUVZRJXLGDQFHZJXLGDQFH00%%,6&&$RXUVZRJXLGDQFHZJXLGDQFHUnder review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

the two types of content. During sampling, while CFG helps improve sample quality by contrasting
conditional and unconditional models, it requires two model inferences per visual token, which
doubles the sampling cost. During training, CFG requires randomly masking text conditions to learn
the unconditional distribution, preventing the simultaneous training of text tokens (Team, 2024).

In contrast to visual generation, LLMs rarely rely on guided sampling. Instead, the surge of LLMs’
instruction-following abilities has largely benefited from fine-tuning-based alignment methods (Schul-
man et al., 2022). Motivated by this observation, we seek to study: “Can we avoid guided sampling
in AR visual generation, but attain similar effects by directly fine-tuning pretrained models?”

In this paper, we derive Condition Contrastive Alignment (CCA) for enhancing visual AR performance
without guided sampling. Unlike CFG which necessitates altering the sampling process to achieve a
more desirable sampling distribution, CCA directly fine-tunes pretrained AR models to fit the same
distribution target, leaving the sampling scheme untouched. CCA is quite convenient to use since
it does not rely on any additional datasets beyond the pretraining data. Our method functions by
contrasting positive and negative conditions for a given image, which can be easily created from the
existing pretraining dataset as matched or mismatched image-condition pairs. CCA is also highly
efficient given its fine-tuning nature. We observe that our method achieves ideal performance within
just one training epoch, indicating negligible computational overhead (∼1% of pretraining).

In Sec. 4, we highlight a theoretical connection between CCA and guided sampling techniques
(Dhariwal & Nichol, 2021; Ho & Salimans, 2022). Essentially these methods all target at the same
sampling distribution. The distributional gap between this target distribution and pretrained models is
related to a physical quantity termed conditional residual (log p(x|c)
p(x) ). Guidance methods typically
train an additional model (e.g., unconditional model or classifier) to estimate this quantity and enhance
pretrained models by altering their sampling process. Contrastively, CCA follows LLM alignment
techniques (Rafailov et al., 2023; Chen et al., 2024a) and parameterizes the conditional residual
with the difference between our target model and the pretrained model, thereby directly training a
sampling model. This analysis unifies language-targeted alignment and visual-targeted guidance
methods, bridging the gap between the two previously independent research fields.

We apply CCA to two state-of-the-art autoregressive (AR) visual models, LLamaGen (Sun et al.,
2024) and VAR (Tian et al., 2024), which feature distinctly different visual tokenization designs.
Both quantitative and qualitative results show that CCA significantly and consistently enhances the
guidance-free sampling quality across all tested models, achieving performance levels comparable
to CFG (Figure 1). We further show that by varying training hyperparameters, CCA can realize a
controllable trade-off between image diversity and fidelity similar to CFG. This further confirms their
theoretical connections. We also compare our method with some existing LLM alignment methods
(Welleck et al., 2019; Rafailov et al., 2023) to justify its algorithm design. Finally, we demonstrate
that CCA can be combined with CFG to further improve performance.

Our contributions: 1. We take a big step toward guidance-free visual generation by significantly
improving the visual quality of AR models. 2. We reveal a theoretical connection between alignment
and guidance methods. This shows that language-targeted alignment can be similarly applied to
visual generation and effectively replace guided sampling, closing the gap between these two fields.

2 BACKGROUND

2.1 AUTOREGRESSIVE (AR) VISUAL MODELS

Autoregressive models. Consider data x represented by a sequence of discrete tokens x1:N :=
{x1, x2, ..., xN }, where each token xn is an integer. Data probability p(x) can be decomposed as:

p(x) = p(x1)

N
(cid:89)

p(xn|x<n).

(1)

n=2
AR models thus aim to learn pϕ(xn|x<n) ≈ p(xn|x<n), where each token xn is conditioned only
on its previous input x<n. This is known as next-token prediction (Radford et al., 2018).

Visual tokenization.
Image pixels are continuous values, making it necessary to use vector-
quantized tokenizers for applying discrete AR models to visual data (Van Den Oord et al., 2017;

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Esser et al., 2021). These tokenizers are trained to encode images x into discrete token sequences
x1:N and decode them back by minimizing reconstruction losses. In our work, we utilize pretrained
and frozen visual tokenizers, allowing AR models to process images similarly to text.

2.2 GUIDED SAMPLING FOR VISUAL GENERATION

Despite the core motivation of developing a unified model for language and vision, the AR sampling
strategies for visual and text contents differ in one key aspect: AR visual generation necessitates a
sampling technique named Classifier-Free Guidance (CFG) (Ho & Salimans, 2022). During inference,
CFG adjusts the sampling logits ℓsample for each token as:

ℓsample = ℓc + s(ℓc − ℓu),
(2)
where ℓc and ℓu are the conditional and unconditional logits provided by two separate AR models,
pϕ(x|c) and pϕ(x). The condition c can be class labels or text captions, formalized as prompt tokens.
The scalar s is termed guidance scale. Since token logits represent the (unnormalized) log-likelihood
in AR models, Ho & Salimans (2022) prove that the sampling distribution satisfies:

psample(x|c) ∝ pϕ(x|c)

(cid:20) pϕ(x|c)
pϕ(x)

(cid:21)s

.

(3)

At s = 0, the sampling model becomes exactly the pretrained conditional model pϕ. However,
previous works (Ho & Salimans, 2022; Podell et al., 2023; Chang et al., 2023; Sun et al., 2024) have
widely observed that an appropriate s > 0 is critical for an ideal trade-off between visual fidelity and
diversity, making training another unconditional model pϕ necessary. In practice, the unconditional
model usually shares parameters with the conditional one, and can be trained concurrently by
randomly dropping condition prompts c during training.

Other guidance methods, such as Classifier Guidance (Ho & Salimans, 2022) and Energy Guidance
(Lu et al., 2023) have similar effects of CFG. The target sampling distribution of these methods can
all be unified under Eq. 3.

2.3 DIRECT PREFERENCE OPTIMIZATION FOR LANGUAGE MODEL ALIGNMENT

Reinforcement Learning from Human Feedback (RLHF) is crucial for enhancing the instruction-
following ability of pretrained Language Models (LMs) (Schulman et al., 2022; OpenAI, 2023).
Performing RL typically requires a reward model, which can be learned from human preference data.
Formally, the Bradley-Terry preference model (Bradley & Terry, 1952) assumes.

p(xw ≻ xl|c) :=

er(c,xw)
er(c,xl) + er(c,xw)

= σ(r(c, xw) − r(c, xl)),

(4)

where xw and xl are respectively the winning and losing response for an instruction c, evaluated
by human. r(·) represents an implicit reward for each response. The target LM πθ should satisfy
πθ(x|c) ∝ µϕ(x|c)er(c,x)/β to attain higher implicit reward compared with the pretrained LM µϕ.

Direct Preference Optimization (Rafailov et al., 2023) allows us to directly optimize pretrained LMs
on preference data, by formalizing rθ(c, x) := β log πθ(x|c) − β log µϕ(x|c):

θ = −E{c,xw≻xl} log σ
LDPO

(cid:18)

β log

πθ(xw|c)
µϕ(xw|c)

− β log

πθ(xl|c)
µϕ(xl|c)

(cid:19)

.

(5)

DPO is more streamlined and thus often more favorable compared with traditional two-stage RLHF
pipelines: first training reward models, then aligning LMs with reward models using RL.

3 CONDITION CONTRASTIVE ALIGNMENT

Autoregressive visual models are essentially learning a parameterized model pϕ(x|c) to approximate
the standard conditional image distribution p(x|c). Guidance algorithms shift the sampling policy
psample(x|c) away from p(x|c) according to Sec. 2.2:

psample(x|c) ∝ p(x|c)

(cid:20) p(x|c)
p(x)

(cid:21)s

.

(6)

3

Under review as a conference paper at ICLR 2025

At guidance scale s = 0, sampling from psample(x|c) = p(x|c) ≈ pϕ(x|c) is most straightforward.
However, it is widely observed that an appropriate s > 0 usually leads to significantly enhanced
sample quality. The cost is that we rely on an extra unconditional model pϕ(x) ≈ p(x) for sampling.
This doubles the sampling cost and causes an inconsistent training paradigm with language.

In this section, we derive a simple approach to directly model the same target distribution psample using
a single AR model psample
. Specifically, our methods leverage a singular loss function for directly
optimizing pretrained models pϕ(x|c) ≈ p(x|c) to become psample
(x|c) ≈ psample(x|c). Despite
having similar effects as guided sampling, our approach does not require altering the sampling process.
We theoretically derive our method in Sec. 3.1 and discuss its practical implementation in Sec. 3.2.

θ

θ

3.1 ALGORITHM DERIVATION

The core difficulty of directly learning psample
is that we cannot access datasets under the distribution
of psample. However, we observe the distributional difference between psample(x|c) and p(x|c) is
related to a simple quantity that can be potentially learned from existing datasets. Specifically, by
taking the logarithm of both sides in Eq. 6 and applying some algebra, we have1:

θ

1
s

log

psample(x|c)
p(x|c)

= log

p(x|c)
p(x)

,

(7)

of which the right-hand side (i.e., log p(x|c)
probability and unconditional probability for an image x, which we term as conditional residual.

p(x) ) corresponds to the log gap between the conditional

Our key insight here is that the conditional residual can be directly learned through contrastive
learning approaches (Gutmann & Hyvärinen, 2012), as sated below:
Theorem 3.1 (Noise Contrastive Estimation, proof in Appendix A). Let rθ be a parameterized model
which takes in an image-condition pair (x, c) and outputs a scalar value rθ(x, c). Consider the loss
function:

LNCE
θ

(x, c) = −Ep(x,c) log σ(rθ(x, c)) − Ep(x)p(c) log σ(−rθ(x, c)),

where σ(·) is the standard logistic function: σ(w) := 1/(1 + e−w).
Given unlimited model expressivity for rθ, the optimal solution for minimizing LNCE

θ

satisfies

r∗
θ (x, c) = log

p(x|c)
p(x)

.

(8)

(9)

Now that we have a tractable way of learning rθ(x, c) ≈ log p(x|c)
p(x) , the target distribution psample can
be jointly defined by rθ(x, c) and the pretrained model pϕ. However, we would still lack an explicitly
parameterized model psample
if rθ(x, c) is another independent network. To address this problem, we
draw inspiration from the widely studied alignment techniques in language models (Rafailov et al.,
2023) and parameterize rθ(x, c) with our target model psample
(x|c) and pϕ(x|c) according to Eq. 7:

θ

θ

rθ(x, c) :=

1
s

log

(x|c)

psample
θ
pϕ(x|c)

.

Then, the loss function becomes

θ = −Ep(x,c) log σ
LCCA

(cid:104) 1
s

log

(x|c)

psample
θ
pϕ(x|c)

(cid:105)

− Ep(x)p(c) log σ

(cid:104)

−

1
s

log

(x|c)

psample
θ
pϕ(x|c)

(cid:105)

.

(10)

(11)

During training, psample

θ

is learnable while pretrained pϕ is frozen. psample

θ

can be initialized from pϕ.

This way we can fit psample with a single AR model psample
unconditional model for guided sampling. Sampling strategies for psample
language model decoding methods, which unifies decoding systems for multi-modal generation.

, eliminating the need for training a separate
are consistent with standard

θ

θ

1We ignore a normalizing constant in Eq. 7 for brevity. A more detailed discussion is in Appendix B.

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

Figure 2: An overview of the CCA method. Given a training batch of K <image, label> pairs,
CCA treats these as positive samples, and generates K negative samples by randomly assigning a
negative label from K − 1 remaining labels for each image. CCA then fine-tunes pretrained models
by contrasting positive and negative data using an alignment loss. Pseudo code in Appendix D.

3.2 PRACTICAL ALGORITHM

Figure 2 illustrates the CCA method. Specifically, implementing Eq. 11 requires approximating
two expectations: one under the joint distribution p(x, c) and the other under the product of its two
marginals p(x)p(c). The key difference between these distributions is that in p(x, c), images x and
conditions c are correctly paired. In contrast, x and c are sampled independently from p(x)p(c),
meaning they are most likely mismatched.

In practice, we rely solely on the pretraining dataset to estimate LCCA
. Consider a batch of K data
pairs {x, c}1:K. We randomly shuffle the condition batch c1:K to become cneg
1:K, where each cneg
represents a negative condition of image xk, while the original ck is a positive one. This results in
our training batch {x, c, cneg}1:K. The loss function is

k

θ

LCCA
θ

(xk, ck, cneg

k ) = − log σ

(cid:104)
β log

psample
θ
pϕ(xk|ck)
(cid:125)
(cid:123)(cid:122)
(cid:124)
relative likelihood for positive conditions ↑

(xk|ck)

(cid:105)

−λ log σ

(cid:104)

− β log

psample
(xk|cneg
k )
θ
pϕ(xk|cneg
k )

(cid:105)

, (12)

(cid:125)
(cid:123)(cid:122)
(cid:124)
relative likelihood for negative conditions ↓

where β and λ are two hyperparameters that can be adjusted. β replaces the guidance scale parameter
s, while λ is for controlling the loss weight assigned to negative conditions. The learnable psample
is
initialized from the pretrained conditional model pϕ, making LCCA

a fine-tuning loss.

θ

θ

We give an intuitive understanding of Eq. 12. Note that log σ(·) is monotonically increasing. The
first term of Eq. 12 aims to increase the likelihood of an image x given a positive condition, with a
similar effect to maximum likelihood training. For mismatched image-condition data, the second
term explicitly minimizes its relative model likelihood compared with the pretrained pϕ.

We name the above training technique Condition Contrastive Alignment (CCA) due to its contrastive
nature in comparing positive and negative conditions with respect to a single image. This naming
also reflects its theoretical connection with Noise Contrastive Estimation (Theorem 3.1).

4 CONNECTION BETWEEN CCA AND GUIDANCE METHODS

As summarized in Table 1, the key distinction between CCA and guidance methods is how to model
log p(x|c)
p(x) , which defines the distributional gap between the target psample(x|c) and p(x|c) (Eq. 7).
In particular, Classifier Guidance (Dhariwal & Nichol, 2021) leverages Bayes’ Rule and turn
log p(x|c)
p(x)

into a conditional posterior:

log

p(x|c)
p(x)

= log p(c|x) − log p(c) ≈ log pθ(c|x) − log p(c),

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

…{𝑥2,𝑐1}{𝑥3,𝑐1}{𝑥𝐾,𝑐1}{𝑥1,𝑐2}{𝑥3,𝑐2}{𝑥𝐾,𝑐2}{𝑥1,𝑐3}{𝑥2,𝑐3}{𝑥𝐾,𝑐3}{𝑥1,𝑐𝐾}{𝑥2,𝑐𝐾}{𝑥3,𝑐𝐾}{𝑥𝐾,𝑐𝐾}𝑥1𝑥2𝑥3𝑥𝐾…𝑐1<Cat>𝑐2<Dog>𝑐3<Bird>𝑐𝐾<Van>{𝑥1,𝑐1}{𝑥2,𝑐2}{𝑥3,𝑐3}…max𝜃log𝜎log𝑝𝜃ȁ𝑥𝑐𝑝𝜙ȁ𝑥𝑐log𝜎log𝑝𝜃𝑝𝜙log𝜎−log𝑝𝜃𝑝𝜙smax𝜃init{𝑥1,𝑐1}{𝑥1,𝑐1}{𝑥1,𝑐1}𝑥1𝑥2𝑥3𝑥𝑁…𝑐1<Cat>𝑐2<Dog>𝑐3<Bird>𝑐𝑁<Van>{𝑥1,𝑐1}……………………(b) AR model likelihoodNegative dataPositive data(a) Training batch(c) Alignment loss…𝑝𝑥𝑝𝑐𝑝𝑥,𝑐𝑝𝑥𝑝𝑐Under review as a conference paper at ICLR 2025

Method

Classifier Guidance

Classifier-Free Guidance

Condition Contrastive Alignment

Modeling of log p(x|c)
p(x)

Training loss

log pθ(c|x) − log p(c)
maxθ Ep(x,c) log pθ(c|x)

log pϕ(x|c) − log pθ(x)
maxθ Ep(x) log pθ(x)

Sampling policy

log pϕ(x|c) + s log pθ(c|x)

(1 + s) log pϕ(x|c) − s log pθ(x)

β[log psample

(x|c) − log pϕ(x|c)]

θ
minθ LCCA

θ
log psample
θ

in Eq. 11

(x|c)

Extra training cost

Sampling cost

Applicable area

∼9% of learning pϕ
×1.3

∼10% of learning pϕ
×2

∼1% of pretraining pϕ
×1

Diffusion

Diffusion & Autoregressive

Autoregressive

Table 1: Comparison of CCA (ours) and guidance methods in visual generative models.

where p(c|x) is explicitly modeled by a classifier pθ(c|x), which is trained by a standard classification
loss. p(c) is regarded as a uniform distribution. CFG trains an extra unconditional model pθ(x) to
estimate the unknown part of log p(x|c)
p(x) :

log

p(x|c)
p(x)

≈ log pϕ(x|c) − log pθ(x).

Despite their effectiveness, guidance methods all require learning a separate model and a modified
sampling process compared with standard autoregressive decoding. In comparison, CCA leverages
Eq. 7 and models log p(x|c)

p(x) as

log

p(x|c)
p(x)

≈ β[log psample

θ

(x|c) − log pϕ(x|c)],

which allows us to directly learn psample

θ

instead of another guidance network.

Although CCA and conventional guidance techniques have distinct modeling methods, they all target
at the same sampling distribution and thus have similar effects in visual generation. For instance, we
show in Sec. 5.2 that CCA offers a similar trade-off between sample diversity and fidelity to CFG.

5 EXPERIMENTS

We seek to answer the following questions through our experiments:

1. How effective is CCA in enhancing the guidance-free generation quality of pretrained AR

visual models, quantitatively and qualitatively? (Sec. 5.1)

2. Does CCA allow controllable trade-offs between sample diversity (FID) and fidelity (IS)

similar to CFG? (Sec. 5.2)

3. How does CCA perform in comparison to alignment algorithms for LLMs? (Sec. 5.3)
4. Can CCA be combined with CFG to further improve performance? (Sec. 5.4)

5.1 TOWARD GUIDANCE-FREE AR VISUAL GENERATION

Base model. We experiment on two families of publicly accessible AR visual models, LlamaGen
(Sun et al., 2024) and VAR (Tian et al., 2024). Though both are class-conditioned models pretrained
on ImageNet, LlamaGen and VAR feature distinctively different tokenizer and architecture designs.
LlamaGen focuses on reducing the inductive biases on visual signals. It tokenizes images in the
classic raster order and adopts almost the same LLM architecture as Llama (Touvron et al., 2023a).
VAR, on the other hand, leverages the hierarchical structure of images and tokenizes them in a multi-
scale, coarse-to-fine manner. VAR adopts a GPT-2 architecture but tailors the attention mechanism
specifically for visual content. For both works, CFG is a default and critical technique.

Training setup. We leverage CCA to finetune multiple LlamaGen and VAR models of various sizes
on the standard ImageNet dataset. The training scheme and hyperparameters are mostly consistent
with the pretraining phase. We report performance numbers after only one training epoch and find

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Model

n
o
i
s
u
f
f
i

D

ADM (Dhariwal & Nichol, 2021)
LDM-4 (Rombach et al., 2022)
U-ViT-H/2 (Bao et al., 2023)
DiT-XL/2 (Peebles & Xie, 2023)
MDTv2-XL/2 (Gao et al., 2023)

s
a

k MaskGIT (Chang et al., 2022)
MAGVIT-v2 (Yu et al., 2023)
MAGE (Li et al., 2023)

M

e
v
i
s
s
e
r
g
e
r
o
t
u
A

VQGAN (Esser et al., 2021)
ViT-VQGAN (Yu et al., 2021)
RQ-Transformer (Lee et al., 2022)
LlamaGen-3B (Sun et al., 2024)

+CCA (Ours)

VAR-d30 (Tian et al., 2024)

+CCA (Ours)

w/o Guidance

w/ Guidance

IS↑

Precision↑ Recall↑

FID↓

IS↑

127.5
103.5
–
121.5
155.6

182.1
200.5
195.8

74.3
175.1
134.0
112.9
276.8
175.6
264.2

0.72
0.71
–
0.67
0.72

0.80
–
–

–
–
–
0.69
0.80
0.75
0.83

0.63
0.62
–
0.67
0.66

0.51
–
–

–
–
–
0.67
0.59
0.62
0.56

3.94
3.60
2.29
2.27
1.58

–
1.78
–

5.20
3.04
3.80
2.18
–
1.92
–

215.8
247.7
263.9
278.2
314.7

–
319.4
–

280.3
227.4
323.7
263.3
–
323.1
–

FID↓

7.49
10.56
–
9.62
5.06

6.18
3.65
6.93

15.78
4.17
7.55
9.38
2.69
5.25
2.54

Table 2: Model comparisons on class-conditional ImageNet 256 × 256 benchmark.

LlamaGen (w/o Guidance)
IS=64.7

LlamaGen + CCA (w/o G.)
IS=384.6

LlamaGen (w/ CFG)
IS=404.0

VAR (w/o Guidance)
IS=154.3

VAR + CCA (w/o G.)
IS=350.4

VAR (w/ CFGv2)
IS=390.8

Figure 3: CCA and CFG can similarly enhance the sample fidelity of AR visual models. The base
models are LlamaGen-L (343M) and VAR-d24 (1.0B). We use s = 3.0 for CFG, and β = 0.02, λ =
104 for CCA. Figure 7 and Figure 8 contain more examples.

this to be sufficient for ideal performance. We fix β = 0.02 in Eq. 12 and select suitable λ for
each model. Image resolutions are 384 × 384 for LlamaGen and 256 × 256 for VAR. Following the
original work, we resize LlamaGen samples to 256 × 256 whenever required for evaluation.

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Figure 4: CCA can achieve similar FID-IS trade-offs to CFG by adjusting training parameter λ.

FID↓
Model
LlamaGen-L 19.00
61.69
12.22
3.43

+DPO
+Unlearn
+CCA

IS
64.7
30.8
111.6
288.2

sFID↓ Precision Recall
0.67
0.61
8.78
0.40
0.36
44.98
0.64
0.66
7.99
0.81
7.44
0.52

Model
VAR-d24
+DPO
+Unlearn
+CCA

FID↓
6.20
7.53
5.55
2.63

IS
154.3
232.6
165.9
298.8

sFID↓ Precision Recall
0.62
0.74
8.50
0.85
0.34
19.10
0.61
0.75
8.41
7.63
0.55
0.84

Table 3: Comparision of CCA and LLM alignment algorithms in visual generation.

Experimental results. We find CCA significantly improves the guidance-free performance of all
tested models (Figure 1), evaluated by metrics like FID (Heusel et al., 2017) and IS (Salimans et al.,
2016). For instance, after one epoch of CCA fine-tuning, the FID score of LlamaGen-L (343M)
improves from 19.07 to 3.41, and the IS score from 64.3 to 288.2, achieving performance levels
comparable to CFG. This result is compelling, considering that CCA has negligible fine-tunning
overhead compared with model pretraining and only half of sampling costs compared with CFG.

Figure 3 presents a qualitative comparison of image samples before and after CCA fine-tuning.
The results clearly demonstrate that CCA can vastly improve image fidelity, as well as class-image
alignment of guidance-free samples.

Table 2 compares our best-performing models with several state-of-the-art visual generative models.
With the help of CCA, we achieve a record-breaking FID score of 2.54 and an IS score of 276.8
for guidance-free samples of AR visual models. Although these numbers still somewhat lag behind
CFG-enhanced performance, they demonstrate the significant potential of alignment algorithms to
enhance visual generation and indicate the future possibility of replacing guided sampling.

5.2 CONTROLLABLE TRADE-OFFS BETWEEN DIVERSITY AND FIDELITY

A distinctive feature of CFG is its ability to balance diversity and fidelity by adjusting the guidance
scale. It is reasonable to expect that CCA can achieve a similar trade-off since it essentially targets
the same sampling distribution as CFG.

Figure 4 confirms this expectation: by adjusting the λ parameter for fine-tuning, CCA can achieve
similar FID-IS trade-offs to CFG. The key difference is that CCA enhances guidance-free models
through training, while CFG mainly improves the sampling process.

It is worth noting that VAR employs a slightly different guidance technique from standard CFG, which
we refer to as CFGv2. CFGv2 involves linearly increasing the guidance scale s during sampling,
which was first proposed by Chang et al. (2023) and found beneficial for certain models. The FID-IS
curve of CCA more closely resembles that of standard CFG. Additionally, the hyperparameter β also
affects CCA performance. Although our algorithm derivation shows that β is directly related to the
CFG scale s, we empirically find adjusting β is less effective and less predictable compared with
adjusting λ. In practice, we typically fix β and adjust λ. We ablate β in Appendix C.

5.3 CAN LLM ALIGNMENT ALGORITHMS ALSO ENHANCE VISUAL AR?

Intuitively, existing preference-based LLM alignment algorithms such as DPO and Unlearning
(Welleck et al., 2019) should also offer improvement for AR visual models given their similarity to
CCA. However, Table 3 shows that naive applications of these methods fail significantly.

8

,6),'=0=104s=0s=3/ODPD*HQ/&&$&)*,6),'=0=104s=0s*=3.0s=1.09$5G&&$&)*&)*YUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Figure 5: The impact of training parameter λ on the performance of CCA+CFG.

Figure 6: Integration of CCA+CFG yields improved performance over CFG alone.

DPO. As is described in Eq. 5, one can treat negative image-condition pairs as dispreferred data
and positive ones as preferred data to apply the DPO loss. We ablate βd ∈ {0.01, 0.1, 1.0, 10.0}
and report the best performance in Table 3. Results indicate that DPO fails to enhance pretrained
models, even causing performance collapse for LlamaGen-L. By inspecting training logs, we find
that the likelihood of the positive data continuously decreases during fine-tuning, which may explain
the collapse. This phenomenon is a well-observed problem of DPO (Chen et al., 2024a; Pal et al.,
2024), stemming from its focus on optimizing only the relative likelihood between preferred and
dispreferred data, rather than controlling likelihood for positive and negative image-condition pairs
separately. We refer interested readers to Chen et al. (2024a) for a detailed discussion.

Unlearning. Also known as unlikelihood training, this method maximizes log pθ(x|c) through
standard maximum likelihood training on positive data, while minimizing log pθ(x|cneg) to unlearn
negative data. A training parameter λu controls the weight of the unlearning loss. We find that with
small 0.01 ≤ λu ≤ 0.1, Unlearning provides some benefit, but it is far less effective than CCA. This
suggests the necessity of including a frozen reference model.

5.4

INTEGRATION OF CCA AND CFG

If extra sampling costs and design inconsistencies of CFG are not concerns, could CCA still be
helpful? A takeaway conclusion is yes: CCA+CFG generally outperforms CFG (Figure 6), but it
requires distinct hyperparameter choices compared with CCA-only training.

Implementation. After pretraining the unconditional AR visual model by randomly dropping
conditions, CFG requires us to also fine-tune the unconditional model during alignment. To achieve
this, we follow previous approaches by also randomly replacing data conditions with [MASK] tokens
at a probability of 10%. These unconditional samples are treated as positive data during CCA training.
We provide pseudo-code in Appendix D.

Comparison of CCA-only and CCA+CFG. They require different hyperparameters. As shown
in Figure 5, a larger λ is typically needed for optimal FID scores in guidance-free generation. For
models optimized for guidance-free sampling, adding CFG guidance does not further reduce the FID
score. However, with a smaller λ, CCA+CFG could outperform the CFG method.

9

*XLGDQFHVFDOHs),'),'ZR&&$*XLGDQFHVFDOHs,6,6ZR&&$3e3&&$HH&&$2SWLPDO&)*VFDOHs*IRU),'ZJXLGHGVDPSOLQJZRJXLGHGVDPSOLQJHH&&$2SWLPDO),'*IRU&&$RQO\*IRU&&$&)*),'ZR&&$ZJXLGHGVDPSOLQJZRJXLGHGVDPSOLQJ000%%),'&&$&)*&&$RQO\&)*RQO\000%%,6&&$&)*&&$RQO\&)*RQO\Under review as a conference paper at ICLR 2025

6 RELATED WORKS

Visual generative models. Generative adversarial networks (GANs) (Goodfellow et al., 2014;
Brock et al., 2018; Karras et al., 2019; Kang et al., 2023) and diffusion models (Ho et al., 2020;
Song & Ermon, 2019; Song et al., 2020; Dhariwal & Nichol, 2021; Kingma & Gao, 2024) are
representative modeling methods for visual content generation, widely recognized for their ability to
produce realistic and artistic images (Sauer et al., 2022; Ho et al., 2022; Ramesh et al., 2022; Podell
et al., 2023). However, because these methods are designed for continuous data like images, they
struggle to effectively model discrete data such as text, limiting the development of unified multimodal
models for both vision and language. Recent approaches seek to address this by integrating diffusion
models with language models (Team, 2024; Li et al., 2024; Zhou et al., 2024). Another line of works
(Chang et al., 2022; 2023; Yu et al., 2023; Xie et al., 2024; Ramesh et al., 2021; Yu et al., 2022)
explores discretizing images (Van Den Oord et al., 2017; Esser et al., 2021) and directly applying
language models such as BERT-style (Devlin et al., 2018) masked-prediction models and GPT-style
(Radford et al., 2018) autoregressive models for image generation.

Language model alignment. Different from visual generative models which generally enhance
sample quality through sampling-based methods (Dhariwal & Nichol, 2021; Ho & Salimans, 2022;
Zhao et al., 2022; Lu et al., 2023), LLMs primarily employ training-based alignment techniques
to improve instruction-following abilities (Touvron et al., 2023b; OpenAI, 2023). Reinforcement
Learning (RL) is well-suited for aligning LLMs with human feedback (Schulman et al., 2017; Ouyang
et al., 2022). However, this method requires learning a reward model before optimizing LLMs, leading
to an indirect two-stage optimization process. Recent developments in alignment techniques (Rafailov
et al., 2023; Cai et al., 2023; Azar et al., 2024; Ethayarajh et al., 2024; Chen et al., 2024a; Ji et al.,
2024) have streamlined this process. They enable direct alignment of LMs through a singular
loss. Among all LLM alignment algorithms, our method is perhaps most similar to NCA (Chen
et al., 2024a). Both NCA and CCA are theoretically grounded in the NCE framework (Gutmann &
Hyvärinen, 2012). Their differences are mainly empirical regarding loss implementations, particularly
in how to estimate expectations under the product of two marginal distributions.

Visual alignment. Motivated by the success of alignment techniques in LLMs, several studies
have also investigated aligning visual generative models with human preferences using RL (Black
et al., 2023a; Xu et al., 2024) or DPO (Black et al., 2023b; Wallace et al., 2023). For diffusion
models, the application is not straightforward and must rely on some theoretical approximations, as
diffusion models do not allow direct likelihood calculation, which is required by most LLM alignment
algorithms (Chen et al., 2024b). Moreover, previous attempts at visual alignment have primarily
focused on enhancing the aesthetic quality of generated images and necessitate a different dataset
from the pretrained one. Our work distinguishes itself from prior research by having a fundamentally
different optimization objective (replacing CFG) and does not rely on any additional data input.

7 CONCLUSION

In this paper, we propose Condition Contrastive Alignment (CCA) as a fine-tuning algorithm for
AR visual generation models. CCA can significantly enhance the guidance-free sample quality
of pretrained models without any modification of the sampling process. This paves the way for
further development in multimodal generative models and cuts the cost of AR visual generation by
half in comparison to CFG. Our research also highlights the strong theoretical connection between
language-targeted alignment and visual-targeted guidance methods, facilitating future research of
unifying visual modeling and language modeling.

REPRODUCIBILITY

We provide experimental details in Appendix E. We submit our source code in the supplementary
material. Code and model weights will be publicly accessible upon publication.

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal
Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from
In International Conference on Artificial Intelligence and Statistics, pp.
human preferences.
4447–4455. PMLR, 2024.

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth
words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition, pp. 22669–22679, 2023.

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models

with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023a.

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models

with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023b.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method

of paired comparisons. Biometrika, 39(3/4):324–345, 1952.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural

image synthesis. arXiv preprint arXiv:1809.11096, 2018.

Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified
language model alignment with demonstration and point-wise human preference. arXiv preprint
arXiv:2312.02554, 2023.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative
image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 11315–11325, 2022.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang,
Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation
via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.

Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models

with explicit rewards. Advances in neural information processing systems, 2024a.

Huayu Chen, Kaiwen Zheng, Hang Su, and Jun Zhu. Aligning diffusion behaviors with q-functions
for efficient continuous control. Advances in neural information processing systems, 2024b.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances

in neural information processing systems, 34:8780–8794, 2021.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image
synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp. 12873–12883, 2021.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model

alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a
strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pp. 23164–23173, 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information
processing systems, 27, 2014.

Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical
models, with applications to natural image statistics. Journal of machine learning research, 13(2),
2012.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans
trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural
information processing systems, 30, 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in

neural information processing systems, 33:6840–6851, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning
Research, 23(1):2249–2281, 2022.

Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie
Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint
arXiv:2402.00856, 2024.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung
Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 10124–10134, 2023.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative
adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern
recognition, pp. 4401–4410, 2019.

Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data

augmentation. Advances in Neural Information Processing Systems, 36, 2024.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image
generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 11523–11532, 2022.

Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage:
Masked generative encoder to unify representation learning and image synthesis. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142–2152, 2023.

Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image

generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024.

Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy
prediction for exact energy-guided diffusion sampling in offline reinforcement learning.
In
International Conference on Machine Learning, pp. 22825–22855. PMLR, 2023.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim
Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), pp. 14297–14306, June 2023.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. Advances in neural information processing systems, 35:27730–
27744, 2022.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.
Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint
arXiv:2402.13228, 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of

the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe
Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image
synthesis. arXiv preprint arXiv:2307.01952, 2023.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.

Improving language

understanding by generative pre-training. article, 2018.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
In
Finn. Direct preference optimization: Your language model is secretly a reward model.
Thirty-seventh Conference on Neural Information Processing Systems, 2023.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen,
and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine
Learning, pp. 8821–8831. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-
conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-
resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer-
ence on computer vision and pattern recognition, pp. 10684–10695, 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training gans. Advances in neural information processing systems, 29,
2016.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse

datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1–10, 2022.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy

optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe-
lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language
models for dialogue. OpenAI blog, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv

preprint arXiv:2010.02502, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.

Advances in neural information processing systems, 32, 2019.

Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan.
Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint
arXiv:2406.06525, 2024.

Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint

arXiv:2405.09818, 2024.

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling:
Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in

neural information processing systems, 30, 2017.

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam,
Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using
direct preference optimization. arXiv preprint arXiv:2311.12908, 2023.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural

text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019.

13

Under review as a conference paper at ICLR 2025

Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin,
Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer
to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024.

Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.
Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances
in Neural Information Processing Systems, 36, 2024.

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong
Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan.
arXiv preprint arXiv:2110.04627, 2021.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan,
Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-
rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong
Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–
tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation
via energy-guided stochastic differential equations. Advances in Neural Information Processing
Systems, 35:3609–3623, 2022.

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob
Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and
diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

w/o Guidance

+CCA (w/o Guidance)

w/ CFG Guidance

Figure 7: Comparison of LlamaGen-L samples generated with CCA or CFG.

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

w/o Guidance

+CCA (w/o Guidance)

w/ CFG Guidance

Figure 8: Comparison of VAR-d24 samples generated with CCA or CFG.

16

Under review as a conference paper at ICLR 2025

A THEORETICAL PROOFS

In this section, we provide the proof of Theorem 3.1.
Theorem A.1 (Noise Contrastive Estimation ). Let rθ be a parameterized model which takes in an
image-condition pair (x, c) and outputs a scalar value rθ(x, c). Consider the loss function:

LNCE
θ

(x, c) = −Ep(x,c) log σ(rθ(x, c)) − Ep(x)p(c) log σ(−rθ(x, c)).

Given unlimited model expressivity for rθ, the optimal solution for minimizing LNCE

θ

satisfies

r∗
θ (x, c) = log

p(x|c)
p(x)

.

(13)

(14)

Proof. First, we construct two binary (Bernoulli) distributions:

Qx,c := {

p(x, c)
p(x, c) + p(x)p(c)

,

p(x)p(c)
p(x, c) + p(x)p(c)

} = {

p(x|c)
p(x|c) + p(x)

,

p(x)
p(x|c) + p(x)

}

P θ

x,c := {

erθ(x,c)
erθ(x,c) + 1

,

1
erθ(x,c) + 1

} = {σ(rθ(x, c)), 1 − σ(rθ(x, c))}

Then we rewrite LNCE

θ

(x, c) as

LNCE
θ

(x, c) = −Ep(x,c) log σ(rθ(x, c)) − Ep(x)p(c) log σ(−rθ(x, c))

(cid:90) (cid:104)

(cid:105)
p(x, c) log σ(rθ(x, c)) + p(x)p(c) log σ(−rθ(x, c))

dxdc

(cid:90) (cid:104)

(p(x, c) + p(x)p(c))

(cid:105)

= −

= −

(cid:104)

p(x, c)
p(x, c) + p(x)p(c)

log σ(rθ(x, c)) +

p(x)p(c)
p(x, c) + p(x)p(c)

log (cid:2)1 − σ(rθ(x, c))(cid:3)(cid:105)

dxdc

(cid:90) (cid:104)

(cid:90) (cid:104)

=

=

(p(x, c) + p(x)p(c))

(cid:105)

H(Qx,c, P θ

x,c)dxdc

(p(x, c) + p(x)p(c))

(cid:105)(cid:104)

DKL(Qx,c∥P θ

(cid:105)
x,c) + H(Qx,c)

dxdc

x,c) represents the cross-entropy between distributions Qx,c and P θ
Here H(Qx,c, P θ
x,c. H(Qx,c) is
the entropy of Qx,c, which can be regarded as a constant number with respect to parameter θ. Due to
the theoretical properties of KL-divergence, we have
(cid:90) (cid:104)

(cid:105)(cid:104)

LNCE
θ

(x, c) =

(p(x, c) + p(x)p(c))

DKL(Qx,c∥P θ

x,c) + H(Qx,c)

(cid:105)
dxdc

(cid:90) (cid:104)

≥

(p(x, c) + p(x)p(c))

(cid:105)
H(Qx,c)dxdc

constantly hold. The equality holds if and only if Qx,c = P θ

x,c, such that

σ(rθ(x, c)) =

erθ(x,c)
erθ(x,c) + 1

=

p(x, c)
p(x, c) + p(x)p(c)

rθ(x, c) = log

p(x, c)
p(x)p(c)

= log

p(x|c)
p(x)

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

B THEORETICAL ANALYSES OF THE NORMALIZING CONSTANT

We omit a normalizing constant in Eq. 7 for brevity when deriving CCA. Strictly speaking, the target
sampling distribution should be:

psample(x|c) =

1
Z(c)

p(x|c)[

p(x|c)
p(x)

]s,

such that

1
s

log

psample(x|c)
p(x|c)

= log

p(x|c)
p(x)
that psample(x|c)

1
s

−

log Z(c).

The normalizing constant Z(c) ensures
(cid:82) psample(x|c)dx = 1. We have Z(c) = (cid:82) p(x|c)[ p(x|c)

p(x) ]sdx = Ep(x|c)[ p(x|c)

p(x) ]s.

is properly normalized,

i.e.,

To mitigate the additional effects introduced by Z(c), in our practical algorithm, we introduce a new
training parameter λ to bias the optimal solution for Noise Contrastive Estimation. Below, we present
a result that is stronger than Theorem 3.1.
Theorem B.1. Let λc > 0 be a scalar function conditioned only on c. Consider the loss function:

LNCE
θ

(x, c) = −Ep(x,c) log σ(rθ(x, c)) − λcEp(x)p(c) log σ(−rθ(x, c)).

Given unlimited model expressivity for rθ, the optimal solution for minimizing LNCE

θ

satisfies

r∗
θ (x, c) = log

p(x|c)
p(x)

− log λc.

(15)

(16)

Proof. We omit the full proof here, as it requires only a redefinition of the distributions Qx,c from
the proof of Theorem A.1:

Qx,c := {

p(x, c)
p(x, c) + λcp(x)p(c)

,

λcp(x)p(c)
p(x, c) + λcp(x)p(c)

} = {

p(x|c)
p(x|c) + λcp(x)

,

λcp(x)
p(x|c) + λcp(x)

}

Then we can follow the steps in the proof of Theorem A.1 to arrive at the result.

s = (cid:2)Ep(x|c)[ p(x|c)

If let λc := Z(c) 1
to psample.
However, in practice estimating Z(c) could be intricately difficult, so we formalize λc as a training
parameter, resulting in our practical algorithm in Eq. 12.

s , we could guarantee the convergence of psample

p(x) ]s(cid:3) 1

θ

C ADDITIONAL EXPERIMENTAL RESULTS

We provide more image samples to compare CCA and CFG in Figure 7 and Figure 8.

We illustrate the effect of training parameter β on the FID-IS trade-off in Figure 9. Overall, β affects
the fidelity-diversity trade-off similar to λ and the CFG method.

Figure 9: Effect of varying β of CCA for the LlamaGen-L model. In our CCA experiments, we either
fix λ = 1e3 and ablate β ∈ [2, 5e − 3] (from left to right) or fix β = 0.02 and ablate λ ∈ [0, 1e4]. In
our CFG experiments, we ablate s ∈ [0, 3].

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

,6),'0RGHO&&$DGMXVWLQJ&&$DGMXVWLQJ&)*DGMXVWLQJsUnder review as a conference paper at ICLR 2025

Figure 10: Training curves of CCA for LlamaGen-L model (β = 0.02, λ = 300). Left: CCA loss.
Right: Relative likelihood log pθ(x|c)

pϕ(x|c) for positive and negative data during training.

D PSEUDO CODE

Algorithm 1 CCA

Input: Pretraining dataset {x, c}, pretrained AR model pϕ, target model pθ. Initialize θ = ϕ
For each gradient step do

Sample K data pairs {x, c}1:K from the dataset as positive samples // p(x, c)
Randomly shuffle c1:K to become cneg
If CCA+CFG then

1:K and form negative samples {x, cneg}1:K . // p(x)p(c)

For each label ck in c1:K and cneg

1:K do
Replace ck with ∅ with a probability of 10% // Random masking

Lθ = 0
For For each data {xk, ck} in training batch {x, c}1:K and {x, cneg}1:K do

Lθ = Lθ − log σ

if {xk, ck} is positive sample or ck = ∅

(cid:105)

(cid:104)

(xk|ck)

β log psample
(cid:104)

θ
pϕ(xk|ck)
− β log psample

(xk|ck)

θ
pϕ(xk|ck)

Lθ = Lθ − λ log σ
θ ← θ − η∇θLθ (Eq. 12)

(cid:105)

if {xk, ck} is negative sample and ck ̸= ∅

We provide an example of training curves for CCA in Figure 10.

E TRAINING HYPERPARAMETERS

Table 4 reports hyperparameters for chosen models in Figure 1 and Figure 6. Other unmen-
tioned design choices and hyperparameters are consistent with the default setting for LlamaGen
https://github.com/FoundationVision/LlamaGen and VAR https://github.
com/FoundationVision/VAR repo. All models are fine-tuned for 1 epoch on the ImageNet
dataset. We use a mix of NVIDIA-H100, NVIDIA A100, and NVIDIA A40 GPU cards for training.

Type

Model
Size

LlamaGen

VAR

B

d30
3B
111M 343M 775M 1.4B 3.1B 310M 600M 1.0B 2.0B

XXL

d16

d20

d24

XL

L

CCA β
CCA λ
CCA+CFG β
CCA+CFG λ
Learning rate
Dropout?
Batch size

0.02
1000
0.1
1
1e-5
Yes
256

0.02
300
0.02
1
1e-5
Yes
256

0.02
1000
0.1
1
1e-5
Yes
256

0.02
1000
0.1
1
1e-5
Yes
256

0.02
500
0.1
1
1e-5
Yes
256

0.02
50
-
-
2e-5
None
256

0.02
50
-
-
2e-5
Yes
256

0.02
100
-
-
2e-5
Yes
256

0.02
1000
-
-
2e-5
Yes
256

Table 4: Hyperparameter table.

All our reported models are trained individually for each hyperparameter. However, we note that
hyperparameters like λ and β can serve as input for our target AR visual model using existing

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

010002000300040005000Gradient Steps0.0050.010.050.10.5CCA Loss010002000300040005000Gradient Steps6004002000Relative LikelihoodPositive Likelihood RatioNegative Likelihood RatioUnder review as a conference paper at ICLR 2025

distillation techniques (Meng et al., 2023) so that we can tune them only during inference. This
way CCA can allow test-time flexibility just like CFG. We present an initial result conditioning the
LlamaGen-L model on parameter λ in Table 5. In order to additionally condition on an extra scalar
input λ, we use the same embedding method as the one used by DiT (Peebles & Xie, 2023) and directly
add the λ embedding on the class token embeddings. We randomly sample λ ∈ [e0, e9 ≈ 10000]
during training. The model is trained for 3 epochs.

Inference-time λ
FID
IS

10
7.23
153.1

100
4.18
218.8

300 (Chosen)
3.59
256.2

1000
3.73
277.5

3000
4.12
307.7

10000
5.33
341.2

Table 5: Performance for different inference-time λ values. For reference, the pretrained LlamaGen
model has an IS of 64.3 and an FID of 19.07. After CCA finetuning with fixed λ = 300, the finetuned
model has an IS 288.2 of and FID of 3.43.

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

20

