Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
Under review as a conference paper at ICLR 2025

PAINTING WITH WORDS: ELEVATING DETAILED IM-
AGE CAPTIONING WITH BENCHMARK AND ALIGNMENT
LEARNING

Anonymous authors
Paper under double-blind review

ABSTRACT

Image captioning has long been a pivotal task in visual understanding, with recent
advancements in vision-language models (VLMs) significantly enhancing the
ability to generate detailed image captions. However, the evaluation of detailed
image captioning remains underexplored due to outdated evaluation metrics and
coarse annotations. In this paper, we introduce DECAPBENCH along with a novel
metric, DCSCORE, specifically designed for detailed captioning tasks. DCSCORE
evaluates hallucinations and fine-grained comprehensiveness by deconstructing
responses into the smallest self-sufficient units, termed primitive information units,
and assessing them individually. Our evaluation shows that DCSCORE aligns
more closely with human judgment than other rule-based or model-based metrics.
Concurrently, DECAPBENCH exhibits a high correlation with VLM arena results
on descriptive tasks, surpassing existing benchmarks for vision-language models.
Additionally, we present an automatic fine-grained feedback collection method,
FEEDQUILL, for preference optimization based on our advanced metric, showing
robust generalization capabilities across auto-generated preference data. Extensive
experiments on multiple VLMs demonstrate that our method not only significantly
reduces hallucinations but also enhances performance across various benchmarks,
achieving superior detail captioning performance while surpassing GPT-4o.

1

INTRODUCTION

Vision-Language Models (VLMs) (Zhu et al., 2023; Liu et al., 2024b; Ye et al., 2023; Bai et al., 2023)
have risen to prominence by integrating the strengths of pre-trained large language models (LLMs)
and vision models, leveraging large-scale multi-modal corpora (Liu et al., 2024b; Dai et al., 2023;
Li et al., 2024a). These models have demonstrated remarkable capabilities across a diverse array of
tasks. To assess their visual understanding capability, numerous benchmarks have been developed,
focusing on question-answering tasks, such as MMVet (Yu et al., 2023), MMStar (Chen et al., 2024a),
and MMMU (Yue et al., 2024). However, these benchmarks often rely on manually defined queries
and questions, which may only cover a limited domain and lead to biased evaluations (Chen et al.,
2024a). Additionally, Chen et al. (2024a) highlights that poorly constructed questions could make the
models rely more on textual knowledge from their training data, thus neglecting actual visual input.

In this context, the image captioning has been a fundamental task to evaluate the visual perception
capabilities of VLMs. Yet, traditional image captioning benchmarks suffer from two significant
limitations: (1) The evaluation metrics (Vedantam et al., 2015; Papineni et al., 2002; Lin, 2004; Hessel
et al., 2021) are unreliable and show low correlation with human judgment and model capability,
and (2) The captions are typically short and lack informative visual details, missing fine-grained
descriptions. In contrast, modern VLMs are capable of generating hyper-detailed image captions
rich in fine-grained visual information (OpenAI., 2024a; Liu et al., 2024b). These models can even
extend and infer non-descriptive elements, which are often not covered by the conventional short
ground-truth captions, leading to unsatisfying detail caption evaluation results. Additionally, many of
the existing image captioning datasets (Lin et al., 2014; Sidorov et al., 2020) focus on short captions
and have become outdated, necessitating a more rigorous evaluation framework for modern VLMs.
To address these limitations, it is crucial to develop new benchmarks and evaluation metrics that align
closely with human judgment and accurately reflect the advanced capabilities of modern VLMs.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

In this paper, we aim to assess the capabilities of modern VLMs in producing detailed image captions.
We introduce a novel metric, DCSCORE, and a comprehensive evaluation benchmark, DECAPBENCH,
designed to address the challenges of hallucination and fine-grained comprehensiveness in image cap-
tioning. Our approach involves breaking down captions into the smallest self-sufficient units, termed
primitive information units. This decomposition reduces ambiguity and enhances the transparency
and interpretability of the evaluation process. By individually assessing these units, we can accurately
measure both descriptive and non-descriptive parts of captions with fine granularity. Additionally,
decomposing captions allows us to evaluate their coverage with high-quality, hyper-detailed reference
captions. Our experiments reveal that DCSCORE achieves the highest consistency with human
expert evaluations, outperforming all existing rule-based and model-based metrics. Furthermore, we
present DECAPBENCH as a detailed captioning dataset that excels in measuring hallucination and
fine-grained comprehensiveness. It demonstrates superior correlation with the VLM description tasks
compared to other benchmarks such as MMVet and MMStar.

In addition, we embrace the concept of breaking down responses into primitive information units
and introduce FEEDQUILL, a fine-grained feedback collection strategy for preference optimization.
Specifically, we generate several candidate responses and decompose them into verifiable statements.
Using open-source VLMs (Liu et al., 2024a; Chen et al., 2024b), we then validate the correctness
of these statements and calculate a preference score to measure precision. To avoid bias towards
overly concise responses, we also factor in the number of primitive information units as feedback
signals. Leveraging proximal policy optimization (PPO) (Schulman et al., 2017), we optimize
preferences using a reward model trained on the collected preference data. Extensive experiments
demonstrate that FEEDQUILL consistently enhances performance across various VLM models on
both comprehensive and task-specific benchmarks, significantly reducing hallucinations by 40.5%
relative points in mmHal-V. Furthermore, our model not only outperforms GPT-4o in detailed image
captioning but also exceeds GPT-4V in visual chatting, underscoring its potential and effectiveness.

The contribution of this work can be summarized as: (1) We present DCSCORE, a novel metric for
image detail caption evaluation with both hallucination and comprehensiveness, and it achieves the
highest consistency with human experts among existing caption metrics. (2) We introduce a new
detailed caption benchmark DECAPBENCH for evaluating the captioning capability of modern VLMs,
which has the highest correlation with human judgement on description task compared to other
public benchmarks. (3) We propose a simple but effective fine-grained feedback collection method
FEEDQUILL by decomposing responses into primitive information units and verify them individually,
which is scalable for automatically collecting preference data. (4) Extensive experimental results
demonstrate the efficacy of FEEDQUILL, showing reduced hallucinations, superior performance in
visual chat compared to GPT-4v, and better detailed image captioning capabilities than GPT-4o.

2 RELATED WORK

Image Captioning Evaluation Metrics
Image captioning tasks are fundamental to visual-language
understanding, as they assess a model’s ability to comprehend and describe images accurately.
Modern vision-language models (Ye et al., 2024; Chen et al., 2024b; Liu et al., 2024a; Bai et al.,
2023) equipped with massive data pre-training, are capable of generating diverse and detailed image
captions. Despite these advancements, evaluating captions accurately and comprehensively remains
challenging. Traditional metrics, such as BLEU (Papineni et al., 2002), METEOR (Banerjee &
Lavie, 2005), and CIDEr (Vedantam et al., 2015), leverage N-gram and lexical similarity with
human-annotated captions but suffer from instability due to variability in phrasing. To address
this issue, model-based metrics like SPICE (Anderson et al., 2016) and CAPTURE (Dong et al.,
2024) parse captions using scene graphs to match ground-truth captions. Additionally, CLIPScore
(Hessel et al., 2021) and PACScore (Sarto et al., 2023) utilize pre-trained vision-language models
like CLIP (Radford et al., 2021) to measure the similarity between images and captions, as well
as between generated and reference captions. Recently, researchers have leveraged the powerful
zero-shot capabilities of large language models (LLMs) to prompt LLMs for assessing the alignment
between model-generated and human-annotated captions (Chan et al., 2023; Lee et al., 2024; Liu
et al., 2024b). Despite their potential, LLM-based evaluation methods face challenges in maintaining
objectivity and comprehensiveness, particularly in extending evaluation to aspects such as knowledge
and atmosphere. To alleviate these problems, we propose DCSCORE, a novel image caption metric
that evaluates image captions by incorporating both hallucination and comprehensiveness thoroughly.

2

Under review as a conference paper at ICLR 2025

Learning from Feedback for VLMs Learning from feedback (Yu et al., 2024a; Sun et al., 2023;
Zhou et al., 2024a;b) is a core technique in the post-training stage of vision language models (VLMs).
This approach enhances model performance on various tasks, such as question answering (Yue et al.,
2024; Liu et al., 2023; Chen et al., 2024a) and reducing hallucinations (Li et al., 2023b), through
alignment learning techniques like PPO (Schulman et al., 2017), DPO (Rafailov et al., 2024), and
RLOO (Ahmadian et al., 2024). The quality of feedback is crucial for aligning models with human
preferences. Early works, such as LLaVA-RLHF (Sun et al., 2023) and RLHF-V (Yu et al., 2024a),
relied heavily on human-intensive labeling to collect high-quality feedback and correct mistakes
in model responses. To alleviate the demand for intensive human labeling, various approaches (Li
et al., 2023a; Zhao et al., 2023; Yu et al., 2024b) have been proposed to collect or construct feedback
with preferences automatically. For instance, Bai et al. (2023) prompt GPT-4v (OpenAI., 2024b) to
collect preference pairs and distill them into a pre-trained VLM. While this method offers ease and
convenience, the preference judgment of GPT-4v is not manually verified, posing risks of bias and
unreliability. Approaches like HA-DPO (Zhao et al., 2023), POVID (Zhou et al., 2024a), and STIC
(Deng et al., 2024) perturb the image and text prompts or inject false statements into model responses
to heuristically construct preference pairs. Other techniques, such as RLAIF-V (Yu et al., 2024b) and
CSR (Zhou et al., 2024b), employ self-rewarding mechanisms to attain correctness scores or vision-
language alignment scores for preferences. In contrast, we propose a fine-grained, verifiable feedback
approach that links specific categories of undesired behavior (e.g., false or irrelevant responses) to
detailed text spans (e.g., sentences or sub-sentences), which provides more generalizable and reliable
automatic feedback for improving learning through feedback.

Figure 1: Overview of the proposed DCSCORE for evaluating detailed image captioning. (1) Given
the image and prompt, model generated responses and human written responses are decomposed
into sets of primitive information units. (2) We match the primitive information units of generated
response P and those of human written response O. (3) Each primitive information unit in P is
verified individually by VLM given the content of images.

3 DECAPBENCH: IMAGE CAPTIONING TESTBED FOR MODERN VLMS

Recent open-source VLMs have been significantly improved, narrowing their performance gap
compared with GPT-4V on various benchmarks. However, this progress does not always translate
into better image captioning abilities. The issue lies in the fact that while current VLMs can generate
detailed captions with many fine-grained elements, existing metrics rely on coarse-grained ground-
truth captions that overlook these details. Furthermore, traditional automatic evaluation metrics show
lower correlation with human evaluations, raising questions about their effectiveness. To address these
limitations, we propose DECAPBENCH, a new image captioning evaluation benchmark, along with a
novel metric DCSCORE, as illustrated in Figure 1, that better captures the descriptive capabilities of
VLMs. Our metric ensures that model rankings align more closely with results from the VLM arena,
which is based on diverse, crowd-sourced user votes for image description tasks.

3.1 DCSCORE EVALUATION METRIC

Previous image caption evaluation metrics (Papineni et al., 2002; Vedantam et al., 2015; Banerjee &
Lavie, 2005; Hessel et al., 2021; Anderson et al., 2016) are designed for short caption evaluation.
When applied to detailed captioning, these metrics suffer from limitations such as low-quality and
uninformative annotations, as well as biased captioning patterns, resulting in failures to adequately
assess hallucinations and the comprehensiveness of captions generated by VLMs. To address

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Pleasedescribe the image in detail.The image features a whitecat sitting on a plastictablenext to a lamp. The cat is positioned on the right side of the table,while the lamp is located on the left side. The table appears to be a dining table, and there is a bookplaced on the table near the cat. Image & QueryGenerated CaptionA lamp, cat, and notebook are on a polished wooden table against a vertically paneled wall in the photo. The lamp's base resembles stone and has a muted brown undertone. It is urn-shaped and has four distinct stone rings: two around its base and two at its widest point. A beige lampshade made of textured fabric rests on top of the base …Hyper-Detailed Human Caption!"(1) DecompositionThe image features a cat.The cat is white.The cat is sitting on a table.The table is plastic.The cat is next to a lamp.…There is a lamp.There is a cat.There is a notebook.There is a table.…Primitive Information Units(Human Caption)Primitive Information Units(Generated Caption)"!(2) Matching•The image features a cat.•The cat is white.•The cat is sitting on a table.•The table is plastic.•There is a lamp.•…•There is a lamp.•There is a cat•There is a notebook.•There is a table.•The table is wooden.•…(3) Verification•The image features a cat.•The cat is white.•The cat is sitting on a table.•The table is plastic.•The cat is next to a lamp.•…✅✅✅✅❌Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

this issue, we propose DCSCORE, a novel metric for detailed image captioning that accounts for
both hallucinations and fine-grained comprehensiveness. DCSCORE evaluates the quality of image
captions by generating and assessing primitive information units, which are the smallest self-sufficient
units of information within a caption. This method reduces ambiguity and enhances the transparency
of the evaluation process. The evaluation process consists of three steps, described as following.

Step 1: Decomposition. The extraction of primitive information units involves splitting the model-
generated caption into distinct components, which can be done either manually or by a large language
model (LLM). For the ground-truth caption, we use human experts to decompose it into a set of
primitive information units, denoted as O = {o1, o2, · · · , oM }, where M is the total number of
extracted units. On the other hand, we prompt the LLM to decompose the model-generated caption
on a sentence-by-sentence basis into a set P = {p1, p2, · · · , pN }, where N represents the number
of units extracted from the model’s description. Since image captions can include elements that
are not directly descriptive of the image, which may influence the overall quality and style of the
caption, it is essential to evaluate these non-descriptive elements as part of the VLMs’ captioning
capabilities. To differentiate between descriptive and non-descriptive units, we prompt LLMs to
perform a binary classification for each unit pi ∈ P during decomposition. Detailed instructions for
extracting primitive information units can be found in the Appendix.

Step 2: Matching. High-quality model-generated captions should incorporate all key elements from
the reference captions without omissions. To evaluate this, we prompt LLMs to assess whether each
primitive information unit pi ∈ P from the generated caption is mentioned or can be logically inferred
from the reference caption oj ∈ O. The matching process is formally computed as Q = P ∩ O,
where Q is the overlap of primitive information units between the generated and reference captions.

Step 3: Verification. To verify the correctness of the primitive information units pi in the generated
captions P, we use modern VLMs. Specifically, we employ GPT-4o (OpenAI., 2024a) to assess
the accuracy of each unit by referencing the corresponding image. GPT-4o is prompted to provide
a simple "yes" or "no" answer regarding the correctness of each unit, without requiring further
explanation, following the approach used by Li et al. (2023b).

After obtaining the model-generated set P, the reference set O, and their overlap Q, we compute
both a precision score sp (non-hallucination) and a recall score sr (comprehensiveness) as follows:

sp =

|Ptrue|
|P|

,

sr =

|Q| + |Ptrue \ Q|
|O| + |Ptrue \ Q|

,

(1)

where Ptrue = {pi|pi ∈ P, pi is correct} represents the set of correct units in the set P.

We assess the overall caption quality using the F1 score sf , which balances the precision score sp
and recall score sr. Additionally, we evaluate the descriptive elements of the caption by computing
the F1 score s′

f for only the descriptive units. The final assessment score F is computed as:

F =

1
2

(sf + s′

f ).

(2)

3.2 DECAPBENCH: A DETAILED IMAGE CAPTIONING EVALUATION BENCHMARK

Dataset. We consider the recently released ImageInWords dataset (Garg et al., 2024), and leverage
400 high-quality, human-curated public image detailed captions from as the ground-truth captioning.
Compared with ImageInWords, traditional caption datasets such as COCO (Sidorov et al., 2020; Lin
et al., 2014; Agrawal et al., 2019) often contains short, coarse-grained captions, and lack detailed
information, making them inadequate for measuring the correctness and comprehensiveness of
the models’ generated detailed captions. In contrast, ImageInWords considers a human-in-the-
loop framework produces hyper-detailed and hallucination-free image descriptions, by combining
human annotators and seeded machine generations. Consequently, we constructed DECAPBENCH,
by applying the proposed DCSCORE evaluation metric to the ImageInWords images and their
corresponding hyper-detailed image captions.

4

Under review as a conference paper at ICLR 2025

Metric
Rule-Based Evaluation
BLEU-4 (Papineni et al., 2002)
ROUGE (Lin, 2004)
METEOR (Banerjee & Lavie, 2005)
CIDEr (Vedantam et al., 2015)
Model-Based Evaluation
SPICE (Anderson et al., 2016)
CLIP-Score (Hessel et al., 2021)
PAC-Score (Sarto et al., 2023)
CAPTURE (Dong et al., 2024)
CLAIR (Chan et al., 2023)
FLEUR (Lee et al., 2024)
GPT4-Eval (Liu et al., 2024b)
Faithscore (Jing et al., 2023)
RLAIF-V (Yu et al., 2024b)
DCSCORE

PCC (ρ) ↑

1 − R2 ↓

Kd τ ↑

Sp τ ↑

0.3439
0.2509
0.3593
0.0522

0.2218
0.2183
0.1525
0.3521
0.3815
0.4230
0.3976
0.1937
0.3547
0.6605

62.78
156.05
111.95
3.3e7

156.11
26.04
20.93
7.62
1.98
3.01
2.95
3.22
5.32
1.54

0.2693
0.1886
0.2417
0.0635

0.1731
0.1724
0.1117
0.2801
0.3847
0.4246
0.3447
0.1626
0.2774
0.5328

0.2931
0.1893
0.2536
0.0601

0.1907
0.1480
0.1260
0.3449
0.4552
0.5325
0.3866
0.1115
0.2544
0.6166

Table 1: Correlation of image captioning evaluation metrics and human judgements. All p-values
< 0.001. The bold number indicates the highest human consistency among all caption metrics.

Human consistency of DCSCORE. To demonstrate consistency with human expert judgments,
we randomly selected 500 captions generated by different models and employed X experienced
annotators to rate each caption. We then computed the statistical metrics to compare the proposed
DCSCORE with human ratings, including the Pearson correlation coefficient (PCC) ρ, coefficient
of determination R2, Kendall’s τ (Kd τ ) and Sample-wise τ (Sp τ ). The correlation statistics, as
presented in Figure 2 (Left), highlight the significant improvements brought by our proposed metric,
DCSCORE. Compared to the state-of-the-art, DCSCORE enhances PCC ρ by 0.2375 and boosts
Kendall τ by 0.1082. These advancements suggest that our metric achieves superior linear correlation
and pairwise ranking accuracy with human judgments. Hence, DCSCORE holds great potential for
optimizing detailed captions produced by VLMs.

High-quality and hyper-detailed image descriptions are crucial for evaluating model-generated
captions, as they closely mirror the content of the image. To investigate this, we assess the impact of
varying quality of ground-truth descriptions on our proposed DCSCORE. As shown in Figure 2 (Left),
descriptions with finer granularity achieve higher consistency with human judgments compared
to COCO-style concise captions. Specifically, detailed captions annotated by either humans or
GPT-4o (OpenAI., 2024a) demonstrate a superior alignment with human evaluators, highlighting the
importance of granularity in image description for more reliable and accurate evaluation.

Source of Captions PCC (ρ) ↑ 1 − R2 ↓ Kd τ ↑ Sp τ ↑
0.4375 0.5093
COCO-Style
Instruct-BLIP
0.4745 0.5620
0.5194 0.5745
GPT-4o
0.5328 0.6166
Human Annotated

0.5468
0.6062
0.6497
0.6605

14.10
5.50
2.03
1.54

Figure 2: (Left) Comparison of four sources for
ground-truth captions in terms of correlation between
DCSCORE and human judgments. All p-values are
less than 0.001. (Right) DECAPBENCH achieves the
highest correlation with Arena Elo, with a Spear-
man’s correlation of 0.90 among different VLM
benchmarks.

Human consistency of DECAPBENCH. To further study the consistency between the proposed
DECAPBENCH and human judgement in the wild, we select the subset of image description from the
VLM arena, and compute the ranking correlation. Note that VLM arena is a public VLM evaluation
platform, where two model responses for the same task prompt are voted by humans to reflect their

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Arena EloDeCapBenchMMVetMMStarMathVistaMMMUHallusionBenchMMBench ENMMBench CNCCBenchArena EloDeCapBenchMMVetMMStarMathVistaMMMUHallusionBenchMMBench ENMMBench CNCCBench1.000.900.880.860.840.790.660.630.590.470.901.000.840.700.730.840.470.630.620.440.880.841.000.600.910.910.690.810.800.640.860.700.601.000.650.520.520.410.310.250.840.730.910.651.000.840.820.670.630.550.790.840.910.520.841.000.720.640.680.580.660.470.690.520.820.721.000.330.360.480.630.630.810.410.670.640.331.000.950.690.590.620.800.310.630.680.360.951.000.840.470.440.640.250.550.580.480.690.841.000.30.40.50.60.70.80.91.0Under review as a conference paper at ICLR 2025

preferences. Specifically, we compute human preferences using Elo ratings, derived from over 1,000
pairwise comparisons with around 800 images across 13 different VLMs on image captioning tasks.

In Figure 2 (Right), we visualize the Spearman correlation heatmap among various automatically
evaluated multi-modal benchmarks (Chen et al., 2024a; Liu et al., 2023; Yue et al., 2024; Kembhavi
et al., 2016) and human-voted preference benchmarks (Lu et al., 2024). From the figure, we observe
that DECAPBENCH achieves the highest correlation with Arena Elo at 0.90, indicating a high level
of alignment with human preferences and a strong consistency in ranking. This high correlation
demonstrates the effectiveness of DECAPBENCH in capturing the nuances of human judgment,
making it a reliable benchmark for evaluating the image captioning capabilities of VLMs.

Compared with existing multimodal benchmark, the proposed DECAPBENCH is unique in its
dedication to the task of detailed captioning, verified by the highest correlation with Arena captoin
subset. Note that MMVet (Yu et al., 2023) evaluates the models’ ability to solve complex vision-
language tasks. MMMU (Yue et al., 2024) and MathVista (Lu et al., 2023) assess subject knowledge
and mathematical reasoning in visual contexts, respectively, while HallusionBench focuses on
understanding visually misleading figures. The MMBench-series (Liu et al., 2023) (e.g., MMBench-
EN, MMBench-CN, and CCBench) concentrates on fine-grained perception and reasoning tasks using
multiple-choice questions. Additionally, MMStar (Chen et al., 2024a) corrects the misjudgments of
actual multi-modal performance.

4 LEARNING FROM FINE-GRAINED FEEDBACK
4.1 FINE-GRAINED FEEDBACK COLLECTION

The feedback collected for preference learning consists of comparison pairs, where each pair includes
a preferred response and a less preferred response to the same input. The model learns from this
preference data to distinguish differences among its own generated candidate responses. To gather
these candidate responses, we generate multiple outputs for given images and prompts using nucleus
sampling (Holtzman et al., 2019), varying the random seed to ensure diversity. By learning to rank
these candidate responses based on the preference data, the model becomes capable of assessing the
quality of its outputs and deriving appropriate signals for preference optimization.

However, judging the quality of different responses is complex, even for experienced human annota-
tors (Sun et al., 2023), due to the semantic intricacies involved. Previous methods (Zhou et al., 2024a;
Zhao et al., 2023) attempted to address this by manually modifying responses and injecting noise
to create negative samples. However, these approaches suffer from poor generalization because of
implicit patterns in the data. In contrast, by adapting the concept of primitive information units and
step-by-step verification (Lightman et al., 2023), we propose FEEDQUILL for feedback collection,
which leverages modern VLMs to generate fine-grained feedback in the following three steps:

• Decomposition. We prompt an LLM to decompose the response into a set of N primitive
i=1 on a sentence-by-sentence basis, rewriting them into self-sufficient

information units {pi}N
and verifiable statements.

• Scoring. We use several powerful VLMs (Chen et al., 2024b; Liu et al., 2024a) to ver-
ify these rewritten statements using the prompt: "{STATEMENT} Is the statement
correct? Please only answer ’yes’ or ’no’". To increase confidence in our
judgments, we ensemble the results from multiple open-source VLMs for verification.

• Preference. After obtaining the verification results for each primitive information unit, we
calculate the preference score cp as the fraction of correct units: cp = 1
i=1 1{pi = 1},
N
where a higher score indicates fewer hallucinations in the response. Given the scores of each
response, we construct a preference dataset D = (xi, y+
i ) by treating the response with the
higher score as the preferred response y+
i and the one with the lower score as the non-preferred
response y−
i .

i , y−

(cid:80)N

As discussed in Zhu et al. (2023), responses with fewer hallucinations are often inherently less helpful.
Specifically, models are more likely to hallucinate when producing longer responses compared to
shorter ones. To address this issue, we construct a preference dataset Dr using the number of primitive
information units as the preference score cr. A response with a higher score cr — indicating more
primitive information units — is considered more preferable. This approach encourages the model to
generate responses that are not only accurate but also rich in helpful and detailed information.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

4.2 PREFERENCE OPTIMIZATION

Preference optimization (Ouyang et al., 2022; Rafailov et al., 2024) has shown promise in fine-tuning
language models and aligning their behavior with desired outcomes. Specially, we train the reward
model rϕ with the preference set D and Dr respectively, with the a pairwise comparison loss (Ouyang
et al., 2022) as LRM = −E(x,y+,y−)∼D [log (σ(rϕ(x, y+) − rϕ(x, y−)))], where σ(·) is the sigmoid
function and rϕ(·, ·) is the output score of the reward model. To mitigate biased preferences, such as
unhelpful responses, we opt against using a single scalar reward to represent response quality. Instead,
we leverage rewards derived from multiple reward models, each contributing to distinct behaviors
like hallucination (cp) and richness (cr). To optimize these preferences, we utilize proximal policy
optimization (PPO) (Schulman et al., 2017), a widely adopted reinforcement learning algorithm. In
To fully exploit the characteristics of preferences related to hallucination and comprehensiveness, we
select captioning as the optimization task. For additional details, please refer to the Appendix.

5 EXPERIMENTS

5.1 EXPERIMENTAL SETUP

Model. We conduct our experiments based on a series of LLaVA models (Liu et al., 2024b) with
different sizes and capabilities. We initialize both the policy model and reward model with same
parameters as well as same size for validating the effectiveness of our proposed method. For the main
results, we report the performance of our model FEEDQUILL-7B trained on LLaVA-Onevision-7B,
one of the most capable models in the < 10B size category.

Training Dataset for PPO. The PPO is performed with the detailed captioning task. To ensure the
model learns robust generalization capabilities, diversity in image distributions is crucial. Therefore,
we randomly sample images from a wide range of datasets, including MSCOCO (Lin et al., 2014),
OpenImages (Kuznetsova et al., 2020), and ShareGPT4V (Chen et al., 2023). Additionally, to
maintain diversity of instructions during training, we prompt GPT-4o (OpenAI., 2024a) to generate a
variety of caption prompts, and provide in Appendix.

5.2 ABLATIONS

Preference Data for Reward Model. To assess the ability of various preference data to generalize,
we trained multiple reward models using the same SFT model. For evaluation, we randomly sampled
portions of the preference data that were held out. The findings, presented in Table 2, reveal that
our model achieved the highest accuracy across diverse preference datasets. Notably, with the same
scale of training data, our reward model outperformed the human-labeled dataset RLHF-V by 9.9%
in accuracy. It also surpassed the RLAIF-V dataset, which, despite having over 80k training samples,
was outperformed by our model that utilized a smaller data size. Additionally, we observed that
increasing the amount of training data led to an improvement in average accuracy from 71.3% to
75.2%, highlighting the scalability of our approach.

Train Data

HA-DPO (Zhao et al., 2023)
RLHF-V (Yu et al., 2024a)
POVID (Zhou et al., 2024a)
CSR (Zhou et al., 2024b)
RLAIF-V (Yu et al., 2024b)
STIC (Deng et al., 2024)
FEEDQUILL*
FEEDQUILL

Held-Out Eval Dataset
HA-DPO RLHF-V POVID CSR RLAIF-V STIC Average
53.5
44.2
59.4
87.5
55.5
43.3
59.7
55.2

51.0
48.7
52.5
51.8
68.1
50.1
64.7
69.4

42.0
67.8
59.5
23.6
66.8
99.9
74.1
84.9

93.5
82.0
32.5
62.5
69.5
48.0
78.0
76.5

81.1
94.7
30.6
51.8
49.5
59.7
64.1
71.9

23.7
30.7
99.5
60.3
77.6
26.8
87.4
93.2

57.5
61.4
55.7
56.3
64.5
54.6
71.3
75.2

Table 2: Reward model zero-shot accuracy on the held-out validation set trained with different
preference data on LLaVA-1.5-7B. * indicates that we only utilize 10k preference data to match the
size of other training set.

Preference Data for Preference Optimization. We delve into how varying types of preference
data impact preference optimization. Using LLaVA-1.5-7B as our baseline model, we trained it with

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

a variety of preference datasets. The performance of these models was then assessed through a range
of downstream benchmarks in a zero-shot context. As showcased in Table 3, our approach not only
excels in captioning performance but also substantially cuts down on hallucinations, achieving a
notable 0.75 improvement on mmHal-V compared to the baseline.

Method
LLaVA-1.5

w/ HA-DPO
w/ POVID
w/ CSR
w/ RLAIF-V
w/ FEEDQUILL

MMBench ↑ VizWiz ↑ MMStar ↑ WildVision ↑

LLaVA-W ↑ DECAPBENCH ↑ mmHal-V ↑ CHAIRS ↓ CHAIRI ↓

64.8
64.3
64.7
64.2
62.7
66.3

50.0
54.1
47.9
52.8
50.9
55.2

33.1
33.5
35.4
33.8
34.7
35.8

14.48
15.17
13.25
13.85
15.65
19.68

65.3
65.1
71.5
70.3
76.0
76.0

24.50
22.45
23.54
23.70
28.21
34.52

1.85
2.12
1.90
2.12
2.59
2.60

47.8
49.3
31.8
15.7
8.5
5.1

25.3
25.5
5.4
7.9
4.3
2.6

Table 3: The performance of different preference data on LLaVA-1.5-7B across different benchmarks.

Data Size. We scale up the training set of the reward model, and investigate the correlation between
downstream performance through preference optimization. We evaluate different checkpoints ranging
from 5,000 to 200,000 training samples, using models of sizes 7B and 13B. The results are illustrated
in Figure 3. As the size of the preference data increased, the performance of mmHal-V improves
from 2.05 to 2.6. Similarly, MMStar, which focuses on image understanding, shows a consistent
increase from 34.7 to 35.8, yielding a 1.1 point lift. This demonstrates that as the size of preference
data for the reward model grows, the model’s performance consistently improves since the better
reward model provides more accurate signals for preference optimization.

Figure 3: Impact of the preference dataset size in terms of downstream performance.

Source of Responses. We explore the effect of the source of model responses on preference data,
based on the hypothesis that improvements might arise from the model’s ability to generalize across
varying sources. To test this hypothesis, we use LLaVA-1.5-13B as the base model and examine
responses sampled either from the same model or from other models such as LLaVA-1.5-7B, LLaVA-
1.6-7B, and LLaVA-1.6-13B. Furthermore, we assess the impact of combining responses from these
different sources. The results of these experiments are summarized in Table 4. We observe that
integrating responses generated by the same model only leads to a significant performance boost
compared to the baseline model. Conversely, integrating responses from different models only leads
to larger performance gains on DECAPBENCH by providing diverse responses, while smaller gains
on other benchmarks. When combining responses from both sources, the model achieves superior
performance, surpassing the use of either source alone. Specifically, this combination results in an
improvement of 13.0 points on LLaVA-W and 13.23 points on DECAPBENCH compared to baseline.

Source of Response
Same Model Other Models

MMStar LLaVA-W mmHal-V DECAPBENCH

✓

✓

✓
✓

33.1
37.6
38.0
38.3

65.3
75.1
71.5
78.3

1.85
2.74
2.53
2.83

24.50
26.32
34.84
37.73

Method

Base
Only cp
Only cr
cp + cr

LLaVA-1.5-7B
LLaVA-W DECAPBENCH

LLaVA-1.5-13B
LLaVA-W DECAPBENCH

65.3
67.3
46.2
76.0

24.50
25.21
10.03
34.52

72.8
74.3
56.9
78.3

25.55
26.23
15.11
37.73

Table 4: Comparison of performance by varying
sources of preference data.

Table 5: Ablation of using different reward
scores during preference optimization.

Source of Rewards. Table 5 provides a comparative analysis of incorporating the preference score
for the number of primitive information units (cr) alongside the preference score for the proportion
of correct units (cp). Each preference score is obtained separately from different reward models,
summed to a final reward in PPO training procedure. We specifically evaluate our method against
three distinct variants: (1) the base model without any preference optimization (Base); (2) a model
optimized solely with the cp score (Only cp); and (3) a model optimized exclusively with the cr

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

5K10K20K50K100K200KData Size35363738MMStar34.7034.4534.8134.9735.5235.8136.6037.1637.5137.4437.4138.28LLaVA-1.5-7BLLaVA-1.5-13B5K10K20K50K100K200KData Size687072747678LLaVA-W71.5068.4071.1070.9074.7076.0073.7074.1074.5075.7077.5078.305K10K20K50K100K200KData Size2.02.22.42.62.83.0mmHal-V2.052.152.262.272.442.602.382.412.402.442.562.875K10K20K50K100K200KData Size32333435363738DeCapBench31.7932.2333.3532.8033.5534.5235.3835.8436.5036.7736.9337.73Under review as a conference paper at ICLR 2025

score (Only cr). This comparison enables a thorough examination of the impact of each optimization
strategy on model performance. Notably, models trained with the cp score consistently enhance
performance on both LLaVA-W and DECAPBENCH. Conversely, models trained with the cr score
yield poorer results on both datasets due to the absence of a precision constraint. Furthermore, when
both cp and cr are incorporated, our method exhibits significant improvements, notably a 10.7%
increase on LLaVA-1.5-7B and a 5.5% boost on LLaVA-1.5-13B.

Comprehensive Benchmark

Visual Hallucination

Visual Chat and Captioning

Method

MMBench MMStar

VizWiz

LLaVA-1.5-7B

+ FEEDQUILL
LLaVA-1.5-13B
+ FEEDQUILL

LLaVA-1.6-7B

+ FEEDQUILL
LLaVA-1.6-13B
+ FEEDQUILL

LLaVA-Onevision-7B

+ FEEDQUILL

64.8
66.3 (+1.7)
68.7
69.2 (+0.5)
67.1
67.9 (+0.8)
69.3
69.9 (+0.6)
80.8
80.5 (+0.3)

33.1
35.8 (+2.7)
34.3
38.3 (+4.0)
37.6
38.6 (+1.0)
40.4
41.1 (+0.7)
61.7
62.4 (+0.7)

50.0
55.2 (+5.2)
53.6
56.8 (+3.2)
57.6
63.4 (+5.8)
60.5
66.7 (+6.2)
60.0
60.4 (+0.4)

SciQAI

66.8
68.9 (+2.1)
71.6
73.4 (+1.8)
70.2
70.3 (+0.1)
73.6
73.5 (+0.1)
96.0
95.9(-0.1)

mmHal-V

LLaVA-W WildVision DECAPBENCH

1.85
2.60 (+0.75)
2.33
2.83 (+5.00)
2.58
2.93 (+0.35)
2.95
3.76 (+0.81)
2.94
3.10 (+0.16)

65.3
76.0 (+10.7)
72.8
78.3 (+5.5)
79.8
82.4 (+2.6)
85.2
87.1 (+1.9)
90.7
100.5 (+9.8)

14.48
17.68 (+3.20)
16.17
18.15 (+1.98)
26.15
44.16 (+18.01)
33.69
49.69 (+16.00)
54.50
59.60 (+5.10)

24.50
34.52 (+10.02)
25.55
37.73 (+12.18)
35.74
52.69 (+16.95)
36.28
53.26 (+16.98)
43.49
55.65 (+12.16)

Table 6: Performance of FEEDQUILL with various VLM models on downstream tasks.

Compatibility Analysis. To validate the applicability of FEEDQUILL across various VLMs, we
conduct experiments on various models. The summarized results in Table 6 reveal that FEEDQUILL
is effective regardless of model size, consistently enhancing performance on downstream tasks such
as MMBench, mmHal-V, and DECAPBENCH. This underscores the robust generalization capability
of our proposed FEEDQUILL. Notably, LLaVA-1.6-13B trained with FEEDQUILL exhibits large
improvement on mmHal-V, increasing from 2.95 to 3.76. Simultaneously, it significantly boosts
performance on WildVision and DECAPBENCH, with gains of +16.0% and +16.98%, respectively.

5.3 MAIN RESULTS

AI2D ChartQA MMBench SEEDBench MME MMMU MMVet MMStar SciQA LLaVA-W WildVision DECAPBENCH

94.7
94.4
78.2
94.2

Model
Proprietary Model
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4V
GPT-4o
Open-Source Model
79.7
Cambrian-34B
VILA-40B
-
81.5
XComposer-2.5-7B
83.8
InternVL-2-8B
InternVL-2-26B
84.5
LLaVA-Onevision-7B 81.4
81.3
FEEDQUILL-7B

90.8
87.2
78.5*
85.7

73.8
-
82.2
83.3
84.9
80.0
80.3

78.5
73.9
79.8
80.5

81.4
82.4
82.2
81.7
83.4
80.8
80.5

-
-
49.9
76.2

-
75.8
75.4
76.0
76.8
75.4
75.8

-/-
-/-
1409/517
-/-

-/-
1762
2229
2210
2260
1580/418
1515/450

68.3
62.2
56.8
69.1

49.7
51.9
42.9
49.3
48.3
48.8
47.9

75.4
64.0
57.1
76.2

53.2
51.2
51.7
60.0
65.4
57.5
59.3

60.2
58.7
75.7
59.8

85.6
54.2
59.9
59.4
60.4
61.7
62.4

80.5
-
75.7
83.5

67.8
-
-
97.0
97.5
96.0
95.9

102.9
-
98.0
106.1

-
-
78.1
84.5
99.6
90.7
100.5

50.00
35.45
80.01
89.41

-
-
-
-
-
54.50
59.60

52.37
46.34
48.52
53.44

35.12
38.02
29.60
45.55
49.59
43.49
55.65

Table 7: Main experimental results of our method and other open-sourced state-of-the-art VLMs.
*GPT-4V reports 4-shot results on ChartQA. All results are presented in the 0-shot setting.

We evaluate FEEDQUILL-7B across a variety of multi-modal large language model benchmarks,
including AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), MMBench (Liu et al.,
2023), SEEDBench (Li et al., 2024b), MME (Fu et al., 2023), MMMU (Yue et al., 2024), MMVet
(Yu et al., 2023), MMStar (Chen et al., 2024a), ScienceQA (Lu et al., 2022), LLaVA-W Liu et al.
(2024b), WildVision (Lu et al., 2024), and DECAPBENCH. These datasets are specifically designed to
measure various capabilities of VLMs, including document understanding, question answering, visual
chatting, visual perception, and detailed image captioning. Table 7 presents a comparative analysis
of FEEDQUILL-7B against state-of-the-art VLMs, encompassing both proprietary and open-source
models including Claude-3.5-Sonnet (Anthropic., 2024), Gemini-1.5-Pro (Team et al., 2023), GPT-4v
(OpenAI., 2024b), GPT-4o (OpenAI., 2024a), Cambrian-34B (Tong et al., 2024), VILA-40B (Lin
et al., 2024), XComposer-2.5-7B (Zhang et al., 2024), and InternVL-2-8B/26B (Chen et al., 2024b).

FEEDQUILL-7B achieves state-of-the-art performance in detailed image captioning, surpassing
GPT-4o by 2.21 points. Remarkably, it also outperforms GPT-4v on LLaVA-W, showing strong

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

capability in visual chatting. Despite being trained solely on the captioning task, our model maintains
its strong performance while achieving a 1.8-point improvement on MMVet and a 0.7-point increase
on MMStar compared to LLaVA-Onevision-7B. Additionally, it retains most of its capabilities after
preference optimization – a feat that many aligned models, such as BHDS (Amirloo et al., 2024),
CSR (Zhou et al., 2024b), and RLAIF-V (Yu et al., 2024b), fail to accomplish.

5.4 CASE STUDY

Figure 4: Qualitative results of FEEDQUILL-7B compared with LLaVA-Onevision-7B (Li et al.,
2024a) in terms of image captioning.

We provide qualitative results of LLaVA-Onevision-7B and FEEDQUILL-7B in Figure 4 to illustrate
the effectiveness of our proposed method. In the example above, LLaVA-Onevision-7B incorrectly
identifies the red wine in the glasses as a vibrant screen. In contrast, our model correctly identifies
it as a red liquid with fewer instances of hallucination. Additionally, while LLaVA-Onevision-7B
generically names both phone as "cell phone", FEEDQUILL-7B specifically identifies them as a
Blackberry device and a flip phone, showcasing its strong fine-grained captioning capabilities. We
refer readers to the Appendix for more qualitative results.

6 CONCLUSION

We have described a novel metric, DCSCORE, designed to evaluate both hallucination and compre-
hensiveness, the two critical challenges in detailed image captioning. Empirical validations show
that DCSCORE achieves the highest consistency with human judgments, underscoring its reliability.
Additionally, we present a new detailed caption benchmark, DECAPBENCH, specifically for assessing
the captioning capabilities of modern VLMs. Our results demonstrate that the correlation of DE-
CAPBENCH with human judgment surpasses that of any other public benchmark in description tasks.
Furthermore, we propose an effective fine-grained feedback collection method, FEEDQUILL, which
decomposes responses into primitive information units for individual verification and subsequently
learns an improved model through preference optimization. Comprehensive experiments reveal that
FEEDQUILL is applicable across various models, achieving superior image captioning performance
while reducing hallucinations, and setting new state-of-the-art. We believe that both DECAPBENCH
and FEEDQUILL will serve as invaluable foundations for advancements in detailed image captioning
and preference optimization.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Explain the visual content of the image in great detail.In the dimly lit room, a group of people are engrossed in their phones. The person closestto us, a man wearing a black shirt, is holding a phone with a vibrant red screen, which is prominently displayed in the foreground. The phone, a sleek device with a small screen, is held securely in their hand.A little further away, another person is seen holding a phone with a blackscreen, which is positioned in the upper right corner of the image. This phone, slightly larger than the one in the foreground, is also held in a hand.In the background, there's a person wearing a black and whiteshirt, their hand holding a phone with a white screen, which is positioned in the upper left corner of the image. The phone is slightly larger than the one in the foreground, and it's held in a hand.The room they're in is dimly lit, creating an atmosphere of focus and concentration. The people are all dressed in dark clothing, adding to the overall mood of the scene. The image captures a moment of modern life, where technology and social interaction intersect.LLaVA-OneVision-7BFEEDQUILL-7B (This work)The image is a photograph taken from an overhead perspective, focusing on the hands and upper bodies of two individuals. The main subjects are the hands of the individuals, which are holding two different types of mobile devices. The person on the leftis holding a flip phone, which is open, revealing its screen and keypad. The person on the rightis holding a Blackberry device, which is also open, displaying its screen and keyboard.The person on the right is also holding a glass containing a red liquid, which appears to be wine. The glass is transparent and has a stem, suggesting it is a wine glass. The individual is wearing a black and white patterned garment, which could be a scarf or a piece of clothing with a distinctive design.The background of the image is dark and out of focus, drawing attention to the hands and the objects they are holding. The lighting in the image is dim, with the main source of light coming from the screens of the mobile devices, which illuminate the hands and the glass of wine.There are no visible texts or logos on the mobile devices, and the image does not provide any additional context about the location or the event taking place. The style of the photograph is candid and informal, capturing a moment in time without any staging or posing. The composition of the imageemphasizes the interaction between the individuals and their devices, as well as the contrast between the modern technology and the traditional act of holding aglass of wine.Wrong Elements (Hallucinations)Correct ElementsAdditional Correct ElementsUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra,
Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In
Proceedings of the IEEE/CVF international conference on computer vision, pp. 8948–8957, 2019.

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and
Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human
feedback in llms. arXiv preprint arXiv:2402.14740, 2024.

Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu
Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. Understanding alignment in
multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477, 2024.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional
image caption evaluation. In Computer Vision–ECCV 2016: 14th European Conference, Ams-
terdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382–398. Springer,
2016.

Anthropic. The claude 3 model family: Opus, sonnet, haiku., 2024.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv preprint arXiv:2308.12966, 2023.

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic
evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.

David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. CLAIR: Evalu-
ating image captions with large language models. In Houda Bouamor, Juan Pino, and Kalika
Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing, pp. 13638–13646, Singapore, December 2023. Association for Computational Linguis-
tics. doi: 10.18653/v1/2023.emnlp-main.841. URL https://aclanthology.org/2023.
emnlp-main.841.

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua
Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint
arXiv:2311.12793, 2023.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi
Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language
models? arXiv preprint arXiv:2403.20330, 2024a.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial
multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024b.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-
language models with instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate
Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing
Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,
New Orleans, LA, USA, December 10 - 16, 2023, 2023.

Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang.
Enhancing large vision language models with self-training on image comprehension. arXiv preprint
arXiv:2405.19716, 2024.

Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Bench-

marking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin,
Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation
benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023. doi: 10.48550/
ARXIV.2306.13394. URL https://doi.org/10.48550/arXiv.2306.13394.

Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa
Imageinwords:

Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut.
Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793, 2024.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu
Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng,
Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu,
Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao,
Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu,
Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu,
Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen
Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models
from glm-130b to glm-4 all tools, 2024.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and
Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608–3617,
2018.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A
reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang,
Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical
Methods in Natural Language Processing, pp. 7514–7528, Online and Punta Cana, Dominican
Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.
emnlp-main.595. URL https://aclanthology.org/2021.emnlp-main.595.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text

degeneration. arXiv preprint arXiv:1904.09751, 2019.

Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. Faithscore: Evaluating

hallucinations in large vision-language models. arXiv preprint arXiv:2311.01477, 2023.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.
A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference,
Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp. 235–251.
Springer, 2016.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab
Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset
v4: Unified image classification, object detection, and visual relationship detection at scale.
International journal of computer vision, 128(7):1956–1981, 2020.

Yebin Lee, Imseong Park, and Myungjoo Kang. Fleur: An explainable reference-free evaluation
metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004,
2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei
Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint
arXiv:2408.03326, 2024a.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.
Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 13299–13308, 2024b.

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou
Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv
preprint arXiv:2312.10665, 2023a.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating ob-
In Proceedings of the 2023 Conference
ject hallucination in large vision-language models.
on Empirical Methods in Natural Language Processing, pp. 292–305, Singapore, December
2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.20. URL
https://aclanthology.org/2023.emnlp-main.20.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint
arXiv:2305.20050, 2023.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization

branches out, pp. 74–81, 2004.

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On
pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 26689–26699, 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–
ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings,
Part V 13, pp. 740–755. Springer, 2014.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.

Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in

neural information processing systems, 36, 2024b.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi
Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?
arXiv preprint arXiv:2307.06281, 2023.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord,
Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for
science question answering. Advances in Neural Information Processing Systems, 35:2507–2521,
2022.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng,
Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning
of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin.
Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint
arXiv:2406.11069, 2024.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench-
mark for question answering about charts with visual and logical reasoning. arXiv preprint
arXiv:2203.10244, 2022.

OpenAI. Hello gpt-4o., 2024a. https://openai.com/index/hello-gpt-4o/.

OpenAI. Gpt-4v., 2024b. https://openai.com/index/gpt-4v-system-card/.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. Advances in neural information processing systems, 35:27730–
27744, 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association
for Computational Linguistics, pp. 311–318, 2002.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International conference on machine learning, pp.
8748–8763. PMLR, 2021.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. Advances
in Neural Information Processing Systems, 36, 2024.

Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Positive-
augmented contrastive learning for image and video captioning evaluation. In Proceedings of the
IEEE/CVF conference on computer vision and pattern recognition, pp. 6914–6924, 2023.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy

optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for
image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European
Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 742–758. Springer,
2020.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and
Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pp. 8317–8326, 2019.

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan,
Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with
factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu
Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable
multimodal models. arXiv preprint arXiv:2312.11805, 2023.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha
Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open,
vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image
description evaluation. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pp. 4566–4575, 2015.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li,
Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint
arXiv:2408.01800, 2024.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu,
Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with
multimodality. arXiv preprint arXiv:2304.14178, 2023.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and
Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality
collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 13040–13051, 2024.

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu,
Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment
from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 13807–13816, 2024a.

Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He,
Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning mllms through open-source ai feedback for
super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024b.

14

Under review as a conference paper at ICLR 2025

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang,
and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv
preprint arXiv:2308.02490, 2023.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu
Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under-
standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 9556–9567, 2024.

Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong
Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language
model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024.

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond
hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv
preprint arXiv:2311.16839, 2023.

Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in
vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024a.

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang,
Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv
preprint arXiv:2405.14622, 2024b.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En-
hancing vision-language understanding with advanced large language models. arXiv preprint
arXiv:2304.10592, 2023.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 DISCUSSION

A.1.1 RELATED WORKS

Descriptive/Non-Descriptive
Response Evaluation Coverage
Hallucination
Comprehensiveness
Decomposition Method
For Evaluation
For Preference Learning

Faithscore
/
Full

RLAIF-V
/
Partial

Ours
/
Full

Rewrite Question-Answer Pairs Rewrite

Human Correlation (PCC ρ)
Human Correlation (Kd τ )
Human Correlation (Sp τ )

0.1937
0.1626
0.1115
Table 8: The comparison among related works.

0.3547
0.2274
0.2544

0.6605
0.5328
0.6166

We have compared Faithscore (Jing et al., 2023) and RLAIF-V (Yu et al., 2024b), two metrics built
on a similar conceptual foundation, and the distinctions are detailed in Table 8. Below, we summarize
these differences to highlight our main contributions:

• Granularity: While Faithscore and RLAIF-V evaluate the descriptive aspects of responses, they
neglect the non-descriptive elements, which are crucial for caption quality. For example, incor-
rect assertions about the image’s context and inferences can significantly impair understanding.
However, in the realm of detailed image captioning, comprehensiveness is equally critical, as
shorter captions may indeed exhibit lower hallucination rates but often suffer from a lack of
informative value. Our approach uniquely addresses this by simultaneously considering both
descriptive and non-descriptive components.

• Decomposition Method: Like Faithscore, our method decomposes responses sentence-by-
sentence, yet it also includes non-descriptive elements. RLAIF-V, on the other hand, generates
question-answer pairs for verification, potentially omitting crucial details.

• Score Generation: Faithscore rates the proportion of correct statements, while RLAIF-V counts
incorrect statements, which may encourage the model to avoid making any assertions or to state
irrelevant but correct information. Conversely, our approach evaluates both the proportion of
correct statements for hallucination and the number of valid statements for comprehensiveness.

• Application: Our method, designed for detailed image captioning, serves both evaluation
and preference learning within a unified framework. Faithscore and RLAIF-V are limited to
evaluating or optimizing hallucinations independently.

• Human Consistency: Our approach demonstrates the highest correlation with human judgment
across various aspects, as shown in the table, validating its effectiveness for detailed image
captioning.

In essence, our method introduces a more granular, comprehensive, and human-aligned evaluation
framework that surpasses existing methods for detailed image captioning.

A.1.2 CASE STUDY

As instances in Figure 5 and Figure 6 indicates, FEEDQUILL-7B not only significantly reduces
hallucinations, but also remarkably improves the granularity and richness of descriptions compared
with LLaVA-Onevision-7B (Li et al., 2024a), which is the initial model of FEEDQUILL-7B. From
these case we can see the preference score of precision (cp) and the preference of recall (cr) jointly
determine the direction of preference optimization in FEEDQUILL, leading the descriptions of
the images more precise and more comprehensive. Additionally, we present qualitative results of

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 5: Qualitative results of FEEDQUILL-7B compared with LLaVA-Onevision-7B (Li et al.,
2024a) in terms of image captioning.(1)

FEEDQUILL-7B and GPT4o (OpenAI., 2024a) in Figure 7. In these cases GPT4o still introduce
hallucinations while FEEDQUILL-7B describe them precisely. From these examples we can get an
intuitive understanding of the superior image captioning performance FEEDQUILL-7B achieves.

A.1.3 THE PERFORMANCE OF VLMS ON DECAPBENCH

We present the performance of various current VLMs on DECAPBENCH in Table 9. As shown, the
performance in detailed image captioning consistently improves with an increase in model size. For
instance, notable improvements are observed in the InternVL-2 series (8/26/40B) (Chen et al., 2024b)
and the LLaVA-series (7/13/34B) (Liu et al., 2024a).

17

Describe the following image.The image captures a vibrant city street, bathed in the soft glow of daylight. Dominating the scene is a red and white bus, its yellow poles standing tall and proud. The bus is parked on the right side of the street, patiently waiting for its next journey.On the left side of the street, a red brick building stands. It's adorned with a mural, a splash of color against the urban landscape. The building and the bus seem to be in a silent conversation, each telling their own story in this urban symphony.In the background, awhite car is parked, adding to the city's hustle and bustle. Astreet lamp stands sentinel, ready to bathe the street in light when night falls. A traffic light hangs overhead, its colors unseen in the daylight but ever ready to guide the flow of traffic.Above it all, the sky is a canvas of blue, dotted with birds in flight. They add a touch of life to the scene, their wings cutting through the air as they soar above the cityscape. The image is a snapshot of urban life, each element telling its own tale in this urban tapestry.LLaVA-OneVision-7BFEEDQUILL-7B (This work)The image is a color photograph depicting an urban street scene. In the foreground, there is a sidewalk with a curb and a street. The sidewalk is adjacent to a building with a wall covered in colorful graffiti, featuring various shapes, patterns, and text in a range of colors including red, blue, yellow, and purple. The graffiti appears to be vibrant and covers the entire visible portion of the wall.On the sidewalk, there are several people standing and walking. They are dressed in casual clothing, and some are carrying items like backpacks and handbags. The individuals are dispersed along the sidewalk, with some closer to the camera and others further away.In the middle ground, there is a bus stopped at a bus stop. The bus is predominantly white with blue and yellow accents, and it has the number "2242" visible on its front. The bus is connected to overhead wires, suggesting it is an electric or hybrid vehicle. There are also several cars parked along the street, and a few more in motion.In the background, there are more buildings, some of which have banners hanging from them. The banners are rectangular and feature text and images, but the details are not clear enough to read. There are also several birds in flight, captured against the sky, which appears to be overcast.The overall style of the image is candid and captures a moment of everyday city life. The photograph has a naturalistic colorpalette and is taken from a perspective that places the viewer on the sidewalk, looking down the street.Can you describe what is happening in this picture?In the heart of a cozy diner, a woman in a vibrant blue shirt is seated at a table, her attention captivated by the plate of donuts before her. She holds one in her hand,poised for a bite, while two othersrest on the plate, untouched. The table, draped in a pristine white tablecloth, also holds a black cell phone, lying idle. The backdrop is a cheerful yellow wall, adorned with a window and a painting, adding to the warm ambiance of the scene. The woman's anticipation is palpable as she prepares to enjoy her sweet treat.LLaVA-OneVision-7BFEEDQUILL-7B (This work)The image is a color photograph featuring a woman seated at a table with a bowl of donuts in front of her. The woman appears to be in her mid-30s to 40s, with short dark hair, and is wearing a light blue t-shirt. She is holding a donut in her right hand and appears to be in the process of eating it. Her expression is focused on the donut, and she is looking down at it.The bowl in front of her contains several donuts, some of which are visible. The donuts vary in color and decoration, with at least one having colorful sprinkles. The bowl is white and appears to be made of plastic.The table is wooden and has a dark finish. On the table, to the left of the bowl, there is a black cell phone lying flat. The background of the image shows a yellow wall witha small, square-shaped window or opening near the top. There is also a colorful, abstract painting or artwork hanging on the wall to the right of the woman.The setting appears to be indoors, possibly in a casual dining environment or a home. The lighting in the room is bright, and the overall atmosphere seems relaxed and informal. There are no visible texts or distinctive brands in the image.Additional Correct ElementsCorrect ElementsWrong Elements (Hallucinations)Additional Correct ElementsCorrect ElementsWrong Elements (Hallucinations)Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 6: Qualitative results of FEEDQUILL-7B compared with LLaVA-Onevision-7B (Li et al.,
2024a) in terms of image captioning.(2)

A.2

IMPLEMENTATION

A.2.1 TRAINING DETAILS

Reward Model We initialize the reward model with the parameters of the SFT model and adopt the
pairwise comparison loss (Ouyang et al., 2022) for training. The training is conducted for 1 epoch,
with learning rates set to 2e-5 for the 7B model and 5e-6 for the 13B model. The weight decay is set
to 0. The training size of the reward model is set to 200,000 pairs unless otherwise specified. During
inference, the reward model produces scalar outputs to provide the score for the responses.

PPO Our implementation of the PPO algorithm is a variant of (Ouyang et al., 2022). We adopt two
reward models: a cp RM and a cr RM. The cp RM is trained with the preference for the proportion
of correct units, which measures the precision or hallucination rate of the description of the image.
The cr RM is trained with the preference for the number of primitive information units, which

18

What's happening in the scene?In the heart of a bustling kitchen, a man in a white shirt and black hat is engrossed in his work. He stands in front of a large, white brick oven, its curved top reaching towards the ceiling. The oven, a symbol of warmth and comfort, is the centerpiece of this culinary space.To his left, two large pizzas rest on stacks of white plates, their vibrant colors popping against the neutral tones of the kitchen. One pizza is adorned with a medley of vegetables, their fresh hues hinting at the flavors within. The other pizza is generously topped with meat, promising a hearty and satisfying meal.The kitchen itself is a symphony of white and gray, with white tiles covering the walls and a gray counter providing ample workspace. The man's focus is unwavering as he prepares to slide one of the pizzas into the oven, the anticipation of a delicious meal palpable in the air. This image captures not just a moment, but an entire story of dedication, passion, and the art of cooking.LLaVA-OneVision-7BFEEDQUILL-7B (This work)The image is a color photograph depicting a scene inside a pizzeria. In the foreground, there are two stacks of white plates,each with a pizza on top. The pizzas appear freshly baked with various toppings, including what looks like tomato sauce, cheese, and possibly vegetables and meats. The crust of the pizzas is golden brown, indicating they have been cooked.Behind the stacks of plates, there is a man wearing a white t-shirt and a cap. He is facing away from the camera and appears to be engaged in an activity related to pizza making, possibly checking on the pizzas in the oven or preparing to serve them. The man is standing in front of a large, dome-shaped brick oven, which is a common feature in traditional pizzerias. The oven has a rustic appearance with visible bricks and a curved arch.On the left side of the image, there is a framed poster on the wall with the text "Napoletona" and an illustration of a person in a chef's hat. The wall behind the oven is tiled in white, and there are various kitchen items and containers visible on the counter and shelves, suggesting an active and functional kitchen environment.The lighting in the photograph is bright, with natural light coming from the left side, illuminating the scene and creating a warm ambiance. The focus of the image is on the pizzas and the man, with the background slightly blurred, which draws attention to the main subjects. The style of the photograph is candid and seems to capture a moment in the daily operations of the pizzeria.Wrong Elements (Hallucinations)Correct ElementsAdditional Correct ElementsWhat is this photo about?In the heart of a cozy living room, a beige couch and a matching armchair form a comfortable seating area. The couch, adorned with a blue and white patterned throw pillow, is positioned against the wall, while the armchair sits invitingly in the foreground. A black and white dog, the room's fourth occupant, is lying on the floor next to the armchair, adding a touch of warmth to the scene.A coffee table sits in front of the couch, hosting a blue plate with a slice of cake and a cup of coffee, perhaps indicating a relaxed afternoon tea time. The walls of the room are painted in a soothing light blue color, providing a calm backdrop for the scene.Two paintings hang on the wall, adding an artistic touch to the room. A lamp stands on a side table next to the couch, ready to cast a soft glow when needed. The room is further illuminated by natural lightstreaming in through a windowwithwhite curtains.The curtains are drawn back, suggesting it might be daytime outside.The room exudes a sense of tranquility and comfort, making it an ideal place for relaxation and leisure.LLaVA-OneVision-7BFEEDQUILL-7B (This work)The image is a color photograph depicting an indoor scene in a living room. The room is dimly lit, with the primary light source coming from a lamp on a side table to the left. The walls are painted in a light gray tone, and there are two framed paintings hanging on the wall. The larger painting is positioned above the couch and features a landscape scene with buildings and a body of water. The smaller painting is hung above a side table and depicts a figure in alandscape setting.In the foreground, there is a beige sofa with a person sitting on it, facing away from the camera. This person appears to be engaged in an activity, possibly using a device or reading. On the sofa, there is a patterned throw pillow. In front of the sofa, there is a woodencoffee table with a blue tray on top, which holds a few items, including what appears to be a plate with food.To the right of the sofa, there is a woman sitting on a wooden chair with a dark finish. She is wearing a black top and dark pants, and she is looking towards the camera. Her posture suggests she is in a relaxed position.On the floor, there are two dogs. One dog is lying down near the woman, and the other is closer to the camera, also lying down. Both dogs appear calm and are looking in different directions.In the background, there is a set of French doors with multiple panes of glass, allowing some light to enter the room. The doors are closed, and the glass reflects the interior lighting.The style of the photograph is candid and appears to capture a casual, everyday moment in a home setting. The composition of theimage is balanced, with the furniture and people arranged in a way that creates a sense of space and comfort.Wrong Elements (Hallucinations)Correct ElementsAdditional Correct ElementsUnder review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Figure 7: Qualitative results of FEEDQUILL-7B compared with GPT4o (OpenAI., 2024a) in terms of
image captioning.

measures the richness of the description of the image. We sum the two RM outputs to a final reward:
r = cp + αrcr. The hyper-parameter αr controls the trade-off between accuracy and richness, we set
it to 0.5 in our experiments. We set temperature to 1.0 and top-P to 0.7 when sampling trajectories
for the diversity of responses. The PPO training data is entirely composed of captioning task data,
containing 100k images. Other PPO hyper-parameters are presented in Table 10.

A.2.2 EVALUATION METRICS AND BENCHMARKS

• MMBench (Liu et al., 2023) introduces a diversity of evaluation questions, and use circular
evaluation protocol for multiple choices that leverage GPT to transform free-form answer into
the choice.

• MMStar (Chen et al., 2024a) is a vision-critical multi-modal benchmark with 1,500 human-
curated challenge samples designed to evaluate 6 core capabilities and 18 detailed axes of VLMs.
It is enhanced by strict human review to ensure visual dependency.

19

What do you see happening in this image?The image features a street scene on a relatively busy urban sidewalk. Dominating the foreground is a bright yellow fire hydrantwith a blue top and nozzle, firmly planted on a concrete sidewalk. Surrounding the base of the hydrant is a patch of grass and a few scattered weeds, indicating it is usually exposed to the elements.In the background, slightly to the left of the hydrant, there is a public phone booth with the recognizable Bell logo at the top. The booth has a clear glass exterior with a blueframe. A man in a light colored jacket is engaged in a call, visible through the glass. To the right of the booth, a silversedan is parked on the road, slightly angled and positioned just behind the phone booth. Further down the sidewalk, you can see more pedestrians going about their daily lives.The street itself is lined with tall, modern buildings, hinting at a downtown area. Some of the buildings have large glass windows, while others are adorned with metal and concrete facades. The weather appears overcast with a cloudy sky, which casts a mutedlight over the entire scene. Trees with autumnal leaves line the street, their golden and brown foliage providing a contrast to the grey and metallic urban structures.Overhead, the wires and poles suggest a well-developed neighborhood infrastructure. The overall atmosphere of the image is busy but routine, capturing a typical moment in an urban setting where people go about their daily activities amidst the steady flow of city life.GPT4oFEEDQUILL-7B (This work)The image is a color photograph depicting an urban street scene. In the foreground, there is a fire hydrant painted in brightyellow with a blue top and a blue side cap. The hydrant is situated on a concrete sidewalk and appears to be slightly tilted to the right. There is some vegetation growing around the base of the hydrant, including small shrubs and grass.Behind the fire hydrant, there is a public telephone booth with a dark green exteriorand a bluesign on top that reads "Bell" in white letters. The booth has a glass door and a glass panel on the side, through which the interior is partially visible. There are people walking on the sidewalk in the background, and they appear to be going about their daily activities.Further back, there is a street with vehicles, including a black car parked on the side of the road. The buildings in the background are modern, with reflective glass windows and concrete facades. The sky is overcast, suggesting a cloudy day.The image has a realistic style, capturing the everyday elements of city life with a focus on the fire hydrant as the main subject. The composition of the photograph places the fire hydrant in the center, drawing the viewer's attention to it, while the background elements provide context and a sense of place.Wrong Elements (Hallucinations)Correct ElementsAdditional Correct ElementsAnalyze the image in a comprehensive and detailed manner.The image captures a dynamic moment during a tennis match on an outdoor tennis court. The central figure is a tennis player dressed in a white shirt and white shorts. He is holding a tennis racket in his right hand, preparing to hit a yellow tennis ball that is visible in the foreground to the left. His left arm is extended outwards for balance, and he is moving forward, exhibiting an intense focus on the ball.The tennis court is blue, with visible white lines marking the playing area. Behind the player, there is a dark blue wall that features several pieces of text and logos in white. On the left side, the text reads, ”KSWelcomCenter.com," while the right side bears the name "Western & Southern Financial Group Masters" and "CINCINNATI 2004."In the background, several spectators are seated in what appears to be a stadium setting, watching the match intently. The weather seems to be sunny and clear, indicated by the bright lighting on the court and the spectators wearing casual summer attire like hats and sunglasses.Overall, the image exudes an energetic and competitive atmosphere typical of a professional tennis match.GPT4oFEEDQUILL-7B (This work)The image is a color photograph featuring a male tennis player in action. The player is dressed in white athletic attire, including a headband, shirt, shorts, and wristbands. He is holding a tennis racket in his right hand and appears to be in the process of hitting a tennis ball, which is visible in the air to the left of the player. His body is turned sideways to the camera, and his left arm is extended behind him, suggesting a follow-through motion.The player‘s facial expression is focused, and he is looking intently at the ball. His hair is short, and he has a light complexion. The background of the image includes a blue wall with text and logos. The text includes “WelcomeCenter.com,” “Western & Southern Financial Group Masters," and " CINCINNATI 2004," indicating the event and location where the photo was taken. There are also spectators visible in the background, suggesting that the photograph was taken during a tennis match.The style of the image is dynamic, capturing a moment of athletic action. The focus is on the player and his interaction with the ball, with the background elements providing context for the event. The colors in the image are vibrant, with the blue of the wall contrasting with the white of the player's outfit and the yellow of the tennis ball.Wrong Elements (Hallucinations)Correct ElementsAdditional Correct ElementsUnder review as a conference paper at ICLR 2025

Model
Qwen-VL-Chat-7B (Bai et al., 2023)
mPLUG-Owl2 (Ye et al., 2024)
LLaVA-1.5-7B (Liu et al., 2024b)
LLaVA-1.5-13B (Liu et al., 2024b)
XComposer2.5-7B (Zhang et al., 2024)
Cambrian-34B (Tong et al., 2024)
LLaVA-1.6-7B (Liu et al., 2024a)
MiniCPM-Llama3-V-2.5-8B (Yao et al., 2024)
LLaVA-1.6-13B (Liu et al., 2024a)
ViLA-40B (Lin et al., 2024)
InternVL-1.5-20B (Chen et al., 2024b)
LLaVA-1.6-34B (Liu et al., 2024a)
LLaVA-Onevision-7B (Li et al., 2024a)
Gemini-Pro-1.5 (Team et al., 2023)
InternVL-2-8B (Chen et al., 2024b)
GPT-4v (OpenAI., 2024b)
InternVL-2-26B (Chen et al., 2024b)
GLM-4v-9B (GLM et al., 2024)
InternVL-2-40B (Chen et al., 2024b)
Claude-3.5-Sonnet (Anthropic., 2024)
GPT-4o (OpenAI., 2024a)

FEEDQUILL-7B

Language Model DCSCORE F

Qwen-7B
LLaMA-2-7B
Vicuna-v1.5-7B
Vicuna-v1.5-13B
InternLM2.5-7B
Yi-34B
Vicuna-v1.5-7B
LLaMA-3-8B
Vicuna-v1.5-13B
Yi-34B
InternLM2-20B
Yi-34B
Qwen2-7B
-
InternLM2.5-7B
-
InternLM2.5-20B
GLM-4-9B
Yi-34B
-
-

Qwen2-7B

19.16
23.27
24.50
25.55
29.60
35.12
36.21
36.36
37.98
38.02
39.28
40.46
43.49
46.34
47.39
48.52
49.59
49.85
51.17
52.37
53.44

55.65

Table 9: The performance of various VLMs on DECAPBENCH.

Hyper-parameter
Optimizer
Learning Rate
Scheduler
Batch Size
β (KL Penalty Coefficient)
γ (discount factor)
λ (TD trade-off factor)
Number of Mini-batches
ϵ (Policy Clipping Coefficient)
ϵv (Value Clipping Coefficient)

Default Value
AdamW (ϵ = 1e − 8)
1e-6 (actor), 5e-6 (critic)
Linear
256
0.05
1.0
0.95
1
0.2
0.2

Table 10: PPO hyper-parameters

• TextVQA (Singh et al., 2019) measures the capability of VLMs for answering question about

the text in the natural images.

• VizWiz (Gurari et al., 2018) comes from a natural visual question answering dataset for blinding

people.

• ScienceQA (Lu et al., 2022) consists of approximate 21K multi-modal multiple choice questions
with a diverse set of science topics and annotations of their answers with corresponding lectures
and explanations.

• mmHal-V (Amirloo et al., 2024) is a visual hallucination evaluation benchmarks for VLMs,
which consists object attribute, adversarial object, comparison, counting, spatial relation, envi-
ronment, holistic description, and other types.

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

• LLaVA-W (Liu et al., 2024b) aims to evaluate the model’s capability in visual chatting, which
including memes, indoor and outdoor scenes, painting, sketches, etc. Each each image is
associated with a highly-detailed and manually-curated description and a proper selection of
questions, and utilize GPT to score the model’s response.

• WildVision (Lu et al., 2024) simulates the arena and evaluate the model with various real-world

questions, while benchmarking human preference.

• CHAIRS and CHAIRI (Chan et al., 2023) a widely-recognized tool for evaluating the incidence
of object hallucination in image captioning tasks which assess object hallucination at the
instance-level and sentence-level respectively.

• MME (Fu et al., 2023) is a comprehensive benchmark for evaluating the capabilities of VLMs in
multi-modal tasks. It systematically assesses models across two primary dimensions: perception
and cognition, through 14 meticulously designed subtasks that challenge the models’ interpretive
and analytical skills.

• SeedBench (Li et al., 2024b) consists of 19K multiple choice questions with accurate human
annotations, and it spans 12 evaluation dimensions including the comprehension of both the
image and video modality.

• MMMU (Yue et al., 2024) includes 11.5K meticulously collected multi-modal questions from
college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business,
Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.

A.2.3 PREFERENCE OPTIMIZATION

The following algorithm demonstrates how to leverage PPO (Schulman et al., 2017) to optimize the
base model (SFT Model) with reward models trained with preference data D for cp and preference
data Dr for cr.

Algorithm 1 Preference Optimization with FEEDQUILL
Input initial policy model Pθinit; initial value model Vψinit ; reward models Rϕp/r trained from cp or
cr; PPO training prompts Dt; PPO hyperparameters γ, λ, ϵ, β.
1: policy model Pθ ← Pθinit, value model Vψ ← Vψinit
2: for step = 1, . . . , T do
3:
4:

Sample a batch B from Dt
Sample output sequence yn ∼ Pθ(· | xn) for each prompt xn ∈ B
Compute rewards {rn
+ rn
pt
rt
Compute advantages {At}|yn|
for PPO iteration = 1, . . . , µ do

}|yn|
t=1 from the reward model Rϕp and Rϕr for each yn.
t=1 and value targets {V est(st)}|yn|
t=1 for each yn with Vψ.

Update the policy model by maximizing the PPO clipped surrogate objective:

θ ← arg max

θ

1
|B|

|B|
(cid:88)

n=1

1
|yn|

|yn|
(cid:88)

t=1

min

(cid:18) Pθ(at | st)
Pθold(at | st)

At, clip(vt, 1 − ε, 1 + ε)At

(cid:19)

Update the value model by minimizing a L2 objective:

ψ ← arg min

ψ

1
|B|

|B|
(cid:88)

n=1

1
|yn|

|yn|
(cid:88)

t=1

(cid:0)Vψ(st) − V est(st)(cid:1)2

5:

6:
7:
8:

9:

end for

10:
11: end for
Output Pθ

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

A.2.4 EVALUATION PROMPT FOR DCSCORE

To measure the quality of the generated captions, we present prompts for decomposition in Table 11,
matching in Table 12, and verification in Table 13. We utilize GPT-4o (OpenAI., 2024a) through the
whole evaluation process.

You are a linguistic expert in extracting primitive information units in the image caption. In specific,
"primitive information units" refer to the smallest standalone pieces of information that collectively
represent the entire meaning of the sentence without losing any detail, which typically describe
various properties of the visual elements in an image. The primitive information unit should be a
simple statement. The fact must represent the smallest piece of information that cannot be further
broken down without loss of meaning. Abstract concepts or broad interpretations should be reduced
to more basic, constituent observations if possible. The primitive information unit should only
contain ONE primary element.

When extracting primitive information units from image caption, it is useful to assign unique
identifiers to the primary objects or entities being discussed. This will help in maintaining clarity and
preventing confusion, especially when there are multiple similar objects or entities. For example,
if the caption mentions two cats, you can assign unique identifiers such as "cat1" and "cat2" to
distinguish them. Besides, for each attribute, you should also assign the identifier to the object it
belongs to. Meanwhile, for spatial relationships, you can assign the identifier to the object that is the
subject of the relationship in the primitive information unit.

For each primitive information unit, you should also need to justify whether the primitive information
unit directly describe the image or not.

**IMPORTANT**: Please extract ALL of the primitive information units in the image caption. DO
NOT omit any information!

The output should be a list of dict [{"fact": [PRIMITIVE INFORMATION UNIT], "identifier":
[UNIQUE ID], "relevance": 1/0}, ...] into JSON format. The "identifier" would be optional, if the
item in the fact has already been identified with ids. The "relevance" would be 1 if the primitive
information unit directly describe the content of the image. Otherwise it would be 0 if the primitive
information unit is inference or extension to the description and not directly describe to the content of
image.
> > > Caption: {Caption Here}

Table 11: The prompt for decomposing the generated captions into set of primitive information
units.

A.2.5 TRAINING PROMPT FOR PPO

We prompt GPT-4o (OpenAI., 2024a) to generate a series of image captioning prompts for PPO
training, as listed in Table 14.

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

22

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

You are now a visual-linguistic expert in matching two set of primitive information units generated
from two captions.

You will be received a set of predicted primitive information units across a variety of categories and a
set of oracle primitive information units (ground truth). The set of primitive information units is
represented as a list of dict [{"fact": [PRIMITIVE INFORMATION UNIT], "identifier": [UNIQUE
ID]}, ...] within JSON format. In addition, each primitive information unit in the oracle set would be
accompanied with a unique "id" to identify the oracle primitive information unit.

To match primitive information units from a predicted set in terms of the given image with oracle set
of primitive information units. Here is the step by step instruction:
1. Preliminary Review: Conduct an initial review of both sets of primitive information units,
considering all primitive information units. Understand the details and context presented within each
primitive information unit.
2. Inferring Identifier Mappings: Closely examine both sets to deduce potential correlations and
mappings based on the content of the primitive information units. Determine if there are any unique
identifiers or descriptors that hint at matching entities between the sets. For example, "cat0" in the
predicted set’s primitive information units may be mapped to "cat1" in the oracle set’s primitive
information units. Consider the attribute and spatial relation in both sets for possible mapping. Please
note that there might be some attribute and spatial errors when mapping the objects. Try find the most
similar mapping if exists (not need exact matching). If no oracle primitive information unit matches,
simply set matched oracle id to "None".

**IMPORTANT**: Please consider each primitive information unit in the set individually, and
MUST NOT omit any primitive information units from the predicted set.

You should only output the matching results which will be formatted as a list of dict as [{"fact":
[PRIMITIVE INFORMATION UNIT], "identifier": [UNIQUE ID], "matched_oracle_id": [CORRE-
SPONDING ORACLE ID]}, ...] in JSON format. The "identifier" would be optional, if the item in
the fact has already been identified with ids as illustrated in the predicted primitive information units.
For key named "matched_oracle_id", the value of "matched_oracle_id" should be the corresponding
"id" of the primitive information unit in the oracle set. For the primitive information unit in the
predicted set which cannot be matched with any oracle primitive information unit, set the value of
"matched_oracle_id" to "None".

> > > Set of Primitive information units: {set of units for generated caption}

> > > Oracle Set of Primitive information units: {set of units for human-written caption}

> > > Matching Result:

Table 12: The prompt for verifying the correctness of each primitive information units by utilizing
both image and human-written caption.

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

You are an extraordinary visual-linguistic expert in verifying the correctness of a set of primitive
information units given the image and the corresponding reference caption. The set of primitive
information units are extracted from a paragraph of machine-generated image caption of that image.

The set of primitive information units is represented as a list of dict ["fact":
[PRIMITIVE
INFORMATION UNIT], "identifier": [UNIQUE ID], ...] within JSON format. The identifier is
unique and to identify the primary objects or entities being discussed. This will help in maintaining
clarity and preventing confusion, especially when there are multiple similar objects or entities. For
example, if the caption mentions two cats, we would assign unique identifiers such as "cat1" and
"cat2" to distinguish them. Besides, for each attribute, it also assigned the identifier to the object it
belongs to. Meanwhile, for spatial relationships, it assigned the identifier to the object that is the
subject of the relationship in the primitive information unit.

You should first go through all of the primitive information units, and understand the details and
context presented within each primitive information unit. Then you need to verify the correctness
of each individual primitive information units by asking yourself: Statement: "[PRIMITIVE
INFORMATION UNIT]" Does the statement correct according to image or reference caption?

The output for the predicted set of primitive information units should be formatted as a list of dict as
["fact": [PRIMITIVE INFORMATION UNIT], "identifier": [UNIQUE ID], "verification": 1/0, ...] in
JSON format, where 1 represents the fact is correct and 0 represents the fact is incorrect. Other keys
in the dictionary are the same as the input. The "identifier" would be optional, if the item in the fact
has already been identified with ids as illustrated in the input.

> > > Reference Caption: {reference caption}

> > > Primitive Information Units: {primitive information units}

Table 13: The prompt for verifying the correctness of each primitive information units by utilizing
both image and human-written caption.

24

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

• What do you see happening in this image?

• Can you describe what is happening in this picture?

• What events are taking place in this image?

• What do you observe in this photo?

• Can you explain the scene depicted in this image?

• What is this photo about?

• What is the subject of this picture?

• Can you explain the theme of this image?

• What is the focus of this photo?

• What is the central topic of this picture?

• What is the main idea of this image?

• What is the essence of this photo?

• What is the core subject of this picture?

• What is the primary focus of this image?

• What is the overall theme of this photo?

• What is the main topic depicted in this picture?

• Can you elaborate on the elements of the picture provided?

• Can you give more details about the components of this image?

• What are the various elements in this picture?

• Can you describe the different parts of this photo?

• What are the individual components of this image?

• Can you break down the elements of this picture?

• What are the distinct features of this photo?

• Can you provide more information on the elements in this image?

• What are the specific parts of this picture?

• Can you detail the elements present in this photo?

• are the various aspects of this image?

• Analyze the image in a comprehensive and detailed manner.

• Provide a thorough analysis of this picture.

• Can you give an in-depth examination of this image?

• What is your detailed analysis of this photo?

• Can you break down this image comprehensively?

• What is your extensive analysis of this picture?

Table 14: Part of example prompts for preference optimization.

25

