LIME: LESS IS MORE FOR MLLM EVALUATION
Under review as a conference paper at ICLR 2025

LIME: LESS IS MORE FOR MLLM EVALUATION

Anonymous authors
Paper under double-blind review

ABSTRACT

Multimodal Large Language Models (MLLMs) are measured on numerous bench-
marks like image captioning, visual question answer, and reasoning. However,
these benchmarks often include overly simple or uninformative samples, making it
difficult to effectively distinguish the performance of different MLLMs. Addition-
ally, evaluating models across many benchmarks creates a significant computational
burden. To address these issues, we propose LIME (Less Is More for MLLM Eval-
uation), a refined and efficient benchmark curated using a semi-automated pipeline.
This pipeline filters out uninformative samples and eliminates answer leakage by
focusing on tasks that require image-based understanding. Our experiments show
that LIME reduces the number of samples by 76% and evaluation time by 77%,
while it can more effectively distinguish different models’ abilities. Notably, we
find that traditional automatic metrics like CIDEr are insufficient for evaluating
MLLMs’ captioning performance, and excluding the caption task score yields a
more accurate reflection of overall model performance. All code and data are
available at https://anonymous.4open.science/r/LIME-49CD.

1

INTRODUCTION

In order to better understand the model’s capabilities and guide addressing the shortcomings of
MLLMs, researchers develop numerous benchmarks for various tasks (Antol et al., 2015; Wei et al.,
2023; Fu et al., 2023; Yue et al., 2024; Wu et al., 2024a). These benchmarks thoroughly explore the
capabilities of MLLMs in various tasks such as image captioning, image question answering, and
multimodal retrieving.

However, existing MLLM benchmarks and unified evaluation frameworks cannot effectively and
efficiently reflect the ability of MLLMs. Current benchmarks include numerous relatively simple
samples (i.e., how many chairs are in the picture) and some incorrect questions caused by annotation
issues. Most MLLMs consistently perform on these samples (i.e., all correct or all wrong). Therefore,
those benchmarks cannot fully reflect the gap between different MLLMs and across various tasks.
Besides, the current unified multimodal evaluation frameworks require significant computational
resources, necessitating integrating much evaluation data from various benchmarks. The selection of
effective evaluation data is largely overlooked by current researchers.

As shown in Figure 1, to address the aforementioned issues, we propose to use a general data process
pipeline and curate a LIME, which contains 9403 samples and is refined across 10 tasks within 6
domains. We select six major tasks in the multimodal domain and use 9 MLLMs to refine those 10
benchmarks within the corresponding domain. To eliminate bias introduced by individual models,
we choose 9 models as judges and filter samples based on their performance. On the one hand, we
remove samples that most models answer correctly due to the fact that they cannot distinguish the
capabilities among different models. On the other hand, we use a method that combines humans
and MLLMs to filter out some abnormally difficult samples. Meanwhile, we use LLMs to filter out
samples that can be answered directly from the context of the question. After that, we obtain a smaller
yet higher-quality unified bench (i.e., LIME).

We conduct various experiments on LIME using both MLLMs and LLMs on different input settings,
such as QA + image inputs, QA input (text-only input), and the QA + image description experiment.
We make several valuable findings:

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

• LIME can better reflect the performance differences of MLLMs. On our LIME benchmark,
under consistent conditions (same model series, same model size), different MLLMs demon-
strate a wider score range, indicating that LIME is more effective at reflecting performance
differences between models with a smaller amount of data.

• MLLMs exhibit varying capabilities across different subtasks. Specifically, they excel in the
Visual Question Answering (VQA) subtasks, showcasing relatively high performance when
answering questions directly related to factual information depicted in images. However,
their performance is comparatively lower in tasks that necessitate the application of addi-
tional commonsense knowledge or complex reasoning. This highlights the significant image
content recognition capabilities of current MLLMs.

• Through the correlation analysis of scores across different tasks, we find that using traditional
automatic metrics for the captioning task makes it difficult to reasonably evaluate the model’s
performance. Different tasks have varying requirements for factual perception and the
application of additional commonsense knowledge in images.

2 METHOD

Most benchmarks contain low-quality, noisy data. Figure 2 shows the statistics of different
subtasks within our LIME benchmark. It is worth mentioning that the proportion of easy and wrong
samples exceeds 30Out of the 10 subtasks, 6 have proportions exceeding 50%. Notably, for the
POPE dataset, 95% of the data can be classified as noisy or erroneous. This indicates that existing
benchmarks are filled with a large amount of low-quality data, which does not accurately reflect the
true capabilities of MLLMs.

Inspired by MMStar (Chen et al., 2024a), we utilize open-source MLLMs and LLMs as the judges for
filtering, specifically, we remove the existing annotation errors. The overall pipeline consists of three
main stages: (1) Using open-source models as judges, (2) A semi-automated screening process, and
(3) Eliminating answer leakage. Our approach aims to improve existing benchmarks by removing
inaccurate and oversimplified data.

Figure 1: Pipeline of the Data Curation. The left half part is the Open-Source Models as Judges
module, which uses several Multimodal LLMs to answer questions for each sample and assess their
difficulty. The upper right part is the Semi-Automated Screening Process module filtering some
samples that are too simple or difficult. As for the Eliminating Answer Leakage, we filter the sample
that can be answered without the image.

2

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

s…BenchmarksLLaVA…XcomposerInstructBLIPOpen-Socurce Models as JudgesMLLMsSample 1Sample 2Sample n-1…………EasyMiddleHardClassified SamplesRemoving Bad CaseGPT4VHumanWhat is the capital of Massachusetts?Image:Question:What is the shape of the Province in the map?Text OnlyDon’t knowYesLLMsResponse:Eliminating Answer LeakageEasy SetSemi-Automated Screening ProcessSample n…ZeroPassBad CaseCategoryData withLIME Filtering Easy Sample…Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 2: Overall data statics about selected subtasks. Easy: questions that most models can answer
correctly, Bad Case: questions that may contain errors, Remained: questions that finally remain.

2.1 OPEN-SOURCE MODELS AS JUDGES

To avoid potential biases that may exist in individual MLLMs, we select ten different types of open-
source models as judges. To categorize the difficulty of each sample, we analyze the performance of
all judge models on each question and label the difficulty based on the number of models that answer
correctly. We define N as the number of models that correctly answer the sample. If N ≥ 6, the
question is classified as the easy set. If 3 ≤ N ≤ 5, it is classified as the middle set. Conversely, if
N ≤ 2, it is classified as the hard set.

2.2 SEMI-AUTOMATED SCREENING PROCESS

Easy samples do not effectively differentiate the capabilities of various models, as most models can
answer them correctly. Therefore, we remove the easy samples to assess model performance better.

Furthermore, we find that some questions are not correctly answered by any model, which can be
due to potential errors in the question design. To mitigate these potential errors and filter out totally
incorrect questions, we implement a semi-automated screening process, which consists of two stages.
In the first stage, all questions with zero passes are reviewed by GPT-4V to assess their correctness
in terms of logic and meaning. In the second stage, questions deemed correct by GPT-4V are then
manually screened. This strategy helps us eliminate meaningless or erroneous data from the dataset,
thereby reducing its size and improving its quality.

2.3 ELIMINATING ANSWER LEAKAGE

Although the previous two stages have filtered out potential errors and assessed the quality of the
questions, we still need to address the potential issue of ANSWER LEAKAGE. Multimodal Answer
Leakage can be summarized into two main categories: 1.Text Answerable Questions: The textual
information contains all the necessary details to answer the question, making the corresponding visual
information redundant. 2.Seen Questions: The MLLMs have encountered a specific question during
training and has memorized the question along with its corresponding ground truth.

As for the Seen Questions, it has been removed in the Filtering Easy Sample module in Sec. 2.2.
Therefore, we conduct a text-only check using pure text LLMs to eliminate the ANSWER LEAK-
AGE. Specifically, based on LLMs’ responses, we remove the samples that can be directly answered
without using the image. After that, we proportionally sample 1,200 samples from these categories
based on their difficulty levels. For benchmarks with fewer than 1,200 entries, we adapt all samples.

3

COCO-CaptionTextCapsPOPEOK-VQATextVQAInfoVQAChartQAAI2DScienceQAOCRBenchEasyBad CaseRemained30.7%5.0%64.3%21.2%9.3%69.5%93.8%4.9%1.3%53.2%10.9%36.0%50.3%11.9%37.8%27.5%8.4%64.2%28.7%7.6%63.7%71.5%4.0%24.5%59.7%2.7%37.7%44.3%9.7%46.0%Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

3 LIME: A COMPREHENSIVE MLLMS BENCHMARK

In this section, we propose LIME, a comprehen-
sive benchmark for Multimodal Large Language
Models (MLLMs). LIME streamlines existing
mainstream benchmarks. Tab 1 shows the main
datasets included in our benchmark, as well as
the data scale after careful pruning. For each
sub-dataset, we aim to keep the size around 1k
samples.

3.1 TASK DEFINITION

We have categorized the existing mainstream
tasks into six domains: Captioning, T/F Reason-
ing, Normal VQA, Infographic Understanding
QA, Science QA, and OCR. Below are the task
definitions for each domain

Table 1: Data statics: Full Size: the size of the
original dataset, Lite Size: the final size of the
LIME. For the COCO-Caption dataset, we selected
the 2017 subset, and for the ScienceQA dataset,
we chose the ScienceQA-IMG subset.

Task Domain

Dataset

Split Full Size Lite Size

Captioning

T/F reasoning

Normal VQA

Infographic QA

Science QA

OCR

TextCaps
COCO-Caption

POPE

OK-VQA
TextVQA

infoVQA
ChartQA

ScienceQA
AI2D

OCRBench

val
val

val

val
val

val
val

val
val

val

3166
5000

9000

5046
5000

2801
2500

2097
3088

1000

1200
1200

443

1200
1200

1200
1200

300
1000

460

Image understanding and captioning: The Captioning task focuses on the fundamental image-
text understanding ability, requiring MLLMs to accurately describe and understand the content of
images. This ability is commonly learned by most multimodal models during the pre-training stage.
For example, the CLIP model aligns image and text features through contrastive learning, making
Captioning a measure of the basic capabilities of MLLMs.

T/F reasoning: T/F Reasoning requires the model to judge the truthfulness of textual statements
based on the image content. This not only demands basic image understanding from the MLLMs but
also requires a certain level of reasoning ability.

Normal VQA: Normal VQA, or Visual Question Answering, comprehensively evaluates the model’s
ability to answer questions based on visual input. MLLMs are required to select the most appropriate
answer from specific options.

Infographic Understanding QA: This task differs from Normal VQA as it tests the model’s ability
to retrieve details from images. MLLMs need to find the most relevant information in the image
related to the question and then provide a reasoned answer.

Science QA: Science QA includes questions and answers related to natural science knowledge. This
requires the model to have domain-specific knowledge in natural sciences, mainly assessing the
MLLMs’ mastery of knowledge within a specific domain.

OCR: The OCR task requires the precise extraction of textual content from images.

3.2 DATA STATISTICS

LIME is composed of 10 open-source multimodal evaluation benchmarks, with scales ranging from
1,000 to 9,000. After our three-stage data curation, the data scale of each benchmark is significantly
reduced. Figure 1 shows the number of samples removed at each stage compared to the original
dataset. The amount of data removed varies at each stage, with the most being removed in the first
stage, reflecting a large number of low-difficulty or data-leakage samples in the existing 9 MLLMs.
Comparing the data volumes before and after the second stage of semi-automated screening, we can
see that many datasets, such as OK-VQA and TextVQA, have a high rate of low-quality data leading
to MLLMs’ incorrect answers. Additionally, some datasets, such as ScienceQA and AI2D, have
a significant amount of data removed after the third stage, indicating that many questions in these
datasets may contain potential answer leakage. The statistics of the curated data are shown in Tab 1.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 3: The number of samples removed at each stage compared to the original data, including
three stages of filtering and the final sampling stage.

4 EXPERIMENT

4.1 EXPERIMENT SETTING

To evaluate the quality of LIME, we conduct a series of experiments across various open-source and
closed-source models. These experiments primarily encompass the following three settings:

Main experiment: To demonstrate the performance of LIME, we evaluate mainstream open-source
and closed-source models using a standardized process to reflect their overall performance differences.

Text-only set: To prevent potential data leakage issues, we conduct validation experiments using
text-only QA pairs. This verifies whether LLMs can correctly answer questions based on text-only
information.

Text-only question with Image Description set: Image Description (ID) refers to simple descriptions
of images that represent superficial information contained within them. For most MLLMs, questions
containing only superficial information are easy to answer; however, questions requiring complex
visual inference are significantly more challenging. To further validate whether LIME can reflect
the capabilities of MLLMs, we input text-only QA pairs combined with ID into LLMs and test their
ability.

4.2 BASELINES

We select LLaVA-1.5 (Liu et al., 2023a;b), LLaVA-1.6 (Liu et al., 2024), Tinny-LLaVA (Zhou et al.,
2024), MiniCPM (Hu et al., 2024), Idefics-2 1, Deepseek-VL(Lu et al., 2024), CogVLM (Wang
et al., 2023; Hong et al., 2023), XComposer-4KHD (Zhang et al., 2023), Mantis (Jiang et al., 2024),
InternVL-1.5 and InternVL-2 (Chen et al., 2023; 2024b) as our MLLMs baseline, and LLaMA3,
Yi, Yi-1.5 (AI et al., 2024), Qwen-1.5 (Bai et al., 2023a) and Qwen2 (Yang et al., 2024) as LLMs
baseline. To ensure fairness in the evaluations, we use the unified evaluation framework provided
by lmms-eval (Zhang et al., 2024b) to conduct evaluation experiments on LIME. For models not
supported by lmms-eval, we refine the inference code provided by the model developers to make it
compatible with the new models for the sake of aligning the results of different models.

Metrics For most tasks included in LIME, we reference the metrics computation methods used in
lmms-eval. Specifically, for tasks such as AI2D, ScienceQA, OCRBench, and POPE, we calculate
the accuracy of the extracted responses. For tasks such as OK-VQA and TextVQA, we calculate the
metric scores based on the overlap between the response and the candidate answers. For tasks like
TextCaps and COCO-Caption2017, we use CIDEr as the score. The ANLS metric is used for the
infoVQA task, and the Relaxed Overall metric is used for the ChartQA task.

We calculate the sub-scores for each task category by taking a weighted average of the subtask scores,
and then compute the overall score by weighted averaging the scores of all tasks except for the caption
tasks. The details of the metrics calculation are provided in Tab 7.

1https://huggingface.co/blog/idefics2

5

COCO-CaptionTextCapsPOPEOK-VQATextVQAInfoVQAChartQAScienceQAAI2DOCRBench02000400060008000SizeOrigin sizeAfter stage1(LMMs filter)After stage2(gpt filter)After stage3(text only filter)After sampleUnder review as a conference paper at ICLR 2025

5 RESULTS

5.1 MAIN RESULT

Table 2: Left half of the table:Comparing overall scores of LIME and Original. The arrow next to
the LIME score indicates the change in ranking on LIME compared to the original dataset. ↑: upward
shift, ↓: downward shift, and -: no change. Right half of the table: performance on six domains

Model

Size

LIME

Original Reasoning VQA InfoQA SciQA OCR Caption

GPT-4O
claude-3-5-sonnet
Gemini-1.5-Pro-Vision
GPT-4-Vision-Preview

InternVL-2 2023
Qwen2-VL 2023b
InternVL-1.5 2024b
InternVL-2 2023
InternVL-2 2023
LLaVA-OneVision 2024
XComposer2-4KHD 2023
InternVL-2 2023
CogVLM-2 2024
Qwen2-VL 2023b
InternVL-2 2023
CogVLM-1 2023
Cambrian 2024
Cambrian 2024
InternVL-2 2023
Cambrian 2024
LLaVA-1.6 2024
MiniCPM-LLaMA3-2.5 2024
LLaVA-OneVision 2024
LLaVA-LLaMA3 2023
Mantis-Idefics-2 2024
Deepseek-VL 2024
LLaVA-1.6-vicuna 2024
Idefics-2 2024
LLaVA-1.6-vicuna 2024
Mantis-SigLIP 2024
MiniCPM 2024
LLaVA-1.5 2023a
LLaVA-1.5 2023a
InstructBLIP-vicuna 2023
Tiny-LLaVA-1 2024

-
-
-
-

40B
7B
26B
26B
8B
7B
7B
4B
19B
2B
2B
17B
34B
13B
1B
8B
34B
8B
0.5B
8B
8B
7B
13B
8B
7B
8B
1.0
13B
7B
7B
1.4B

52.63
51.99
49.46
42.23

66.85 ( - )
65.28 (↑ 1)
64.12 (↓ 1)
63.98 ( - )
62.00 ( ↑ 1 )
61.95 ( ↓ 1 )
57.52 (↑ 4)
57.22 (↓ 1)
54.44 (↑ 6)
54.00 (↑ 5)
53.64 (↓ 2)
51.03 (↑ 1)
50.17 (↓ 5)
48.57 (↓ 4)
48.21 (↑ 3)
47.95 (↓ 4)
44.06 (↑ 3)
42.61 (↓ 3)
41.40 (↑ 4)
40.90 (↓ 3)
39.25 ( - )
38.10 (↑ 2)
37.08 (↓ 4)
36.39 (↓ 2)
30.15 ( - )
29.13 (↑ 1)
26.15 (↑ 2)
20.38 (↓ 2)
17.20 (↓ 1)
15.55 ( - )
13.95 ( - )

-
-
-
-

80.31
79.14
79.49
78.82
77.84
78.71
71.93
73.97
69.93
70.86
73.00
71.34
73.26
72.39
68.46
71.84
67.22
71.22
65.65
69.74
66.91
65.62
67.29
66.73
64.80
58.96
56.18
59.58
57.27
47.87
34.30

47.18
35.89
54.63
42.44

51.69
53.05
51.69
54.63
49.21
52.37
46.28
47.18
51.02
50.79
50.79
55.10
49.44
50.79
52.82
49.89
47.00
43.10
48.98
44.24
44.24
48.50
43.10
42.00
41.10
45.60
44.00
36.60
32.51
45.10
37.00

42.95
50.33
37.71
33.86

48.72
51.37
52.68
45.64
45.15
51.27
44.22
39.89
37.19
43.78
40.71
51.45
39.66
41.53
36.46
42.12
30.80
43.55
35.87
37.36
36.79
34.90
30.00
46.05
25.75
29.39
21.60
25.80
19.97
16.75
9.80

57.63
56.38
55.33
48.00

81.12
80.83
78.96
79.12
76.00
74.50
73.29
71.21
69.92
66.25
62.88
59.46
57.50
56.04
56.04
53.55
53.21
58.55
48.04
43.33
39.75
38.50
41.63
18.50
32.88
25.79
24.58
8.96
7.17
6.04
8.33

56.15
44.69
50.15
42.39

75.92
62.08
63.32
70.54
68.54
66.77
58.38
63.31
54.00
46.38
56.54
36.54
60.23
49.23
47.92
49.46
53.08
6.60
36.23
45.56
43.69
44.23
41.54
47.46
31.77
35.77
35.46
31.08
29.81
24.77
27.85

72.39
73.91
73.26
55.22

75.87
77.61
60.65
71.09
70.65
47.83
53.04
67.17
68.26
68.04
67.39
41.96
39.13
42.39
65.00
43.04
37.17
55.87
42.83
30.22
32.17
25.43
31.96
42.61
23.70
10.65
14.57
5.87
4.78
4.35
3.48

47.84
28.00
41.38
29.14

56.02
89.67
90.93
66.54
34.00
106.46
87.57
28.83
28.84
88.39
47.27
33.92
4.62
6.96
14.19
6.13
66.25
35.89
93.34
74.03
82.44
68.72
62.23
77.87
62.20
74.69
72.80
74.81
72.47
77.61
61.05

As shown in Tab 2, we evaluate both open-source and closed-source MLLMs using our LIME
benchmark. Overall, for closed-source models, GPT-4O achieves the best performance with a score
of 52%, while for open-source models, models with larger parameter sizes and newer versions tend to
have higher overall scores. InternVL-1.5, InternVL-2-Large (26B, 40B), and LLaVA-OneVision-7B
achieve the best overall performance, with their overall scores all surpassing 60%. The performance
of InternVL-2-Small (1B-8B), the CogVLM series, and the Cambrian series follows, with their
overall scores ranging from 45% to 60%.

Comparing the overall scores of LIME and Origin benchmarks, we observe that certain model
series, such as Cambrian and LLaVA-1.5, experience a decline in overall scores. Conversely, the
CogVLM and LLaVA-OneVision series show an improvement, with CogVLM2 and XComposer-
4KHD experiencing significant increases of 4% and 6%, respectively.

Tab 6 provides more detailed experimental results. Regarding caption subtasks, most models demon-
strate good performance. These tasks involve generating or assessing descriptions of the content in
images, which indicates that current MLLMs possess strong image content recognition capabilities.
As for the VQA task, current MLLMs perform relatively well on TextVQA, ChatQA, and ScienceQA,
where the questions directly ask about facts in the picture. However, their performance is relatively
lower on OK-VQA, infoVQA, and AI2D, which require additional commonsense knowledge or
complex reasoning to answer the questions. This demonstrates that current MLLMs exhibit significant
image content recognition capabilities but are limited in their ability to perform complex reasoning
using additional knowledge. We believe this limitation may be due to constraints in the language
model component of MLLMs.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Figure 4: Correlation distribution between LIME and Wildvison Elo.

5.2 CORRELATION ANALYSIS

Figure 4 illustrates the correlation between the various sub-tasks in LIME and WildVision Bench.
Most tasks in LIME exhibit a strong positive correlation with WildVision Bench. Six subtasks
have correlation scores exceeding 80%. Additionally, the overall score of LIME correlates at 91%
with WV-Elo, which is higher than any individual sub-task and the original bench’s correlations,
demonstrating that the overall score of LIME provides a more comprehensive reflection of MLLMs’
capabilities.

Automated evaluation metrics (e.g., CIDEr) cannot effectively assess the performance of
MLLMs in captioning tasks. As an early foundational problem, the captioning task is exten-
sively studied, and MLLMs demonstrate exceptional ability in this task. For instance, earlier models
like InstructBlip perform exceptionally well on captioning tasks, and there is a broad presence of
training data for image captioning in MLLMs’ training processes. However, the captioning task
shows a negative correlation with all other sub-tasks. This indicates that previous metrics (e.g., BLEU,
CIDEr) only focus on the overlap between the model-generated responses and the ground truth, but
do not consider that MLLMs might generate content that is semantically close to the ground truth
(i.e., the model-generated response may be semantically similar to the ground truth but expressed
differently, or the model may generate more detailed content about the image). Consequently, we
exclude it from the overall score calculation.

There is a certain degree of correlation between the sub-tasks in LIME. On the one hand, the
relevance of TextVQA, InfographicVQA, and OCRBench is relatively high. As shown in Fig. 4, the
correlation of these tasks all surpasses 85%, and these two VQA tasks require MLLMs to understand
fine-grained content in images to answer questions. This demonstrates that OCR tasks also rely on
the ability of MLLMs to perceive fine-grained objective facts in images. On the other hand, POPE,
ChartQA, and InfographicVQA all require reasoning abilities using extra commonsense knowledge.
The correlation scores of these tasks are all above 70%, and POPE requires the model to use extra

7

WV EloLIMEOriginalOCRBenchChartQAInfoVQATextVQAAI2DPOPEScienceQA OK-VQATextCapsCOCO-CaptionWV EloLIMEOriginalOCRBenchChartQAInfoVQATextVQAAI2DPOPEScienceQA OK-VQATextCapsCOCO-Caption10091708685848480595511-25-3991100799386919576775934-18-5870791007362617060445160-1-2786937310069829578606937-4-50858662691007774687037-8-37-3984916182771008759763125-47-5884957095748710066696734-5-70807660786859661003872105-1159774460707669381002428-29-505559516937316772241002863-4211346037-8253410282810028-27-25-18-1-4-37-47-55-296328100-8-39-58-27-50-39-58-70-11-50-42-27-8100604020020406080100Under review as a conference paper at ICLR 2025

knowledge to solve the hallucination of MLLMs. We assume that ChartQA and infoVQA may also
necessitate the use of additional common knowledge by the models to solve problems.

5.3 EFFECTIVENESS OF LIME

Figure 5: with the same series of models, the distribution differences of various Parameter sizes.
Left(⋆): LLaVA-1.6 series, Right(▲): InternVL-2 series

Table 3: Statistics on the score distributions
across different model series.

Table 4: Statistics on the score distributions
across different model sizes.

Model series Dataset GiNi

stdev

Model size Dataset GiNi

stdev

InternVL-2

Cambrian

LLaVA-1.6

LIME
Original

LIME
Original

LIME
Original

0.061
0.030

0.006
0.002

0.042
0.004

6.972
4.421

1.227
0.715

6.730
1.418

7B

8B

13B

LIME
Original

LIME
Original

LIME
Original

0.271
0.086

0.128
0.046

0.174
0.043

19.041
10.836

10.685
6.270

13.536
6.446

LIME provides a more challenging evaluation for MLLMs. As shown in Tab 2, the MLLMs’
performances on LIME are less than those on the Original Bench for most tasks. Compared to the
Origin benchmark, different MLLMs show a larger score range on our LIME, indicating that our
LIME can better reflect the performance differences between models with a smaller amount of data.

Furthermore, we compare the score variations across different model series and model sizes. Figure 5
illustrates a clear positive correlation between model performance and model size within the same
model series. Notably, LIME exhibits a more dispersed score distribution, effectively highlighting the
differences in model performance. In Tab 3 and 4, the Gini coefficient and standard deviation are used
to measure the differences in overall score distribution across the same model series and model sizes.
The larger the Gini coefficient and standard deviation, the greater the disparity in data distribution. It
can be observed that, whether within the same model series or the same model size, LIME achieves
higher Gini and standard deviation values compared to the original bench. This indicates that LIME
can better differentiate the performance differences between various models.

LIME eliminates potential data leakage. For multimodal question answering tasks, visual infor-
mation input is essential, and LLMs are unable to provide correct answers due to they cannot perceive
the content within the image. However, as shown in Figure 6 (right), there are severe data leakage
issues in the original Bench for the AI2D and ScienceQA tasks. The average score for AI2D is close
to 55%, and for ScienceQA, it exceeds 60%, which shows that data from AI2D and ScienceQA
in Original are highly likely to have been exposed to the training data of LLMs. In contrast, the
LIME has eliminated this potential threat, achieving scores below 25% in AI2D and close to 40% in
ScienceQA.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

30.1537.0844.0664.867.2967.22010203040OriginalLIMEModel Parameter SizeScore(B)48.2153.6457.2262.0063.9866.8568.467373.9777.8478.8280.3101020304050OriginalLIMEModel Parameter SizeScore(B)Under review as a conference paper at ICLR 2025

Model

LLaMA3-8B
LLaMA3-70B
Qwen1.5-32B
Qwen1.5-72B
Qwen2-7B
Qwen2-72B
Yi-1.5-9B-Chat
Yi-1.5-34B-Chat
Yi-1.5-34B-Chat

AI2D

ScienceQA

LIME

Original

LIME

Original

18.10
25.70
24.10
19.80
21.00
20.60
20.10
23.60
25.20

46.76
62.05
61.14
57.45
57.09
69.95
23.22
54.15
60.69

33.33
56.00
43.67
35.00
43.00
38.67
17.33
42.00
46.00

59.35
69.91
67.97
61.13
67.38
63.36
23.60
65.20
70.55

Figure 6: Comparing text only results of LIME and original bench. Left: text only results between
LIME and Original on AI2D and ScienceQA; Right: average score comparison of Original and
LIME.

5.4 THE IMPACT OF DETAIL IMAGE PERCEPTION

Table 5: Text-only with VD results: With the condition of providing only text QA information and
VD information, the performance comparison between vlms-bench and origin bench.

Setting

Models

AI2D ChQA COCO IVQA OCRBen OK VQA POPE

SciQA TCaps TVQA

LIME

Original

LLaMA3-8B
LLaMA3-70B
Qwen1.5-32B
Qwen1.5-72B
Qwen2-7B
Qwen2-72B
Yi-1.5-9B-Chat

LLaMA3-8B
LLaMA3-70B
Qwen1.5-32B
Qwen1.5-72B
Qwen2-7B
Qwen2-72B
Yi-1.5-9B-Chat

23.5
24.0
28.8
25.4
27.6
26.3
22.1

49.0
52.0
60.5
58.8
59.2
60.4
24.7

6.4
7.7
6.7
2.5
6.7
6.9
2.3

11.4
12.4
10.7
6.4
12.7
10.5
3.1

2.8
3.3
6.5
3.2
6.9
2.7
0.3

3.1
3.6
8.1
3.8
7.4
3.5
0.0

12.9
12.3
9.4
10.1
11.2
10.8
3.1

18.6
17.6
15.0
16.6
19.7
15.7
5.9

9.2
9.3
8.7
8.9
8.9
9.6
0.0

19.3
19.5
20.2
20.2
19.6
20.5
0.5

17.4
21.8
4.7
7.8
15.0
10.6
7.8

32.5
36.4
15.8
21.1
30.5
24.2
7.8

32.1
38.1
39.3
42.7
44.2
36.3
40.0

46.9
5.2
47.4
35.1
44.6
34.3
32.7

16.4
39.4
46.6
44.2
45.5
45.2
0.0

59.5
64.6
68.8
68.4
69.0
67.9
31.7

5.3
6.0
9.2
6.0
12.5
5.2
0.2

6.5
7.8
10.6
7.1
15.4
6.8
0.2

17.9
22.0
13.7
15.2
19.0
16.8
5.8

26.4
36.2
22.1
27.4
33.3
28.7
5.8

In our data cleaning process, we remove many questions that most models can answer, as well as a
small number of questions that are difficult for both humans and GPT-4V to answer, in order to make
the benchmark better highlight the differences in model capabilities. As shown in Tab 5, to investigate
whether the remaining samples need to be answered by using textual and image information, we
conduct experiments using LLMs to generate answers on both the Original Benchmark and MLLMs
Benchmark under QID (question + image description) setting.

LIME requires MLLMs to perceive deeper levels of image information. Especially in tasks
such as AI2D, OCRBench, and TCaps, the scores of LLMs on LIME are significantly lower than on
the Original Benchmark when provided with only the questions and simple image descriptions. This
indicates that, after removing some of the simpler questions, LIME is better at testing the models’
ability to perceive image details.

5.5 EXISTING BENCHMARK STILL DIFFERS FROM REAL-WORLD QUERY.

To further investigate the gap between LIME and real-world users’ queries, we construct a similarity
search system that compares them. MixEval (Ni et al., 2024) uses SentenceTransformers(Reimers,
2019) as the retrieval model, while Uniir (Wei et al., 2023) employs multimodal models like CLIP
and BLIP. We use WildVision-Chat as the query data source, which contains 45.2k high-quality user
questions, and employ SentenceTransformers to retrieve the top 10 most similar samples from LIME.
To fully incorporate image information, we combine the question and image description as the query
input. Additionally, we utilize Qwen2-72B to ensure a high level of relevance in the final results. As

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

AI2DScienceQA0102030405060Avg ScoreLIMEOriginalUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

a result, we obtain a LIME-fit dataset containing 1.1k relevant samples. Existing benchmark can’t
cover all types of real-world query.

In Figure 9, we compare the category distribution differences between LIME-fit and the WildVision
Bench. It is evident that LIME-fit concentrates in a few specific categories (e.g., data analysis,
general description, object recognition). However, it does not include instructions for solving real-
world problems, such as Face Recognition, Problem Solving, and Scene Description. Furthermore,
Figure 10 shows the frequency distribution of each subcategory in LIME-fit, which follows a long-
tail distribution. This indicates that the current benchmark does not fully cover the instruction
requirements of real-world scenarios.

6 RELATED WORK

In recent years, there has been increasing attention on establishing evaluation benchmarks to assess
the performance of MLLMs in different scenarios to guide the development of MLLMs. Early
multimodal evaluation benchmarks primarily focused on single tasks, such as Visual Question
Answering (VQA)(Antol et al., 2015; Goyal et al., 2017; Kafle & Kanan, 2017; Singh et al., 2019;
Marino et al., 2019), Image Captioning(Agrawal et al., 2019), and Information Retrieval (Wei et al.,
2023). As MLLMs develop, simple benchmarks are no longer sufficient to evaluate the versatile
capabilities of these models comprehensively, since most MLLMs demonstrate exceptional ability on
those benchmarks. Consequently, numerous more difficult and diverse benchmarks have emerged
in recent years to assess the capabilities of MLLMs comprehensively. For instance, MMMU (Yue
et al., 2024) and CMMMU (Zhang et al., 2024a) are comprehensive benchmark tests for university-
level multidisciplinary multimodal understanding and reasoning. MMBench (Liu et al., 2023c)
has developed a comprehensive evaluation pipeline that offers fine-grained capability assessment
and robust evaluation metrics. MMRA (Wu et al., 2024b) systematically establishes an association
relation system among images to assess the multi-image relation mining ability of MLLMs.

However, those benchmarks cannot distinguish the performance gaps among different models ex-
cellently, as they still contain some too simple or difficult samples that most models yield the same
results on. Furthermore, training datasets across different models may contain the samples of those
benchmarks, which results in data leakage issues (Fu et al., 2023). Mmstar (Chen et al., 2024a)
and MMLU Redux (Gema et al., 2024) have identified several issues within current benchmarks.
Mmstar (Chen et al., 2024a) proposes an automated pipeline to filter benchmark data, aiming to detect
potential data leakage, while MMLU Redux (Gema et al., 2024) focuses on correcting annotation
errors. However, there is still a pressing need for a comprehensive pipeline that fully addresses the
challenges posed by multimodal datasets. In response to this, we introduce LIME: LESS IS MORE
FOR MLLM EVALUATION. We have carefully selected six task types from existing mainstream
benchmarks and scaled them down according to clear guidelines. This streamlined version retains the
core elements of mainstream MLLM benchmarks, providing a more efficient and focused evaluation.

7 CONCLUSION

As MLLMs continue to advance, a notable absence of convenient and high-quality multimodal
benchmarks has emerged. In response to this, we propose a pipeline aimed at semi-automatically
refining existing benchmarks to enhance their quality, culminating in the development of LIME,
which comprises 9,403 evaluation samples across 6 types of tasks and 10 different benchmark
datasets. By refining the original benchmarks to filter question difficulty and eliminate potentially
problematic items, LIME offers a more rigorous evaluation for MLLMs, necessitating a deeper
understanding of image information. The outcomes of our evaluation experiments demonstrate the
heightened challenge posed by LIME for MLLMs. We anticipate that our approach will contribute
to the advancement of MLLM evaluation systems, and we are committed to continually enriching
LIME with an expanded array of datasets through regular updates and expansions. Our ultimate goal
is to provide the community with a simpler, more efficient, and more accurate evaluation method and
suite for MLLMs.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra,
Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In
Proceedings of the IEEE/CVF international conference on computer vision, pp. 8948–8957, 2019.

01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li,
Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin
Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu,
Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and
Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick,
and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international
conference on computer vision, pp. 2425–2433, 2015.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu,
Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan,
Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin
Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng
Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou,
Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609,
2023a.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv preprint arXiv:2308.12966, 2023b.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi
Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language
models? arXiv preprint arXiv:2403.20330, 2024a.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong
Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl:
Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint
arXiv:2312.14238, 2023.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial
multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024b.

XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/

InternLM/xtuner, 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language
models with instruction tuning, 2023. URL https://arxiv.org/abs/2305.06500.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu
Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal
large language models. arXiv preprint arXiv:2306.13394, 2023.

Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria
Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani,
et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127, 2024.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa
matter: Elevating the role of image understanding in visual question answering. In Proceedings of
the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan
Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents,
2023.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng,
Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video
understanding. arXiv preprint arXiv:2408.16500, 2024.

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang,
Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models
with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.

Dongfu Jiang, Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis:

Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024.

Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In
Proceedings of the IEEE international conference on computer vision, pp. 1965–1973, 2017.

Hugo Laurenc¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building

vision-language models? arXiv preprint arXiv:2405.02246, 2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei
Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint
arXiv:2408.03326, 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
llava-vl.github.io/blog/2024-01-30-llava-next/.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi
Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?
arXiv preprint arXiv:2307.06281, 2023c.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren,
Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan.
Deepseek-vl: Towards real-world vision-language understanding, 2024.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual
question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf
conference on computer vision and pattern recognition, pp. 3195–3204, 2019.

Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and
Yang You. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint
arXiv:2406.06565, 2024.

N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint

arXiv:1908.10084, 2019.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and
Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pp. 8317–8326, 2019.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha
Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open,
vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang,
Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.
Cogvlm: Visual expert for pretrained language models, 2023.

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen.
Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint
arXiv:2311.17136, 2023.

12

Under review as a conference paper at ICLR 2025

Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran
Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, and Chenghua
Lin. SciMMIR: Benchmarking scientific multi-modal information retrieval. In Findings of the
Association for Computational Linguistics ACL 2024, pp. 12560–12574, 2024a.

Siwei Wu, Kang Zhu, Yu Bai, Yiming Liang, Yizhi Li, Haoning Wu, Jiaheng Liu, Ruibo Liu,
Xingwei Qu, Xuxin Cheng, et al. Mmra: A benchmark for multi-granularity multi-image relational
association. arXiv preprint arXiv:2407.17379, 2024b.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint
arXiv:2407.10671, 2024.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li,
Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding
Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong
Sun. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint 2408.01800, 2024.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu
Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under-
standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 9556–9567, 2024.

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang
Cheng, Chunpu Xu, Shuyue Guo, et al. Cmmmu: A chinese massive multi-discipline multimodal
understanding benchmark. arXiv preprint arXiv:2401.11944, 2024a.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu,
Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation
of large multimodal models. arXiv preprint arXiv:2407.12772, 2024b.

Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan-
grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm-xcomposer: A vision-
language large model for advanced text-image comprehension and composition. arXiv preprint
arXiv:2309.15112, 2023.

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava:
A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 OVERALL DATA STATICS

Figure 7shows the overall data distribution in LIME, and figure 8 shows an example for each category
title

Figure 7: The overall percentage distribution of LIME.

Figure 8: The overview of LIME.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

T/F ReasoningOcrCaptioningScience qaNormal VQAInforgraphic QASubcategory: pope(cid:0)Difficulty Level: middleSubcategory: ocrbench(cid:0)Difficulty Level: middleSubcategory: TextCaps(cid:0)Difficulty Level: hardSubcategory: scienceqa(cid:0)Difficulty Level: middleSubcategory: OK-VQA(cid:0)(cid:0)Difficulty Level: middleSubcategory: Chart-QA(cid:0)Difficulty Level: hardQuestion: Which of these states is farthest east?Question: Is there a bottle in the image?Question: Where is the location that is a quarter miles far from here?Question: Please carefully observe the image and come up with a caption for the image.Options: (cid:0)A. Florida(cid:0)B. New York(cid:0)C. New Hampshire(cid:0)D. IowaAnswer: (cid:0)(1) motocross (cid:0)(cid:0)(2) racing (cid:0)(cid:0)(3) riding(cid:0)(cid:0)Answer: 13(cid:0)Answer: East Dunne Ave(cid:0)Answer: FalseAnswer: 3 young children in karate uniforms named Gracie Barra are raising arms in victory.Question: What sport can you use this for?(cid:0)(cid:0)(cid:0)Question:How many years are represented on this graph?(cid:0)(cid:0)(cid:0)Under review as a conference paper at ICLR 2025

A.2 MORE EXPERIMENT RESULT

Table 6: Comparing overall scores of LIME and Original. Top: results on LIME, Bottom: results
on the original dataset. The arrow next to the model name indicates the change in ranking on LIME
compared to the original dataset. ↑: upward shift, ↓: downward shift, and -: no change.

InfoVQA

ScienceQA

OCR

Captioning

ChQA ↑
88.33
83.17
87.00
87.67
83.75
80.83
80.42
81.92
80.33
67.50
71.75
61.67
71.83
69.25
65.83
69.42
64.33
69.00
55.42
64.42
59.00
54.67
54.50
13.08
43.08
35.33
35.75
5.50
5.25
3.00
4.50

85.52
83.70
83.32
84.44
80.12
82.48
81.04
74.60
74.72
72.60
80.60
73.44
67.00
72.90
73.12
79.84
69.30
71.40
62.20
67.40
63.56
26.40
61.36
60.60
55.00
18.10
42.56
18.20
15.40
12.50
11.10

IVQA ↑ AI2D ↑
69.20
73.92
58.20
78.50
54.81
70.92
62.80
70.58
60.40
68.25
59.20
68.17
54.90
66.17
54.00
60.50
45.00
59.50
42.90
65.00
45.80
54.00
31.40
57.25
54.80
43.17
45.20
42.83
37.00
46.25
43.60
37.67
49.60
42.08
6.60
48.10
31.20
40.67
40.72
22.25
35.80
20.50
37.00
22.33
38.10
28.75
38.10
23.92
27.10
22.67
27.70
16.25
27.90
13.42
25.90
12.42
24.05
9.08
21.90
9.08
22.80
12.17

76.08
72.50
78.86
72.72
70.69
70.65
65.19
51.48
57.69
50.73
72.80
48.05
63.30
56.90
67.15
62.62
37.60
52.02
41.50
51.90
31.17
37.00
46.23
34.30
37.00
29.50
26.56
25.80
20.10
22.90
22.20

85.88
78.90
80.73
83.16
81.38
82.25
78.08
80.41
72.70
73.93
34.40
72.99
61.90
71.70
70.21
72.41
71.60
62.56
70.40
76.10
66.81
69.20
57.09
63.40
65.30
59.40
57.84
55.20
56.90
34.00
32.30

SciQA ↑ OCRBen ↑

98.33
75.00
91.67
96.33
95.67
92.00
70.00
94.33
84.00
58.00
92.33
53.67
78.33
62.67
84.33
69.00
64.67
6.60
53.00
61.67
70.00
68.33
53.00
78.67
47.33
62.67
60.67
48.33
49.00
34.33
44.67

98.56
94.50
85.57
97.47
95.88
97.03
96.03
85.52
94.25
79.08
96.00
80.32
70.50
53.00
77.89
90.93
73.30
89.59
73.50
82.70
81.80
87.20
67.03
81.70
70.20
72.80
75.36
69.50
43.00
36.40
58.20

75.87
77.61
60.65
71.09
70.65
47.83
53.04
67.17
68.26
68.04
67.39
41.96
39.13
42.39
65.00
43.04
37.17
55.87
42.83
30.22
32.17
25.43
31.96
42.61
23.70
10.65
14.57
5.87
4.78
4.35
3.48

79.90
71.40
81.20
77.60
62.10
76.50
75.00
59.00
75.50
61.40
66.90
61.60
59.10
69.40
75.30
76.60
55.00
74.20
55.00
58.60
54.20
61.60
57.60
43.30
52.40
33.60
34.50
31.50
60.00
25.90
17.20

COCO ↑
63.10
68.74
69.24
76.18
42.55
104.74
97.07
35.99
23.67
75.06
51.95
29.28
4.27
7.40
15.19
5.85
84.25
31.91
96.40
99.35
61.82
54.22
76.62
61.23
76.05
68.16
68.96
80.89
79.20
102.08
63.19

99.15
95.80
92.13
110.30
140.45
89.77
54.08
8.18
79.52
14.33
134.00
9.13
28.40
35.50
103.52
24.10
135.00
49.34
101.90
114.40
79.42
71.90
131.90
67.60
100.00
115.40
91.37
109.00
25.90
141.40
80.90

TCaps ↑
48.94
110.60
112.63
56.91
25.44
108.18
78.07
21.67
34.01
101.72
42.59
38.56
4.97
6.52
13.19
6.41
48.25
39.86
90.28
48.71
103.07
83.21
47.83
94.51
48.35
81.21
76.65
68.73
65.73
53.14
58.91

62.03
148.10
144.36
80.10
136.97
36.70
30.17
6.08
59.81
9.44
111.40
7.97
44.70
52.90
131.92
42.23
69.60
18.03
67.30
69.10
134.08
119.10
120.81
110.10
72.00
104.00
111.43
98.00
41.60
74.00
83.10

Model

Size

Overall

InternVL-2 2023 ( - )
Qwen2-VL 2023b (↑ 1)
InternVL-1.5 2024b (↓ 1)
InternVL-2 2023 ( - )
InternVL-2 2023 ( ↑ 1 )
LLaVA-OneVision 2024 ( ↓ 1 )
XComposer2-4KHD 2023 (↑ 4)
InternVL-2 2023(↓ 1)
CogVLM-2 2024 (↑ 6)
Qwen2-VL 2023b (↑ 5)
InternVL-2 2023 (↓ 2)
CogVLM-1 2023 (↑ 1)
Cambrian 2024 (↓ 5)
Cambrian 2024 (↓ 4)
InternVL-2 2023 (↑ 3)
Cambrian 2024 (↓ 4)
LLaVA-1.6 2024 (↑ 3)
MiniCPM-LLaMA3-2.5 2024 (↓ 3)
LLaVA-OneVision 2024 (↑ 4)
LLaVA-LLaMA3 2023 (↓ 3)
Mantis-Idefics-2 2024 ( - )
Deepseek-VL 2024 (↑ 2)
LLaVA-1.6-vicuna 2024 (↓ 4)
Idefics-2 2024 (↓ 2)
LLaVA-1.6-vicuna 2024 ( - )
Mantis-SigLIP 2024 (↑ 1)
MiniCPM 2024 (↑ 2)
LLaVA-1.5 2023a (↓ 2)
LLaVA-1.5 2023a (↓ 1)
InstructBLIP-vicuna 2023 ( - )
Tiny-LLaVA-1 2024 ( - )

InternVL-2
InternVL-1.5
Qwen2-VL
InternVL-2
LLaVA-OneVision
InternVL-2
InternVL-2
Cambrian
InternVL-2
Cambrian
XComposer-4KHD
Cambrian
CogVLM-1
MiniCPM-LLaMA3-2.5
Qwen2-VL
CogVLM-2
LLaVA-LLaMA3
InternVL-2
LLaVA-1.6-vicuna
LLaVA-1.6
Mantis-Idefics-2
Idefics-2
LLaVA-OneVision
Deepseek-VL
LLaVA-1.6-vicuna
LLaVA-1.5
Mantis-SigLIP
LLaVA-1.5
MiniCPM
InstructBLIP-vicuna
Tiny-LLaVA-1

40B
7B
26B
26B
8B
7B
7B
4B
19B
2B
2B
17B
34B
13B
1B
8B
34B
8B
0.5B
8B
8B
7B
13B
8B
7B
8B
1.0
13B
7B
7B
1.4B

40B
26B
7B
26B
7B
8B
4B
34B
2B
13B
7B
8B
17B
8B
2B
19B
8B
1B
13B
34B
8B
8B
0.5B
7B
7B
13B
8B
7B
1.0
7B
1.4B

66.85
65.28
64.12
63.98
62.00
61.95
57.52
57.22
54.44
54.00
53.64
51.03
50.17
48.57
48.21
47.95
44.06
42.61
41.40
40.90
39.25
38.10
37.08
36.39
30.15
29.13
26.15
20.38
17.20
15.55
13.95

80.31
79.49
79.14
78.82
78.71
77.84
73.97
73.26
73.00
72.39
71.93
71.84
71.34
71.22
70.86
69.93
69.74
68.46
67.29
67.22
66.91
66.73
65.65
65.62
64.80
59.58
58.96
57.27
56.18
47.87
34.30

T/F
POPE ↑
51.69
53.05
51.69
54.63
49.21
52.37
46.28
47.18
51.02
50.79
50.79
55.10
49.44
50.79
52.82
49.89
47.00
43.10
48.98
44.24
44.24
48.50
43.10
42.00
41.10
45.60
44.00
36.60
32.51
45.10
37.00

89.23
88.90
88.17
88.64
89.17
87.90
87.71
88.46
88.90
88.53
87.00
88.24
88.90
88.00
87.78
87.56
87.80
87.94
87.50
85.60
86.90
86.80
88.33
87.10
87.60
87.10
81.47
87.00
85.10
85.00
56.30

Common VQA
TVQA ↑ OK VQA ↑

77.98
74.56
69.88
75.20
66.10
65.22
60.30
62.29
69.46
70.70
59.56
71.20
57.28
58.93
57.55
59.00
51.20
61.80
48.61
40.01
44.51
44.80
43.90
56.50
39.00
26.34
37.00
19.50
16.50
11.40
18.70

82.59
79.00
80.92
82.06
76.02
77.00
74.51
72.11
72.39
73.07
74.30
72.47
79.70
75.00
78.70
77.59
65.40
69.67
67.00
68.90
63.51
71.30
65.85
63.20
64.90
48.70
49.59
46.10
55.30
33.20
38.50

19.45
28.18
35.47
16.08
24.20
37.32
28.13
17.48
4.92
16.87
21.87
31.70
22.03
24.13
15.37
25.23
10.40
25.30
23.13
34.72
29.07
25.00
16.10
35.60
12.50
32.43
6.20
32.10
23.43
22.10
0.90

50.98
60.70
55.68
48.50
60.98
52.02
38.43
52.07
43.74
53.28
51.90
52.17
46.90
52.30
40.59
18.51
60.20
33.84
46.30
31.00
52.50
53.90
44.17
48.70
44.20
58.30
52.90
53.40
47.30
45.20
3.80

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A.3 PIPELINE DETAILS

A.3.1

PROMPT TEMPLATE DETAILS

Semi-Automated Screening Process Prompt We selected GPT-4V as the basis for automatic
judgment and interacted with the GPT-4V API using specific prompt templates for different subtasks.

Semi-Automated Screening Process Prompt(VQA tasks)

Please judge whether the <Answer>is the golden answer to the <Question>. If it is, please reply YES,
otherwise reply NO.

<Question>: {question} <Answer>: {answer}

<Your judgement> : <YES or NO>

Semi-Automated Screening Process Prompt(captioning tasks)

Now there is an image captioning task.
Please first describe the content of the image, then compare the image content with the provided captions.
If the captions are suitable as captions for the image, please answer YES; if they are not suitable, please
answer NO.
Respond with NO if any of the captions are unsuitable. Respond with YES only if all captions are
suitable.

<Captions>: {answer}
<Description>: <Content of the image>

<Your judgement>: <ONLY YES or NO>

Exact Vision Description Prompt For the QVD experiment, we use LLaVA-NEXT-110B to extract
information from the images, with the following prompt:

Exact Vision Description Prompt

<image> Please provide a description of the following image, You should consider elements in the
image.

A.3.2 METRICS

Subtask metrics: As shown in the Tab 7, different metrics are used for different subtasks. It is
important to note that, except for the CIDEr metric, all other metrics have a range between 0 and 1.
The final score for each subtask is calculated by taking the average of these metrics.

Table 7: Metrics for different subtask

Metric

Accuracy

Subtask

AI2D, ScienceQA-IMG,
OCRBench, POPE

Accuracy =

Formula
if the prediction is correct
if the prediction is incorrect

(cid:26)1,
0,
(cid:80)m

CIDEr = 1
m

i=1

(cid:80)N

n=1 wn ·

g(n)
·r(n)
i
i
∥g(n)
i ∥∥r(n)
i ∥
SCORE = min (cid:0)1, match nums
(cid:1)
ANLS(X, Y ) = 1 − Lev(X,Y )
SCORE = |prediction−SCORE|

max(|X|,|Y |)

3

|target|

CIDEr

TextCaps,COCO-Caption

Match score

OK-VQA,TextVQA

ANLS

Relaxed Overall

InfoVQA

ChartQA

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Overall metric: For the overall metric, we explored two mainstream calculation methods: arithmetic
mean 1 and weighted mean 2.

Arithmetic Mean =

1
n

n
(cid:88)

i=1

xi

Weighted Mean =

(cid:80)n
i=1 wixi
(cid:80)n
i=1 wi

(1)

(2)

The arithmetic mean directly calculates the average of each subtask’s scores, while the weighted
mean takes into account the number of samples in each subtask. We compare the results of these two
calculation methods, as shown in the Tab 8. weighted average method achieves a higher correlation
with WV-ELO. This suggests that the weighted average method is slightly superior to the arithmetic
mean, as it considers the impact of the number of data points on the overall score, thereby avoiding
potential errors caused by uneven data distribution. Therefore, in our work, we ultimately chose the
weighted average as the method for calculating the overall score.

Table 8: Comparison of different overall metrics method

model

overall weighted

overall sum overall cider WV bench

LLaVA-1.6-vicuna-7B

LLaVA-1.6-vicuna-13B

LLaVA-1.6-34B

CogVLM

Deepseek-VL

Idefics2

MiniCPM-v-1.0

Tinny-LLaVA-1-hf

LLaVA-1.5-13B

InstructBLIP-vicuna-7B

correlation score

30.15

37.08

44.06

51.03

38.1

36.39

26.15

13.95

20.38

15.55

0.91

30.46

36.52

43.30

47.66

39.04

38.43

28.95

17.79

22.88

18.61

0.90

36.07

41.04

47.12

44.03

43.31

43.83

35.79

24.15

32.3

29.56

0.87

992

956

1059

1016

979

965

910

879

891

862

1

A.3.3 DIFFICULTY CLASSIFICATION DETAILS

For subtasks using the accuracy (acc) metric, where the scores are binary, with only 1 or 0, other
tasks may have various possible score distributions (e.g., COCO-Caption, OK-VQA). Therefore, we
determine the threshold score based on the overall distribution of subtask scores, and choose the
cutoff value that offers the greatest distinction, as shown in Tab 9, for the metrics ANLS, Relaxed
Overall and Accuracy (Acc), the threshold is set to 1.0, for BLEU-4 (for the captioning task, we use
the BLEU-4 metric to represent the score for each question), the threshold is set to 0.2, while for
Match Score, it is set to 0.6. When the score is greater than the threshold, it is marked as correct;
otherwise, it is marked as incorrect.

Metrics

bleu4 Match score ANLS Relaxed Overall Acc

Threshold

0.2

0.6

1.0

1.0

1.0

Table 9: Thresholds for Different Metrics

17

Under review as a conference paper at ICLR 2025

A.3.4 RETRIEVE FROM REAL WORLD QUERYD

Qwen2-72B Judge Prompt

Your task is to compare the content of two questions along with their corresponding image descriptions
to determine if they are the same or aligned. Analyze from multiple perspectives, such as theme,
question type, and description content.

Please adhere to the following guidelines:

1. Theme Consistency:
- Compare whether the themes of the two questions and their corresponding image descriptions match.
If they focus on entirely different topics, they should be marked as not aligned.

2. Question Type:
- Analyze whether the question types (e.g., technical, artistic, textual) of both questions match with
each other and align with their respective image descriptions. If they are of different natures, note the
mismatch.

3. Description Alignment:
- Compare the task or content expected in each question with what is visually or descriptively present in
both image descriptions. If the questions or image content require specific actions (e.g., reading text or
coding) that differ from each other or the descriptions, they should be marked as misaligned.

4. Evaluate Similarity:
- Rate the similarity between the two questions and their respective descriptions on a scale from 1 to 5,
where 1 means entirely different and 5 means highly similar.

5. Output Clarification:
- You should return whether the two questions and their image descriptions align or not in a simple
”True” or ”False” result. - Provide a brief reason for your conclusion. - Include a similarity rating from 1
to 5, based on how well the questions and descriptions match. - The output should only contain the
”result,” ”reason,” and ”similarity rating” fields.

### Example:
<Question 1>: Can you write codes to load this 3D object?
<Description 1>: The image shows a stone sculpture of an angel sitting on a pedestal. The angel has
large, feathered wings that spread out behind it, and its head is bowed down, as if in deep thought or
prayer. The angel’s body is draped in flowing robes, and its arms are crossed over its lap. The pedestal
is ornately carved with intricate designs, and the entire sculpture is set against a dark background, which
makes the white stone stand out even more. The overall mood of the image is one of solemnity and
reverence.

<Question 2>: What is written in the image?
<Description 2>: The image shows the word ”ART” in white capital letters on a blue background.
The letters are bold and have a slight shadow effect, giving them a three-dimensional appearance. The
overall design is simple and modern, with a focus on the text itself.

Result: False
Reason: The first question asks for coding assistance to load a 3D object, but its description is about an
angel sculpture. The second question is focused on reading text from an image, which is aligned with its
description showing the word ”ART.” The themes, questions, and descriptions are entirely different.
Similarity Rating: 1

<Input Question 1>: {Question 1}
<Input Description 1>: {Description 1}

<Input Question 2>: {Question 2}
<Input Description 2>: {Description 2}

<Output>:

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Figure 9: category difference between LIME-fit and wildvision bench

Figure 10: subcategory distrubution of LIME-fit.

19

DescriptiveRecognitionAnalyticalInteractiveInstructiveCreativeComprehensiveExplanationGeneral DescriptionDetailed DescriptionObject DescriptionScene DescriptionExplanationText RecognitionMovies/TV Shows DescriptionText RecognitionObject RecognitionFace RecognitionLocation RecognitionAttribute-based Question AnswerEmotion RecognitionMovies/TV Shows DescriptionBrand RecognitionArtistic RecognitionScene RecognitionExplanationData AnalysisProblem SolvingCritical ReviewsMathematical ReasoningMeme ComprehensionComparative AnalysisAttribute-based Question AnswerRecommendationsDecision MakingHow-to GuidesStory WritingArt and Design IdeasComparative AnalysisExplanationObject RecognitionData AnalysisGeneral DescriptionText RecognitionHow-to GuidesCultural AnalysisSymbol RecognitionCultural AnalysisAttribute-based Question AnswerLocation IdentificationText and Symbol RecognitionAnalytical - Data AnalysisObject IdentificationText IdentificationTranscriptionLanguage IdentificationTextContent IdentificationProduct IdentificationBrand RecognitionOCR Formula RecognitionData AnalysisActivity DescriptionRecognitionMath Problem Solving0.00.10.20.30.4ProbabilityUnder review as a conference paper at ICLR 2025

A.4 ABLATION STUDY ABOUT DATA SIZE

Table 10: data size ablation study on OK-
VQA.

Table 11:
ChartQA.

data size ablation study on

Model

Full

100

500

1200

Model

Full

100

500

1200

llava1.5-7B

22.71

17.00

20.76

22.92

llava1.5-7B

llava1.5-13B

31.59

36.00

29.60

30.23

llava1.5-13B

4.77

4.71

3.00

5.00

3.80

4.40

4.17

4.33

llava1.6-7B

11.46

13.00

10.40

11.32

llava1.6-7B

42.81

39.00

42.00

42.67

llava-llama3-8B

36.12

32.60

36.92

36.17

llava-llama3-8B

64.78

66.00

66.00

65.75

xcomposer2-4khd

25.91

29.40

26.48

26.90

xcomposer2-4khd

82.11

80.00

83.00

82.92

minicpm

instructblip

idefics2

internvl

Table 12:
TextVQA.

25.76

20.60

29.92

25.30

minicpm

70.37

67.00

71.60

70.75

21.45

20.60

23.60

21.78

instructblip

2.95

3.00

3.00

3.00

32.76

27.60

35.12

33.00

38.36

45.00

39.80

38.28

idefics2

internvl

13.18

16.00

12.80

14.25

87.13

89.00

87.80

86.92

data size ablation study on

Table 13: data size ablation study on In-
foVQA.

Model

Full

100

500

1200

Model

Full

100

500

1200

llava1.5-7B

16.68

14.40

18.34

17.46

llava1.5-7B

9.40

7.00

9.00

8.83

llava1.5-13B

19.54

17.90

22.30

20.14

llava1.5-13B

12.18

16.00

11.60

11.17

llava1.6-7B

38.58

43.00

38.62

39.35

llava1.6-7B

21.30

19.00

23.00

20.33

llava-llama3-8B

39.81

46.40

38.26

40.17

llava-llama3-8B

22.69

25.00

23.40

22.33

xcomposer2-4khd

61.20

59.40

60.98

61.63

xcomposer2-4khd

72.36

72.00

75.80

73.75

minicpm

instructblip

idefics2

internvl

63.07

60.30

63.90

63.39

minicpm

49.22

55.00

48.20

48.75

11.66

8.60

12.00

11.25

instructblip

9.73

11.00

8.40

9.50

55.94

54.90

57.76

56.56

70.28

70.10

70.48

70.74

idefics2

internvl

24.69

23.00

24.00

25.42

72.08

69.00

72.20

72.25

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

20

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

B CASE STUDY

The original dataset contains noise data. In the following figure, we categorize the problematic data
into three types and present specific examples from different datasets.

Text Answerable Questions: Some questions can be answered without the need for visual infor-
mation, mainly focusing on the AI2D and ScienceQA datasets. As shown in figs. 30 and 31, AI2D
and ScienceQA emphasize knowledge in the field of science while overlooking the importance of
visual information. Given the background of domain knowledge, some LLMs are able to provide
answers even without requiring visual input.

Annotation Error Questions: Most benchmarks are manually curated, which inevitably leads to
annotation errors. Problematic questions exist in almost all benchmarks. It can be found in figs. 32,
33 and 39 to 44.

Repeated Question: Some benchmarks also contain a significant amount of duplicate data, where
the question content and image content are completely identical. This issue is mainly found in the
POPE dataset, as shown in the figs. 34 to 38.

List of Case Study Figures

.
.
1 Data Leakage-MMMU-1 .
.
.
2 Data Leakage-MMMU-2 .
.
.
3 Data Leakage-MMMU-3 .
.
.
4 Data Leakage-MMMU-4 .
.
.
.
5 Easy sample-MMMU-1 .
.
.
.
6 Easy sample-MMMU-2 .
.
.
.
7 Easy sample-MMMU-3 .
.
.
.
8 Easy sample-MMMU-4 .
.
9 Easy sample-MMMU-5 .
.
.
.
10 Data Leakage-MMBench-1 .
.
11 Data Leakage-MMBench-2 .
.
12 Data Leakage-MMBench-3 .
.
13 Data Leakage-MMBench-4 .
.
14 Data Leakage-MMBench-5 .
.
15 Easy sample-MMBench-1 .
.
16 Easy sample-MMBench-2 .
.
17 Easy sample-MMBench-3 .
.
18 Easy sample-MMBench-4 .
.
19 Easy sample-MMBench-5 .
.
.
20 Data Leakage-AI2D-1 .
21 Data Leakage-AI2D-2 .
.
.
22 Data Annotation-InfoVQA-1 .
23 Data Annotation-InfoVQA-2 .
24 Repeated questions-POPE-1 .
25 Repeated questions-POPE-2 .
26 Repeated questions-POPE-3 .
27 Repeated questions-POPE-4 .
28 Repeated questions-POPE-5 .
29 Data Annotation-OKVQA-1 .
30 Data Annotation-OKVQA-2 .
31 Data Annotation-OKVQA-3 .
32 Data Annotation-TextVQA-1 .
33 Data Annotation-TextVQA-2 .
34 Data Annotation-TextVQA-3 .

.
.

.
.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22
.
23
.
24
.
25
.
26
.
27
.
28
.
29
.
.
30
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
32
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
.
42
.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
43
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
46
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
47
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
48
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
49
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
52
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
55
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Figure 11: A sample bad case of MMMU
Back to List of figures

22

MMMUQuestion: What vessel(s) serve(s) areas involved in speech in the majority of people? <image 1>Ground Truth:   Left middle cerebral artery.Error Category:  Answer LeakageOptions: ['Right middle cerebral artery.', 'Left middle cerebral artery.', 'Right and left middle cerebral arteries.', 'Right and left posterior cerebral arteries.']Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Figure 12: A sample bad case of MMMU
Back to List of figures

23

MMMUQuestion: Which of the following does the offspring of a pod bug resemble?Ground Truth: Similar to the adult, but shorter and without wingsError Category: Answer LeakageOptions: ['Similar to the adult, but shorter and without wings', 'Grub', 'Maggot', 'Caterpillar', "Don't know and don't want to guess"]Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Figure 13: A sample bad case of MMMU
Back to List of figures

24

MMMUQuestion: <image 1> <image 2> Which of the following Acts of Parliament was passed in direct response to the events of the Boston Tea Party?Ground Truth: Coercive ActsError Category: Answer LeakageOptions: ['Coercive Acts', 'Tea Act', 'Townshend Acts', 'Currency Act']<image 1> <image 2> Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Figure 14: A sample bad case of MMMU
Back to List of figures

25

MMMUQuestion: Which theory of <image 1> focuses on the labels acquired through the educational process?Ground Truth: Symbolic interactionismError Category: Answer LeakageOptions: ['Critical sociology', 'Feminist theory', 'Functionalist theory', 'Symbolic interactionism']<image 1> Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Figure 15: A easy sample of MMMU
Back to List of figures

26

MMMUQuestion: Hicks Products produces and sells patio furniture through a national dealership network. They purchase raw materials from a variety of suppliers and all manufacturing, and assembly work is performed at their plant outside of Cleveland, Ohio. They recorded these costs for the year ending December 31, 2017. What is total revenue?Ground Truth:  A Error Category: Easy QuestionOptions: [A:'$3,100,000’, B:'$2,616,000’, C:'$2,474,000’, D:'$484,000']< 11 >Under review as a conference paper at ICLR 2025

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Figure 16: A easy sample of MMMU
Back to List of figures

27

MMMUQuestion: You are asked to compare two options with parameters as given. The risk-free interest rate should be assumed to be 6%. Assume the stocks on which these options are written pay no dividends. <image 1> Which call option is written on the stock with the higher volatility?Ground Truth: BError Category: Easy QuestionOptions: [A:'A', B:'B', C:'Not enough information']< 28 >Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Figure 17: A easy sample of MMMU
Back to List of figures

28

MMMUQuestion: <image 1> What seems to be the issue with this young citrus tree?Ground Truth: EError Category: Easy QuestionOptions: [A:'Mineral deficiency’, B:'Nematode attack’, C:"Don't know and don't want to guess", D:'There is no problem’, E:'Pot bound']< 33 >Under review as a conference paper at ICLR 2025

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Figure 18: A easy sample of MMMU
Back to List of figures

29

MMMUQuestion: <image 1> What is the common term for the yellow area surrounding the site of an infection?Ground Truth: DError Category: Easy QuestionOptions: [A:’I don’t know and I don't want to guess’, B:'Corona’, C:'Border’, D:'Halo’, E:'Toxin zone']< 45 >Under review as a conference paper at ICLR 2025

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Figure 19: A easy sample of MMMU
Back to List of figures

30

MMMUQuestion: <image 1> What is the substance present on the top surface of these citrus leaves?Ground Truth: CError Category: Easy QuestionOptions: [A:'Algae’, B:"Don't know and I don't want to guess", C:'Honey dew', 'Gummosis-produced resin', 'Bacterial ooze']< 47 >Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Figure 20: A sample bad case of MMBench
Back to List of figures

31

MMBenchQuestion: Complete the statement. Ammonia is ().Ground Truth:  BError Category: Data LeakageOptions: [A:'an elementary substance’, B:'a compound’]< en: 316 >Under review as a conference paper at ICLR 2025

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Figure 21: A sample bad case of MMBench
Back to List of figures

32

MMBenchQuestion: Identify the question that Madelyn and Tucker's experiment can best answer.Ground Truth:  B Error Category: Data LeakageOptions: [A:'Does Madelyn's snowboard slide down a hill in less time when it has a thin layer of wax or a thick layer of wax?’, B:' Does Madelyn's snowboard slide down a hill in less time when it has a layer of wax or when it does not have a layer of wax?’]< en: 241 >Under review as a conference paper at ICLR 2025

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Figure 22: A sample bad case of MMBench
Back to List of figures

33

MMBenchQuestion: Which fish's mouth is also adapted for tearing through meat?Ground Truth:  B Error Category: Data LeakageOptions: [A:'copperband butterflyfish’, B:'tiger moray’]< en: 274 >Under review as a conference paper at ICLR 2025

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Figure 23: A sample bad case of MMBench
Back to List of figures

34

MMBenchQuestion: Which animal's skin is also adapted for survival in cold places?Ground Truth:  B Error Category: Data LeakageOptions: [A:'fantastic leaf-tailed gecko’, B:'polar bear’]< en: 278 >Under review as a conference paper at ICLR 2025

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

Figure 24: A sample bad case of MMBench
Back to List of figures

35

MMBenchQuestion: Which material is this spatula made of?Ground Truth:  AError Category: Data LeakageOptions: [A:'rubber’, B:'cotton’]< en: 293 >Under review as a conference paper at ICLR 2025

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

Figure 25: A easy sample of MMBench
Back to List of figures

36

MMBenchQuestion: 图中所示建筑名称为？Ground Truth: AError Category: Easy QuestionOptions: [A:天坛, B:故宫, C:黄鹤楼, D:少林寺]< CC: 0 >Under review as a conference paper at ICLR 2025

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

Figure 26: A easy sample of MMBench
Back to List of figures

37

MMBenchQuestion: 图中所示建筑名称为？Ground Truth: BError Category: Easy QuestionOptions: [A:东方明珠, B:长城, C:中山陵, D:少林寺]< cc: 1 >Under review as a conference paper at ICLR 2025

1998
1999

2000
2001
2002
2003
2004
2005

2006
2007
2008
2009
2010
2011

2012
2013
2014
2015
2016

2017
2018
2019
2020
2021
2022

2023
2024
2025
2026
2027
2028

2029
2030
2031
2032
2033
2034

2035
2036
2037
2038
2039

2040
2041
2042
2043
2044
2045

2046
2047
2048
2049
2050
2051

Figure 27: A easy sample of MMBench
Back to List of figures

38

MMBenchQuestion: 图中所示景观所在地点为？Ground Truth: DError Category: Easy QuestionOptions: [A:重庆, B:香港, C:青岛, D:上海]< cc: 4 >Under review as a conference paper at ICLR 2025

2052
2053

2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065

2066
2067
2068
2069
2070

2071
2072
2073
2074
2075
2076

2077
2078
2079
2080
2081
2082

2083
2084
2085
2086
2087
2088

2089
2090
2091
2092
2093

2094
2095
2096
2097
2098
2099

2100
2101
2102
2103
2104
2105

Figure 28: A easy sample of MMBench
Back to List of figures

39

MMBenchQuestion: Which of the following could Laura and Isabella's test show?Ground Truth: BError Category: Easy QuestionOptions: [A:’if the concrete from each batch took the same amount of time to dry’, B:’if a new batch of concrete was firm enough to use’]< cc: 1 >Under review as a conference paper at ICLR 2025

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

Figure 29: A easy sample of MMBench
Back to List of figures

40

MMBenchQuestion: Which animal's limbs are also adapted for gliding?Ground Truth: AError Category: Easy QuestionOptions: [A:”northern flying squirrel’, B: ring-tailed lemur’]< cc: 9 >Under review as a conference paper at ICLR 2025

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

Figure 30: A sample bad case of AI2D
Back to List of figures

41

AI2DQuestion: Which stage follows the egg stage of development in a beetle's life cycle?Ground Truth:  Larve Error Category:  Data Leakage Options: ["Nymph", "Larva", "Adule", "Pupa"]Under review as a conference paper at ICLR 2025

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

Figure 31: A sample bad case of AI2D
Back to List of figures

42

AI2DQuestion: In the illustration, if mahi mahi were to die off the large shark population would?Ground Truth:  “decrease” Error Category:  Data Leakage Options: [ "decrease", "remain the same", "can't tell", "increase" ]Under review as a conference paper at ICLR 2025

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

Figure 32: A sample bad case of InfoVQA
Back to List of figures

43

InfographicVQAQuestion: What percent of executives does not use social media daily? Ground Truth:   ‘24%’ , ‘24’ [图片]Error Category: Annotation ErrorUnder review as a conference paper at ICLR 2025

2322
2323

2324
2325
2326
2327
2328
2329

2330
2331
2332
2333
2334
2335

2336
2337
2338
2339
2340

2341
2342
2343
2344
2345
2346

2347
2348
2349
2350
2351
2352

2353
2354
2355
2356
2357
2358

2359
2360
2361
2362
2363

2364
2365
2366
2367
2368
2369

2370
2371
2372
2373
2374
2375

Figure 33: A sample bad case of InfoVQA
Back to List of figures

44

InfographicVQAQuestion: What is the second last solution given? Ground Truth:  ‘access to technical and vocational training’ Error Category:  Annotation ErrorUnder review as a conference paper at ICLR 2025

2376
2377

2378
2379
2380
2381
2382
2383

2384
2385
2386
2387
2388
2389

2390
2391
2392
2393
2394

2395
2396
2397
2398
2399
2400

2401
2402
2403
2404
2405
2406

2407
2408
2409
2410
2411
2412

2413
2414
2415
2416
2417

2418
2419
2420
2421
2422
2423

2424
2425
2426
2427
2428
2429

Figure 34: A sample bad case of POPE
Back to List of figures

45

POPEQuestion: : Is there a tv in the image? Ground Truth:  No Error Category: Annotation ErrorOptions: Yes< 228 >Under review as a conference paper at ICLR 2025

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

Figure 35: A sample bad case of POPE
Back to List of figures

46

POPEQuestion: : Is there a dining table in the image? Ground Truth:  No Error Category: Annotation ErrorOptions: Yes< 934 >Under review as a conference paper at ICLR 2025

2484
2485

2486
2487
2488
2489
2490
2491

2492
2493
2494
2495
2496
2497

2498
2499
2500
2501
2502

2503
2504
2505
2506
2507
2508

2509
2510
2511
2512
2513
2514

2515
2516
2517
2518
2519
2520

2521
2522
2523
2524
2525

2526
2527
2528
2529
2530
2531

2532
2533
2534
2535
2536
2537

Figure 36: A sample bad case of POPE
Back to List of figures

47

POPEQuestion: : Is there a boat in the image? Ground Truth:  No Error Category: Annotation ErrorOptions: Yes< 1412 >Under review as a conference paper at ICLR 2025

2538
2539

2540
2541
2542
2543
2544
2545

2546
2547
2548
2549
2550
2551

2552
2553
2554
2555
2556

2557
2558
2559
2560
2561
2562

2563
2564
2565
2566
2567
2568

2569
2570
2571
2572
2573
2574

2575
2576
2577
2578
2579

2580
2581
2582
2583
2584
2585

2586
2587
2588
2589
2590
2591

Figure 37: A sample bad case of POPE
Back to List of figures

48

POPEQuestion: : Is there a boat in the image? Ground Truth: Repeated with id 940Error Category: Repeated QuestionsOptions: Yes< 6940 >Under review as a conference paper at ICLR 2025

2592
2593

2594
2595
2596
2597
2598
2599

2600
2601
2602
2603
2604
2605

2606
2607
2608
2609
2610

2611
2612
2613
2614
2615
2616

2617
2618
2619
2620
2621
2622

2623
2624
2625
2626
2627
2628

2629
2630
2631
2632
2633

2634
2635
2636
2637
2638
2639

2640
2641
2642
2643
2644
2645

Figure 38: A sample bad case of POPE
Back to List of figures

49

POPEQuestion: : Is there a dining table in the image? Ground Truth: Repeated with id 694Error Category: Repeated QuestionsOptions: Yes< 6694 >Under review as a conference paper at ICLR 2025

2646
2647

2648
2649
2650
2651
2652
2653

2654
2655
2656
2657
2658
2659

2660
2661
2662
2663
2664

2665
2666
2667
2668
2669
2670

2671
2672
2673
2674
2675
2676

2677
2678
2679
2680
2681
2682

2683
2684
2685
2686
2687

2688
2689
2690
2691
2692
2693

2694
2695
2696
2697
2698
2699

Figure 39: A sample bad case of OKVQA
Back to List of figures

50

OK VQAQuestion: : How would you dress for this setting? Ground Truth: [ "shorts", "swimming suit", "bathing suit", "bikini" ]Error Category: Annotation ErrorOptions: [ "shorts", "shorts", "shorts", "shorts", "bathing suit", "bathing suit", "bikini", "bikini", "summer", "summer" ]< 1708495 >Under review as a conference paper at ICLR 2025

2700
2701

2702
2703
2704
2705
2706
2707

2708
2709
2710
2711
2712
2713

2714
2715
2716
2717
2718

2719
2720
2721
2722
2723
2724

2725
2726
2727
2728
2729
2730

2731
2732
2733
2734
2735
2736

2737
2738
2739
2740
2741

2742
2743
2744
2745
2746
2747

2748
2749
2750
2751
2752
2753

Figure 40: A sample bad case of OKVQA
Back to List of figures

51

OK VQAQuestion: : Where are these people?Ground Truth: [ "outside", "riverbank", "grassland", "field", "hill", "outdoors", ”lawn" ]Error Category: Annotation ErrorOptions: [ "outside", "outside", "outside", "outside", "field", "field", "on hill", "on hill", "outdoors", "outdoors" ]< 3981385 >Under review as a conference paper at ICLR 2025

2754
2755

2756
2757
2758
2759
2760
2761

2762
2763
2764
2765
2766
2767

2768
2769
2770
2771
2772

2773
2774
2775
2776
2777
2778

2779
2780
2781
2782
2783
2784

2785
2786
2787
2788
2789
2790

2791
2792
2793
2794
2795

2796
2797
2798
2799
2800
2801

2802
2803
2804
2805
2806
2807

Figure 41: A sample bad case of OKVQA
Back to List of figures

52

OK VQAQuestion: : How is this effect painted on to walls?Ground Truth: [ "whitewash", "paint", "plaster" ]Error Category: Annotation ErrorOptions: [ "sponge", "sponge", "sponge", "sponge", "with sponge", "with sponge", "sponged", "sponged", "sky", "sky" ]< 1269585 >Under review as a conference paper at ICLR 2025

2808
2809

2810
2811
2812
2813
2814
2815

2816
2817
2818
2819
2820
2821

2822
2823
2824
2825
2826

2827
2828
2829
2830
2831
2832

2833
2834
2835
2836
2837
2838

2839
2840
2841
2842
2843
2844

2845
2846
2847
2848
2849

2850
2851
2852
2853
2854
2855

2856
2857
2858
2859
2860
2861

Figure 42: A sample bad case of TextVQA
Back to List of figures

53

Text VQAQuestion: : what is one of the numbers on the buttons of the calculator? Ground Truth: [ "1", ”2", ”3", ”4", ”5", ”6", "7", ”8", ”9", ”0" ]Error Category: Annotation ErrorOptions: [ "1", "1", "1", "1", "1", "7", "7", "5", "1", "5" ]< 35925 >Under review as a conference paper at ICLR 2025

2862
2863

2864
2865
2866
2867
2868
2869

2870
2871
2872
2873
2874
2875

2876
2877
2878
2879
2880

2881
2882
2883
2884
2885
2886

2887
2888
2889
2890
2891
2892

2893
2894
2895
2896
2897
2898

2899
2900
2901
2902
2903

2904
2905
2906
2907
2908
2909

2910
2911
2912
2913
2914
2915

Figure 43: A sample bad case of TextVQA
Back to List of figures

54

Text VQAQuestion: : what is served at this place?Ground Truth: [ "ice cream", ”coffee", ”sandwiches", ”gelato", ”cake", ”yule log", ”gift certificates" , “grilled focaccia sandwiches”]Error Category: Annotation ErrorOptions: [ "gift certificates", "ice cream, coffee, and sandwiches", "ice cream& coffee", "traditional italian ice cream and coffee", "ice cream & coffee", "ice cream, coffee, and grilled focaccia sandwiches", "ice cream & coffee", "traditional italian, ice cream and coffee, grilled focaccia sandwiches", "ice cream & coffee, grilled focaccia sandwiches", "gelato" ]< 37706 >Under review as a conference paper at ICLR 2025

2916
2917

2918
2919
2920
2921
2922
2923

2924
2925
2926
2927
2928
2929

2930
2931
2932
2933
2934

2935
2936
2937
2938
2939
2940

2941
2942
2943
2944
2945
2946

2947
2948
2949
2950
2951
2952

2953
2954
2955
2956
2957

2958
2959
2960
2961
2962
2963

2964
2965
2966
2967
2968
2969

Figure 44: A sample bad case of TextVQA
Back to List of figures

55

Text VQAQuestion: : what is the cell phone carrier? Ground Truth: [ "EDGE " ]Error Category: Annotation ErrorOptions: [ "cingular", "blackberry", "cingular", "cingular", "cingular", "cingular", "at&t", "cingular", "cingular", "cingular" ]< 36711 >