MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos
Under review as a conference paper at ICLR 2025

MMVU:
FACETED VIDEO UNDERSTANDING EVALUATION

TOWARDS MULTI-DISCIPLINE MULTI-

Anonymous authors
Paper under double-blind review

Figure 1: MMVU covers seven broad disciplines and 69 subdisciplines, focusing on the evalua-
tion of multi-faceted reasoning beyond perception (e.g., explanation, counterfactual thinking, future
prediction, domain expertise). On the right are four video samples from the Science, Tech & Engi-
neering, Embodied Tasks, and Game disciplines.

ABSTRACT

Multimodal Language Language Models (MLLMs) demonstrate the emerging
abilities of ”world models”—interpreting and reasoning about complex real-world
dynamics. To assess these abilities, we posit videos are the ideal medium, as they
encapsulate rich representations of real-world dynamics and causalities. To this
end, we introduce MMVU, a new benchmark for multi-discipline, multi-faceted
multimodal video understanding. MMVU distinguishes itself from previous video
(1) multi-discipline,
understanding benchmarks with two unique advantages:
covering various disciplines that often require domain expertise for comprehen-
sive understanding; (2) multi-faceted reasoning, including explanation, counter-
factual thinking, future prediction, etc. MMVU consists of a human-annotated
dataset to evaluate MLLMs with questions about the whole videos and a syn-
thetic dataset to analyze MLLMs within a single modality of perception. Together,
MMVU encompasses 1,910 videos across seven broad disciplines and 69 subdis-
ciplines, complete with 6,627 question-answer pairs and associated captions. The
evaluation includes 4 proprietary and 11 open-source MLLMs, which struggle
on MMVU (e.g., GPT-4o performs the best with only 62.5% accuracy), showing
large room for improvement. Further ablation studies reveal other interesting find-
ings such as models’ different skill sets from humans. We hope MMVU can serve
as an essential step towards world model evaluation in videos.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Q: Which is not a characteristic of Kirby as shown in the video?A: Kirby can inhale objects and spit out a star.Q: What would happen if the man skip the step shown in video?A: The TV bench will not have a second layer.Type: FuturePredictionType: Counterfactual ThinkingQ: How do the pulleys move when the hands are off the pulley system?A: Two static and two moving upward.Type: ExplanationType: DomainExpertiseQ: How does the hair dryer change the flow of the smoke?A: The air pressure at the back of the hair dryer is low, contributing to the formulation of laminar flow of the smoke.Sports & ArtsUnder review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

1

INTRODUCTION

Foundation models, such as Large Language Models (LLMs) (OpenAI, 2023c; Touvron et al.,
2023a; Jiang et al., 2023; Anil et al., 2023) and Multimodal LLMs (MLLMs) (Team et al., 2023;
Lin et al., 2023a; Li et al., 2023c; Maaz et al., 2024; Chen et al., 2023), have demonstrated remark-
able abilities in text and image domains, igniting debates about their potential pathways to Artificial
General Intelligence (AGI). This raises a critical question: how well do these models understand the
dynamics of the real world? Are they equipped with an inherent World Model (LeCun, 2022; Chen
et al., 2024; Ha & Schmidhuber, 2018; Xiang et al., 2024) that can understand and reason about the
underlying principles and causalities of the dynamic, multimodal world?

Videos, with their rich, dynamic portrayal of the real world, are ideally suited for evaluating the
”world modeling” capabilities of MLLMs. Existing video understanding benchmarks (Li et al.,
2023d; Ning et al., 2023b; P˘atr˘aucean et al., 2023; Li et al., 2023d), however, fall short in two key
perspectives for such evaluations. First, as LeCun et al. (LeCun, 2022) discussed, the world model
should be able to (1) estimate missing information about the state of the world not provided by
perception, and (2) predict plausible future states of the world. Evaluation of such capabilities re-
quires multi-faceted reasoning beyond perception level, including explaining the video dynamics,
counterfactual thinking of alternative consequences, and predicting future activities within videos.
Moreover, the multi-discipline nature of the multimodal world necessitates a grasp of diverse funda-
mental principles—ranging from physics and chemistry to engineering and business. Hence, domain
expertise across a variety of disciplines is imperative for a thorough evaluation of a model’s world
understanding towards AGI (Morris et al., 2023; Yue et al., 2023).

Therefore, we introduce MMVU, a multi-discipline multi-faceted multimodal video understanding
benchmark to comprehensively evaluate MLLMs’ abilities in reasoning and interpreting real-world
dynamics 1. MMVU encompasses a wide range of disciplines and presents multi-faceted reasoning
challenges that demand a combination of visual, auditory, and temporal understanding. It consists
of 1,910 videos that span seven common disciplines, including Art & Sports, Business, Science,
Health & Medicine, Embodied Tasks, Tech & Engineering, and Games, and 69 subdisciplines (see
Figure 1) such as Robotics, Chemistry, Trading, and Agriculture, thereby fulfilling the objective
of breadth in discipline coverage. The dataset includes a total of 1,559 question-answer pairs and
video captions annotated and reviewed by humans. Meanwhile, for multi-faceted reasoning, MMVU
mainly contains seven kinds of questions focusing on explanation (explaining the phenomenon in
videos), counterfactual thinking (answering what-if questions), future prediction (predicting future
events), domain expertise (answering domain-specific inquiries), temporal understanding (reason-
ing about temporal information), and etc. Four video examples with these questions from different
disciplines are depicted in Figure 1. To serve as a comprehensive benchmark, MMVU comprises
two datasets: a human-annotated dataset for evaluating MLLMs on the whole video and a synthetic
dataset designed to analyze MLLMs’ perception within single visual or audio modalities. We eval-
uate 15 MLLMs that can handle videos or image sequences on MMVU, including both open-source
(e.g., Video-LLaVA-7B (Lin et al., 2023a)) and proprietary models (GPT-4o (OpenAI, 2024) and
Gemini (Team et al., 2023)).

We summarized the contributions and key findings as follows:

• We introduce MMVU, a new benchmark designed to rigorously evaluate the capabilities
of Multimodal Large Language Models (MLLMs) in world modeling through the realm of
video understanding. MMVU spans a broad spectrum of disciplines, featuring a rich array
of question types for multi-faceted reasoning.

• In addition to the human-annotated dataset, we develop an automatic data collection
pipeline, streamlining video content selection and question-answer generation, and con-
struct a well-controlled synthetic dataset to analyze MLLMs within single visual or audio
modalities.

1Note that the term “world model” in MMVU is broadened from its traditional use in reinforcement learning
to a more generalized sense. MMVU is not a sufficient testbed for world model evaluation, but we believe
overcoming the unique challenges presented in MMVU is essential and necessary towards comprehensive world
modeling.

2

Under review as a conference paper at ICLR 2025

Table 1: Comparison between MMVU and previous benchmarks for real-world video understand-
ing on a variety of criteria. Multi-faceted include Explanation (Explain.), Counterfactual Think-
ing (Counter.), Future Prediction (Future.) and Domain Expertise (Domain.) MMVU is
the first multi-discipline and multitask video understanding benchmark that covers wider reasoning
questions, and also included first-party data annotations.

Benchmarks

Multi-
Discipline

Multi-
Task

Multi-Faceted Reasoning

Counter.

Future.

Domain.

First-Party
Annotation

Explain.
✓
✓

MovieQA (Tapaswi et al., 2016)
TVQA (Lei et al., 2018)
ActivityNet-QA (Yu et al., 2019b)
MSVD-QA (Xu et al., 2017) (Xu et al., 2016)
MSRVTT-QA (Xu et al., 2016)
Sports-QA (Li et al., 2024)
VaTeX (Wang et al., 2019)
VALUE (Li et al., 2021)
Video-Bench (Ning et al., 2023a)
MVBench (Li et al., 2023d)
Perception Test (P˘atr˘aucean et al., 2023)
VideoMME (Fu et al., 2024)
MMBench-Video (Fang et al., 2024)
TempCompass (Liu et al., 2024c)
ViLMA (Kesen et al., 2023)
VITATECS (Li et al., 2023e)
NExT-QA (Xiao et al., 2021)
CVRR (Khattak et al., 2024)
Causal-VidQA (Li et al., 2022)
MMVU (Ours)

✓
✓
✓
✓
✓

✓
✓

✓

✓

✓

✓
✓
✓
✓

✓

✓

✓
✓

✓

✓

✓
✓

✓

✓

✓
✓
✓
✓
✓

✓

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓
✓
✓
✓
✓
✓

✓
✓
✓
✓
✓
✓
✓
✓
✓

• We observe that existing MLLMs still face substantial challenges posed by MMVU. Even
the best performer, GPT-4o, can only achieve a 62.54% overall accuracy, and four MLLMs
particularly trained on videos perform worse than random chance.

• Although there is stll a clear gap between open-source and proprietary models, the open-
source model Video-LLaVA-7B achieves the best on Embodied Tasks. It outperforms GPT-
4V and Gemini Pro on Embodied Tasks by a large margin and performs similarly on Art
& Sports, where spatiotemporal dynamics play a more crucial role in video understanding.
This is further validated with its leading results on Temporal Understanding question type.
• In our study comparing MLLMs with average humans (non-experts), we notice some cor-
relation between question difficulties as perceived by humans and MLLMs. However,
MLLMs present different skill sets than humans in that they can answer reasonable amount
of difficult questions that humans completely fail but also struggle at easy questions that
humans excel at. This indicates different perception, cognition, and reasoning abilities
between MLLMs and humans.

2 RELATED WORK

2.1 MULTIMODAL LARGE LANGUAGE MODELS (MLLMS)

Emerging MLLMs Recent advancements in Large Language Models (LLMs) (OpenAI, 2023a;
Google, 2023; Touvron et al., 2023a; Chiang et al., 2023; Touvron et al., 2023b; Bai et al., 2023a)
have paved the way for several multimodal counterparts in the vision-and-language domain (Dai
et al., 2023; Liu et al., 2023b;a; Li et al., 2023a; Zhu et al., 2023b; Zheng et al., 2023; Bai et al.,
2023b), and recently released GPT-4V (OpenAI, 2023b), followed by Gemini Vision family (Team
et al., 2023). As LLMs have been applied to world modeling and simulation (Wang et al., 2024a),
MLLMs now extend their capabilities beyond text and image inputs. Pretrained on large-scale,
diverse datasets, these models are equipped with commonsense, domain-specific knowledge, and
broad generalizability.

VideoChat (Li et al., 2023c) leverages the QFormer (Li et al., 2023b) to map visual representa-
tions to LLM (Chiang et al., 2023), and performs a multi-stage training pipeline. Otter (Li et al.,
2023a) proposes to conduct instruction finetuning based on Openflamingo (Awadalla et al., 2023).
PandaGPT (Su et al., 2023) employs the ImageBind (Han et al., 2023) as the backbone and fine-
tunes it. The mPLUG-Owl (Ye et al., 2023) introduces an abstractor module to perform visual and
language alignment. VideoLLaMA (Zhang et al., 2023a) introduces a frame embedding layer and

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

also leverages ImageBind to inject temporal and audio information into the LLM backend. Chat-
UniVi (Jin et al., 2023) uses clustering to do feature fusion. LWM (Liu et al., 2024b) collects a large
video and language dataset from public books and video datasets and trains a world model that is
capable of processing more than millions of tokens.

These MLLMs demonstrate emerging abilities in multi-disciplinary world knowledge and excel at
multi-faceted reasoning tasks, such as inverse dynamic prediction—predicting intermediate steps
between previous and next states, a crucial auxiliary task for next-state prediction (Devlin, 2018; Lu
et al., 2019; Paster et al., 2020) in real-world scenarios. In response to the emerging capabilities of
MLLMs, we propose MMVU to evaluate their ability to understand real-world dynamics, underlying
principles, and causalities, with the ultimate goal of achieving world modeling.

Benchmarking MLLMs To evaluate MLLMs, there is a flourishing of analysis (Liu et al., 2024a;
Zhang et al., 2023b; Jiang et al., 2022; Lu et al., 2024; Fan et al., 2024; Cui et al., 2023; Guan
et al., 2024; Yu et al., 2023; Fu et al., 2023a) and the establishment of innovative benchmarks such
as VisIB-Bench (Bitton et al., 2023) which evaluates models with real-world instruction-following
ability given image inputs, MMMU (Yue et al., 2023) designed to access models on college-level
image-question pairs that span among different disciplines, and VIM (Lu et al., 2023) which chal-
lenges the model’s visual instruction following capability.

However, these recent analyses and benchmarks only cover the image input. Recently, video bench-
marks such as Perception Test (P˘atr˘aucean et al., 2023) is proposed to focus on perception and
skills like memory and abstraction. However, it uses scenarios with a few objects manipulated by
a person, which limits the variety of contexts. In contrast, MMWorld operates in an open-domain
scenario with diverse scenes; MVBench (Li et al., 2023d), TempCompass (Liu et al., 2024c) centers
on temporal understanding, while MMVU not only includes temporal reasoning but also evaluates
other multi-faceted reasoning abilities such as counterfactual thinking and domain-specific exper-
tise; EgoSchema Mangalam et al. (2023) focuses on natural human activity and behavior, but it
does not cover the broad range of disciplines that MMWorld does. MLLMs that can perfectly solve
MMWorld would unlock the ability to perform multifaceted, multidisciplinary reasoning and the
potential to serve as a world model.

2.2 VIDEO UNDERSTANDING BENCHMARKS

Previous video benchmarks, as shown in Table 1, focus on video understanding tasks, including
activity-focused on web videos (Yu et al., 2019a), description-based question answering (Zeng et al.,
2017), video completion (Fu et al., 2023b), and video infilling (Himakunthala et al., 2023). Recently,
Video-Bench (Ning et al., 2023b) introduces a benchmark by collecting videos and annotations from
multiple existing datasets. Mementos (Wang et al., 2024b) builds a benchmark for MLLM reasoning
for input image sequences. STAR (Wu et al., 2021) builds a benchmark for situated reasoning in
real-world videos. CLEVER (Yi et al., 2020) builds a benchmark containing videos focusing on ob-
jects with simple visual appearance. None of these benchmarks match the multi-discipline coverage
that MMWorld provides. MMWorld, in contrast, presents a new benchmark designed to encom-
pass interdisciplinary coverage, task diversity, and multifaceted reasoning capabilities—including
future prediction, counterfactual thinking, and more—underpinned by original human annotations
and integrated domain knowledge.

3 THE MMVU BENCHMARK

The MMVU benchmark is built on three key design principles: multi-discipline coverage, multi-
faceted reasoning, and temporal reasoning. It spans various disciplines that require domain expertise
and incorporates diverse reasoning skills such as explanation, counterfactual thinking, and future
prediction. The benchmark consists of two parts: a human-annotated dataset and a synthetic dataset.
The human-annotated dataset serves as the main testbed to evaluate MLLMs from multiple
perspectives. The synthetic dataset is divided into two subsets, each designed to assess MLLMs’
perception behavior based on visual and audio inputs, respectively.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

3.1 MANUAL DATA COLLECTION

We collect videos from YouTube with the Creative Licence in seven disciplines: Art & Sports
(18.5%), Business (12.0%), Science (20.4%), Health & Medicine (12.0%), Embodied Tasks
(12.0%), Tech & Engineering (12.9%), and Game (12.2%). For Art & Sports, 29 videos are col-
lected from the SportsQA dataset (Li et al., 2024). And for Embodied Tasks, 24 videos are sourced
from IKEA Assembly (Ben-Shabat et al., 2021), RT-1 (Brohan et al., 2022), and Ego4D (Grauman
et al., 2022) datasets to increase video diversity.

Our manual benchmark collection takes two stages. In the first stage, we conduct a detailed exami-
nation of each of the seven primary disciplines to identify a comprehensive range of subdisciplines
for inclusion in our benchmark. Our selection of videos is driven by three key principles:

1. The first principle, multi-discipline coverage, emphasizes the requirement for domain knowl-
edge—selecting videos that inherently demand an understanding of specialized content across vari-
ous disciplines; 2. The second principle, multi-faceted annotation, involves collecting videos that
enable the creation of question-answer pairs from multiple perspectives to evaluate world model
properties comprehensively; 3. The third principle, temporal information, prioritizes the inclu-
sion of videos that provide meaningful content over time, as understanding temporal information
is crucial for grasping world dynamics. This allows models to engage in temporal reasoning and
answering questions in MMWorld requires implicit temporal reasoning, e.g., the model needs to
understand temporal information to explain “why does the robot need to do the step shown in the
video”. We also design a “temporal understanding” question type to explicitly test models’ ability
to reason about temporal information (more examples can be found in Section F in the Appendix).

During the second stage, our team began the task of annotating questions, answers, and options.
All annotators were asked to carefully watch the collected videos and create questions with corre-
sponding answers and options, ensuring that understanding the video content and applying temporal
reasoning were necessary to determine the correct answers. We also ensured that the clarity, cor-
rectness, and grammatical accuracy of the questions and answers were verified using GPT-4o, and
that the questions could not be correctly answered without video input. We craft questions that
primarily test seven aspects of multimodal video understanding also from the perspective of multi-
faceted reasoning: 1) Explanation: Questions ask the model to elucidate the underlying logic or
purpose within the video; 2) Counterfactual Thinking: Tests the model’s ability to hypothesize and
consider alternative outcomes; 3) Future Prediction: Aims to predict future events based on the
current scenario, challenging the model’s foresight; 4) Domain Expertise: Evaluates the model’s
depth of knowledge in specific fields, such as how to assemble a coffee table; 5) Temporal Un-
derstanding: Assesses the model’s capability to reason about temporal sequences and dynamics;
6) Attribution Understanding: These questions focus on identifying cause-and-effect relationships
within the video, including tasks like counting; 7) Procedure Understanding: Tests the model’s abil-
ity to comprehend and explain procedural tasks shown in the video. The detailed distribution and
examples are shown in Figure 2. For quality control, we ensure each annotation is cross-checked by
at least two professional researchers to ensure accuracy and prevent annotation errors.

3.2 AUTOMATED DATA COLLECTION

Understanding real-world dynamics requires models to process both audio and visual modalities. To
evaluate MLLMs’ perception abilities in these modalities, we designed an automated data collection
pipeline. This pipeline collects targeted videos and generates QA pairs based on either audio or
visual information, ensuring the model’s capabilities are assessed independently for each modality.
By using information from a single modality to generate QA pairs, our pipeline ensures that the
synthetic data remains unbiased regarding input modality.

The synthetic data generation pipeline is illustrated in Figure 3. We employ a systematic approach
to gather videos with Creative Commons licenses from YouTube and the extensive YouTube-8M
dataset (Abu-El-Haija et al., 2016). This method ensures a diverse and comprehensive collection of
video data, which is important for the robust evaluation of multimodal video understanding models.

Video Collection and Processing We start with the video Query Generator. We start with the same
seven disciplines as the manually collected dataset. For each discipline, a set of subdisciplines is de-
fined to encapsulate a wide spectrum of topics, ensuring a diverse and comprehensive dataset. Once

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 2: The questions in MMVU are designed to evaluate seven primary understanding and rea-
soning abilities of models. Each question is annotated with all relevant categories. The figure
showcases one example question for each reasoning category, based on its main category.

Figure 3: Schematic diagram of the synthetic data generation pipeline in MMVU. It starts with
generating subdiscipline-specific queries, followed by video retrieval from YouTube-8M (Abu-El-
Haija et al., 2016) and YouTube. Keyframes are extracted for visual-based QA generation, and
videos are transcribed using an ASR module for audio-based QA generation.

the queries are generated, the Video Mapping and Filtering step is initiated. We perform mapping
of videos to YouTube-8M and online videos, constrained by a strict time limit of two minutes per
query, keeping only the most pertinent videos that satisfy the predefined criteria. Simultaneously,
the works in conjunction with the video transcripts to extract key terms and concepts. This iterative
process refines the search parameters and enhances the semantic richness of the dataset by identify-
ing and encoding the salient themes present in the videos. The Video Summarization module utilizes
Query-focused video summarization techniques based on Katna2 and UniVTG (Lin et al., 2023b).
This module selects ten representative frames from each video, distilling the essence of the content
while preserving the narrative context. This summarization facilitates efficient storage and quicker
processing times, which are crucial for large-scale analysis.

QA Generation The final stage in our pipeline is the QA / Caption Generation module, where we
leverage the capabilities of GPT-4V to generate accurate and contextually relevant questions and
answers, as well as captions, based on the video frames and transcripts. This step not only provides
rich annotations for each video but also equips the dataset with a multimodal dimension that supports
various downstream tasks such as video QA, captioning, and more.

Quality of the Synthetic Dataset Human evaluators were engaged to ascertain the reasonableness
of automatically generated questions and answers, ensuring that the synthetic dataset maintains a
high standard of quality and relevance. The findings from this human evaluation phase are detailed
in Section D of the Appendix, offering insights into the dataset’s efficacy and the realism of its
constructed queries and responses.

2https://github.com/keplerlab/katna

6

Q:What would happen if the man skipped the step shown in the video?A:The desktop of the coffee table will be upside down, which will make it impossible to mount the legs.Q:What is the reason that the lady decides to use the easy frost?A:Because it has no-fuss frosting.Q:What was first added into the milk?A:Cocoa powder.Q:What hasbeenchangedinthevideo?A:Thebottomdrawerhasbeenclosed.Q:What willhappennextas the price is below the blue and red lines?A:Thepricewillgodown.Q:How many animals appear in the video?A:Two. There are a horse and a dog.Multi-facetedReasoningTemporal UnderstandingProcedure UnderstandingCounterfactual ThinkingFuturePredictionDomainExpertiseAttribution Understanding22.1%10.8%10.9%14.1%16.5%11.9%13.7%Q:How does the girl actively change her spinning speed?A:Stretchingout or contracting legs and arms.ExplanationUnder review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 2: Key Statistics of the MMVU Benchmark. The main subset is the human-annotated subset.
Synthetic Subset I contains generated QA pairs focused exclusively on the audio content, while
Synthetic Subset II contains QA pairs focused exclusively on the visual content of the video.

Statistics

Main Subset

Synthetic I

Synthetic II

#Discipline/#Subdiscipline
#Videos
#QA pairs
Avg Video Lengths (s)
Avg #Questions per Video
Avg #Options
Avg Question Length
Avg Option Length
Avg Answer Length
Avg Caption Length
# Unique Words in Questions
# Unique Words in Answers

7/61
417
1,559
102.3
4.05
3.90
11.39
7.27
6.42
27.00
1,913
2,292

7/51
746
2,969
103.4
3.98
4.00
15.12
6.01
6.71
71.87
2,528
2,981

7/54
747
2,099
115.8
2.81
4.00
17.56
5.19
5.67
82.33
2,279
2,657

Finally, the statistics of automated curated data, which is used for the ablation study, are shown in
Table 2. The taxonomy of our dataset is shown in Figure 1. We note that only a portion of the
subdisciplines are shown due to space concerns. Please refer to the Appendix for full information.

4 EXPERIMENTS

4.1 EXPERIMENTAL SETTINGS

In our study, we compare MLLM’s performance on the MMVU benchmark, including GPT-
4o (OpenAI, 2024), GPT-4V (OpenAI, 2023b), Gemini Pro (Team et al., 2023), Claude-3.5-
Sonnet Anthropic (2024), Video-Chat (Li et al., 2023c), Video-ChatGPT (Maaz et al., 2024),
Video-LLaMA (Zhang et al., 2023a), Video-LLaVA (Lin et al., 2023a), ChatUnivi (Jin et al.,
2023), mPLUG-Owl (Ye et al., 2023), Otter (Li et al., 2023a), ImageBind-LLM (Han et al., 2023),
PandaGPT (Su et al., 2023), LWM (Liu et al., 2024b), and X-Instruct-BLIP (Panagopoulou et al.,
2023). For proprietary model, we adhere to the default settings provided by their official APIs.
They both take ten image frames extracted from the video content as the input. The Gemini Pro is
set to process visual input and configured with safety settings to filter a range of harmful content.
The configuration thresholds are set to ‘BLOCK NONE’. For PandaGPT, we set ‘top p’ to 0.7 and
‘temperature’ to 0.5. For VideoChat, we set ‘max frames’ to 100. For X-Instruct-BLIP, the model
is implemented using four image frames. We use GPT-4-32K as the judge for judging whether the
model answer is correct when it can not mapped to the option letter using the rule-based method.
For others, we all use the default setting. All inferences are run on a NVIDIA A6000 workstation.
The detailed implementation is given in the Appendix.

4.2 EVALUATION STRATEGY

Our dataset contains multiple-choice questions and captions corresponding to each video, supporting
tasks such as video question answering and video captioning. In our evaluation setup, we focus on
video question answering by measuring a model’s accuracy in selecting the correct answer from
the provided options. This method is straightforward to quantify and provides objective assessment.
However, one challenge is reliably mapping the model’s predictions to one of the predefined choices.

To address this, we employ two mapping strategies. We employ two mapping strategies. The first
method employs automated scripts to parse the models’ predictions and compare the parsed results
with the ground truth, similar to the approach used in (Yue et al., 2023); The second method involves
models freely generating answers, which are then evaluated by GPT-4. Given the question, correct
answer, and model’s prediction, GPT-4 returns a True or False judgment. This approach is based on
recent works in model evaluation (Maaz et al., 2024; Hsu et al., 2023; Hackl et al., 2023; Liu et al.,
2023c).

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 3: MLLM accuracy across diverse disciplines (averaging over three runs). GPT-4V and Gem-
ini Pro lead at most disciplines and achieve the best overall accuracy. The best open-source model
Video-LLaVA-7B outperforms them on Embodied Tasks and perform similarly on Art & Sports. All
data are annotated by humans.

Model

Random Choice

Art&
Sports

25.03

Business Science

Health& Embodied Tech&
Medicine Tasks

Engineering

Game

Average

25.09

26.44

25.00

26.48

30.92

25.23

26.31

Proprietary MLLMs

GPT-4o (OpenAI, 2024)
Claude-3.5-Sonnet (Anthropic, 2024)
GPT-4V (OpenAI, 2023b)
Gemini Pro (Team et al., 2023)

47.87 ±1.47 91.14 ±0.87 73.78 ±2.88 83.33 ±1.47 62.94 ±3.47
54.58 ±0.45 63.87 ±0.40 59.85 ±1.28 54.51 ±1.28 30.99 ±0.40
36.17 ±0.58 81.59 ±1.74 66.52 ±1.86 73.61 ±0.49 55.48 ±2.70
37.12 ±2.68 76.69 ±2.16 62.81 ±1.83 76.74 ±1.30 43.59 ±0.33

75.53 ±2.61
58.87 ±0.61
61.35 ±1.00
69.86 ±2.01

80.32 ±2.05 62.54 ±0.79
59.44 ±0.68 54.54 ±0.29
73.49 ±1.97 52.30 ±0.49
66.27 ±2.60 51.02 ±1.35

Open-source MLLMs

35.91 ±0.96 51.28 ±0.87 56.30 ±0.76 32.64 ±0.49 63.17 ±1.44
Video-LLaVA-7B (Lin et al., 2023a)
39.53 ±0.06 51.05 ±0.00 30.81 ±0.21 46.18 ±0.49 40.56 ±0.57
Video-Chat-7B (Li et al., 2023c)
24.47 ±0.49 60.84 ±1.51 52.00 ±0.73 61.11 ±1.96 46.15 ±2.06
ChatUnivi-7B (Jin et al., 2023)
29.16 ±1.62 64.10 ±1.84 47.41 ±3.29 60.07 ±1.30 23.78 ±3.47
mPLUG-Owl-7B (Ye et al., 2023)
26.84 ±0.69 39.16 ±3.02 36.45 ±1.31 53.12 ±0.00 36.60 ±3.25
Video-ChatGPT-7B (Maaz et al., 2024)
25.33 ±0.54 42.66 ±3.02 39.41 ±2.67 38.54 ±3.07 35.43 ±0.87
PandaGPT-7B (Su et al., 2023)
ImageBind-LLM-7B (Han et al., 2023)
24.82 ±0.16 42.66 ±0.99 32.15 ±1.11 30.21 ±1.47 46.85 ±1.14
X-Instruct-BLIP-7B (Panagopoulou et al., 2023) 21.08 ±0.27 15.85 ±0.87 22.52 ±1.11 28.47 ±0.49 18.41 ±1.44
12.04 ±0.53 17.48 ±0.57 15.41 ±0.91 20.49 ±0.98 25.87 ±1.98
LWM-1M-JAX (Liu et al., 2024b)
Otter-7B (Li et al., 2023a)
6.94 ±0.98 13.29 ±1.51
9.33 ±0.36
17.12 ±1.17 18.65 ±0.87
6.15 ±0.44 21.21 ±0.66 22.22 ±1.45 31.25 ±1.70 15.38 ±1.14
Video-LLaMA-2-13B (Zhang et al., 2023a)

58.16 ±1.00
39.36 ±0.00
56.74 ±1.33
41.84 ±5.09
41.49 ±1.74
41.84 ±2.79
41.49 ±1.50
22.34 ±0.87
21.99 ±2.19
15.96 ±1.74
19.15 ±1.74

49.00 ±3.16 44.60 ±0.58
44.98 ±0.57 40.11 ±0.06
52.61 ±2.84 39.47 ±0.42
62.25 ±3.16 38.94 ±1.52
36.55 ±2.27 33.27 ±0.97
40.16 ±4.65 32.48 ±0.45
41.37 ±0.57 31.75 ±0.14
26.10 ±0.57 21.36 ±0.18
11.65 ±3.01 15.39 ±0.32
15.26 ±0.57 14.99 ±0.77
24.90 ±5.93 14.03 ±0.29

Table 4: Results of different MLLMs on multi-faceted reasoning. All data are annotated by humans.

Model

Explanation

Counterfactual
Thinking

Future

Domain

Attribution

Temporal

Prediction Expertise Understanding Understanding

GPT-4o (OpenAI, 2024)
GPT-4V (OpenAI, 2023b)
Claude-3.5-Sonnet (Anthropic, 2024)
Gemini Pro (Team et al., 2023)

Video-LLaVA (Lin et al., 2023a)
Video-Chat-7B (Li et al., 2023c)
Video-ChatGPT-7B (Maaz et al., 2024)
ImageBind-LLM-7B (Han et al., 2023)
PandaGPT-7B (Su et al., 2023)
ChatUnivi-7B (Jin et al., 2023)
Video-LLaMA-2-13B (Zhang et al., 2023a)
X-Instruct-BLIP-7B (Panagopoulou et al., 2023)
LWM-1M-JAX (Liu et al., 2024b)
Otter-7B (Li et al., 2023a)
mPLUG-Owl-7B (Ye et al., 2023)

Proprietary MLLMs

75.88 ±1.47
64.90 ±0.58
62.75 ±0.16
65.49 ±0.42

82.48 ±0.69
78.59 ±1.55
71.78 ±0.40
65.45 ±1.05

69.05 ±0.49
61.07 ±0.17
66.79 ±0.45
53.87 ±1.31

Open-source MLLMs

42.55 ±0.85
43.73 ±0.32
39.02 ±1.12
26.86 ±0.58
37.45 ±1.80
48.82 ±0.48
23.92 ±0.97
15.29 ±0.28
18.82 ±0.55
10.98 ±0.42
49.61 ±1.31

64.96 ±0.69
45.74 ±0.20
47.45 ±2.09
50.61 ±0.20
46.47 ±1.05
61.80 ±0.53
25.30 ±1.11
27.25 ±0.53
30.66 ±0.34
15.82 ±0.20
55.47 ±1.58

47.86 ±0.58
40.95 ±0.10
33.69 ±1.08
33.93 ±0.17
33.93 ±0.45
45.95 ±0.68
16.31 ±1.03
21.07 ±0.51
17.98 ±0.26
13.10 ±0.68
47.74 ±1.07

56.68 ±0.72
44.90 ±0.07
51.94 ±0.23
48.58 ±1.07

42.46 ±0.61
41.66 ±0.06
32.13 ±0.38
29.51 ±0.27
29.55 ±0.41
33.91 ±0.31
10.55 ±0.29
23.05 ±0.24
11.62 ±0.39
16.91 ±0.54
35.20 ±1.17

65.10 ±1.15
59.61 ±0.85
40.00 ±0.55
43.92 ±1.40

36.86 ±1.95
30.59 ±0.00
21.18 ±2.00
34.90 ±1.40
26.27 ±2.24
33.33 ±0.64
8.63 ±0.85
24.31 ±0.64
21.57 ±0.85
17.65 ±0.00
24.71 ±2.00

40.90 ±2.42
27.17 ±1.00
25.77 ±0.46
24.65 ±1.00

34.45 ±1.19
25.77 ±0.23
23.53 ±0.76
19.89 ±0.91
28.01 ±0.82
22.97 ±0.91
6.16 ±1.00
11.20 ±0.82
7.00 ±0.46
9.52 ±1.00
20.17 ±0.69

We validated the second GPT-4-based evaluation approach with human evaluators, showing an error
rate of only 4.76% across 189 examples, demonstrating its reliability as an evaluator. Detailed results
for human evaluation and both evaluation strategies are provided in Appendix. All results presented
in the main paper are based on the second evaluation approach.

4.3 MAIN EVALUATION RESULTS ON HUMAN-ANNOTATED DATA

We show in Table 3 the main evaluation results of different MLLMs. Among these, GPT-4o emerges
as the top performer, followed by Claude-3.5-Sonnet. Video-LLaVA also demonstrates strong re-
sults, primarily due to the extensive training data which consists of 558K LAION-CCSBU image-
text pairs and 702K video-text pairs from WebVid (Bain et al., 2021). Its superior performance may
also be attributed to the adoption of CLIP ViT-L/14 trained in LanguageBind (Lin et al., 2023a) as
its vision model and the inclusion of a large volume of image-video-text pairings within the train-
ing data. On the other hand, models like Otter and LWM perform poorly across most disciplines,
possibly due to their weaker backbone and architecture used. Otter uses the LLaMA-7B language
encoder and a CLIP ViT-L/14 vision encoder, both of which are frozen, with only the Perceiver
resampler (Awadalla et al., 2023) module fine-tuned, which may lead to the lower performance.
Additionally, four MLLMs perform even worse than random, highlighting the challenging nature of
MMVU.

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

(a) Accuracy of MLLMs at difficulty levels for average
humans.

(b) GPT-4V results by disciplines at different difficulty
levels for average humans.

Figure 4: Model performance at different difficulty levels for average humans. Average human
difficulty levels are defined by 3 turkers’ performance per question: Easy (3/3 correct answers),
medium (2/3 correct), hard (1/3 correct), and expert (0/3 correct).

Study on Multi-faceted Reasoning Table 4 illustrates the multi-faceted reasoning performance
of each MLLM. GPT-4o emerges as the strongest model across all facets. Notably, in temporal
understanding, the open-sourced Video-LLaVA outperforms all other models except GPT-4o, likely
due to its extensive training on high temporal resolution video data, enhancing its spatio-temporal
reasoning abilities. This is further reflected in its high scores on Embodied Tasks (the best) and Art
& Sports, both of which involve dense spatio-temporal information, as shown in Table 3.

Study on MLLM Performance at Different Difficulty Levels for Average Humans

Figure 4a indicate some correlation between the difficulty levels as perceived by humans and the per-
formance of MLLMs. The difficulty levels are defined based on the average human performance.
MLLMs generally follow a trend where accuracy decreases as the difficulty level increases, which
aligns with human performance patterns. However, the correlation is not perfect, suggesting that
while models and humans share some common ground in understanding question difficulty, there
are also notable differences in their capabilities. The data reveals that MLLMs exhibit different skill
sets compared to humans. As highlighted in Figure 4b, models like GPT-4V can correctly answer
expert-level questions that humans often get wrong, particularly in disciplines such as Business and
Health & Medicine, where humans often struggle, yet they sometimes falter on easier questions,
likely due to the lack of contextual understanding. Notably, discrepancies in disciplines like Art &
Sports and Tech & Engineering highlight areas where MLLMs’ performance does not align with hu-
man results, suggesting different perception, cognition, and reasoning abilities in handling abstract
concepts. These differences suggest that MLLMs can complement human capabilities, offering po-
tential for enhanced task performance by combining the data-driven insights of models with human
intuition and contextual knowledge.

Error Analysis To gain deeper insights into the limitations of current open-sourced MLLMs and
provide guidance for developing next-generation models, we prompted the models to explain their
reasoning, particularly when errors occurred. We grouped and identified common error patterns into
seven distinct categories. We conducted a comparative test by posing the error-inducing questions
for GPT-4V to other MLLMs, as GPT-4V was used as a representative model due to its strong
performance and its ability to highlight errors common across MLLMs.

Our analysis revealed that Video-LLaVA exhibited the lowest error frequencies among open-source
MLLMs Its superior performance, particularly in reducing Visual Perception Errors (PE), Hallu-
cination Errors (HE), and Reasoning Errors (RE), can also be linked to its use of the CLIP ViT-
L/14 model in LanguageBind (Zhu et al., 2023a). In contrast, mPLUG-Owl showed higher rates
of Visual Perception Errors, possibly due to its reliance on weaker video embedder architectures.
Furthermore, VideoChat outperformed Video-LLaMA due to its GMHRA (Li et al., 2023c) module
for temporal aggregation, demonstrating the importance of effective temporal aggregation in reduc-
ing errors. Common trends across all models included frequent hallucination errors and a lack of
domain-specific knowledge, highlighting the need for accurate, noise-free training data and sug-
gesting that techniques like Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al.,
2022) could help mitigate these issues. While current MLLMs demonstrate strong multi-disciplinary

9

EasyMediumHardExpert0.10.20.30.40.50.60.7ChatUniviClaude-3.5-SonnetGemini ProGPT-4VGPT-4oImageBind-LLMLWMmPLUG-OwlOtterPandaGPTVideo-ChatVideo-ChatGPTVideo-LLAMAVideo-LLAVAX-Instruct-BLIPEasyMediumHardExpertBusinessEmbodiedTasksGameHealth& MedicineScienceSports& ArtsTech& Engineering9285848060874651899169637178776483717360263639326059616930405060708090Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 5: The frequency of different error types across various MLLMs. For each error type, 100
examples were evaluated. Error types are abbreviated as follows: QUE (Question Understanding
Error), AUE (Audio Understanding Error), VPE (Visual Perception Error), HE (Hallucination Er-
ror), RE (Reasoning Error), LDK (Lack of Domain Knowledge), and RA (Reject to Answer).

Table 5: Performance on Synthetic Subset I (Audio) and II (Visual). Synthetic Subset I contains
QAs based solely on the audio content, while Synthetic Subset II focuses exclusively on the visual
content of the video. We evaluated four MLLMs processing both audio and visual inputs along with
Gemini Pro (for the audio setting, only providing the question).

Model

Art&Sports

Business

Science

Health&Medicine Embodied Tasks Tech&Engineering

Game

Average

Audio Visual Audio Visual Audio Visual Audio

Visual

Audio Visual Audio

Visual

Audio Visual Audio Visual

Random Choice
Video-Chat (Li et al., 2023c)
ChatUnivi (Jin et al., 2023)
Video-LLaMA (Zhang et al., 2023a)
Otter (Li et al., 2023a)
Gemini Pro (Team et al., 2023)

31.59
33.98
30.03
30.15
14.22
20.88

30.14
32.48
43.22
30.23
16.82
61.38

31.18
46.47
30.19
36.18
16.77
29.43

26.58
41.46
52.85
33.17
14.24
77.35

36.98
41.86
38.75
31.33
16.12
30.62

32.89
39.15
54.59
31.34
17.00
74.26

38.74
45.95
34.76
30.90
19.82
30.14

32.64
36.81
50.69
32.78
13.19
81.53

32.81
32.81
20.14
33.13
10.94
22.57

31.25
46.88
40.63
30.05
12.50
70.31

27.23
37.48
24.17
31.18
15.63
18.83

32.60
35.91
46.41
30.55
12.43
66.22

32.01
32.98
29.98
20.49
6.65
29.96

30.78
46.70
45.44
27.20
10.44
65.01

32.44
38.82
31.82
29.08
12.83
24.45

30.91
39.07
48.44
30.47
13.41
69.97

world knowledge, they could benefit from enhanced domain-specific expertise, potentially through
retrieval-based methods. Detailed qualitative examples and further analysis are provided in the Ap-
pendix.

4.4 STUDY ON MODALITY OF PERCEPTION ON SYNTHETIC DATA

We conducted ablation studies to evaluate how well MLLMs can perceive the world when limited to
a single modality (audio or visual) using the synthetic dataset of MMVU. In these experiments, we
isolated scenarios where only one modality—either audio or visual—was available. Table 5 presents
the results, which assess the models’ ability to interpret spoken language, background noises, and
other audio elements without visual context, as well as their visual perception without any audio
input. For the visual perception test, Gemini Pro performed the best, demonstrating its strong abil-
ity to process visual information. Interestingly, Video-Chat exhibited better audio perception than
ChatUnivi, despite its poorer visual perception. This may be attributed to its use of the Whis-
per (Radford et al., 2022) speech recognition model. It also explains that in Table 3, Video-Chat
outperforms ChatUnivi in the Art & Sports discipline, which requires a greater understanding of
music, voice, and background audio. However, in other disciplines such as Science and Health &
Medicine, Video-Chat’s performance is significantly worse.

5 CONCLUSION

Our MMVU Benchmark represents a significant step forward in the quest for advanced multi-modal
language models capable of understanding complex video content. By presenting a diverse array
of videos across seven disciplines, accompanied by questions that challenge models to demonstrate
explanation, counterfactual thinking, future prediction, and domain expertise, we have created a
rigorous testing ground for the next generation of AI. While using LLMs for data generation can
introduce hallucination issues, these challenges are manageable and are commonly addressed (Wang
et al., 2024c; Shen et al., 2023). Another potential risk is the misuse of MLLMs for surveillance or
privacy invasion. The ability of models to understand video content and perform reasoning could be
exploited to monitor individuals without their consent, leading to serious ethical and legal concerns
regarding privacy.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Ethics Statement
In line with the ICLR Code of Ethics, we acknowledge our responsibility to
adhere to ethical principles throughout the entirety of our research. Our work does not involve hu-
man subjects, and the datasets we used are available in the submitted supplementary material and
do not raise any concerns regarding privacy or security issues. The evaluation of models in this
paper focuses on publicly available multimodal larger language models, and no sensitive or per-
sonally identifiable information was involved in this process. While our work benchmarks existing
multimodal large language models via multi-discipline, multi-faceted world model evaluation, we
recognize the potential risks of unintended bias and fairness issues in these models, which may have
inherited biases from their training data. We encourage future research to address these concerns in
the development of more inclusive and fair models. There are no conflicts of interest or sponsorship
influencing this research, and our work fully complies with legal and ethical standards.

Reproducibility Statement To ensure the reproducibility of our work, we provide extensive de-
tails on our methodology, datasets, and evaluation setup in the main paper and the Appendix. The
datasets used are available in the supplementary material, and their collection and annotation steps
are described in Section 3 of the paper. We also provide detailed descriptions of the experimen-
tal setup, including hyperparameters, model architectures, in the main paper and Appendix. All
evaluation protocols and metrics are explained to facilitate replication of our results.

REFERENCES

Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan
Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification
benchmark. arXiv preprint arXiv:1609.08675, 2016.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos,
Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark,
Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark
Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang,
Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad-
bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo-
pher A. Choquette-Choo, Aakanksha Chowdhery, Cl´ement Crepy, Shachi Dave, Mostafa De-
hghani, Sunipa Dev, Jacob Devlin, Mark D´ıaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu
Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy
Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy
Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy,
Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li,
Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar-
cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary
Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex
Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros,
Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov,
David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli,
Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun-
han Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang
Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.

Anthropic.

Introducing the next generation of Claude. https://www.anthropic.com/

news/claude-3-family, 2024. Accessed: 2024-07-29.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani
Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-
arXiv preprint
source framework for training large autoregressive vision-language models.
arXiv:2308.01390, 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv preprint arXiv:2308.12966, 2023a.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local-
ization, text reading, and beyond, 2023b.

Max Bain, Arsha Nagrani, G¨ul Varol, and Andrew Zisserman. Frozen in time: A joint video and
image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision,
2021.

Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hong-
dong Li, and Stephen Gould. The ikea asm dataset: Understanding people assembling furniture
through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Appli-
cations of Computer Vision, pp. 847–859, 2021.

Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard-
ner, Rohan Taori, and Ludwig Schimdt. Visit-bench: A benchmark for vision-language instruction
following inspired by real-world use. arXiv preprint arXiv:2308.06595, 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn,
Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics
transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Kr-
ishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large lan-
guage model as a unified interface for vision-language multi-task learning, 2023.

William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide
promptable representations for reinforcement learning. arXiv preprint arXiv:2402.02651, 2024.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An
open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//lmsys.org/blog/2023-03-30-vicuna/.

Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu
Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv
preprint arXiv:2311.03287, 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language
models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023.

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.

arXiv preprint arXiv:1810.04805, 2018.

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, and
Xin Eric Wang. Muffin or chihuahua? challenging large vision-language models with multipanel
vqa, 2024.

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen.
Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv
preprint arXiv:2406.14515, 2024.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu
Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation
benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023a.

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu
Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua-
tion benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024.

Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yang Fu, Jong-Chyi Su, William Yang Wang, and Sean
Bell. Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked
Video Generation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Google. Bard - chat based ai tool from google, powered by palm 2. https://bard.google.com/?hl=en,

2023.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird-
har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in
3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 18995–19012, 2022.

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang
Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An
advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-
language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2024.

David Ha and J¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.

Veronika Hackl, Alexandra Elena M¨uller, Michael Granitzer, and Maximilian Sailer.

Is gpt-4 a
reliable rater? evaluating consistency in gpt-4 text ratings. arXiv preprint arXiv:2308.02575,
2023.

Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu,
Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint
arXiv:2309.03905, 2023.

Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay
Sonar, Michael Saxon, and William Yang Wang. Let’s think frame by frame with vip: A video
infilling and prediction dataset for evaluating video chain-of-thought, 2023.

Ting-Yao Hsu, Chieh-Yang Huang, Ryan Rossi, Sungchul Kim, C Lee Giles, and Ting-Hao K
Huang. Gpt-4 as an effective zero-shot evaluator for scientific figure captions. arXiv preprint
arXiv:2310.15405, 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap-
lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier,
L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas
Wang, Timoth´ee Lacroix, and William El Sayed. Mistral 7b, 2023.

Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. Comclip: Training-free compositional

image and text matching. arXiv preprint arXiv:2211.13854, 2022.

Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified vi-
sual representation empowers large language models with image and video understanding. arXiv
preprint arXiv:2311.08046, 2023.

Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Par-
calabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, et al. Vilma: A zero-shot
benchmark for linguistic and temporal grounding in video-language models. arXiv preprint
arXiv:2311.07022, 2023.

Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Fed-
erico Tombari, Fahad Shahbaz Khan, and Salman Khan. Complex video reasoning and robustness
evaluation suite for video-lmms. arXiv preprint arXiv:2405.03690, 2024.

Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open

Review, 62(1), 2022.

Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video

question answering. arXiv preprint arXiv:1809.01696, 2018.

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A
multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele,
and Chen Chen. Sports-qa: A large-scale video question answering benchmark for complex and
professional sports. arXiv preprint arXiv:2401.01505, 2024.

Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence
In Proceedings of the IEEE/CVF

and commonsense reasoning for video question-answering.
Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
arXiv preprint

image pre-training with frozen image encoders and large language models.
arXiv:2301.12597, 2023b.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang,
and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355,
2023c.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen,
Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video under-
standing benchmark. arXiv preprint arXiv: 2311.17005, 2023d.

Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou,
Xin Eric Wang, William Yang Wang, et al. Value: A multi-task benchmark for video-and-
language understanding evaluation. arXiv preprint arXiv:2106.04632, 2021.

Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou.
Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models.
arXiv preprint arXiv:2311.17404, 2023e.

Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united
visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023a.

Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin-
peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal
grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2794–2804, 2023b.

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating
hallucination in large multi-modal models via robust instruction tuning. In Proceedings of the
International Conference on Learning Representations, 2024a.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and

language with ringattention. arXiv preprint arXiv:2402.08268, 2024b.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning. arXiv preprint arXiv:2310.03744, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv

preprint arXiv:2304.08485, 2023b.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg
evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023c.

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun,
arXiv preprint

and Lu Hou. Tempcompass: Do video llms really understand videos?
arXiv:2403.00476, 2024c.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin-
guistic representations for vision-and-language tasks. Advances in neural information processing
systems, 32, 2019.

Yujie Lu, Xiujun Li, William Yang Wang, and Yejin Choi. Vim: Probing multimodal large language

models for visual embedded instruction following, 2023.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Wang, Yejin Choi, and Yuchen Lin. Wild-
vision arena: Benchmarking multimodal llms in the wild, February 2024. URL https:
//huggingface.co/spaces/WildVision/vision-arena/.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt:
Towards detailed video understanding via large vision and language models. In Proceedings of
the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench-
mark for very long-form video language understanding. Advances in Neural Information Process-
ing Systems, 36:46212–46244, 2023.

Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Alek-
sandra Faust, Clement Farabet, and Shane Legg. Levels of agi: Operationalizing progress on the
path to agi. arXiv preprint arXiv:2311.02462, 2023.

Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan.
Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language
models. arXiv preprint arXiv:2311.16103, 2023a.

Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan.
Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language
models. arXiv preprint arXiv:2311.16103, 2023b.

OpenAI. Gpt-4: Technical report. arXiv preprint arXiv:2303.08774, 2023a.

OpenAI. Gpt-4v(ision) system card. https://openai.com/research/gpt-4v-system-card, 2023b.

OpenAI. Gpt-4 technical report, 2023c.

OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed:

2024-07-29.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol-
low instructions with human feedback. Advances in neural information processing systems, 35:
27730–27744, 2022.

Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio
Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning
x-modal instruction-aware representations to llms and emergent cross-modal reasoning. arXiv
preprint arXiv:2311.18799, 2023.

Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics

models. arXiv preprint arXiv:2012.02419, 2020.

Viorica P˘atr˘aucean, Lucas Smaira, Ankush Gupta, Adri`a Recasens Continente, Larisa Markeeva,
Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Do-
ersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak,
Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen,
Andrew Zisserman, and Jo˜ao Carreira. Perception test: A diagnostic benchmark for multi-
In Advances in Neural Information Processing Systems, 2023. URL
modal video models.
https://openreview.net/forum?id=HYEGXFnPoq.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, C. McLeavey, and I. Sutskever. Ro-
bust speech recognition via large-scale weak supervision. International Conference on Machine
Learning, 2022. doi: 10.48550/arXiv.2212.04356.

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ”do anything now”:
Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv
preprint arXiv: 2308.03825, 2023.

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to

instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023.

15

Under review as a conference paper at ICLR 2025

Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja
Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings
of the IEEE conference on computer vision and pattern recognition, pp. 4631–4640, 2016.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly
capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda-
tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Cˆot´e, Peter Clark, and
arXiv preprint

Peter Jansen. Can language models serve as text-based world simulators?
arXiv:2406.06485, 2024a.

Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A
large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of
the IEEE/CVF International Conference on Computer Vision, pp. 4581–4591, 2019.

Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon,
Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for mul-
timodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529,
2024b.

Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer:
Evaluating safeguards in LLMs. In Yvette Graham and Matthew Purver (eds.), Findings of the
Association for Computational Linguistics: EACL 2024, pp. 896–911, St. Julian’s, Malta, March
2024c. Association for Computational Linguistics. URL https://aclanthology.org/
2024.findings-eacl.61.

Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark
In Thirty-fifth Conference on Neural Information

for situated reasoning in real-world videos.
Processing Systems Datasets and Benchmarks Track (Round 2), 2021.

Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua
Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Pandora: Towards
general world model with natural language actions and video states. 2024.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-
answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com-
puter vision and pattern recognition, pp. 9777–9786, 2021.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.
Video question answering via gradually refined attention over appearance and motion. In Pro-
ceedings of the 25th ACM international conference on Multimedia, pp. 1645–1653, 2017.

and Yong Rui.

Jun Xu, Tao Mei, Ting Yao,

on Computer Vision

for bridging video and language.

A large video descrip-
In IEEE International Confer-
tion dataset
ence
IEEE International
2016.
June
Conference
https://www.microsoft.com/en-us/research/publication/
URL
msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/.

and Pattern Recognition

and Pattern Recognition

on Computer Vision

(CVPR),

(CVPR).

Msr-vtt:

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen
Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models
with multimodality. arXiv preprint arXiv:2304.14178, 2023.

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B.
Tenenbaum. CLEVRER: collision events for video representation and reasoning. In ICLR, 2020.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang,
and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv
preprint arXiv:2308.02490, 2023.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-
qa: A dataset for understanding complex web videos via question answering. In AAAI, pp. 9127–
9134, 2019a.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-
qa: A dataset for understanding complex web videos via question answering. In Proceedings of
the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127–9134, 2019b.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens,
Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal
understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023.

Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and
Min Sun. Leveraging video descriptions to learn video question answering. Proceedings of the
AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/aaai.v31i1.11238.
URL https://ojs.aaai.org/index.php/AAAI/article/view/11238.

Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language

model for video understanding. arXiv preprint arXiv:2306.02858, 2023a.

Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan,
William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as a generalist evaluator for vision-
language tasks, 2023b.

Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language genera-

tion via generative vokens. arXiv preprint arXiv:2310.02239, 2023.

Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang,
Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-
modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023a.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En-
hancing vision-language understanding with advanced large language models. arXiv preprint
arXiv:2304.10592, 2023b.

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

A OVERVIEW OF THE APPENDIX

This Appendix is organized as follows:

• Section B contains additional experimental results;

• Section C contains the implementation details;

• Section D contains the settings and results from human evaluations;

• Section E contains the error analysis;

• Section F contains the data examples from MMVU;

• Section G contains additional data statistics of MMVU;

B ADDITIONAL RESULTS

B.1 RESULTS ACROSS DIFFERENT SEED FOR EACH MODEL

In Table 6, we show detailed results using three different seeds for each evaluated models.

Table 6: Detailed results (%) of model performance, measured as accuracy percentages across di-
verse disciplines for three runs. The random choice baseline involves shuffling candidate answers
for each video question before consistently selecting answer ‘a’. GPT-4V and Gemini Pro utilize 10
image frames extracted from the video content.

Model

GPT-4o-seed 1 (OpenAI, 2024)
GPT-4o-seed 2 (OpenAI, 2024)
GPT-4o-seed 3 (OpenAI, 2024)
Claude-3.5-seed 1 (Anthropic, 2024)
Claude-3.5-seed 2 (Anthropic, 2024)
Claude-3.5-seed 3 (Anthropic, 2024)
GPT-4V-seed 1 (OpenAI, 2023b)
GPT-4V-seed 2 (OpenAI, 2023b)
GPT-4V-seed 3 (OpenAI, 2023b)
Gemini Pro-seed 1 (Team et al., 2023)
Gemini Pro-seed 2 (Team et al., 2023)
Gemini Pro-seed 3 (Team et al., 2023)
Video-LLaVA-seed 1 (Lin et al., 2023a)
Video-LLaVA-seed 2 (Lin et al., 2023a)
Video-LLaVA-seed 3 (Lin et al., 2023a)
Video-Chat-seed 1 (Li et al., 2023c)
Video-Chat-seed 2 (Li et al., 2023c)
Video-Chat-seed 3 (Li et al., 2023c)
mPLUG-Owl-seed 1 (Ye et al., 2023)
mPLUG-Owl-seed 2 (Ye et al., 2023)
mPLUG-Owl-seed 3 (Ye et al., 2023)
ChatUnivi-seed 1 (Jin et al., 2023)
ChatUnivi-seed 2 (Jin et al., 2023)
ChatUnivi-seed 3 (Jin et al., 2023)
Video-ChatGPT-seed 1 (Maaz et al., 2024)
Video-ChatGPT-seed 2 (Li et al., 2023c)
Video-ChatGPT-seed 3 (Li et al., 2023c)
PandaGPT-seed 1 (Su et al., 2023)
PandaGPT-seed 2 (Su et al., 2023)
PandaGPT-seed 3 (Su et al., 2023)
ImageBind-LLM-seed 1 (Han et al., 2023)
ImageBind-LLM-seed 2 (Han et al., 2023)
ImageBind-LLM-seed 3 (Han et al., 2023)
X-Instruct-BLIP-seed 1 (Panagopoulou et al., 2023)
X-Instruct-BLIP-seed 2 (Panagopoulou et al., 2023)
X-Instruct-BLIP-seed 3 (Panagopoulou et al., 2023)
LWM-seed 1 (Liu et al., 2024b)
LWM-seed 2 (Liu et al., 2024b)
LWM-seed 3 (Liu et al., 2024b)
Otter-seed 1 (Li et al., 2023a)
Otter-seed 2 (Li et al., 2023a)
Otter-seed 3 (Li et al., 2023a)
Video-LLaMA-seed 1 (Zhang et al., 2023a)
Video-LLaMA-seed 2 (Zhang et al., 2023a)
Video-LLaMA-seed 3 (Zhang et al., 2023a)

Art&
Sports
47.10
46.58
49.94
54.32
54.32
55.10
36.90
35.48
36.13
40.90
35.10
35.35
34.58
36.77
36.39
39.48
39.48
39.61
31.35
28.65
27.48
24.13
25.16
24.13
26.71
27.74
27.74
26.06
24.77
25.16
24.77
25.03
24.65
21.42
20.77
21.03
11.35
12.13
12.65
18.45
17.29
15.61
5.55
6.58
6.32

Health& Embodied
Medicine
81.25
84.38
84.38
53.12
54.17
56.25
73.96
73.96
72.92
78.12
75.00
77.08
32.29
32.29
33.33
46.88
45.83
45.83
61.46
58.33
60.42
62.50
62.50
58.33
53.12
53.12
53.12
41.67
34.38
39.58
31.25
31.25
28.12
29.17
28.12
28.12
19.79
19.79
21.88
8.33
6.25
6.25
29.17
31.25
33.33

Tasks
65.03
65.73
58.04
30.77
30.77
31.47
51.75
58.04
56.64
43.36
44.06
43.36
61.54
65.03
62.94
39.86
41.26
40.56
28.67
21.68
20.98
48.95
44.06
45.45
37.76
39.86
39.86
35.66
34.27
36.36
46.85
45.45
48.25
16.78
20.28
18.18
24.48
24.48
28.67
14.69
13.99
11.19
15.38
13.99
16.78

Tech&
Engineering
72.34
75.53
78.72
59.57
58.51
58.51
60.64
60.64
62.77
71.28
71.28
67.02
57.45
57.45
59.57
39.36
39.36
39.36
48.94
37.23
39.36
56.38
58.51
55.32
41.49
43.62
43.62
39.36
40.43
45.74
43.62
40.43
40.43
21.28
22.34
23.40
24.47
22.34
19.15
15.96
18.09
13.83
21.28
17.02
19.15

Game

Average

78.31
83.13
79.52
59.04
59.04
60.24
71.08
75.90
73.49
65.06
69.88
63.86
50.60
51.81
44.58
44.58
45.78
44.58
65.06
57.83
63.86
56.63
50.60
50.60
34.94
39.76
39.76
42.17
44.58
33.73
40.96
40.96
42.17
26.51
25.30
26.51
10.84
8.43
15.66
14.46
15.66
15.66
18.07
32.53
24.10

62.22
61.77
63.63
54.27
54.52
54.84
51.64
52.79
52.47
52.92
50.16
49.97
43.94
45.35
44.52
40.03
40.15
40.15
41.05
37.52
38.23
39.77
39.77
38.87
32.59
34.64
34.64
32.97
31.88
32.58
31.62
31.69
31.94
21.23
21.62
21.23
15.20
15.14
15.84
15.84
15.14
13.98
13.66
14.05
14.37

Business

Science

75.11
69.78
76.44
59.11
61.33
59.11
64.00
68.44
67.11
60.44
63.11
64.89
57.33
56.00
55.56
30.67
30.67
31.11
45.78
44.44
52.00
52.00
51.11
52.89
34.67
36.89
36.89
38.22
36.89
43.11
30.67
32.44
33.33
22.22
24.00
21.33
16.44
15.56
14.22
8.89
9.33
9.78
24.00
20.44
22.22

92.31
90.91
90.21
64.34
63.64
63.64
79.72
83.92
81.12
79.72
75.52
74.83
51.05
52.45
50.35
51.05
51.05
51.05
65.73
65.03
61.54
60.14
62.94
59.44
34.97
41.96
41.96
44.06
45.45
38.46
41.96
41.96
44.06
14.69
16.78
16.08
18.18
17.48
16.78
19.58
17.48
18.88
21.68
20.28
21.68

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Table 7: Performance (%) of different set of turkers

Model

Turker Set 1
Turker Set 2
Turker Set 3

Art&
Sports
25.224
30.452
26.710

Business

Science

39.860
46.154
41.958

32.444
35.556
36.889

Health& Embodied
Medicine
40.625
42.708
46.875

Tasks
51.049
53.846
53.147

Tech&
Engineering
50.000
51.064
42.553

Game& Average

40.964
46.988
38.554

33.227
37.652
34.830

Table 8: Performance (%) of different MLLMs across different disciplines.

Model

Video-Chat (Open-ended) (Li et al., 2023c)
Video-Chat (Li et al., 2023c)
Video-LLaMA (Open-ended) (Zhang et al., 2023a)
Video-LLaMA (Zhang et al., 2023a)
ChatUnivi (Open-ended) (Jin et al., 2023)
ChatUnivi (Jin et al., 2023)
Otter (Open-ended) (Li et al., 2023a)
Otter (Li et al., 2023a)
ImageBind-LLM (Open-ended) (Han et al., 2023)
ImageBind-LLM (Han et al., 2023)
PandaGPT (Open-ended) (Su et al., 2023)
PandaGPT (Su et al., 2023)
LWM (Open-ended) (Liu et al., 2024b)
LWM (Liu et al., 2024b)
X-Instruct-BLIP (Open-ended) (Panagopoulou et al., 2023)
X-Instruct-BLIP (Panagopoulou et al., 2023)

Art&
Sports
27.484
39.355
5.419
27.355
21.161
12.387
37.677
17.677
3.355
23.742
22.581
27.613
16.000
16.387
3.613
19.355

Business

Science

9.091
48.951
27.972
31.469
61.538
58.042
32.867
16.783
3.497
34.965
16.084
44.056
20.979
18.182
11.888
13.287

18.137
31.863
24.020
31.373
42.157
50.000
37.255
12.255
14.706
51.471
24.020
39.706
14.706
18.137
14.706
22.549

Health& Embodied
Medicine
10.417
45.833
31.250
48.958
61.458
60.417
32.292
5.208
10.417
33.333
21.875
25.000
16.667
19.792
25.000
29.167

Tasks
29.371
39.161
11.816
16.084
30.070
30.070
22.378
17.483
21.678
48.951
19.580
40.559
19.580
22.378
17.483
18.881

Tech&
Engineering
19.149
38.298
15.957
28.723
37.234
43.617
27.660
15.957
18.085
56.383
21.277
21.277
20.213
21.277
13.830
14.894

Average

22.887
39.588
16.096
28.729
32.646
29.072
34.639
15.876
8.179
33.952
21.718
31.615
16.976
17.938
9.416
19.519

B.2 RESULTS FROM AMAZON TURKERS

Table 7 presents the evaluation results from three sets of Amazon Turkers across various disciplines.
The results indicate that there is slightly variability in performance across different human evalua-
tors.

B.3 RESULTS FOR THE TWO DIFFERENT EVALUATION STRATEGIES

In Table 8, we give additional evaluation results for different MLLMs evaluated in this paper. For
closed-source models, the evaluation pipeline is the one used in the main paper, which involves
utilizing GPT-4V as a judger. The process consists of presenting GPT-4V with the question, a
corresponding answer generated by the baseline model, and the set of possible options. GPT-4V
then assesses whether the model-generated answer is accurate within the given context; Another is
open-ended generation where we employ a two-step methodology. We first prompt each model to
do open-ended generation. Subsequently, we prompt the model to align its generative response with
one of the predefined options: ‘a’, ‘b’, ‘c’, or ‘d’.

B.4 ADDITIONAL EVALUATION RESULTS USING OPEN-SOURCED EVALUATOR AS THE

EVALUATOR

In addition to GPT-4V, we also experimented with using the open-sourced Video-LLaVA model
as an evaluator. The average accuracy of various models evaluated with this method is shown in
Table 10. The rankings are consistent with those obtained using GPT-4V in the main paper, high-
lighting the versatility of our benchmark, which supports multiple evaluator options beyond GPT-4V.

B.5 ADDITIONAL TEMPORAL REASONING EXPERIMENTS

To better understand the impact of temporal coherence on reasoning tasks of different models, we
conducted two experiments focused on temporal reasoning. These experiments were designed to
analyze model performance under varying temporal constraints, including reduced video frames
and shuffled video frames.

• Reduced Video Frames: Videos were processed by reducing the number of frames to 1/5
of the original. This setting evaluates the models’ ability to reason with limited temporal
information.

19

Under review as a conference paper at ICLR 2025

Table 9: Performance (%) of MLLMs on temporal reasoning tasks under different conditions.

Model

Original Videos

Shuffled Videos Reduced Video Frames

GPT-4o (OpenAI, 2024)
GPT-4V (OpenAI, 2023b)
Claude-3.5-Sonnet (Anthropic, 2024)
Gemini Pro (Team et al., 2023)
Video-LLaVA (Lin et al., 2023a)
Video-Chat-7B (Li et al., 2023c)
Video-ChatGPT-7B (Maaz et al., 2024)
ImageBind-LLM-7B (Han et al., 2023)
PandaGPT-7B (Su et al., 2023)
ChatUnivi-7B (Jin et al., 2023)
Video-LLaMA-2-13B (Zhang et al., 2023a)
X-Instruct-BLIP-7B (Panagopoulou et al., 2023)
LWM-1M-JAX (Liu et al., 2024b)
Otter-7B (Li et al., 2023a)
mPLUG-Owl-7B (Ye et al., 2023)

40.90
27.17
25.77
24.65
34.45
25.77
23.53
19.89
28.01
22.97
6.16
11.20
7.00
9.52
20.17

35.11
22.04
21.58
20.19
18.47
21.50
21.62
16.19
24.35
19.41
5.02
9.88
5.75
3.25
18.19

32.19
22.33
19.45
18.97
28.50
20.19
20.17
14.98
22.57
17.14
4.58
8.95
5.56
7.93
16.59

Table 10: Performance of different models across evaluations using Video-LLaVA as the evaluator.

Model

Accuracy (%)

Video-Chat-7B (Li et al., 2023c)
ChatUnivi-7B (Jin et al., 2023)
mPLUG-Owl-7B (Ye et al., 2023)
PandaGPT-7B (Su et al., 2023)
ImageBind-LLM-7B (Han et al., 2023)
X-Instruct-BLIP-7B (Panagopoulou et al., 2023)
LWM-1M-JAX (Liu et al., 2024b)
Otter-7B (Li et al., 2023a)
Video-LLaMA-2-13B (Zhang et al., 2023a)

41.96
39.81
38.01
31.66
31.65
22.02
16.81
12.08
10.84

• Shuffled Video Frames: Videos were processed by shuffling their frames. This setting

tests the models’ ability to reason when the temporal order of the frames is disrupted.

The results of these experiments are summarized in Table 9. From Table 9, there is a significant per-
formance drop when videos are either reduced in frame count or shuffled. These findings highlight
the sensitivity of models to temporal coherence and emphasize the necessity of maintaining suffi-
cient temporal information for accurate reasoning. Notably, proprietary models such as GPT-4o and
GPT-4V demonstrate better resilience under these settings compared to most open-source models.

C IMPLEMENTATION DETAILS

We use the optimum number of video frames and report the performance in the main paper. The
numbers of the sampled frames are 10 for GPT-4V/o and Gemini Pro, 8 for Video-LLaVA, 32
for ChatUniVi. For closed-source models, for both Gemini Pro and GPT-4V, we use the default
settings provided by their official APIs. We use Katna 3 to extract key video frames as input to
these two models. The Gemini Pro is set to process visual input and configured with safety settings
to filter a range of harmful content. The configuration thresholds are set to ‘BLOCK NONE’. For
PandaGPT, we set ‘top p’ to 0.7, and ‘temperature’ to 0.5. For VideoChat, we set ‘max frames’ to
100. For LWM, we use the LWM-Chat-1M variant. For X-Instruct-BLIP, the model is implemented
using four image frames. For Otter, we use the video variant. We use GPT-4-32K as the judge
for judging whether the model answer is correct when it can not mapped to the option letter
using the rule-based method. The prompt provided to GPT-4-32K is structured as follows: "I
will present a response from a question-answering model alongside
several answer options. Your task is to evaluate the response and
determine which of the following options it most closely aligns

3https://github.com/keplerlab/katna

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Figure 6: The interface of using Amazon Mechanical Turk to do human evaluation.

Table 11: Category-wise and overall error rates

Category
Sports & Arts
Health & Medicine
Science
Robotics
Business
Tech & Engineering
Overall

Incorrect/Total Error Rate (%)

5/62
2/7
1/52
0/12
0/10
1/46
9/189

8.06
28.57
1.92
0.00
0.00
2.17
4.76

with, denoting the most similar option by its corresponding letter
(a, b, c, or d).".

Query Generation in Synthetic Data Generation Pipeline For the discipline of Science, queries
are generated for subdisciplines such as Geography, Chemistry, Wildlife Restoration, Mycology, Na-
ture, Physics, Weather, Zoology, Math, Botany, Biology, and Geology. In the Tech & Engineering
discipline, our queries span across Electronics, Animal Behavior, Mechanical Engineering, Energy
& Power, Architecture, Agriculture, Nature, Physics, Robotics, Woodworking, and Gardening. The
Sports & Arts discipline encompasses a broad range of cultural and physical activities, including
Music, Drawing and Painting, Football, Volleyball, Aerobic Gymnastics, Basketball, Instrument,
Baking, Dance, Woodworking, Graffiti, Anatomy, and additional Music-related topics. Embodied
Tasks are represented through queries for Assembly, Ego-motion, and Single Object Manipulation,
focusing on the interaction between agents and their physical environment. The Health & Medicine
discipline is segmented into Pharmacy, Public Health, Clinical Medicine, and Basic Medical Sci-
ence, reflecting the multifaceted nature of healthcare and medical studies. The Business discipline
is stratified into fundamental areas such as accounting, finance, management, marketing, and eco-
nomics, each representing key facets of the commercial and economic world. Lastly, the Game
discipline consists of Role Playing Game, First Person Shooting game, Racing Game, Adventure
Game, Real-Time Strategy Game, Tower Defense game, and Fighting Game.

Each generated query retrieves relevant video content, which is then filtered and processed to align
with the specific needs of our research objectives. Videos that meet our criteria in terms of con-
tent, length, and quality are downloaded and incorporated into our dataset, forming the basis for
subsequent analysis and model training.

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Figure 7: Human evaluation interface for GPT judger.

D HUMAN EVALUATION

D.1 QUALITY OF DATA

We hired Amazon Mechanical Turk to do human evaluation on the data with the results shown in
Table 7. Workers were required to have completed more than 1000 Human Intelligence Tasks (HITs)
and have an HIT approval rate greater than 95% to qualify for our tasks. We show in Figure 6 the
human evaluation interface on the generated data. Each worker was compensated 0.20 for complet-
ing an assignment. This amount was determined based on the estimated time and effort required to
complete each task. We set the number of unique workers per task to 3 to collect diverse perspec-
tives while avoiding redundancy. Workers were given 1 hour to complete each assignment. This
time frame was chosen to enable thoughtful responses from workers.

We also hired students from campus to do human evaluation on subset of the data. The results are
shown in Table 12. The performance of the human evaluators did not surpass that of GPT-4V and
Gemini-Pro. This outcome underscores the challenging nature of the dataset, which often necessi-
tates specialized domain knowledge that our evaluators—primarily non-experts—found demanding.

22

Under review as a conference paper at ICLR 2025

Figure 8: Examples from MMVU in the Embodied Tasks discipline.

Figure 9: Examples from MMVU in the Tech & Engineering discipline.

These results highlight the complexity of the questions and the potential necessity for discipline-
specific understanding to achieve high accuracy

D.2 QUALITY OF USING GPT AS THE JUDGER

For a comprehensive assessment of GPT-4V’s accuracy when using it as the judger, we devised a
human evaluation protocol also resort to Amazon Mechanical Turk, as visualized in Figure 7. The
evaluators present a series of statements derived from the video, and GPT-4V is tasked with selecting
the most accurate answer from a set of multiple-choice questions. Through this interface, human
evaluators can efficiently gauge GPT-4V’s performance across different types of questions—when
using it as the judger.

The results obtained from this human evaluation process are shown in Table 11, across 189 examples,
there are only 9 incorrect ones with the error rate of 4.76%, validating the effectiveness of using
GPT-4V as the judger.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Figure 10: Examples from MMVU in the Science discipline.

Figure 11: Examples from MMVU in the Business discipline.

E ERROR ANALYSIS

In this section, we delve into the analysis of errors from evaluated MLLMs. We summarized error
types as follows:

Question Understanding Error (QUE): Models misinterpret the question’s intent, such as misun-
derstanding how a pendulum’s period would change if a condition in the scenario is altered.

Audio Understanding Error (AUE): Models fail to interpret audio cues correctly, shown by their
failure to recognize blue and red lines on a stock chart.

Visual Perception Error (VPE): There is a misinterpretation of visual content, leading to incorrect
assumptions about the visual data presented in the video.

Hallucinations (HE): Models generate content or details that are not present in the actual data,
essentially ‘hallucinating’ information.

24

Under review as a conference paper at ICLR 2025

Figure 12: Examples from MMVU in the Arts & Sports discipline.

Figure 13: Examples from MMVU of explicit temporal understanding and implicit temporal under-
standing (e.g., in explanation).

Reasoning Error (RE): Models demonstrate a lack of logical reasoning, leading to incorrect con-
clusions based on the given data.

Lack of Domain Knowledge (LDK): Models show an inability to answer questions that require
specific domain expertise, indicating a gap in their knowledge.

Reject to Answer (RA): An example of this error was observed when the model was asked to select
an answer regarding the outcome of an experiment involving liquid nitrogen. Instead of choosing
an option, the model provided an unrelated response concerning a light bulb, indicating either a
misunderstanding or a cautious approach due to the potential for the question to be interpreted as
pertaining to a sensitive topic, which can trigger content filters focused on safety and compliance
policies.

We show in Figure 18, 19, 20, 21 some error cases of Question Understanding Error, Audio Under-
standing Error, Visual Perception Error, Hallucinations, Reasoning Error, Lack of Domain Knowl-
edge, and Reject to Answer respectively from MLLMs evaluated on MMVU.

25

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Figure 14: Examples from the Sports & Arts discipline, illustrating tailored Q&A pairs.

F DATA EXAMPLES

F.1 MAIN SUBSET

We present additional examples from the main subset of MMVU in Figures 8, 9, 10, 11, 12, and 13.

26

Q: What might be the reason to start with the eyes when drawing a face? A: You want to start with the short distances which are easier to measure compared to longer distances. And then you can relate the rest of everything to the distance of the eyes.Q: What would happen if you started with the nose? A: It would be more difficult to relate the rest of the face and maintain proportional accuracy.Q: What will likely happen to the cheesecake after this video? A: The cheesecake will be eaten.Q: What influenced the making of this cheesecake? A: Spanish burnt Basque cheesecakes.Q: How will the canvas look if the experiment is left forever? A: The center of the canvas will become a solid color.Q: According to the video, what happens when the person takes their foot off the pedal? A: The hi-hat opens.Q: How many players are in the video? A: One.Q: What do the players perform before performing transition flight from high bar to low bar? A: Giant circle forward with 1 turn on one arm before handstand phase.Under review as a conference paper at ICLR 2025

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Figure 15: Examples from the Science discipline, illustrating tailored Q&A pairs.

Table 12: Comparison of Human Evaluation on subset of 75 videos.

Model

Human Evaluation
GPT-4V (OpenAI, 2023b)
Gemini-Pro (Team et al., 2023)

Art&
Sports
31.183
30.399
28.745

Business

Science

59.782
89.203
80.909

42.103
68.731
69.425

Health& Embodied
Medicine
48.858
80.059
80.023

Tasks
56.429
38.432
50.987

Tech&
Engineering
50.134
69.108
80.479

Average

43.758
48.793
48.083

Furthermore, Figures 14, 15, and 16 demonstrate how Q&A pairs in MMVU are carefully tailored
to specific disciplines, including Sports & Arts, Science, and Business. Each example is designed to
reflect the unique reasoning and understanding required within its respective discipline.

F.2 SYNTHETIC I AND SYNTHETIC II

We present in Figure 17 additional examples from Synthetic I and Synthetic II of MMVU. The
examples correspond to various disciplines: Business, Health & Medicine, Science, and Gaming,
respectively. For each discipline, the first example showcases an audio-only generated QA from
Synthetic I, while the second example represents a visual-only generated QA from Synthetic II.
These examples highlight the multi-disciplinary reasoning capabilities evaluated in our benchmark,
even for the synthetic dataset.

27

Q: How does the girl actively change her spinning speed? A: By stretching out or contracting her legs and arms.Q: What would happen if the woman let go of the rope in the middle of the video? A: She would fall onto the platform, lose her balance, and eventually fall.Q: What is the physical law that makes the girl spin faster when she contracts her legs and arms? A: Conservation of angular momentum.Q: What is the reason that the girl on the right is floating in the air? A: The woman on the left is pulling a rope connected to the girl.Q: How does the solution color change over time? A: Purple → Green → Light brown.Q: What will happen once the stirrir is turned off? A: The solution will settle, no further change.Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Figure 16: Examples from the Business discipline, illustrating tailored Q&A pairs.

G ADDITIONAL DATA STATISTICS

For human annotated dataset, the length of each video was capped at approximately two minutes.
The statistical distribution of the disciplines within the dataset for this part is as follows:

• Sports & Arts: The subset that consists of 77 videos, showcasing a vibrant collection that
covers a wide range of topics from athletic endeavors to various forms of artistic expression.

• Science: A subset of 75 videos, which delves into the empirical world of scientific inquiry,
spanning a multitude of specializations from fundamental physics to advanced biological
studies.

• Tech & Engineering: Encompassing 54 videos, this segment captures the cutting-edge ad-
vancements and foundational concepts that drive innovation and infrastructure in the mod-
ern world.

• Embodied Tasks: With 50 videos, the dataset provides a focused insight into the dynamic
field of Embodied Tasks, highlighting the intersection of AI, mechanics, and automation.

28

Q: What is the primary goal of the man showing the two coins with both sides? A: To encourage people to purchase the coins.Q: What does it tell us if the price is below the blue and red lines? A: The price is moving downward.Q: What do the blue and red lines in the video represent? A: The average price over the last 20 and 50 days.Q: What might be a possible subsequent action taken by the person in the video after drawing on the paper? A: Proceeding to enact a scene related to shopping.Q: What process is shown in the video? A: The process of making coins.Under review as a conference paper at ICLR 2025

• Health & Medicine: This essential discipline is well-represented with 50 videos, offering

perspectives on medical breakthroughs, healthcare practices, and life sciences.

• Business: This discipline includes 50 videos, reflecting on the multifaceted nature of com-

merce, from economics to management sciences.

• Game: This discipline includes 51 videos, reflecting various aspects of gaming.

Altogether, the MMVU Benchmark’s diversity is visually encapsulated in Figure 22, which delin-
eates the distribution of videos across 61 subdisciplines. The horizontal bar chart provides a quan-
tified representation of the dataset’s range, reflecting the careful curation process that has gone into
ensuring breadth across various knowledge areas.

MMWorld also has additional annotations such as ”Requires Audio”, ”Requires Video”, and ”Ques-
tion Only”. The world we live in is rich with both audio and visual information, and effective
world modeling requires an understanding of how these modalities interact and convey meaning.
To achieve this, we annotated additional attributes such as ”Requires Audio”, ”Requires Video”,
and ”Question Only” during data collection. These annotations help determine whether correctly
answering a question necessitates audio information, visual cues from the video, or can be ad-
dressed based solely on the question itself. By doing so, we ensure that our benchmark tests the full
spectrum of multimodal comprehension, reflecting the complex, sensory-rich environment in which
real-world understanding takes place. The statistics of these annotations are shown in Figure 23.

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

29

Under review as a conference paper at ICLR 2025

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Figure 17: Examples from Synthetic Subset I and Synthetic Subset II of MMVU.

30

Q: From the video, between the periods 1925-1935 and 2000-2010, which had a higher increasing amount of printed money in the United States? A: 2000-2010.Q: The speaker mentions borrowing to buy a tractor versus a TV. What reasoning does he use to explain which is better? A: Borrowing for a tractor is better because it generates income to repay the debt, while a TV does not produce income.Q: Based oj video, what substajce surroujds the orgajelles outside the jucleus, ajd what does it look like? A: The organelles are surrounded by cytoplasm, which appears as a jelly-like substance within the cell.Q: According to the speaker, what if a eukaryotic cell lost its nucleus? A: The cell would no longer be able to regulate its activities or reproduce properly since the nucleus contains the genetic material required for these functions.Q: Why does the phejomejoj ij the video occur? A: Adding sugar releases CO₂ rapidly, creating pressure that inflates the balloon.Q: According to the speaker, why does adding salt to the lower egg top help the upper egg? A: Adding salt creates a rough surface, increasing friction and providing micro-support points that stabilize the upper egg, preventing it from sliding.Q: From the video, if a player cojtijues usijg the techjique showj ij the A site, what is likely to happej ij a close-quarters battle? A: The attacking team can leverage the reduced visibility created to surprise defenders and secure eliminations while staying concealed.Q: According to the speaker, what if you take a Macbeth shot in your nightclub on a stormy night between 1 and 4 AM? A: There is a chance you will spawn on top of Mount Chiliad and see a UFO flying away.Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Figure 18: Error pattern of lack of domain knowledge and reasoning error. In the first case, the
model does not give the correct answer because of lacking the domain knowledge. In the second
case, the model makes the wrong reasoning.

Figure 19: Error pattern of question understanding, visual perceptron error, and hallucinations. In
the first case, the model does not understand the question correctly where the model fails to ac-
In the second scenario, the model
curately discern the query regarding the pendulum’s period.
erroneously identifies objects within the visual input, leading to the hallucination of non-existent
elements, such as a red box.

31

Lack of Domain KnowledgeQ:How many eggs can a frog lay at once?A:I'm sorry, but the images provided are of mushrooms in a forest setting, which do not provide any information about the egg-laying habits of frogs. ReasoningErrorQ:How do the pulleys move when the hands are off the pulley system?A:When the hands are removed from the pulley system, as shown in the series of images, there is no force being applied to any of the pulleys to cause movement. Therefore, all four pulleys remain static. Question UnderstandingErrorQ: What wouldhappen to the pendulum period if the professor sit up straight on the ball instead?A:The pendulum period would not afford it.Visual PerceptionError, HallucinationsQ:What would happen if the man skipped the step shown in the video?A:If the man skipped the step of rolling the blue tube on the floor to uncover the red box, he would likely have encountered difficulty in accessing the contents of the box. This step is important because it helps the man to create a flat surface to work with the red box, ensuring that the box is visible and accessible. Under review as a conference paper at ICLR 2025

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Figure 20: Error pattern of reject to answer and audio understanding error. In the first case, the
model will reject to answer because of the dangerous action showing in the video. In the second
case, the model does not understand the audio correctly as the answer to the question is given in the
audio of the video.

Figure 21: Error pattern due to visual perception inaccuracies and insufficient domain knowledge.
The first case demonstrates a visual perception error where the model incorrectly identifies the num-
ber of capacitors present. The second case showcases a compound error where the model not only
fails to discern the colors indicative of different gases but also lacks the domain knowledge neces-
sary to infer their identity correctly.

32

Reject to AnswerQ:What if we replace the boiling water with iced water? The question will trigger the content filter and there will be no answerAudio Understanding ErrorQ:What does the blue and red lines in the stock chart represents?A: The blue and red lines in the stock chart represent the support and resistance levels, respectively. The support level is where the stock price tends to stop falling, while the resistance level is where the stock price tends to stop rising. In the video, the blue line indicates a level of support, while the red line indicates a level of resistance.Visual Perception ErrorQ: How many capacitors are there in the final circuit?A:There are several capacitors in the final circuit. We can see at least 14 of them in the video.Visual PerceptionError, Lack of Domain KnowledgeQ:What could be the three gases in order?A: I'm unable to provide assistanceas the request requires information to be extracted from images/videos which is beyond my current capabilities. Under review as a conference paper at ICLR 2025

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Figure 22: The number of videos per subdiscipline in MMVU. Each horizontal bar indicates the
quantity of videos corresponding to a subdiscipline, showcasing the dataset’s diversity and coverage
across various domains of knowledge. Synthetic Subset I is collected with audio-only data and
Synthetic Subset II is collected with visual-only data.

33

010203040506070Number Of VideosReal-time Strategy GameElectronicsWoodworkingDanceAerobic GymnasticsGeologyDiagnosticPublic HealthRacing GamePharmacyAssemblyBiologyMoosicEconomicsInvestmentFinanceVolleyballManagementStock ChartEgo-motionRobotics BehaviorE-commerceTower Defense GameBakingWarMarketingGraffitiAnimal BehaviorMathAgricultureNatureCommercial PromotionWildlife RestorationAdventure GameRoboticsFighting GameWeatherBasic Medical ScienceChemistrySupply Chain ManagementQuantitative FinanceRole Playing GameGymnasticFootballZoologyTradingTaxationMusicMaterialsAnatomyMycologyClinical MedicineWealth ManagementFirst Person Shooting GameGeographyBotanyEnergy & PowerMechanical EngineeringInstrumentAccountingArchitecturePhysicsElectroincsSingle Object ManipulationDrawing And PaintingBasketballGardeningManual Collected SubsetSynthetic Subset ISynthetic Subset IIUnder review as a conference paper at ICLR 2025

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Figure 23: The distribution statistics of questions in the MMVU benchmark by annotations.

34

020040060080010001200CountRequiringAudioRequiringQuestionOnlyRequiringDomainKnowledgeRequiringVisualInformationAttribute