OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
Under review as a conference paper at ICLR 2025

OMNI R: EVALUATING OMNI-MODALITY LANGUAGE
MODELS ON REASONING ACROSS MODALITIES

Anonymous authors
Paper under double-blind review

ABSTRACT

We introduce Omni×R, an evaluation suite designed to benchmark state-of-the-
art Omni-modality Language Models (OLMs), such as GPT-4o and Gemini. Eval-
uating OLMs, which integrate multiple modalities such as text, vision, and audio,
presents unique challenges. Particularly, the user message might often consist of
multiple modalities, such that OLMs have to establish holistic understanding and
reasoning across modalities to accomplish the task. Existing benchmarks are lim-
ited to single-modality or dual-modality tasks (e.g., image+text or video+text),
overlooking comprehensive multi-modal assessments of model reasoning. To ad-
dress this, Omni×R offers two evaluation variants: (1) Omni×RSYNTH: a syn-
thetic dataset generated automatically by translating text into multiple modali-
(2) Omni×RREAL: a real-
ties—audio, images, video, and hybrids (Omnify!).
world dataset, manually curated and annotated by experts, for evaluating cross-
modal reasoning in natural settings. Omni×R presents a unique evaluation to-
wards assessing OLMs over a diverse mix of modalities, such as a question that in-
volves video, audio, and text, providing a rigorous cross-modal reasoning testbed
than any existing benchmarks. Our experiments ﬁnd that all state-of-the-art OLMs
struggles with Omni×R questions that require integrating information from mul-
tiple modalities to answer. Further analysis highlight differences in reasoning
behavior and underscoring the challenges of omni-modal AI alignment.

1

INTRODUCTION

Recent advances in Omni-modality Language Models (OLMs) (OpenAI, 2024b; Gemini-Team,
2024b) has pushed the boundaries of AI by enabling a more comprehensive understanding of real-
world inputs across diverse modalities, e.g., text, vision, audio, (Lu et al., 2019; Gan et al., 2020;
Akbari et al., 2021; Zellers et al., 2021) and generating outputs that are more aligned with human
communications (Lu et al., 2024; Zhang et al., 2024; Gao et al., 2024).

However, the evaluation of these sophisticated OLMs presents unique challenges. While traditional
benchmarks (Christopher Chou, 2024) have predominantly focused on models that handle single or
dual modalities, such as vision-language or video-text pairs, they fail to capture the complexities
that arise when multiple modalities are involved. In real-world scenarios, user inputs are rarely con-
ﬁned to one or two modalities. Instead, they often consist of diverse combinations of text, images,
videos, and audio, necessitating a holistic understanding and reasoning across information presented
in these modalities for OLMs to effectively perform tasks. This mismatch between existing evalua-
tion methods and the multimodal capabilities of state-of-the-art OLMs has left a signiﬁcant gap in
the assessment of these models.

One common ﬂaw in existing OLMs is their inconsistent behavior when presented with the same
question in different modalities or mixtures of modalities. Figure 1 presents an example on the Gem-
ini 1.5 Flash (Gemini-Team, 2024a) (similar behaviour also observed in other OLMs, see Section 3.2
for analysis). Particularly, when the same math question is presented different modalities, such as
rendered as image input, or spoke out as audio input, the model produces varying responses that ex-
hibit signiﬁcant performance discrepancies, i.e., different reasoning bevhiours or different answers.
This observation indicates a lack of robust cross-modal information integration and reasoning capa-
bilities in existing OLMs. Such inconsistency not only undermines the reliability of these models

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Reasoning Behavior of a OLM Varies across Modalities. Taking Gemini-1.5-Flash as an example,
on text question, the reasoning behaviour is expected and the answer is correct. When the same question is
rendered to an image, the model generate a reasonable reasoning but incorrect answer. On the video or audio
representation of the same question, the model generates no reasoning and produces incorrect answers.

but also highlights the limitations of current evaluation benchmarks that do not adequately assess
performance across diverse modality combinations.

To bridge this critical evaluation gap, we introduce Omni×R, an evaluation suite speciﬁcally de-
signed to benchmark the reasoning performance of OLMs across a wide range of modalities. Unlike
existing benchmarks that are limited to a maximum of two modalities, Omni×R provides a com-
prehensive testbed that includes complex modality combinations such as video + audio + text
and image + audio + text, offering a more rigorous and holistic evaluation of these models’
capabilities. Speciﬁcally, Omni×R contains two subsets of the data:

• Omni×RSYNTH: a synthetic reasoning dataset constructed with a scalable and low-cost automatic
method (i.e., Omnify!) to translate information embedded in text to various modalities — audio,
images, video, and hybrids of them.

• Omni×RREAL: a real-world reasoning dataset manually collected and annotated with expert an-

notators, for evaluating cross-modal reasoning in the realistic distribution.

In construction of Omni×RSYNTH, Omnify! translates text-based inputs into various other modali-
ties, such as images, audio, and video, as well as their hybrid combinations, using programmatic text
rendering services, programmatic video construction pipeline, and state-of-the-art text-to-speech
service. This scalable synthetic dataset ensures a diverse and robust dataset that challenges OLMs
to demonstrate their cross-modal reasoning abilities. Meanwhile, Omni×RREAL develops a realistic
test environment for evaluating omnimodal reasoning. Particularly, we crawled 100 Youtube videos
whose topics are related to math, physics, chemistry and coding, and manually curate, convert and
annotate the quiz questions from those videos, ensuring that each question is associated with mul-
tiple variants, each in one modality or a hybrid of many modalities. With both complementary
subsets, Omni×R allows us to better assess how well OLMs can reason across different modalities
and integrate information in a way that mirrors human-like understanding.

Our evaluation of state-of-the-art OLMs on Omni×R has yielded several important ﬁndings. No-
tably, Omni×R is the ﬁrst benchmark that quantitatively measured the aforementioned omni-modal
behaviour discrepancy, especially in scenarios requiring deep reasoning across multiple modali-
ties. Moreover, we also observe that some simple prompting strategy that exploits the underly-
ing data creation logic in Omni×RSYNTH, i.e., Extract the information and Then Answer
(ETA prompting), could signiﬁcantly improve every omini-modality language model’s behaviour
consistency and ﬁnal answer accuracy on Omni×RSYNTH. These results suggest that the main strug-
gle of current model is to establish a holistic understanding across modality, where the need to inte-
grate and reason across different forms of data becomes crucial. When evaluated on Omni×RREAL,
where the information across modality is naturally distributed and blended with noises, OLMs can
no longer rely on a simple prompting strategy to alleviate the omnimodal behaviour inconsistency,
indicating the demand of further training for future omnimodal language models.

2

The total cost of producing x cameras is C(x) = 2 + x^3. What is the average cost if 10 cameras are made? What is the marginal cost of producing 10 cameras?A. Average Cost: $1000/unit, Marginal Cost: $100B. Average Cost: $300/unit, Marginal Cost: $100 C. Average Cost: $300/unit, Marginal Cost: $1000D. Average Cost: $102/unit, Marginal Cost: $301……[Prompt] Think step by step then output the answer in the format of "The answer is (X)" at the end.[Response(to Text)] To find the average cost of producing 10 cameras given the cost function $C(x) = 2 + x^3$: 1. Calculate the total cost of producing 10 cameras:  $C(10) = 2 + (10)^3 = 2 + 1000 = 1002$  2. The average cost is the total cost divided by the number of units produced:   $\text{Average Cost} = \frac{C(10)}{10} = \frac{1002}{10} = 100.2$. The closest option is: The answer is J. [Response(to Video)] The answer is C. [Response(to Audio)] The total cost of producing 10 cameras is $C(10) = 2 + 10^3 = 1002$. The average cost is $1002 / 10 = 102$ per unit. The marginal cost is the derivative of the cost function, which is $C'(x) = 3(2 + x)^2$. The marginal cost of producing 10 cameras is $C'(10) = 3(2 + 10)^2 = 363$. The answer is D.[Response(to Image)] The total cost of producing x cameras is given by the function $C(x) = 2 + x^3$. The average cost of producing 10 cameras is the total cost divided by the number of cameras, or $C(10)/10 = (2 + 10^3)/10 = 1002/10 = \$100.20/camera$ The marginal cost is the derivative of the total cost function, or $C'(x) = 3x^2$. The marginal cost of producing 10 cameras is $C'(10) = 3(10)^2 = \$300/camera$. Therefore, the correct answer is: **The answer is E**.Text Input:Audio Input:Video Input:Image Input:OmniLMorUnder review as a conference paper at ICLR 2025

Figure 3: We propose Omnify! to create the synthetic omni-modality evaluation data from the
original text benchmarks.

2 OMNI×R BENCHMARK

In this section, we introduce Omnify! a scalable and low-cost automatic method designed to
translate text into various modalities, including audio, image, video, and combinations thereof.
The overarching goal of Omnify! is to build up a scalable method to generate omni-modality
data while keeping information the same across them for evaluating OLMs’ reasoning capabilities
across modalities. We construct the Omni×R benchmark in two subsets: (1) Omni×RSYNTH: a
synthetic omni-modal reasoning evaluation dataset derived from applying Omnify! on the MMLU-
Pro (Wang et al., 2024). (2) Omni×RREAL: a real-world omni-modal reasoning evaluation derived
from Youtube, which is then processed and annotated by human experts.

2.1 OMNIFY!

Text to image. Though there are many ways
to convert text into images, like using image
generation models (e.g., Imagen-3 (Baldridge
et al., 2024), DALLE-3 (OpenAI, 2024a)),
the seemingly appealing text-to-
however,
image generation models make it challenging
to control quality; they cannot ensure the gen-
eration contains all the information we need to
answer a question. Before ﬁguring out how to
judge the quality of and information in the gen-
erated images, it is not viable to use image gen-
erators to scale up the mapping from text to im-
ages. Since our main goal is to evaluate models’ reasoning capability, we start from the simplest
approach in this work: rendering a canvas and then write the words on it. Given the images as in-
put, we expect the models can achieve the same performance as they read text in this ideal scenario,
where no extra noises, information losses, or variations are introduced by the text-to-image mapping
process. Speciﬁcally, we use PIL1 to create a new image with a white background and the text is
drawn onto the image with black color. The engineering details/efforts can be found in Appendix G.

The overview of Omni×RSYNTH and

Figure 2:
Omni×RREAL.

Text to Audio We initially attempted to use Google Text-to-Speech2 (TTS) for text-to-audio con-
version. However, we encountered challenges with the mathematical equations. To address this,
we developed a two-step process. First, we convert the original text, if it contains mathematical
equations, into a format that is easy to speak orally. The details for the conversion could be found
in Table 7. Then, we use a TTS engine to generate the audio, which contains the full information of
the original text question.

Text to Video Like text-to-image generation models, there exist Sora (Brooks et al., 2024) and
Veo (Google, 2024) we could leverage to map text to videos. However, they would incur the same
problems as described in the text to image: quality control, time consumption, and computational
cost. The main objective with videos here is to evaluate a model’s capabilities on understanding
a video input, which is a series of images from a model’s view, and then reasoning to solve the
problems. We fulﬁll this objective again using a simple approach to generating the video data from
text as follows. Based on our image generation process, we render a series of images where each

1https://github.com/python-pillow/Pillow
2https://cloud.google.com/text-to-speech?hl=en

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Text QuestionAudioImageOmnify!Question: Euglena is a common green flagellate protozoan found in fresh waterponds. Describe briefly the method of locomotion, nutrition, and asexual reproduction in this organism.Question:VideoInterleaved ( Video + Audio and Image + Audio)Euglenais…Text, Image, Audio, Video,Video+Audio, Image + AudioMath, Physics, Chemistry, Computer Science…100 Examples in each category. 1400 exmaplesin each modality.6 Modalities14 Categories1400 Samples25%25%30%20%MathCodingChemistryPhysicsText, Image, Audio, Video,#Test Samples: 400#EachModality: 1004Modalities𝑂𝑚𝑛𝑖𝑋𝑅!"#$%𝑂𝑚𝑛𝑖𝑋𝑅&’()Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

(a) Coding

(b) Physics

(c) Calculus

(d) Chemistry

Figure 4: Visualization of Examples in the Omni×R-Real set.

image contains one or several words from the text. We ensure that the information in the text is fully
translated to the video. The input text is split into individual words ﬁrst. Then we use OpenCV to
create a video writer object with a speciﬁed frame rate, i.e., 1 FPS, and frame size (300x100 pixels).
Each word is converted into an image using the text-to-image method. Finally, these images are
combined sequentially to create video frames.

2.2 OMNI×RSYNTH: SCALABLE SYNTHETIC OMINI-MODAL REASONING EVALUATION

Our initial choices of the text benchmark for Omnify! are Arc-Challenge (Clark et al., 2018) and
GSM8K (Cobbe et al., 2021), but we identify the potential data contamination problems on these two
benchmarks as Gemini-1.5-pro (Gemini-Team, 2024a) can achieve over 99% on GSM8K (results are
shown in Table 9). It is very likely that contaminated OLMs just capture the part of the information
they need from the video/audio questions and use their ‘memory’ to give correct answers, which
cannot reﬂect the actual reasoning ability of the models. Thus, we choose MMLU-Pro (Wang et al.,
2024), which is augmented from MMLU with ten options per question and released in June after
the Gemini-1.5-Pro-0013 release, as the text benchmark to Omnify!. In this way, we minimize the
contamination inﬂuence, enabling a more accurate study of OLMs’ omni-reasoning. We randomly
sample 100 questions from each of the 14 categories in MMLU-Pro to construct Omni×RSYNTH.
Some examples for Audio and Video modalities are available4.

2.3 OMNI×RREAL: HIGH-QUALITY REAL-WORLD OMINI-MODAL REASONING EVALUATION

We crawl the video data from youtube and then transcribe it into different modalities to develop a
realistic set as a valuable addition to the Omni×R.

Video: We select four categories that require dense reasoning in real-world scenarios: Mathematics,
Coding, Physics, and Chemistry. Videos are sourced from popular educational channels, such as
MIT OpenCourse. Two human annotators, spend approximately 30 hours each to review 100 videos
(200 in total) and identify those containing non-trivial questions that demand substantial reasoning
to solve. From these, 100 videos are carefully selected to construct a high-quality set, Omni×RREAL.
Each video clip is curated based on the following criteria: (1) it must contain one or more key
frames that provide all the necessary information to solve the question; (2) the clip should exclude
the answer to maintain the challenge; (3) some misleading or irrelevant frames are intentionally
included to assess the model’s robustness in reasoning. Image: We manually ﬁnd the key frame(s)
which contain the question information. It should be noted that in some cases, there might be several
frames containing the relevant information, where we will crawl two or three frames and merge them
together into one image. Text: Five human annotators transcribe the text from the video with the
help of the tools, e.g., Gemini. All the open-ended generation questions are transferred into multiple
choice questions to make the benchmark easy-to-use. Audio: The original audio will be checked
ﬁrst, which is extracted from the video we crawled. If it contains all the information for OLMs
to answer the question, then we will just keep and use it. However, there are many cases where
the audio does not contain the enough information for answering the questions, e.g., the instructor
shows a slide and asks “solve the problems in the slide”, where the problem is shown in image. In
that scenario, we will use the same method in Omnify! to transfer the transribed text into audio by
Google TTS.

3https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions
4https://anonymous.4open.science/r/OmnixR-Examples-7961/

4

Under review as a conference paper at ICLR 2025

3 EXPERIMENTS AND FINDINGS

3.1 EXPERIMENT SETUP

Models. We mainly test three series of models: Gemini (Gemini-Team, 2024a), i.e., Gemini-
1.5-Pro, and Gemini-1.5-Flash, OpenAI-GPT (OpenAI, 2024c), i.e., GPT-4o and GPT-4o-mini,
Anthropic-Claude (Anthropic, 2024), i.e., Claude-3-Opus, Claude-3-Sonnet, Claude-3-Haiku. More
details about the test models are shown in Appendix C.

CoT Prompting. The standard setting in MMLU-Pro (Wang et al., 2024) is to use Chain-of-
Thought(CoT) prompting to elicit the reasoning ability of the OLMs for a more comprehensive
evaluations. Following them, we use CoT with 0-shot, as our standard setting, i.e., the prompt used
for evaluation is “Think step by step then output the answer in the format of “The answer is (X)” at
the end.”

Extract-Then-Answer (ETA) Prompting. In addition, we employ Extract-Then-Answer (ETA)
prompting, leveraging the benchmark’s inherent structure. This method involves ﬁrst extracting the
textual content and then using the OLMs’ language capabilities for reasoning to provide answers
based on the transcriptions. To prevent potential hackings on Omni×R, we transparently demon-
strate this approach in our benchmark, aiming for a comprehensive evaluation of OLMs. Speciﬁ-
cally, the prompt ’Please extract the text from image/audio/videos’ instructs the OLMs to function
as text extractors. The extracted text from this initial step is subsequently fed back into the same
OLM with Chain-of-Thought (CoT) prompting to obtain the ﬁnal answer. Consequently, the model’s
performance reﬂects two key abilities: OCR/Transcription and Text Reasoning.”

Video/Audio/Image. We ﬁrst process the video to 1-fps to meet the requirements for both the
Gemini and GPT models. For testing with Claude, we used the API available before August 10th,
which only supported a maximum of 5 image inputs, so video evaluations were not conducted. The
GPT-4o API supports 250 images input at the maximum, so any additional frames were dropped
in the evaluation. In contrast, Gemini had no issues with the video modality and could handle all
frames as input. Image processing is the modality that all models support most effectively, allowing
comprehensive testing across all OLMs. Notably, Gemini is the only model supporting audio input.

Answer Extraction: We use the model to extract the answers. Since the regex parsing may affect
the performance, we sacriﬁce the API cost to trade in the excellent extraction.

Table 1: Results on Omni×RSYNTH show different mixed modalities evaluations, including text,
image, audio, video. Each modality (Image/Audio/Video) combines two input sources: the ’Ques-
tion’ provided by the respective image, audio, or video modality, and the ’CoT instruction’ provided
by the text The numbers in red font, following the downward arrows, shows the drops compared to
the pure text input.

Gemini 1.5

Claude

GPT

Modality

Pro

Flash
Perf. ∆ Perf. ∆

Opus

Haiku
Sonnet
Perf. ∆ Perf. ∆ Perf. ∆

4o

4o-mini
Perf. ∆ Perf. ∆

Text
Image
Audio
Video

-

-

69.9

77.5
57.3 20.2↓ 36.3 33.6↓
56.6 20.9↓ 53.9 16.0↓
36.3 41.2↓ 15.1 54.8↓

-

77.7
26.9 50.8↓ 18.8 58.6↓ 9.9 62.6↓

72.5

77.4

-

-

71.5
60.1 11.4↓ 48.5 24.1↓

72.6

-

-

-
-

-
-

-
-

-
-

-
-

-
-

-

-

-

-

53.1 18.4↓ 18.6 54.0↓

Extract-Then-Answer (ETA) Prompting

Image
Audio
Video

1.8↓
4.0↓ 68.1
73.5
6.3↓
7.6↓ 63.6
69.9
48.6 28.9↓ 42.8 27.1↓

62.6 15.1↓ 48.1 29.3↓ 43.2 29.3↓

-
-

-
-

-
-

-
-

-
-

-
-

66.7
-
25.0

4.8↓ 58.4 14.2↓
-
-
46.5↓ 59.3 13.3↓

-

3.2 MAIN RESULTS ON OMNI×RSYNTH

We show the main experimental results on ominiﬁed MMLU-Pro in Table 1.

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Under review as a conference paper at ICLR 2025

Table 2: Results on Omni×RREAL shows similar behaviour discrapancy of OLMs as indicated in
results on the Omni×RSYNTH. Interestingly, we also observe that simple prompting strategy (ETA
prompting) is not as effective as it was on Omni×RSYNTH, possibly due to the natural noise and
redundancy in real-world image, video, and audio data.

Gemini 1.5

Claude

GPT

Modality

Pro

Flash
Perf. ∆ Perf. ∆

Opus

Haiku
Sonnet
Perf. ∆ Perf. ∆ Perf. ∆

4o

4o-mini
Perf. ∆ Perf. ∆

Text
Image
Audio
Video

-

-
8↓

80
86
65 15↓
78
71 15↓ 64 14↓
64 22↓ 53 27↓

-

-

78
65
66
41 34↓ 39 27↓ 33
-
-
-
-
-
-

-
-

-
-

-
8↓
-
-

-

75
-
85
63 12↓
6↓
79
-
-
-
73 12↓ 66

9↓

Extract-Then-Answer (ETA) Prompting

Image
Audio
Video

7↓

65 15↓
79
55 31↓ 51 29↓
7↓
71 15↓ 73

63 15↓ 52 14↓ 51 14↓
-
-

-
-

-
-

-
-

-
-

-

6↓
-

79
-
66 19↓ 63 12↓

70
-

5↓
-

Model Comparison. Gemini-1.5-Pro demonstrates the most versatile performance across all
modalities, showing results in text, image, audio, video tasks. Claude models struggle with image
tasks and lack audio and video capabilities. GPT models show a balanced performance, with GPT-4o
performing particularly well in direct image and video compare to Gemini and Claude. Generally,
larger models outperform their smaller counterparts across modalities, e.g., Pro > Flash, Opus >
Haiku). But interestingly, GPT-4o-mini outperforms GPT-4o in text and video with ETA prompting.
For video tasks using ETA prompting, GPT-4o’s performance inconsistencies led us to examine the
model’s responses to the extraction, we found that in over 46.8% test samples, the detailed analy-
sis can be found in Appendix F, GPT-series models cannot extract the text from video, which we
identify as the primary cause for the signiﬁcant performance drop compared to CoT prompting. Re-
garding the text modality, two possible explanations emerge: ﬁrst, MMLU-Pro was released before
GPT-4o-mini, suggesting that OAI might have optimized for it. Second, since our dataset uses a
subset sampled from MMLU-Pro, inherent biases may have inﬂuenced the results.

Modality Analysis. Text is the most mature modality across all models, with consistently high
scores (ranging from 69.9% to 77.7%). Image modality shows signiﬁcant variability, with direct
task performance ranging from 9.9% (Claude Haiku) to 60.1% (GPT-4o). However, ETA prompt-
ing on image generally improves performance for all models, particularly for Claude (e.g., Opus
improves from 18.8% to 62.6%). The improvement justiﬁes the inclusion of ETA prompting as
a standard in our benchmark to prevent potential manipulation. Audio modality, only available for
Gemini models, shows moderate performance with notable improvement via ETA prompting. Video
modality presents the most challenges, especially for the small models, i.e., Gemini-1.5-Flash, and
GPT-4o-mini.

There are also additional results on Arc-Challenge and GSM8k benchmarks shown in Table 9 with
different modality input, i.e., text, image, audio, video. Though the models are likely to be data
contaminated on these benchmarks, the performance drops are still signiﬁcant on image/video/audio
compared to the pure text.

3.3 MAIN RESULTS ON OMNI×RREAL

The results on the realistic set generally align with those from the synthetic set, showing signiﬁcant
drops in performance across audio, image, and video tasks compared to the text. One difference
here is that performance on video does not drop a large margin compared to that in the synthetic
set. Though the video is noisy than it is in the synthetic data, we can still capture one key frame and
answer the question according to that key frame which largely reduces the difﬁculties, compared
to the synthetic scenario, if the model can ﬁnd the main frame in the video. Another interesting
ﬁnding is that ETA prompting does not consistently improve performance; for example, there are
performance drops in audio tasks with ETA prompting compared to CoT on both Gemini-Flash and
Gemini-Pro. These ﬁndings conﬁrm that our synthetic set effectively simulates real-world scenar-
ios in a scalable, cost-efﬁcient way, serving as a valuable sanity check for OLMs’ omni-modality
reasoning capabilities.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Key Takeaways. We summarize the following interesting takeaways from our experiments:

1. Multi-modal capabilities vary signiﬁcantly across models, with Gemini 1.5 Pro showing the most

broad support and balanced performance across all modalities.

2. Gaps still exists on other modalities compared to the text modality even just in such easy per-
ception test scenarios. Signiﬁcant room for improvement exists in video processing across all
models, presenting opportunities for future development.

3. ETA prompting generally improves performance on Omni×RSYNTH but OLMs can no longer
solely rely on it for Omni×RREAL, indicating the necessity of the further alignment on omni-
modality.

4. There’s a clear trade-off between model size and performance, but smaller models (e.g., GPT-4o-

mini) can sometimes outperform larger counterparts in speciﬁc tasks.

5. Our Omni×RSYNTH could be a good simulating set for the real-world scenarios, as the results on

Omni×RREAL match the results in the Omni×RSYNTH.

4 MIXED MODALITIES

Table 3: The results of more complex mixed modalities on Omni×RSYNTH. We use the ∆ to denote
the performance drops from the text modality.

Input Modality

Gemini-Pro

Gemini-Flash

Question

CoT Prompt

Perf.

∆

Perf.

∆

Text
Text
Text
Text
Image + Audio
Video + Audio

Text
Video
Audio
Image
Text
Text

77.5
76.1
74.1
74.1
61.8
40.1

-
1.4↓
3.4↓
3.4↓
15.7↓
37.4↓

69.9
66.8
68.3
66.9
49.1
25.9

-
3.1↓
1.6↓
3.0↓
20.8↓
44.0↓

Text to Mixed Modalities. In addition to the types of the Omnify! described in Section 2.1, our
method could also be applied to generating interleaved modalities to better simulate more complex
real-world scenarios, where the information is included in different modalities and requires a model
to reason across the modalities to solve a problem. For example, an instructor can write down
an equation on the blackboard and say “compute the derivative” in a Calculus lecture. Scenarios
like this example require a model to jointly use image perception and audio understanding process
the question, reason across the visual and audio modalities, and then provide a response. Using
our Omnify!, we seamlessly integrate different modalities and create test samples with interleaved
modalities, i.e., “Video + Audio”, and “Image + Audio”, to Omni×RSYNTH, which captures a more
authentic user experience where multiple senses are engaged simultaneously. To be speciﬁc, We
transfer the question into video and all the options are transferred for Audio, to get the modality,
“Video + Audio”, while CoT prompting remains in text form to maintain the model’s reasoning
ability across different modalities.

Transferring CoT prompt to other modalities. All the CoT prompting is in text for all the previous
test cases. Here, we convert the CoT prompt into different modalities while keeping the others, i.e.,
questions and options in MMLU-Pro intact.

Results. As shown in Table 3, there is a noticeable decline in performance when transitioning from
text to mixed-modality tasks. For example, both the Pro and Flash models perform signiﬁcantly
worse in the ”Video + Audio” scenario, achieving scores of 40.1 and 25.9, respectively. This in-
dicates that handling mixed modalities presents a signiﬁcant challenge, likely due to the increased
complexity of integrating video and audio information. For Audio/Image/Video CoT, the model
generally treats these inputs as noise or irrelevant context, having minimal impact on the ﬁnal re-
sults, as performance approaches that observed with text-based CoT. We focus on evaluating the
Gemini-series models since only Gemini supports audio inputs.

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

5 ANALYSIS

5.1 OMNI-MODALITY REASONING BEHAVIOUR ANALYSIS

After investigating the responses, we ﬁnd that in omni-modality cases, Gemini-1.5-Flash models can
only output very short answers though prompted to CoT before giving the answers, which is quite
different from the reasoning behaviour in the pure-text. An example in Figure 1 shows the different
behaviours among modalities, which intrigues us to have a quantitative analysis of the reasoning
paths. We write a simple regex, detecting if the model output starts with ”the answer/response is
(*.)”, with the rule, the total number of words should be less than 40, to evaluating whether the
models’ output contain the reasoning path. The results are shown in Table 4.

Table 4: The percentage of the model outputs containing the reasoning paths on Omni×RSYNTH.

Path(%)

Gemini 1.5

Claude

GPT

Modality

Pro

Flash Sonnet Opus Haiku

4o

4o-mini

Text
Image
Video
Audio

98.9
93.2
91.3
94.0

89.1
54.3
23.4
82.3

100
100
-
-

100
100
-
-

98.6
72.8
-
-

100
100
99.1
-

100
100
95.7
-

Our analysis reveals that smaller models tend to produce reasoning paths less frequently for image,
video, and audio inputs. Notably, for complex modalities like video, Gemini-1.5-Flash generates
reasoning paths for only 23.4% of test examples, substantially lower than Gemini-1.5-Pro. Among
the modalities, audio inputs elicit reasoning paths most similarly to text, while video inputs show the
lowest rate of reasoning path generation. GPT-series models demonstrate excellent performance in
producing reasoning paths across available modalities. However, these results underscore the signif-
icant challenges remaining in cross-modal reasoning. Given that models are expected to exhibit rea-
soning abilities, they should ideally output reasoning paths consistently across all input modalities.

5.2 VISUAL/VIDEO FORMATS INFLUENCES PERCEPTION PRECISION

5.2.1

IMAGE

We ﬁrst analyze how formats affect the performance on images. We show images with two different
text formats in Figure 5. The lower image has a compact format, where the options are not spaced
out; instead, they are presented in a continuous, inline format separated by periods. Compared to
it, each option in the upper image is listed separately, making it easy to read, with letters (A to
J) clearly aligned before each option. The results of CoT and ETA prompting with two different
formats of images are shown in Table 6. The overall trend here is that with better format, we could
signiﬁcantly improve the performance across all the tested models. ETA prompting also boosts
the performance for the both formats in general. For all the other models, the performance can be
signiﬁcantly improved when comparing BF with ETA, only the GPT-4o being an outlier.

We further analyze transcription accuracy using the Character Error Rate (CER), a standard metric
for assessing text recognition performance, especially in OCR tasks. A CER of 0 indicates perfect
accuracy, with higher values reﬂecting more errors. Details of the CER calculation are provided in
Appendix H, and results are shown in Table 5. The results reveal that GPT-4o’s OCR performance is
largely format-independent, whereas other models exhibit considerable format sensitivity, explain-
ing the pronounced improvements seen with ETA prompting for all models except GPT-4o when
format is enhanced.

5.2.2 VIDEO

We create different types of videos, one word per frame, several words per frame, etc. Our ablations
reveal that increasing the number of words per frame generally leads to improved performance for
both Gemini-Flash and Gemini-Pro models under both testing promptings, CoT and ETA prompting.
This trend suggests that providing more context within each frame aids in the models’ understanding
and processing of the video content and narrow the gaps between images and videos.

8

Under review as a conference paper at ICLR 2025

Table 5: The Character Error Rate, the metric for evaluating the OCR, of different models on two
different formats images.

Gemini 1.5

Claude

GPT

Pro

Flash Opus Sonnet Haiku

4o

4o-mini

Image
0.11
Better Image 0.06

0.10
0.03

0.19
0.05

0.28
0.18

0.34
0.26

0.11
0.11

0.12
0.11

Figure 5: We include two ﬁgures to illustrate which is a better format image. The upper one is the
image with better format. The lower one is the image with the original format.

6 RELATED WORK

Large Foundational Models. GPT-4o (OpenAI, 2024b), Gemini (Gemini-Team, 2024a) both claim
their models having omni-modality capabilities, but actually OAI’s model does not support audio(no
audio access via APIs)/video(only 250 frames and the videos should be separated manually be-
fore feeding into the model) while Gemini can take very long videos and has good Audio support.
Claude (Anthropic, 2024) can be viewed as a vision-language model (Bordes et al., 2024) since it
has capabilites to take image but no audio or video support. There are also other open-sourced vi-
sion language models, but they are mostly supporting only two modalities, e.g., the vision-language

)

%

(

e
c
n
a
m
r
o
f
r
e
P

60
50
40
30
20
10

Flash CoT

Flash ETA

Pro CoT

Pro ETA

6
.
8
4

8
.
2
4

3
.
6
3

1
.
5
1

7
.
8
5

4
.
3
5

1
.
2
4

5
.
7
1

3
.
9
5

3
.
5
5

3
.
6
4

9
.
3
2

4
.
1
6

2
.
8
5

9
.
7
4

1
.
7
2

1

2

4

8

Words Per Frame

Figure 6: Video ablation study: Model performance with different words per frame. Pro and Flash
denotes Gemini-1.5-Pro-001 and Gemini-1.5-Flash-001, respectively.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Question: The relatives of a group of pelicans from the same species that separated from each other because of an unsuccessful migration are reunited 150 years later and find that they are unable to produce offspring. This is an example ofOptions: A. gene flow.B. temporal isolation.C. disruptive selection. D. founder effect. E. genetic drift.F. sexual selection.G. sympatric speciation.H. habitat fragmentation.I. bottleneck effect.J. allopatric speciationQuestion: The relatives of a group of pelicans from the same species that separated from each other because of an unsuccessful migration are reunited 150 years later and find that they are unable to produce offspring. This is an example of  Options: A.gene flow. B. temporal isolation. C. disruptive selection. D. founder effect.  E. genetic drift. F. sexual selection. G. sympatricspeciation. H.habitatfragmentation. I. bottleneck effect. J. allopatric speciationUnder review as a conference paper at ICLR 2025

Table 6: The ablations: image with better format. BF: better format. The blue font denotes the
performance gain of the better image compared to the original image format.

Gemini 1.5

Claude

GPT

Prompt

Pro

Flash

Opus

Sonnet

Haiku

4o

4o-mini

Text

Image
Better Image

Image
Better Image

CoT

CoT
CoT

ETA
ETA

77.5

69.9

77.7

77.4

76.5

71.5

72.6

57.3
64.6 7.3↑ 43.6 7.3↑ 33.5 6.6↑ 28.9 10.1↑

36.3

18.8

26.9

68.7
73.5 4.8↑ 68.1 6.8↑ 62.6 26.2↑ 48.1 21.5↑

61.3

26.6

36.4

9.9
19.1 9.2↑ 65.5 5.4↑ 52.1 3.6↑

48.5

60.1

24.9
43.2 18.3↑ 66.9 0.2↑ 61.7 3.3↑

58.4

66.7

models like LLaMA-3.1 and 3.2 (Meta, 2024), Pixtral (Mistral, 2024), LLaVA (Liu et al., 2023b;a);
Audio-LLM like GAMA (Ghosh et al., 2024), LTU (Gong et al., 2023b;a), and SALMONN (Tang
et al., 2024). It is hard to judge them on our benchmark, since the main idea behind our evaluations
are that we expect the model has cross-modality reasoning and would like to encourage the model
improving their cross-modal reasoning, only vision/audio/video would not get a comprehensive re-
sults. We would expect the open-sourced community to release real OLMs in the future and we will
update the results accordingly.

Video/Audio/Image Evaluation benchmarks. Omnibench (Li et al., 2024b) speciﬁcally aimed
at evaluating OLMs’ tri-modal, i.e., text, vision, and audio, processing capabilities with human-
annotated tasks. Compared to it, OmnixR emphasizes the omni-modality reasoning evaluations
with both human-annotated realistic set and scalable synthetic set. MMMU (Yue et al., 2024a),
MMMU-Pro (Yue et al., 2024b), CMMMU (Ge et al., 2024) focuses on evaluating vision-language
models across various college-level disciplines with highly heterogeneous image types, emphasiz-
ing expert-level perception and reasoning across text-image pairs while LMSYS-Vision (Christo-
pher Chou, 2024) evaluates the instruction-following of the large vision-language models (Liu et al.,
2023a; Chen et al., 2023; 2024; Yang et al., 2024). Compared to them, OmnixR has larger scope on
evaluating OLMs on cross-modality reasoning, not only vision input, but audio, video, and mixed
modalities such as image + audio. AiShell-1, AiShell-2 (Du et al., 2018), Clotho-AQA (Lipping
et al., 2022) are audio understanding benchmarks, providing extensive and high-quality real-world
audio data for Mandarin ASR and audio question answering. MVBench (Li et al., 2024a) fo-
cuses on temporal reasoning across 20 challenging video tasks, Video-Bench (Ning et al., 2023)
assesses Video-LLMs across video-exclusive, knowledge-based, and decision-making tasks, while
MMBench-Video (Fang et al., 2024) offers a long-form, multi-shot evaluation of LVLMs with 609
videos and 2,000 human-annotated QA pairs across 26 ﬁne-grained capabilities. In OmnixR, we
also include long video in both synthetic and realistic scenarios and we also have mixed-modality
evals including video + audio.

7 CONCLUSION

In this paper, we introduced Omnify!, a scalable and cost-efﬁcient approach for generating multi-
modal data from text, facilitating the construction of diverse and challenging test scenarios for
omni-modal language models (OLMs). Using this method, we developed Omni×RSYNTH, a syn-
thetic omni-modal reasoning evaluation dataset derived from MMLU-Pro, as well as Omni×RREAL,
a real-world omni-modal reasoning dataset based on YouTube content. Our comprehensive evalu-
ations reveal that OLMs experience substantial performance drops when confronted with complex
multi-modal inputs, particularly in tasks that demand cross-modality reasoning. Notably, we ob-
served that smaller models, e.g., Gemini-1.5-Flash, are less adept at producing reasoning paths for
image, video, and audio inputs compared to text, underscoring the inherent challenges in cross-
modal reasoning. The evaluation results underscore the necessity for enhanced training strategies to
address the complexities of omni-modal tasks. To sum up, Omni×R stands as a critical benchmark
for guiding future advancements in OLMs, providing a foundation for measuring progress toward
more human-aligned and truly omni-modal AI systems.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

REFERENCES

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing
Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and
In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in
text.
Neural Information Processing Systems, 2021. URL https://openreview.net/forum?
id=RzYrn625bu8.

Anthropic. Claude: An ai assistant by anthropic, 2024. URL https://www.anthropic.com/

claude. Accessed: 2024-09-21.

Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan,
Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, et al. Imagen 3. arXiv preprint
arXiv:2408.07009, 2024.

Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne
Petryk, Oscar Ma˜nas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to
vision-language modeling. arXiv preprint arXiv:2405.17247, 2024.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe
Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video
generation models as world simulators. 2024. URL https://openai.com/research/
video-generation-models-as-world-simulators.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing-
long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In-
ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv
preprint arXiv:2312.14238, 2023.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong,
Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to
commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024.

Wei-Lin Chiang Ying Sheng Lianmin Zheng Anastasios Angelopoulos Trevor Darrell Ion Stoica
Joseph E. Gonzalez Christopher Chou, Lisa Dunlap. Multimodal arena. 2024. URL https:
//lmsys.org/blog/2024-06-27-multimodal/.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
arXiv preprint arXiv:1803.05457, 2018.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training veriﬁers to
solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. Aishell-2: Transforming mandarin asr research

into industrial scale, 2018.

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen.
Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv
preprint arXiv:2406.14515, 2024.

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.

Large-
In H. Larochelle,
In-
Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2020/

scale adversarial training for vision-and-language representation learning.
M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural
formation Processing Systems, volume 33, pp. 6616–6628. Curran Associates,
2020.
file/49562478de4c54fafd4ec46fdb297de5-Paper.pdf.

Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao,
Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun
He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for
a family of multi-modal large language models. ArXiv, abs/2402.05935, 2024. URL https:
//api.semanticscholar.org/CorpusID:267547619.

11

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Under review as a conference paper at ICLR 2025

Zhang Ge, Du Xinrun, Chen Bei, Liang Yiming, Luo Tongxu, Zheng Tianyu, Zhu Kang, Cheng
Yuyang, Xu Chunpu, Guo Shuyue, Zhang Haoran, Qu Xingwei, Wang Junjie, Yuan Ruibin,
Li Yizhi, Wang Zekun, Liu Yudong, Tsai Yu-Hsuan, Zhang Fengji, Lin Chenghua, Huang Wen-
hao, and Fu Jie. Cmmmu: A chinese massive multi-discipline multimodal understanding bench-
mark. arXiv preprint arXiv:2401.20847, 2024.

Gemini-Team.
of

tokens
google-gemini-next-generation-model-february-2024/.

Gemini 1.5: Unlocking multimodal understanding across millions of
URL https://blog.google/technology/ai/

context,

2024a.

Gemini-Team. Gemini: A family of highly capable multimodal models, 2024b. URL https:

//arxiv.org/abs/2312.11805.

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi,
Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model
with advanced audio understanding and complex reasoning abilities, 2024. URL https://
arxiv.org/abs/2406.11768.

Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass.

Joint audio
In 2023 IEEE Automatic Speech Recognition and Understanding
and speech understanding.
Workshop (ASRU). IEEE, December 2023a. doi: 10.1109/asru57964.2023.10389742. URL
http://dx.doi.org/10.1109/ASRU57964.2023.10389742.

Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and

understand. arXiv preprint arXiv:2305.10790, 2023b.

Google. Veo: Google’s most capable generative video model. 2024. URL https://deepmind.

google/technologies/veo/.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen,
Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understand-
In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition
ing benchmark.
(CVPR), volume abs/2204.14198, pp. 22195–22206. IEEE, June 2024a. doi: 10.1109/cvpr52733.
2024.02095. URL http://dx.doi.org/10.1109/CVPR52733.2024.02095.

Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng
Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou
Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin.
Omnibench: Towards the future of universal omni-language models, 2024b.

Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-

aqa: A crowdsourced dataset for audio question answering, 2022.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS,

2023b.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis-
tic representations for vision-and-language tasks, 2019. URL https://arxiv.org/abs/
1908.02265.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of
foundation models in visual contexts. In International Conference on Learning Representations
(ICLR), 2024.

Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.

Mistral. Pixtral, 2024. URL https://docs.mistral.ai/capabilities/vision/.

12

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Under review as a conference paper at ICLR 2025

Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan.
Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language
models. arXiv preprint arXiv:2311.16103, 2023.

OpenAI. Dalle 3. https://openai.com/index/dall-e-3/, 2024a.

OpenAI. Hello gpt-4o! OpenAI Research, 2024b. URL https://openai.com/index/

hello-gpt-4o/. Accessed: 2024-09-21.

OpenAI. Gpt-4o mini: advancing cost-efﬁcient intelligence, 2024c. URL https://openai.
com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/. Ac-
cessed: 2024-09-21.

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA,
and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models.
In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview.net/forum?id=14rn7HpKVk.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming
Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi
Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language
understanding benchmark, 2024.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang,
Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren
Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang,
Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin,
Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong
Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu,
Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru
Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.

Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens,
Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun,
Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and
Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning
In 2024 IEEE/CVF Conference on Computer Vision and Pattern
benchmark for expert agi.
Recognition (CVPR), volume 32, pp. 9556–9567. IEEE, June 2024a. doi: 10.1109/cvpr52733.
2024.00913. URL http://dx.doi.org/10.1109/CVPR52733.2024.00913.

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun,
Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more
robust multi-discipline multimodal understanding benchmark, 2024b.

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and

Yejin Choi. Merlot: Multimodal neural script knowledge models, 2021.

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou,
Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the
diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A CONVERT MATH INTO SPOKEN VERSION

For the math equations in the questions, we prompt Gemini-1.5-Pro to convert them into the version
which can be spoken orally. The prompt we used is detailed in Table 7. We also show an example to
explain the transformation: the TTS is hard to read the original question in Table 8 but it can handle
the converted text.

Table 7: The oral conversion prompt designed for Text-to-Audio transfer.

[Prompt] Please transform all the equations in the text into the format that is easy to speak out
orally. [Original text]
Please ﬁrst output a single line of the text in the format ”The transformed text is xxx”

Table 8: An example of the conversion from the original question into the easily spoken text.

[Original Question] For what values of x is it true that x2 − 5x − 4 ≤ 10? Express your answer
in interval notation.
[Converted Text] The spoken version: For what values of x is x squared minus ﬁve x minus
four less than or equal to ten? express your answer in interval notation.

B CATEGORIES IN MMLU-PRO

There are 14 categories in MMLU-Pro, including Math, Physics, Chemistry, Law, Engineering,
Other, Economics, Health, History, Psychology, Business, Biology, Philosophy, Computer Science.

C MODEL SETTINGS/DETAILS

The version of the Geminis we used in this paper are Gemini-1.5-Pro-001 and Gemini-1.5-Flash-
001. The version of the OpenAI models we used are gpt-4o-2024-05-13, and gpt-4o-mini-2024-
07-18. The verison of the Claude models we used are claude-3-sonnet@20240229, claude-3-
opus@20240229, claude-3-haiku@20240307.

The Gemini safety settings we used for video, audio, and images are shown in the following:

1 # Safety Setting
2 generative_models.SafetySetting(
3

category=generative_models.HarmCategory.
HARM_CATEGORY_DANGEROUS_CONTENT,

threshold=generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,

4
5 ),
6 generative_models.SafetySetting(
7

8
9 ),
10 generative_models.SafetySetting(
11

12
13 ),
14 generative_models.SafetySetting(
15

category=generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT,
threshold=generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,

category=generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH,
threshold=generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,

category=generative_models.HarmCategory.
HARM_CATEGORY_SEXUALLY_EXPLICIT,

threshold=generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,

16
17 ),

BLOCK ONLY HIGH is the loosest setting we can use for public Gemini APIs for video, audio,
and images. BLOCK ONLY NONE is the loosest setting we can use for text, so we change all the
Safety Settings for language into BLOCK ONLY NONE.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

For response generation, we follow the commonly used settings, temperature=0.7, top p=0.9, and
output length=1024, for all the models, i.e., Gemini, Claude, GPT models.

D RESULTS ON ARC-CHALLENGE & GSM8K

We also evaluate Gemini models on ARC-Challenge dataset and GSM8K test set. The results are
shown in Table 9.

Table 9: Performance of Gemini Models Across Different Modalities on ARC-Challenge and
GSM8K Benchmarks

Benchmark

Accuracy (%)

Gemini-1.5-Pro Gemini-1.5-Flash

ARC-Challenge

Text
Image
Audio
Video

GSM8K
Text
Image
Audio
Video

95.5
79.5
91.1
63.6

99.1
92.5
86.8
80.3

92.3
75.0
88.0
40.3

96.3
87.9
90.8
63.1

E OMNI×R STATISTICS

Table 10: Statistics for Video and Audio on the Omni×RSYNTH. F: Frames, s: seconds.

Min Max

Mean

Video
Audio

28F
7.2s

552F
251.3s

117.2F
32.3s

Table 11: Statistics for Video and Audio on the Omni×RREAL. F: Frames, s: seconds.

Min Max Mean

Video
Audio

30f
10s

1326f
1326s

255.6f
139.7s

F ANALYZE THE EXTRACTION

We manually check the data ﬁrst, and then ﬁnd the patterns that the extraction failure have are
mostly ”unable to process”, ”can’t extract”, ”I’m sorry”, and ”unable to extract”. So we use these
four patterns to check if the answers contain one of them, and calculate the percentage of the model
answers which do not output the extractions when prompted as ”Please extract the text from video.”

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

G DETAILS OF THE TEXT-TO-IMAGE CONVERSION

We use the Python Imaging Library (PIL) to create a new image with a white background and the text
is drawn onto the image with black color. The tricky part here is that the most commonly used font
”times.ttf” does not support the Unicode well and will encounter the error when we try to convert
the Unicode text, e.g., special mathematical symbols such as ∞, ≥, Π, ∆. Thus, our solution here is
to have a look-up-table to replace these Unicode text with latex code before generating. The details
about the look-up-table is shown in Appendix G.1.

G.1 LOOK-UP-TABLE FOR UNICODE CONVERSION

We show parts of look-up-table here due to the display issues. The full details about the look-up-
table could be referred to our code.

# Alpha
# Beta
# Gamma
# Delta
# Pi
# Sigma
# Phi
# Omega
# Summation
# Product
# Integral
# Capital Delta
# Capital Sigma
# Capital Phi
# Capital Omega

’\u03b1’: r’$\alpha$’,
’\u03b2’: r’$\beta$’,
’\u03b3’: r’$\gamma$’,
’\u03b4’: r’$\delta$’,
’\u03c0’: r’$\pi$’,
’\u03c3’: r’$\sigma$’,
’\u03c6’: r’$\phi$’,
’\u03c9’: r’$\omega$’,
’\u2211’: r’$\sum$’,
’\u220f’: r’$\prod$’,
’\u222b’: r’$\int$’,
’\u0394’: r’$\Delta$’,
’\u03a3’: r’$\Sigma$’,
’\u03a6’: r’$\Phi$’,
’\u03a9’: r’$\Omega$’,
’\u2190’: r’$\leftarrow$’, # Left arrow
"\u2014": r"-",
"\u03a0": r"$\Pi$",
"\u0302": r"$ˆ$",
"\u2228": r"$\vee$",
"\u2013": r"--",
"\u2026": r"$\ldots$",
"\u2283": r"$\supset$",
"\u201c": r"\"",
"\u03c1": r"$\rho$",
"\u03b8": r"$\theta$",
"\u03bb": r"$\lambda$",
"\u2225": r"$\parallel$",
"\u2022": r"$\\bullet$", # Bullet

# En dash

# En dash

# Em dash

# Theta

# Rho

# Capital Pi
# Circumflex accent

# Ellipsis

# Superset

# Left double quotation mark

# Lambda

# Parallel

H CER CALCULATION

The CER is calculated by

CER =

S + I + D
N

Where S = Number of substitutions, I = Number of insertion, D = Number of deletions, N = Total
number of characters in the ground truth text. Since the model output will contain some addition
text, e.g., starting with ”The text in the image reads:”, which will affect the ﬁnal results, we will
detect the ”question:” by regex to eliminate the inﬂuence by that irrelevant text.

16

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Table 12: Examples of different reasoning behaviour across modalities on GPT-4o. We use red font
to mark the key errors in the reasoning path. The correct answer here is F. GPT-4o only respond
correctly on the text modality input.

[Question] Question: Find 4(3/4) % of $24.85. Options: A.$1.50 B.$1.10 C.$0.99 D.$1.25
E.$1.35 F.$1.18 G.$1.00 H.$0.93 I.$1.20 J.$1.05”
[Prompt] Think step by step then output the answer in the format of ”The answer is (X)” at the
end.
[Response(to Text)] To ﬁnd 4 (3/4) % of $24.85, we will follow these steps:
1. Convert the mixed number 4 (3/4) into an improper fraction: - 4 (3/4) = 4 + (3/4) = (4 * 4/4)
+ (3/4) = (16/4) + (3/4) = 19/4.
2. Convert the fraction to a percentage: - (19/4) % = (19/4) * (1/100) = 19/400.
3. Now, calculate 19/400 of $24.85: - First, ﬁnd the value of $24.85 * (19/400): - $24.85 *
(19/400) = $24.85 * 0.0475 (since 19/400 = 0.0475).
4. Perform the multiplication: - $24.85 * 0.0475 = $1.179875.
5. Round to two decimal places: - $1.179875 rounds to $1.18.
Now, we check the options provided: A. $1.50 B. $1.10 C. $0.99 D. $1.25 E. $1.35 F. $1.18 G.
$1.00 H. $0.93 I. $1.20 J. $1.05 The closest match is F. $1.18. Therefore, the answer is F.
[Response(to Video)] 1. The question is asking to ﬁnd 4/3 of $24.85.
2. First, understand the mathematical expression: (4/3) * $24.85.
3. Calculate the multiplication: - Divide $24.85 by 3 = $24.85 / 3 = $8.2833 (approx) - Multiply
the result by 4 = $8.2833 * 4 = $33.1333 (approx)
4. The closest option to $33.13 among the given options is $33.13. The answer is $33.13.
[Response(to Image)] To ﬁnd 3/4 of 24.85, follow these steps:
Calculate 3
4 =
18.6375, Round to the nearest cent: This gives us approximately 18.64. However, since we
need to ﬁnd 4
4 × 24.85 = 18.6375, None of these
options seem to match the calculated value directly. The ﬁnal answer considering the options
is ‘The answer is not found among the options provided.’

4 of that value, we can directly multiply: 3

, 3 × 24.85 = 74.55, Now divide by 4: 74.55

4 × 24.85 = 3×24.85

4 of 24.85: 3

4

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

