#  NTSEBench: Cognitive Reasoning Benchmark for Vision Language Models

Pranshu Pandya1, Agney S Talwarr1, Vatsal Gupta1, Tushar Kataria2, Vivek
Gupta3 , Dan Roth3
1IIT Guwahati, 2University of Utah, 3University of Pennsylvania,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding Author,†Equal Contribution

###### Abstract

Cognitive textual and visual reasoning tasks, such as puzzles, series, and
analogies, demand the ability to quickly reason, decipher, and evaluate
patterns both textually and spatially. While LLMs and VLMs, through extensive
training on large amounts of human-curated data, have attained a high level of
pseudo-human intelligence in some common sense reasoning tasks, they still
struggle with more complex reasoning tasks that require cognitive
understanding. In this work, we introduce a new dataset, NTSEBench, designed
to evaluate the cognitive multi-modal reasoning and problem-solving skills of
large models. The dataset comprises 2,728 multiple-choice questions comprising
of a total of 4,642 images across 26 categories sampled from the NTSE
examination conducted nationwide in India, featuring both visual and textual
general aptitude questions that do not rely on rote learning. We establish
baselines on the dataset using state-of-the-art LLMs and VLMs. To facilitate a
comparison between open source and propriety models, we propose four distinct
modeling strategies to handle different modalities (text and images) in the
dataset instances.

##  1 Introduction

Aptitude and reasoning tests have been essential for assessing intelligence
and are considered strong indicators of problem-solving ability and abstract
reasoning skills. Recent advancements in large language models (LLMs) have
demonstrated their strong performance on IQ test questions, achieving high
scores across five languages King (2023). These results suggest that LLMs are
progressing towards some form of human intelligence at least in textual and
language tasks.

Figure 1: NTSEBench Examples: Shows three samples of textual, direction and
spatial reasoning question from the proposed dataset. Solutions to these
questions are not included here, but are provided in the dataset.

The capabilities of LLMs models to perform on par with or exceed human
baselines on various tasks—such as question answering, sentiment
classification, text generation, visual question answering, coding challenges,
data analytics, mathematical reasoning —continue to improve Srivastava et al.
(2022); Bubeck et al. (2023). LLMs and VLMs have hence become the default
benchmark for many zero or few shot text and vision-based tasks Brown et al.
(2020); Wei et al. (2022). Training on huge amount of data from different
domains has allowed LLMs to acquire human level performance in exams such as
SAT, GRE, AP courses and on platforms leetcode Achiam et al. (2023).

However, the results from King (2023) showed that LLMs performance is heavily
skewed towards textual reasoning tasks like comprehension, textual analogies,
opposites but do not perform good on other types of questions. These large
models have achieved remarkable performance in many tests of human
intelligence, they still fall short of human baselines in tasks requiring
cognitive and logical thinking such as commonsense numerical & scientific
reasoning, puzzles, and analogies Lu et al. (2023); Wang et al. (2023b). This
challenge is evident not only in textual scenarios but also prominently in
visual contexts. Most existing visual and multi-modal reasoning datasets in
the literature are domain-specific, focusing on fields such as science,
engineering, and medicine Yue et al. (2024); Zhang et al. (2024b); Sun et al.
(2024) or they involve reasoning about real-world images. These datasets
primarily assess tasks related to concrete scenarios or specific domains,
which may not necessarily include spatial recognition, visual puzzle solving,
abstract vision logic and pattern recognition. Visual puzzle tasks necessitate
abstract and spatial reasoning abilities that are fundamentally different from
those required for textual or language-based reasoning. However, none of the
existing datasets explicitly focus on testing this aspect. In this study, we
aim to address this research gap by introducing a novel benchmark dataset
(NTSEBench) specifically crafted to evaluate the complex visual, textual, and
multi-modal cognitive reasoning capabilities of large deep learning models
such as LLMs and VLMs. Examples questions from the dataset are shown in Figure
1.

NTSEBench is exclusively dedicated to establishing a benchmark for testing
capabilities that do not rely on domain-specific knowledge or rote learning.
Its primary contribution lies in evaluating the innate problem-solving skills
inherent in human cognitive development and isolating where models lack by
presenting well categorised data. NTSEBench comprises questions sourced from
the nationwide talent search examination(NTSE) conducted in India by the
directorate of education. It includes questions across various problem
categories such as series, calendar & clock, direction sense, analogies, and
more. These questions can be presented in text format, visual format, or both
(multi-modal). We further evaluate the performance of recent LLMs and VLMs,
including both Proprietary Achiam et al. (2023); Reid et al. (2024) and open
source models Achiam et al. (2023); Touvron et al. (2023); Jiang et al.
(2024); Wang et al. (2023a); Bai et al. (2023). This work makes the following
contributions:

  * •

NTSEBench, a dataset to evaluate complex textual, visual and multi-modal
cognitive reasoning capabilities with 2,728 questions in 26 different problem
categories.

  * •

Set up bench mark baselines for multiple state of the art LLMs and VLMs, both
open and proprietary source models.

  * •

Evaluate performance with distinct modelling strategies to handle multi-modal
input handling for reasoning questions.

##  2 NTSEBench

The National Talent Search Examination (NTSE), administered by the National
Council of Educational Research and Training (NCERT) in India since 1963, is a
nationwide exam for 10th-grade students. The exam consists of two sections
designed to assess a wide range of analytical skills: the Mental Ability Test
(MAT) and the Scholastic Aptitude Test (SAT). The MAT section evaluates
students’ general aptitude, critical thinking, logical and spatial reasoning,
and analytical problem-solving skills (for both textual and visual problems).
In contrast, the SAT assesses their domain-specific knowledge in science and
mathematics. All questions in the NTSE are multiple-choice (MCQ) with one
correct option out of four. A set of questions may have a common direction,
which could be common instructions/information about the problems. Questions
and options can be text/image or a combination of both , i.e., multi modal.
Students are given 2 hours to solve 100 questions for each section. We hence
aim to create a dataset focused on cognitive reasoning abilities or MAT-type
questions and developed a benchmark dataset called NTSEBench to evaluate LLMs
and VLMs.

###  2.1 Dataset Sources

To create the dataset, we primarily utilized previous years’ NTSE papers
publicly released along with their solutions provided by Resonance 111Paper
Links, a coaching institute in India that offers paid lessons to students to
help them improve their NTSE scores. Additionally, we used NTSE preparation
materials, such as a reference book titled A Modern Approach to Verbal and
Non-Verbal Reasoning 222A Modern Approach to Verbal and Non-Verbal Reasoning
which includes additional logical reasoning problems. We also incorporated
content from the another book titled Study Guide for NTSE 333Study Guide for
NTSE to construct our dataset.

###  2.2 Extraction Pipeline

As outlined in the dataset sources, we used previous years’ NTSE papers to
create the dataset. We create a pipeline and extract questions from these
sources with human intervention to monitor mistakes.

The data extraction pipeline involves first processing the PDF through MathPix
OCR444MathPix OCR to generate a Word file, which was then manually corrected
for any errors. Next, we used the docxlatex 555docxlatex library to convert
all equations into LaTeX expressions. Finally, we leveraged the PyMuPDF
666PyMuPDF library to extract all text and images, extracting (1) Textual data
i.e. Direction(Extra guidance on the context of the question), the Question
itself,the correct Answer option and the Solution alongside (2) Vision Based
data i.e. relevant images which we segregate into Direction images, Problem
images, Option images, and Solution images.

A total of 2,728 MCQ (multiple-choice questions) comprising of a total of
4,642 images across 26 categories questions are extracted and NTSEBench is
created along with the necessary metadata.

###  2.3 Dataset Details

Problem Categories: NTSEBench encompasses several broad problem categories,
each designed to test a distinct set of skills. Questions from these
categories frequently appear in NTSE exams year after year. The description of
various categories is as in Appendix Table 5.

Problem Sub categories The above categories are further subdivided based on
the different kind of reasoning required. Table 1 lists each sub-category for
Text Only questions and Text+Vision questions, along with the respective count
for each category. As in Table 1 most categories are represented well in the
dataset with mean number of questions being 104 across categories.

Text Only | Vision + Text
---|---
Categories | # Samples | Categories | # Samples
Series | 256 | Non-Verbal Series | 95
Alphabet Test | 94 | Missing Character | 127
Odd one out | 170 | Embedded Figure | 96
Analogy | 151 | Non-Verbal odd one out | 70
Coding-Decoding | 149 | Non-Verbal Analogy | 100
Number and Ranking | 139 | Paper Folding & Cutting | 96
Blood Relation | 126 | Incomplete Figure | 94
Mathematical Operations | 99 | Figure Partition | 71
Puzzle Test | 95 | Cube and Dice | 89
Syllogisms | 44 | Dot problem | 23
Statement & Conclusions | 104 | Direction Sense | 96
Data Sufficiency | 90 | Time and Clock | 51
|  | Mirror, Water and Images | 92
|  | Venn diagrams | 111
Table 1: Problem categories and number of samples

Modality Variations Since NTSEBench has multi-modal questions, options and
solutions, we have results in eight different combination of modality type
that can occur for question-options-solution triplet. Table 2 shows the count
of each triplet option.

Question | Options | Solutions | # Samples
---|---|---|---
✗ | ✗ | ✗ | 1199
✗ | ✗ | ✓ | 381
✗ | ✓ | ✗ | 70
✗ | ✓ | ✓ | 18
✓ | ✗ | ✗ | 330
✓ | ✗ | ✓ | 126
✓ | ✓ | ✗ | 403
✓ | ✓ | ✓ | 201
Table 2: Modality Variations Question Count: Tick(✓) mark indicates whether
question, option or solution contains image.

##  3 Benchmarking using LLMs and VLMs

Figure 2: Examples Showing Input to Different Proposed modelling
strategies.(A) Text Only Standard QA strategy(B) Standard VQA (C) Interleaved
Strategy (D) Image Only.

Lets assume a single($i^{th}$) instance in the dataset is represented by
Di=(Q${}^{J}_{i}$,O${}^{J}_{i}$,S${}^{J}_{i}$), where Q represents the
questions, O represents the options of the MCQ and S represents the solution
to the question. $J\in(T,I)$ represents the modality type which can be either
text($T$) or image($I$).

###  3.1 Modelling Strategies

To accommodate diverse modalities for question, option, and solution triplets,
a single model supporting all combinations can be created by converting all
images to text description via caption generation Li et al. (2022, 2023b) or
convert all text to images and rely on OCR capabilities of the modelFujitake
(2024); Zhao et al. (2023); Shi et al. (2023) to under and answer the question
correctly. Both of these modelling accuracy depends on other sub-tasks, i.e.,
caption generation and OCR.

When questions and options incorporate images, we move into the domain of
utilizing multiple-image inputs for model reasoning. This functionality is
supported through APIs in models such as GPT-4o and GeminiReid et al.
(2024)(proprietary). Due to these sub-task and input dependencies, in order to
assess the reasoning capabilities of the underlying model and achieve a fair
and comprehensive comparison of baseline models, we propose using four
distinct modeling strategies.

####  3.1.1 Standard QA Model for Text-only questions

For instances where question type($J$) for questions($Q$), options($O$) and
solutions($S$) is text($T$), we use a standard text-based QA model like
GPT3.5-Turbo or Llama3-70bAI@Meta (2024) or Mixtral8x7bJiang et al. (2024).

####  3.1.2 Image-Only Model for all Questions

With increasing OCR capabilities of several models Shi et al. (2023); Fujitake
(2024); Zhao et al. (2023) for both document and hand writing recognition,
especially for printed and scanned text.

We propose a modelling approach where all Questions and Options are presented
to the model as a single image. The single image is created by combining all
necessary textual and visual content as it appears in the examination paper,
i.e., a snapshot of the question and options be it textual or visual question.

The model is presented with a single image with a system prompt directing the
model to generate the result in a specified format. The advantage of this
modelling approach is that it can be applied for all modality types of
$D_{i}$.

####  3.1.3 Interleaved model for Text with Multiple Images

In this approach, we integrate text with multiple images to create an
interwoven context. This method involves placing related textual and visual
elements in close proximity, enhancing the model’s ability to draw connections
between them.

####  3.1.4 Standard VQA model with Single Stitched Image and Annotated Text

Open-source models do not support the integration of text and images within a
single prompt. To enable a fair comparison, we propose an alternative modeling
strategy where all question and option images are stitched into a single
image, labeled as Figure 1, Figure 2, etc., and accompanied by a textual
prompt describing different parts of the image. The prompt refers to the
images in a structured manner, guiding the model’s attention to relevant
visual details. This composite image is then used to test the model’s
performance, assessing its ability to understand and respond to questions
based on the integrated visual and textual information.

Example inputs for each of the above modelling strategy proposed are shown in
Figure 2.

###  3.2 Prompting Strategies

We mainly employed two main modelling stragegies for setting up all the
baselines on the proposed dataset, (A) Zero Shot and (B) Few shot, both using
Chain of Thought Wei et al. (2022) prompting to get better results.

Zero Shot COT: Model is given prompt containing question and corresponding
options and asked to choose the right option. The model is also asked to
provide an reasoning behind the choice made by the model. Few Shot COT: In few
shot chain of thought (COT), few samples($N$) with question, options and
solution triplets($D_{i}$) are added to the prompts before asking the question
under test. $N$, the number of exemplars are chosen keep in mind the token
length supported by the model.

###  3.3 Implementation Details

We evaluate NTSEBench using multiple open and closed source LLMs Achiam et al.
(2023); Touvron et al. (2023); Team et al. (2023) and VLMs Reid et al. (2024);
Bai et al. (2023); Dong et al. (2024); Wang et al. (2023a) . We used low
temperature setting to promote reproducibility.

LLMs: GPT-3.5-Turbo, Llama3-70bAI@Meta (2024), Mixtral8x7bJiang et al. (2024)
using the Standard QA strategy with both zero shot COT and few shot COT.

Open-Source VLMs777Evaluated on A6000 machine with half precision used for
internLM: QWEN-VL-chat-7b Bai et al. (2023), CogVLM-2-Llama3-chat-19B Wang et
al. (2023a), internlm-xcomposer2-vl-7b Dong et al. (2024) using the Standard
VQA and Image-Only strategies with zero shot COT. 888Refer to technical
discussion section for details on why few shot COT is challenging with open
source models.

Proprietary VLMs: Gemini-1.5-Pro Reid et al. (2024), GPT-4o have been
evaluated with Interleaving, Standard VQA and Image-Only approaches with both
zero-shot COT and few-shot prompting COT. We provide the prompts and the
hyper-parameters in the appendix A.2 and . We have also evaluated the cheaper
and faster version of Gemini, namely Gemini-1.5-Flash, which has shown
comparable performance to Gemini-1.5-Pro.

##### Answer Extraction and Evaluation.

The answer extraction module is built using a rule-based approach to identify
the correct option from the generated response, handling cases where the
requested format is not followed Gupta et al. (2023). We used percentage
accuracy as our metric for evaluation across all the models. If the model
chooses the right option in the MCQ question label is set of True, else label
is False. There is no partial marking for partially correct answers.

##  4 Results and Analysis

Our experiments answer the following questions:

  * •

How well do large deep learning models perform on advanced textual, visual,
and multi-modal reasoning questions? How challenging is cognitive reasoning
for the current state-of-the-art models?

  * •

Are proprietary models superior to open-source models, and by what margin? Are
there specific categories where proprietary models significantly outperform
open-source models?

  * •

Do different modeling strategies affect a model’s accuracy? Does OCR impact
the reasoning accuracy of models?

###  4.1 Results

Results for text only questions using Standard QA and Image Only(implicit OCR)
modelling strategy are shown in Table 3. For Standard QA strategy we report
results for zero shot setting in all eight models listed in the section above.
For few shot setting, three models are excluded(CogVLM-2, Qwen-VL-Chat,
InternLM-XComposer2) from the results. For Image Only modelling results are
reported for five models which suport vision or multi-modal inputs for zero
shot results. For some categories the solution contains Columns in the table
refers to different types of questions such as SER (Series), ALP (Alphabet
Test), ODO (Odd one out), ANA (Analogy), COD (Coding-Decoding), NUM (Number
and Ranking), BLR (Blood Relation), MTO (Mathematical Operations), PUZ (Puzzle
Test), SYL (Syllogisms), STC (Statement & Conclusions) and DAT (Data
Sufficiency).

| Model | SER | ALP | ODO | ANA | COD | NUM | BLR | MTO | PUZ | SYL | STC | DAT | Avg. Per
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Image Only |
Zero Shot | CogVLM-2 | 14.84 | 17.02 | 20.00 | 19.87 | 24.83 | 16.55 | 23.81 | 20.20 | 20.00 | 22.73 | 22.12 | 15.56 | 19.79
InternLM-XComposer2 | 18.36 | 18.09 | 21.76 | 16.56 | 17.45 | 11.51 | 15.87 | 24.24 | 25.26 | 11.36 | 17.31 | 8.89 | 17.22
Qwen-VL-Chat | 29.69 | 23.4 | 26.47 | 23.84 | 27.52 | 23.19 | 18.25 | 26.26 | 30.53 | 6.82 | 15.38 | 21.59 | 22.74
Gemini 1.5 Pro | 32.42 | 31.91 | 47.65 | 52.32 | 27.52 | 37.41 | 38.10 | 29.29 | 47.37 | 47.73 | 38.46 | 44.44 | 39.55
GPT-4o | 28.12 | 31.91 | 49.41 | 45.03 | 30.87 | 32.37 | 52.38 | 34.34 | 36.84 | 43.18 | 53.85 | 33.33 | 39.30
Few
Shot | Gemini 1.5 Pro | 23.32 | 23.08 | 46.11 | 47.97 | 24.66 | 36.76 | 36.59 | 32.29 | 42.39 | 31.71 | 32.67 | 22.99 | 33.37
GPT-4o | 32.02 | 29.67 | 50.30 | 42.57 | 32.19 | 35.29 | 43.09 | 25.00 | 46.74 | 41.46 | 53.47 | 34.48 | 38.85
| Standard QA |
Zero Shot | Mixtral-8x7B | 19.76 | 19.57 | 24.71 | 45.52 | 14.77 | 26.09 | 29.37 | 29.59 | 32.93 | 24.32 | 53.85 | 33.33 | 29.48
Llama-3 70B | 35.18 | 26.09 | 47.65 | 57.93 | 36.36 | 36.23 | 50.79 | 31.63 | 60.98 | 54.05 | 52.88 | 40.48 | 44.18
GPT-3.5 Turbo | 35.97 | 32.61 | 40.00 | 51.72 | 36.36 | 25.36 | 36.51 | 27.55 | 46.34 | 35.14 | 40.38 | 32.14 | 36.67
CogVLM-2 | 22.27 | 21.28 | 27.65 | 34.44 | 22.82 | 18.71 | 30.95 | 19.19 | 29.47 | 18.18 | 28.85 | 27.78 | 25.13
InternLM-XComposer2 | 21.88 | 24.47 | 19.41 | 36.42 | 25.50 | 28.78 | 25.40 | 27.27 | 45.26 | 40.91 | 34.62 | 28.89 | 29.90
Qwen-VL-Chat | 30.08 | 18.09 | 23.53 | 31.13 | 26.85 | 15.11 | 24.6 | 27.27 | 28.26 | 13.64 | 15.38 | 24.44 | 23.19
Gemini 1.5 Pro | 63.67 | 39.36 | 60.00 | 69.54 | 61.07 | 68.35 | 58.73 | 45.45 | 81.05 | 65.91 | 70.19 | 63.33 | 62.22
GPT-4o | 42.58 | 35.11 | 55.88 | 65.56 | 38.26 | 42.45 | 68.25 | 41.41 | 69.47 | 63.64 | 70.19 | 43.33 | 53.01
Few Shot | Mixtral-8x7B | 27.20 | 24.72 | 28.14 | 50.70 | 29.41 | 27.41 | 33.33 | 29.47 | 18.99 | # | 55.45 | 32.10 | 32.44
Llama-3 70B | 34.00 | 16.85 | 44.91 | 51.41 | 36.47 | 34.81 | 39.84 | 32.63 | 34.18 | # | 50.50 | 34.57 | 37.28
GPT-3.5 Turbo | 30.80 | 32.58 | 20.96 | 47.89 | 30.59 | 31.11 | 30.08 | 29.47 | 36.71 | # | 40.59 | 34.57 | 33.21
Gemini 1.5 Pro | 63.24 | 37.36 | 59.28 | 68.92 | 60.27 | 68.38 | 58.54 | 43.75 | 80.43 | 63.41 | 70.30 | 62.07 | 61.32
GPT-4o | 42.29 | 40.66 | 58.08 | 67.57 | 44.52 | 40.44 | 69.92 | 46.88 | 72.83 | 63.41 | 71.29 | * | 56.17
Table 3: Text Only Results: Zero-shot and Few-shot performance of different
models across various Text-Only categories. We report result using two
different modelling strategies Image Only and Standard QA. italics font for
propriety models, i.e., money or API access is required to run these models.
The # is due to the category’s solution contains thus restricting few shot on
text-only models. Note: (* )In some models, a common issue arises when a model
refrains from providing a response due to safety concerns, often stemming from
misinterpretation of the image’s intent.e.g thinking it as CAPTCHA.

Results for multi-modality question are reported in Table 4. Results are
reported using three different modelling strategies listed in the section
above namely Interleaved, Image Only and  Standard VQA. Column in the table
refer to different categories of visual text reasoning questions in NTSEBench
such as DIR (Direction Sense), VEN (Venn Diagrams), TIM (Time and Clock), MIS
(Missing Character), NVS (Non-Verbal Series), NVO (Non-Verbal odd one out),
NVA (Non-Verbal Analogy), INC (Incomplete Figure), MIR (Mirror, Water and
Images), CUB (Cube and Dice), PAP (Paper Folding & Cutting), EMB (Embedded
Figure), FIG (Figure Partition), DOT (Dot Problem).

Proprietary models outperform Open Source models. From results in table 3 and
4, we can clearly observe that Proprietary models such as Gemini Pro 1.5 and
GPT-4o , outperform other open source models in nearly every category question
category. Proprietary models are generally twice as accurate as open-source
models on NTSEBench questions, including both text-based and multi-modal
questions, across all the different modeling strategies proposed in the
section above. Results from Gemini Pro 1.5 outperform GPT-4o for the majority
of modeling strategies on both text-based and multi-modal questions.
Proprietary models undergo regular updates with refined weights tailored to
specific tasks, as discussed in the literature Reid et al. (2024); Achiam et
al. (2023). Consequently, we posit that proprietary models fine-tuned on
domain-specific reasoning datasets Lu et al. (2023); He et al. (2024) have
acquired advanced reasoning capabilities that contribute to their enhanced
accuracy.

LLAMA-3-70B is the best-performing open-source model for text-only questions,
while Qwen-VL-Chat prevails as the top open-source VLM model for multi-modal
questions.

Modelling Strategy is important. For text-only questions, the Standard QA
strategy clearly outperforms Image Only modeling. Introducing OCR on top of
reasoning tends to confuse models or exacerbate the difficulty of the task.
This effect is particularly noticeable with smaller open-source models, which
struggle to accurately extract characters and integrate them into context.
However, proprietary models like GPT-4o and Gemini-Pro still achieve superior
results using Image Only compared to open-source models employing Standard QA
or text-only processing alone.

Few-shot results present a mixed picture: while models like Mixtral and GPT-4o
show improved performance with added exemplars, others experience a
significant decline. We observe from tables 3 and 4 that proprietary models
generally exhibit a smaller decline in performance compared to open-source
models in such scenarios.

Interleaving text and images performs better than Standard VQA and Image Only
strategy for most categories. This highlights the importance of presenting
text and images separately and in a more fine-grained manner, providing
appropriate context, and integrating the image with text as distinct entities.
This also highlights that decomposing the problems help. Our results in Table
4 show that such an approach can significantly enhance and improve the
performance of Vision-and-Language Models (VLMs). GeminiPro Reid et al. (2024)
achieves the highest accuracy, highlighting the advanced capabilities of
Gemini models in reasoning about different parts of images.

From table 4, we can clearly see that few-shot prompting consistently performs
poorly for all VLM models compared to zero-shot prompting. This suggests that
replicating or inferring underlying reasoning from multi-modal data is notably
challenging.

Multi-modal reasoning is significantly harder. Comparing the best and worst
performing models in Tables 3 and 4, it’s evident that multi-modal reasoning
is considerably more challenging than textual reasoning for current state-of-
the-art VLMs. The highest accuracy on multi-modal questions is below 35%,
whereas for textual reasoning, the best-performing model achieves an accuracy
of over 62%. This underscores the significant gap in reasoning capabilities
between state-of-the-art VLMs and their LLM counterparts, highlighting the
necessity for improved architectures and datasets tailored specifically for
VLM models.

| Model | DIR | VEN | TIM | MIS | NVS | NVO | NVA | INC | MIR | CUB | PAP | EMB | FIG | DOT | Avg. Prec.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Interleaved |
Zero
Shot | Qwen-VL-Chat | 28.12 | 19.82 | 19.61 | 12.6 | 22.11 | 27.14 | 22 | 23.4 | 27.17 | 15.73 | 23.96 | 30.21 | 8.45 | 17.39 | 21.26
Gemini 1.5 Pro | 63.54 | 64.86 | 70.59 | 37.01 | 33.68 | 25.71 | 32 | 38.3 | 35.87 | 43.82 | 30.21 | 36.46 | 46.48 | 30.43 | 42.06
GPT-4o | 37.5 | 50.45 | 41.18 | 29.92 | 16.84 | 22.86 | 26 | 23.4 | 34.78 | 35.96 | 27.08 | 22.92 | 45.07 | 17.39 | 30.81
Few
Shot | Gemini 1.5 Pro | 62.37 | 63.89 | 68.75 | 36.29 | 31.52 | 23.88 | 29.9 | 36.26 | 33.71 | 41.86 | 27.96 | 34.41 | 44.12 | 20 | 39.63
GPT-4o | 39.78 | 52.78 | 52.08 | 27.42 | 17.39 | * | * | 19.78 | * | 38.37 | 33.33 | * | 41.18 | * | 35.79
| Image Only |
Zero Shot | CogVLM-2 | 18.75 | 18.02 | 25.49 | 14.96 | 18.95 | 20 | 8 | 12.77 | 7.61 | 19.1 | 16.67 | 12.5 | 12.68 | 4.35 | 14.98
Qwen-VL-Chat | 21.05 | 26.13 | 27.45 | 22.22 | 26.32 | 21.43 | 17 | 21.28 | 19.57 | 25.84 | 25 | 18.75 | 18.31 | 17.39 | 21.98
InternLM-XComposer2 | 20.83 | 20.72 | 15.69 | 17.32 | 15.79 | 11.43 | 10 | 14.89 | 8.7 | 19.1 | 10.42 | 11.46 | 22.54 | 8.7 | 14.82
Gemini 1.5 Pro | 52.08 | 37.84 | 49.02 | 25.2 | 24.21 | 24.29 | 27 | 26.6 | 29.35 | 32.58 | 23.96 | 23.96 | 42.25 | 34.78 | 32.36
GPT-4o | 40.62 | 31.53 | 33.33 | 22.05 | 22.11 | 25.71 | 19 | 24.47 | 23.91 | 26.97 | 34.38 | 23.96 | 42.25 | 21.74 | 28.00
Few
Shot | Gemini 1.5 Pro | 47.31 | 27.78 | 33.33 | 29.03 | 25 | 23.88 | 21.65 | 23.08 | 21.35 | 37.21 | 32.26 | 19.35 | 22.06 | 25 | 27.73
GPT-4o | 31.18 | 29.63 | 37.5 | 22.58 | 23.91 | 14.93 | 23.71 | 21.98 | 21.35 | 23.26 | 26.88 | 26.88 | 39.71 | 20 | 25.96
| Standard VQA |
Zero Shot | CogVLM-2 | 15.62 | 12.61 | 29.41 | 11.02 | 8.42 | 4.29 | 6 | 3.19 | 11.96 | 15.73 | 9.38 | 10.42 | 8.45 | 17.39 | 11.70
Qwen-VL-Chat | 21.88 | 18.92 | 27.45 | 5.51 | 23.16 | 22.86 | 20 | 24.47 | 26.09 | 8.99 | 20.83 | 19.79 | 8.45 | 8.7 | 18.36
InternLM-XComposer2 | 25 | 20.72 | 25.49 | 17.32 | 18.95 | 8.57 | 15 | 5.32 | 16.3 | 12.36 | 20.83 | 10.42 | 12.68 | 13.04 | 15.85
Gemini 1.5 Pro | 54.17 | 49.55 | 62.75 | 37.8 | 24.21 | 24.29 | 21 | 29.79 | 21.74 | 46.07 | 23.96 | 23.96 | 40.85 | 26.09 | 34.73
| GPT-4o | 50 | 45.95 | 39.22 | 28.35 | 32.63 | 25.71 | 26 | 18.09 | 22.83 | 40.45 | 23.96 | 28.12 | 40.85 | 26.09 | 32.01
Few
Shot | Gemini 1.5 Pro | 61.29 | 47.22 | 68.75 | 32.26 | 17.39 | 16.42 | 18.56 | 27.47 | 20.22 | 44.19 | 20.43 | 25.81 | 44.12 | 25 | 33.50
GPT-4o | 41.94 | 49.07 | 45.83 | 27.42 | 15.22 | 23.88 | 22.68 | 15.38 | 25.84 | 34.88 | 26.88 | 22.58 | 35.29 | 25 | 29.42

Table 4: Multi-modality Results: Zero-shot and Few-shot performance of
different VLM models across various Text+Vision categories. We report results
using all three different modelling strategies proposed to handle
multimodality data, i.e., Interleaved, Image Only and Standard VQA. italics
font for proprity models, i.e., money or API access is required to run these
models. Note: (* )In some models, a common issue arises when a model refrains
from providing a response due to safety concerns, often stemming from
misinterpretation of the image’s intent.e.g thinking it as CAPTCHA.

Question category analysis. The results from Table 3 reveal that while LLMs
generally perform better on the text-only subset of NTSEBench, the high
standard deviation of 11 indicates significant variability in model
performance across different question types. This variability may stem from
overlaps between NTSEBench and other open-source datasets, suggesting that
there are still areas where LLMs exhibit limitations in reasoning
capabilities. This is especially evident in Alphabet Test(ALP) category and
Mathematical Operations(MTO) category, where the accuracy is more than two
standard deviation away from mean accuracy of the model. This could also be
attributed to the potential difficulty of these question types but that
analysis has been left for future work.

VLMs have shown poor performance across all categories for multi-modal
questions, with the best-performing model achieving correct answers only 34%
of the time. Even the standard deviation for accuracy of the best model is 9,
indicating that VLMs struggle more with certain question categories than
others. Specifically, we observe that VLMs perform notably poorly on
categories such as DOT (Dot Problems), NVS (Non-Verbal Series), and NVO (Non-
Verbal Odd One Out). These categories require identifying correlations or
patterns between multiple images, or recognizing emerging patterns in a
sequence of images. This task is akin to identifying similar and evolving
patterns in different parts of an image, and predicting the next possible
pattern. While vision models excel at recognizing existing patterns, they
struggle with predicting new patterns. Improving VLM model performance on
these tasks may necessitate exploring different architectures and training
methodologies in the future.

NTSEBench presents a challenging task for SOTA LLMs and VLMs. Based on the
findings in Tables 3 and 4, it is evident that the proposed dataset presents a
challenging task for all state-of-the-art LLM and VLM models. None of the
open-source models achieve accuracy exceeding 45% on text-only questions and
18% on multi-modal questions, with propriety models achieving 62% and 42%
accuracy respectively. Many of the models tested didn’t every reach random
selection baseline of 24.52%(261 question had 5 options and 2467 question has
4 options), indicating that current LLMs and VLMs have a long way to go to
attain better accuracy on these cognitive reasoning questions. Moreover, these
findings highlight the potential for future advancements in addressing textual
and visual reasoning questions for a variety of categories.

##  5 Related works

Texual and Multimodal Reasoning dataset. There exist multiple datasets to test
domain specific(math, science, medicine) QA and reasoning abilities of LLMs
and VLMs knowledge such as SciBench Wang et al. (2023b), SciEvalSun et al.
(2024), MMMU Yue et al. (2024), MathVista Lu et al. (2023), JEEbench Arora et
al. (2023), MathVerse Zhang et al. (2024a), OlympiadBench He et al. (2024) and
many others on based on real world images or other domains He et al. (2020);
Soni et al. (2022); Liu et al. (2023); Thrush et al. (2022); Li et al.
(2023a). Most existing visual and multi-modal reasoning datasets in the
literature are domain-specific M3Exam Zhang et al. (2024b), RAVENZhang et al.
(2019), or they involve reasoning about real-world images Liu et al. (2023);
Wang et al. (2024); Thrush et al. (2022); Li et al. (2023a) with basic
commonsense reasoning questions such as CLEVR Johnson et al. (2017). However,
current research has not thoroughly explored the capabilities of large models
in addressing cognitive reasoning problems for both textual and multi-modal
data. Datasets such as Liu et al. (2023); Wang et al. (2024); Masry et al.
(2022) may explore mathematical reasoning in visual context but don’t allow
for multiple images within the same prompt or reasoning across images, where
finding a pattern within a sequence of images is required which is the case in
many categories of NTSEBench like Paper Folding and Cutting, Embedded Figure,
Figure Partition, Mirror/Water images etc. NTSEBench is different from all
already existing datasets in the literature because it explicitly focuses on
testing cognitive reasoning abilities of large deep learning models.

Zero shot and Few shot prompting engineering for textual and multi-modal
input. Our work is also related to prompting design and prompt engineering for
LLMs Brown et al. (2020); Chen et al. (2023); Gupta et al. (2023); Khot et al.
(2022); Wei et al. (2022); Sahoo et al. (2024); Ali et al. (2024) and VLMs
gu2023survey; Xu et al. (2024); Bai et al. (2023); Liu et al. (2024); Dai et
al. (2024).There are numerous studies on multi-modal and vision Chain-of-
Thought (CoT) prompting Zhang et al. (2023); Shao et al. (2024). However,
these studies do not explore the potential of state-of-the-art vision-language
models (VLMs) to process multiple images simultaneously. This limitation was
often due to the smaller context windows of older models. However, newer
models now support this functionality and can even handle few-shot examples
for CoT prompting. We explore three distinct prompting strategies made
possible by the extended context windows of large vision-language models
(VLMs).

##  6 Conclusion and Future Work

We have developed a new dataset, NTSEBench, specifically designed to assess
the advanced analytical and logical reasoning capabilities of large deep
learning models (LLMs and VLMs). We also propose four distinct modeling
strategies for handling multi-modal data (text and images) across different
question types in NTSEBench. These strategies enable us to conduct a fair and
comprehensive comparison between closed and open-source models using both
zero-shot and few-shot scenarios. Our findings indicate that both LLMs and
VLMs struggle with advanced visual reasoning tasks, with VLMs performing
particularly poorly on multi-modal questions compared to LLMs on textual
questions. Additionally, proprietary models consistently outperform open-
source models, correctly predicting twice as many questions. Overall, our
results underscore that NTSEBench poses significantly greater challenges for
state-of-the-art LLMs and VLMs.

Future directions. (a) We did not used image-to-text conversion because most
of images in NTSEBench are not real-world images, and caption generation
models are typically trained on real-world images, leading to a domain shift.
This results in subpar captions that are neither relevant nor self-
explanatory. Because LLMs performed significantly better at reasoning task, we
intend to utilize caption generators to investigate whether text-only models,
incorporating image descriptions generated by caption models, can outperform
VLMs. (b)Given the limited data available for cognitive reasoning questions,
we plan to use data augmentation strategies to increase the number of samples.
(c) We also aim to investigate the impact of shuffling multiple-choice
question (MCQ) options on performance in reasoning tasks.

## Limitations

While our dataset is new and sourced for different sources when compared to
dataset already present in the literature, there still might be some overlap
in reasoning questions especially in textual reasoning. So all the dataset
instances might not be independent and exclusive. This dataset is solely
created in English, so no other languages are represented, therefore we cannot
analyze whether language variations can have a significant impact on the
reasoning capabilities of these large models. Our modelling strategies were
limited to zero shot and few shot COT prompting. We did not evaluate whether
finetuning these large models on few examples from each of the categories
would further improve results. This was due to limitation of both GPU
resources and large cost of fine-tuning for Proprietary models. We plan to
address all these limitation in future extension of this work.

## Ethics Statement

As authors of this work, we affirm that our research adheres to the highest
ethical standards in both its conduct and publication. We have carefully
considered and addressed various ethical considerations inherent in
computational linguistics methodologies to ensure responsible and fair
practices. We prioritize transparency by providing detailed information to
facilitate the reproducibility of our results. This includes sharing our code
and datasets, which are sourced from publicly available datasets and handled
in compliance with ethical guidelines established by the original authors.
While our paper accurately reflects our experimental findings, we acknowledge
the inherent variability associated with large language models, which we
mitigate by maintaining a fixed temperature setting. We provide comprehensive
descriptions of our annotations, dataset splits, models used, and prompting
methods employed, ensuring the reproducibility of our work. For grammar
correction, we utilize AI-based writing assistants, and for coding, we
leverage tools like Copilot. Importantly, the genesis of our research ideas
and the execution of our study were entirely independent of AI assistance.

## Acknowledgements

Research was sponsored by the Army Research Office and was accomplished under
Grant Number W911NF-20-1-0080. The views and conclusions contained in this
document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the Army
Research Office or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes notwithstanding any
copyright notation herein. This work was partially funded by ONR Contract
N00014-19-1-2620. We extend our gratitude to the annotators who verified our
flowcharts and corresponding question answer pairs. Lastly, we extend our
appreciation to the reviewing team for their insightful comments.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Ali et al. (2024) Muhammad Asif Ali, Zhengping Li, Shu Yang, Keyuan Cheng, Yang Cao, Tianhao Huang, Lijie Hu, Lu Yu, and Di Wang. 2024.  Prompt-saw: Leveraging relation-aware graphs for textual prompt compression.  _arXiv preprint arXiv:2404.00489_.
  * Arora et al. (2023) Daman Arora, Himanshu Gaurav Singh, et al. 2023.  Have llms advanced enough? a challenging problem solving benchmark for large language models.  _arXiv preprint arXiv:2305.15074_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Chen et al. (2023) Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023.  Unleashing the potential of prompt engineering in large language models: a comprehensive review.  _arXiv preprint arXiv:2310.14735_.
  * Dai et al. (2024) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ , 36.
  * Dong et al. (2024) Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024.  Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.  _arXiv preprint arXiv:2401.16420_.
  * Fujitake (2024) Masato Fujitake. 2024.  Dtrocr: Decoder-only transformer for optical character recognition.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 8025–8035.
  * Gupta et al. (2023) Vatsal Gupta, Pranshu Pandya, Tushar Kataria, Vivek Gupta, and Dan Roth. 2023.  Multi-set inoculation: Assessing model robustness across multiple challenge sets.  _arXiv preprint arXiv:2311.08662_.
  * He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024.  Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.  _arXiv preprint arXiv:2402.14008_.
  * He et al. (2020) Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020.  Pathvqa: 30000+ questions for medical visual question answering.  _arXiv preprint arXiv:2003.10286_.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910.
  * Khot et al. (2022) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022.  Decomposed prompting: A modular approach for solving complex tasks.  _arXiv preprint arXiv:2210.02406_.
  * King (2023) Michael King. 2023.  Administration of the text-based portions of a general iq test to five different large language models.  _Authorea Preprints_.
  * Li et al. (2023a) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023a.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  _arXiv preprint arXiv:2307.16125_.
  * Li et al. (2022) Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. 2022.  mPLUG: Effective and efficient vision-language learning by cross-modal skip-connections.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7241–7259, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International conference on machine learning_ , pages 19730–19742. PMLR.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36.
  * Liu et al. (2023) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023.  Mmbench: Is your multi-modal model an all-around player?  _arXiv preprint arXiv:2307.06281_.
  * Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  _arXiv preprint arXiv:2310.02255_.
  * Masry et al. (2022) Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022.  ChartQA: A benchmark for question answering about charts with visual and logical reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2263–2279, Dublin, Ireland. Association for Computational Linguistics.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_.
  * Sahoo et al. (2024) Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024.  A systematic survey of prompt engineering in large language models: Techniques and applications.  _arXiv preprint arXiv:2402.07927_.
  * Shao et al. (2024) Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024.  Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.  _arXiv preprint arXiv:2403.16999_.
  * Shi et al. (2023) Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. 2023.  Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation.  _arXiv preprint arXiv:2310.16809_.
  * Soni et al. (2022) Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts. 2022.  Radqa: A question answering dataset to improve comprehension of radiology reports.  In _Proceedings of the Language Resources and Evaluation Conference_ , pages 6250–6259, Marseille, France.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_.
  * Sun et al. (2024) Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024.  Scieval: A multi-level large language model evaluation benchmark for scientific research.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 19053–19061.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Thrush et al. (2022) Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023a.  Cogvlm: Visual expert for pretrained language models.
  * Wang et al. (2023b) Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023b.  Scibench: Evaluating college-level scientific problem-solving abilities of large language models.  _arXiv preprint arXiv:2307.10635_.
  * Wang et al. (2024) Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024.  Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning.  _arXiv preprint arXiv:2401.06805_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Xu et al. (2024) Siyu Xu, Yunke Wang, Daochang Liu, and Chang Xu. 2024.  Collage prompting: Budget-friendly visual recognition with gpt-4v.  _arXiv preprint arXiv:2403.11468_.
  * Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9556–9567.
  * Zhang et al. (2019) Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019.  Raven: A dataset for relational and analogical visual reasoning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5317–5327.
  * Zhang et al. (2024a) Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. 2024a.  Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?  _arXiv preprint arXiv:2403.14624_.
  * Zhang et al. (2024b) Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2024b.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_.
  * Zhao et al. (2023) Shuai Zhao, Ruijie Quan, Linchao Zhu, and Yi Yang. 2023.  Clip4str: A simple baseline for scene text recognition with pre-trained vision-language model.  _arXiv preprint arXiv:2305.14014_.

##  Appendix A Example Appendix

###  A.1 Problem Categories

Problem categories with description are shown in Table 5.

Category |  Description
---|---
Series |  Finding the missing element in numerical, alphabetical, or alphanumeric series.
Alphabet Test |  Focusing on operations involving the English alphabet, such as anagrams.
Classification/Odd One Out |  Identifying the item that is different from the others.
Analogy |  Solving problems of the type a:b::c:?
Coding-Decoding |  Deciphering codes and symbols to infer rules and apply them to new examples.
Number and Ranking |  Calculating occurrences or determining order based on certain properties.
Blood Relation |  Solving problems based on family relationships.
Mathematical Operations |  Using mathematical operations like addition, multiplication, subtraction, and division to solve problems.
Direction Sense |  Determining direction and location based on given instructions.
Venn Diagrams |  Using set theory and relationships depicted in Venn diagrams to solve problems.
Time and Clock |  Calculating dates, days, and times based on given information.
Missing Character |  Predicting the missing element in a figure, requiring spatial thinking.
Non-Verbal Series |  Predicting the next element in a sequence of figures.
Non-Verbal Classification/Non-Verbal Odd One Out |  Identifying the figure that is different from the others.
Non-Verbal Analogy |  Solving analogy problems of the type a:b::c:? using figures.
Incomplete Figure |  Identifying the missing part of a figure.
Mirror, Water and Images |  Solving problems related to reflections and image transformations.
Cube and Dice |  Solving problems involving painting, counting, and manipulating cubes and dice.
Paper Folding & Cutting |  Determining the resulting shape after paper folding and cutting.
Embedded Figure |  Finding the alternative which contains a given figure as its part.
Puzzle Test |  Solving general puzzles involving arrangement and deduction.
Figure Partition |  Calculating the number of specific shapes (like triangles) in a figure.
Dot Problem |  Finding similar conditions in alternative figures based on dot arrangements.
Cryptography |  Deciphering codes that involve arithmetic operations.
Syllogisms |  Making inferences based on given statements, often solved using Venn diagrams.
Statement & Conclusions |  Making inferences based on given statements, not typically solved with Venn diagrams.
Data Sufficiency |  Determining whether the given information is sufficient to solve a problem.
Table 5: NTSE Problem Categories. This table lists broad categories of
problems that frequently appear in the NTSE exam. Note: This is not an
exhaustive list; other types of questions may also appear in the NTSE exam.

###  A.2 Prompt Templates

We show the prompt templates across different modeling and prompting
strategies for GPT-4o. We adopt same prompt template for each model withing
the same modeling and prompting strategy.

####  A.2.1 Interleaving

Zero shot

Listing 1: Prompt template for GPT-4o zero shot with interleaving. The
template includes placeholders (‘…‘) for question direction, question text,
answer choices, and their corresponding images, which are encoded in base64
format. All the images are optional and can be there or not based on the
question.

⬇

{

"role": "system",

"content": "You are a brilliant problem solver... Please select the correct
answer choice."

},

{

"role": "user",

"content": [

{"type": "text", "text": "questionDirection: ..."},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestionText: ..."},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nanswerChoices: \n 1. ... \n 2. ... \noptionImages:
\n..."},

{"type": "text", "text": "Answer in the json format as follows: \n {’answer’:
<correct_option_number>, ’explanation’: <explanation>}"}

]

}

Few shot

Listing 2: Few-shot CoT prompt template for GPT-4o. The template includes
placeholders for multiple solved examples (with directions, questions, images,
answer choices, correct answers, and explanations) followed by a new unsolved
question (with directions, questions, images, and answer choices). GPT-4o is
expected to provide a structured answer in JSON format. All the images are
optional and can be there or not based on the question.

⬇

{

"role": "system",

"content": "Understand the following problems carefully... then answer the new
question given at the end."

},

{

"role": "user",

"content": [

{"type": "text", "text": "example 1:"},

{"type": "text", "text": "questionDirection: ... direction image"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestionText: ... question image"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nanswerChoices: \n 1. ... \n 2. ... \noptionImages:
\n..."},

{"type": "text", "text": "{’answer’: <correct_option_number>, ’explanation’:
...}"},

{"type": "text", "text": "So the solution is ... solution image"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\n\n..."},

{"type": "text", "text": "example 2:"},

...

{"type": "text", "text": "example 3:"},

...

{"type": "text", "text": "\n\n..."},

{"type": "text", "text": "now solve this question..."}

{"type": "text", "text": "questionDirection: ..."},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestionText: ..."},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nanswerChoices: \n 1. ... \n 2. ... \noptionImages:
\n..."},

{"type": "text", "text": "Answer in the json format as follows: \n {’answer’:
<correct_option_number>, ’explanation’: <explanation>}"}

]

}

####  A.2.2 Image Only

Zero shot

Listing 3: Zero-shot prompt template for image-only question answering. The
prompt includes a system instruction to solve the problem and provide the
answer in JSON format, followed by the input image encoded in base64.

⬇

{

"role": "system",

"content": "You are a brilliant problem solver... answer the correct option
from the given choices along with a explanation. Answer in form of json in
this format: {’answer’: <correct_option_number>, ’explanation’:
<explanation>}"

},

{

"role": "user",

"content": [

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

]

}

Few shot

Listing 4: Few-shot prompt template for image-only question answering. The
template demonstrates two examples, each with a question image, the correct
answer, an explanation, and potentially solution images. It then presents a
new question image and asks for a structured answer in JSON format.

⬇

{

"role": "system",

"content": "Understand the following problems carefully ..."

},

{

"role": "user",

"content": [

{"type": "text", "text": "You are a brilliant problem solver... Please select
the correct answer choice."},

{"type": "text", "text": "example 1:"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\n so the solution to this example is as
follows\n{’answer’: <correct_option_number>, ’explanation’: ...}"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\n\n..."},

{"type": "text", "text": "example 2:"},

...

{"type": "text", "text": "example 3:"},

...

{"type": "text", "text": "\n\n\n now answer the following question"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

]

}

####  A.2.3 Standard VQA

Zero Shot

Listing 5: Zero-shot prompt template for Standard VQA on GPT-4o. The template
instructs the model to solve a multiple-choice question with reference to an
image. The expected output is a JSON object with the correct answer number and
a corresponding explanation.

⬇

{

"role": "system",

"content": "You are a brilliant problem solver. Solve the given multiple
choice question. Answer in the json format as follows: \n {’answer’:
<correct_option_number>, ’explanation’: <explanation>}"

},

{

"role": "user",

"content": [

{"type": "text", "text": "refer to this image for references in question:
\n"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestion: ... \n answer the question with the
correct option number and explanation in the json format as follows: \n
{’answer’: <correct_option_number>, ’explanation’: <explanation>}"}

]

}

Few shot

Listing 6: Few-shot prompt template for Standard VQA on GPT-4o. The template
showcases a few-shot example with question, images, answers, explanations, and
optional solution images, followed by a new question for the model to answer
in JSON format.

⬇

[

{"type": "system", "content": "You are a brilliant problem solver... First
understand the provided questions and then answer the new question given at
the end."},

{"type": "user", "content": [

{"type": "text", "text": "example 1:"},

{"type": "text", "text": "refer to this image for references in question:
\n"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestion: ... \n answer the question with the
correct option number and explanation in the json format as follows: \n
{’answer’: <correct_option_number>, ’explanation’: <explanation>}"},

{"type": "text", "text": "so the answer to this question is as follows: \n
{’answer’: <correct_option_number>, ’explanation’: ...}"},

{"type": "text", "text": "refer to this image for solution: \n"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\n..."},

{"type": "text", "text": "\n now as you have got the idea of the questions,
let’s answer the following question with a thorough explanation: \n"},

{"type": "text", "text": "refer to this image for references in question:
\n"},

{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},

{"type": "text", "text": "\nquestion: ... \n answer the question with the
correct option number and explanation in the json format as follows: \n
{’answer’: <correct_option_number>, ’explanation’: <explanation>"}

]}

]

###  A.3 Model Hyper parameters

GPT-4o:

  * •

Temparature = 0.0

  * •

Output_format = json

GPT-3.5:

  * •

Temparature = 0.0

  * •

Output_format = json

Gemini-1.5-Pro:

  * •

Temparature = 1.0

Qwen-vl-chat

  * •

seed = 1234

  * •

precision = float16(half)

cogagent2-Llama3-8b

  * •

precision=bf16

interLM-XComposer

  * •

precision=half

The rest of the hyperparameters as default on huggingface.