#  VisReas: Complex Visual Reasoning with Unanswerable Questions

Syeda Nahida Akter1, Sangwu Lee2, Yingshan Chang1, Yonatan Bisk1, Eric Nyberg1
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA,
United States1
Department of Computer Science, University of Rochester, Rochester, NY, United
States2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Verifying a question’s validity before answering is crucial in real-world
applications, where users may provide imperfect instructions. In this
scenario, an ideal model should address the discrepancies in the query and
convey them to the users rather than generating the best possible answer.
Addressing this requirement, we introduce a new compositional visual question-
answering dataset, VisReas, that consists of answerable and unanswerable
visual queries formulated by traversing and perturbing commonalities and
differences among objects, attributes, and relations. VisReas contains 2.07M
semantically diverse queries generated automatically using Visual Genome scene
graphs. The unique feature of this task, validating question answerability
with respect to an image before answering, and the poor performance of state-
of-the-art models inspired the design of a new modular baseline, Logic2Vision
that reasons by producing and executing pseudocode without any external
modules to generate the answer. Logic2Vision outperforms generative models in
VisReas (+4.82% over LLaVA-1.5; +12.23% over InstructBLIP) and achieves a
significant gain in performance against the classification models.

##  1 Introduction

Figure 1: Overview of VisReas dataset construction process. Using scene
graphs, we cluster objects (orange), relations, and attributes of the related
objects (blue) based on the attribute of the corresponding objects (orange).
Then the question engine takes each template as input and traverses all
possible clusters to generate the query as well as the reasoning steps. Each
function in the reasoning steps can return NONE if any object, attribute, or
relation is absent in the image.

In visual question answering (vqa), validating question authenticity with the
corresponding image and then reasoning over it is an important requirement in
real-world application dynamics where users may make errors in judgment,
leading to invalid queries. Confirming a question’s validity becomes pivotal
to maintaining consistency, rectifying mistakes, and preventing misguided
responses Rajpurkar et al. (2018). Following the prior vqa datasets’ Goyal et
al. (2017); Krishna et al. (2016); Hudson and Manning (2019b) focus on
answerable questions only, a system trained solely for answerable questions
may exhibit unstable behaviors when faced with unanswerable queries. For
instance, a delivery robot receiving an incorrect address but a valid
instruction like “place the package by the yellow door” might overlook the
error unless prompted to reevaluate its decision. In contrast, presuming the
correctness of the query would likely lead to unpredictable behaviors.
Therefore, a reliable and responsible system should be able to question the
validity of the instruction it receives before acting upon it.

However, aligning questions with the region of interest in the image breaks
down visual reasoning task into perception (object detection and scene
representation learning) and reasoning (question interpretation and inference
grounded in the scene). Datasets and models proposed to date have shown
significant improvement in the detection task which therefore improved the
perception system Goyal et al. (2017); Krishna et al. (2016); Tan and Bansal
(2019), but they face critical vulnerabilities due to the lack of generalities
in the datasets Zhang et al. (2016a); Agrawal et al. (2016). Recent datasets
Johnson et al. (2017); Selvaraju et al. (2020); Hudson and Manning (2019b)
encourage reasoning beyond surface-level object recognition and focus on
multi-step inference. But they tend to reason about object relations (often
questions revolving around single object) instead of reasoning over clusters
of objects in the image that share common attributes or relations. Reasoning
over general sets of objects requires both identifying objects and
understanding their attributes and relations.Where prior scene-graph based
work assumes reasoning follows from traversing a single path to generate an
answer, our goal is to establish a multi-hop approach of identifying cliques
with shared properties.

Bridging the gap in prior benchmarks, we introduce a new dataset, VisReas
(Visual Reasoning), for studying reasoning over commonalities and differences
across objects. The unnatural assumption in the current VQA datasets - “a
correct answer for every question” causes models to produce an answer even
when the question is inapplicable and has no possible answer. To ensure that
models verify the consistency of question text with the image before
answering, we curate questions that have no answer given the image by altering
relations and attributes among the objects. We design a question generation
engine that takes the information about objects, attributes, and relations
from the Visual Genome scene graphs Krishna et al. (2016) and finds common
features shared among multiple objects. Based on this retrieved information,
we generate $2.07$M unique questions covering vast semantic variations. Each
question is paired with a scene graph and a semantic program that specifies
the series of reasoning steps needed to be performed to produce the answer.
Our generated questions require visual reasoning abilities such as comparing,
differentiating, counting, clustering objects, and performing logical
reasoning. Most importantly, unlike other vqa datasets, VisReas enforces the
vqa models to verify the information in the question with the image in each
reasoning step before predicting an answer.

We find existing vqa models less robust in the reasoning and unanswerable
settings presented by VisReas. Motivated by the shortcomings of existing
models, we propose a new architecture, Logic2Vision that has been trained to
produce logical reasoning steps from the query at first and then predict
answers based on the reasoning steps and the image. Unlike prior generative
models, Logic2Vision is compute and cost-efficient as it does not require any
external expensive APIs or modules and solely relies on the reasoning
capabilities of visual language models (vlm). Experiments on VisReas shows
that Logic2Vision outperforms the current fine-tuned vqa models: obtaining
66.20% (+4.82% over LLaVA-1.5 Liu et al. (2023), +12.23% over InstructBLIP Dai
et al. (2023)) accuracy on VisReas.

In short, our contributions are twofold:

  * •

We introduce VisReas, a dataset containing complex yet natural reasoning. Our
dataset makes the first step towards developing reliable vlm adaptable to
real-world scenarios where user instructions may not always be impeccable.

  * •

We present Logic2Vision, that aims to handle spatial reasoning by executing
consecutive pseudocode with verification in each step.

##  2 Related Works

Recent years have witnessed tremendous progress in visual understanding.
Multiple attempts have been made to mitigate the systematic biases of VQA
datasets Goyal et al. (2017); Zhang et al. (2016b); Agrawal et al. (2018);
Johnson et al. (2017), but they fall short in providing an adequate solution:
Some approaches operate over constrained and synthetic images Zhang et al.
(2016b); Johnson et al. (2017), neglecting the realism and diversity natural
photos provide. Suhr et al. (2019) introduced a dataset for reasoning about
semantically-diverse natural language descriptions of images in the form of a
classification task. While the dataset exhibits diverse semantic phenomena,
this task rarely requires much beyond a single type of object recognition and
its associated relation and attribute. Unlike these datasets, VisReas is open-
ended and consists of both unanswerable and answerable queries based on the
similarity/dissimilarity of multiple objects in the image. VisReas jointly
evaluates vqa models’ alignment, multihop reasoning, and verification ability
which cannot be approximated by simply finding the most likely
object/relation/attribute to answer the question.

Recent transformer-based models have Tan and Bansal (2019); Lu et al. (2020);
Nguyen et al. (2022) achieved promising performance on visual reasoning tasks.
Yet, these models are prone to reproducing spurious correlations without
accurately learning true causal relations Agrawal et al. (2016); Jia and Liang
(2017); Tenenbaum (2018). Neural-symbolic methods Andreas et al. (2016); Hu et
al. (2017); Hudson and Manning (2018, 2019a) explicitly perform symbolic
reasoning on the object and language representations. These models offer
modularity and interpretability in the reasoning process. However, as module
parameters are usually derived solely from end-task supervision, there is a
potential for the program to deviate from accurately explaining the model’s
behavior Ross et al. (2017); Jain and Wallace (2019); Subramanian et al.
(2020).

Conversely, a recent approach to modularity leverages Large Language Models
(llm) to craft code or Python programs using expensive APIs Chen et al.
(2021); Surís et al. (2023); Gupta and Kembhavi (2023); Subramanian et al.
(2023). However, these approaches outsource basic aspects of the reasoning to
external components rather than performing reasoning as part of the model
itself. For example, prior works outsource basic cognitive abilities such as
recognizing objects, counting, and even arithmetic operations. Focusing on
these limitations, our proposed Logic2Vision aims to leverage single vlm to
address complex reasoning in a modular approach that shows promising
performance across models of three different categories.

##  3 VisReas: Visual Reasoning

The VisReas dataset is an attempt towards better aligning model capabilities
with real application circumstances. In parallel, VisReas aims to develop
complex compositional reasoning into the machine that involves consideration
of relations among multiple objects and verification of alignment between
information provided in the question and the image. In the following sections,
we provide details about the VisReas data generation pipeline and a
comprehensive analysis of the VisReas dataset. In the supplementary material,
we conduct a detailed comparative study between VisReas and the well-
established gqa dataset, followed by details of the human evaluation process
using Mechanical Turk.

###  3.1 Data Generation

Our dataset is constructed in three major steps: (1) Process scene graphs, (2)
Define templates and reasoning functions that the question will involve, (3)
Automatically generate corresponding reasoning steps in pseudocodes along with
the final answer from each query as shown in Fig. 1. Finally, to prevent
models from learning statistical biases in attribute, reasoning, or answer
type distributions, we meticulously balance the VisReas dataset across three
distinct paradigms (Appendix A).

####  3.1.1 Scene Graph Processing

To begin with the data construction process, we run two phases of processing
on the scene graphs before passing them to the question engine.

First Phase. We clean up the scene graphs by removing opposite attributes and
discarding object nodes with similar names that share similar attributes and
relations. Our processed scene graphs contain 1703 distinct objects, 14
attributes, and 114 relationships. It is also observed that one object name in
the image might correspond to multiple object IDs and bounding boxes in the
scene graph. This will cause ambiguity in the later question-generation
process. Thus, we merge bounding boxes corresponding to the same object name
with a high IoU ($>$ 0.7). In addition, there can be images where a bigger
bounding box contains multiple small bounding boxes, which can be either parts
of the object represented by the bigger bounding box (e.g., a cat (bigger
bounding box) has a tail, ear, face (small bounding boxes), etc.) or they can
collectively represent the object in the bigger bounding box (e.g., lime and
apple can together be mentioned as fruits). These overlapping bounding boxes
will be problematic while clustering objects based on similar attributes
(e.g., fruits and lime are all green; for ‘What has the same color as the
lime?’ the answer generation module will produce: fruits and apple - which is
ambiguous). To discard these cases, we measure the ratio of intersection area
vs individual bounding box area and check whether the smaller objects are
subclasses of the bigger one using Wordnet Miller (1994). If the ratio is high
and the larger object is a superclass of the smaller one, we discard the
larger bounding box during preprocessing to avoid ambiguity.

Second Phase. We cluster the scene graphs based on the common attributes and
relations among the objects in each image and create several sub-graphs as
seeds for the question engine. Initially, we cluster objects based on a single
relation or attribute, later we merge the clusters recursively if there are
objects with multiple attributes or relations in common. Finally, each cluster
represents a collection of objects that share a similar set of attributes and
relations and the question engine exhaustively traverses all clusters to
generate questions. For each object in a cluster, we also store other objects
that are related to that object along with their relation name. This
information is used to populate nested compositional references for multi-hop
relation traversal.

####  3.1.2 Question Engine

For question generation from the clusters, we manually create 182 templates on
different attributes (Table 1). Our templates cover five categories of
reasoning (query, count, compare, verify, and choose) which can be further
broken down into nine broad categories of reasoning mentioned in Appendix. For
some categories, we have list answers and no-answer cases. All of our
templates are formulated considering clusters of objects to facilitate multi-
object comparison. To generate no-answer cases, we apply two approaches: (1)
We either add an outlier (object not present in the image) to the cluster or
include an object that exists in the image but not in the cluster and has
different relations and attributes from the objects in the cluster. (2) We
perturb the existing relation/attribute of an object inside a cluster (e.g.,
change ‘apple to the left of knife’ to ‘apple to the right of knife’) which
derives no-answer cases.

Attribute | Templates | Train | Validation
---|---|---|---
Color | 12 | 1326086 | 1500
Cleanliness | 8 | 7794 | 900
Material | 15 | 368337 | 1500
Size | 4 | 116438 | 1500
Pose | 18 | 36687 | 1500
Height | 10 | 9894 | 1200
Weather | 6 | 31376 | 1500
Length | 11 | 45764 | 1500
Tone | 11 | 37184 | 1500
Shape | 15 | 30119 | 1500
Activity | 21 | 15639 | 1500
Sport Activity | 21 | 13215 | 1500
Age | 12 | 19594 | 1500
Pattern | 18 | 14313 | 1500
Total | 182 | 2072440 | 20100

Table 1: Question-template distribution over attributes

####  3.1.3 Answer Generation

The answer generation step involves two consecutive phases. Initially, we
formulate the reasoning steps in pseudocode (Figure 1) and produce the
intermediate results for each line of code using our designed parser (Figure
9). For each question template and reasoning type, we have hand-coded the
basic reasoning steps necessary to answer the query. Based on the number of
objects, relations, and attributes, our parser generates all intermediate
reasoning steps along with the answers. Finally, we combine all intermediate
results to come up with the answer. If any intermediate reasoning step results
in ‘NONE’, the final answer becomes ‘the question itself is problematic’
indicating some objects, relations, or attributes mentioned in the question
text cannot be found in the image.

| # QA | # Images | AveQLen |  ListAns |  Grounding |  Counting |  RealWorld |  No Answer |
---|---|---|---|---|---|---|---|---|---
VQA | 614,163 | 204,721 | 6.2 $\pm$ 2.0 |  |  |  |  |  |
Visual7W | 327,939 | 47,300 | 6.9 $\pm$ 2.4 |  |  |  |  |  |
CLEVR | 853,554 | 100,000 | 18.3$\pm$ 3.5 |  |  |  |  |  |
GQA | 1,750,623 | 113,000 | 7.9 $\pm$ 3.1 |  |  |  |  |  |
VisReas | 2,072,437 | 113,000 | 19.4 $\pm$ 4.6 |  |  |  |  |  |

Table 2: Comparisons on existing VQA datasets. VisReas covers a wide variety
of reasoning along with No Answer cases. The average question length is also
higher in VisReas compared to others.

###  3.2 Dataset Analysis and Comparison

The VisReas dataset consists of 113K images from the Visual Genome where each
image is annotated with dense descriptions of the scene stored in the scene
graphs. We refine the existing scene graphs and generate $2,072,437$M unique
questions, twice the size of current vqa datasets (Table 1), that combine
features of multiple objects and their relations and require the
implementation of consecutive complex reasoning skills with an in-depth
understanding of object attributes and relations in the image. Our dataset
covers 14 different attributes and 114 diverse relations among 1703 different
objects from real-life images. We define five major types of reasoning (Figure
2) while generating the corpus based on the overall nature of the query
template. Figure 5 shows details of the query structures along with examples.
However, the intermediate reasoning steps that are necessary to answer the
query can be diverse and can combine all five types of reasoning for a single
query (as in Figure 1). We balance the dataset combinedly across 14 attributes
and 5 reasoning types (Appendix A).

Figure 2: Overview of VisReas statistics. (Top left) The dataset covers 14
attributes in a balanced ratio. (Top right) It consists of five reasoning
types of queries in a balanced distribution. (Bottom left) Comparison of
multi-hop relation traversal for different VQA datasets. Majority questions of
VisReas require multi-hop traversal compared to others. (Bottom right)
Comparison of number of objects mentioned in the question for different
datasets where VisReas questions contain larger amount of objects.

Compared to existing vqa tasks, VisReas emphasizes creating longer reasoning
chains (multi-hop) with a larger number of objects (Figure 2). The average
number of reasoning hops for VisReas is $1.42$ ($95$% CI: $[1.415,1.417]$),
significantly higher than gqa (mean: $0.52$; $95$% CI: $[0.517,0.519]$) and
clevr (mean: $0.84$; $95$% CI: $[0.839,0.843]$). However, to limit the
question length and increase human readability (Figure 6), the majority of the
questions require at most two hops relation traversal for each object.

Reflecting on human clustering ability based on commonalities, VisReas
consists of queries that require consideration of multiple objects based on
their attribute or relation similarities. Therefore, unlike existing datasets,
the majority of VisReas queries are composed of more than three objects from
the image. The average objects per question for VisReas is $3.91$, which is
higher than both gqa ($1.12$) and clevr ($1.63$). Hence, VisReas requires
multiple object detection and consecutive reasoning to answer a single query
(Figure 2). In addition, each query can have multiple attributes associated
with it (7(a)). For example, in question, ‘What is the common material among
the silver and blue utensils?’, both <material> and <color> attributes are
needed to be considered for answer generation that involves multiple attribute
filtering along with the associated objects.

Figure 3: VisReas contains two types of relation traversals. Star relation
states a single object that shares multiple relations with other objects
(Left). Chain relation states multiple objects that share a single relation
with each other (Right).

In contrast to other spatial reasoning datasets that focus primarily on one-
hop relation traversals (Bottom left of Figure 2), we explore two ways of
novel traversals: (1) Star Relation: The target object shares multiple
relations with other objects (e.g. is the center of the star and other objects
are connected to it with a relation – Figure 3 left), and (2) Chain Relation:
The target object is related to an object that is related to another object
and the relation traversal is linear (Figure 3 right). The inclusion of these
traversals adds multi-hop complexity to the corpus and makes the each-step
verification process harder for unanswerable questions (as Figure 10).

##  4 Logic2Vision

Figure 4: Overview of Logic2Vision. In Pseudocode Generation phase, we
generate pseudocode which outlines the reasoning steps. During Pseudocode-
Guided Reasoning, the pseudocodes along with the question and image are
provided to the model. The model executes all intermediate pseudocodes to
arrive at the final answer.

In recent years, llms combined with code generation and chain-of-thought
prompting have shown impressive performance in complex reasoning by generating
intermediate reasoning steps before inferring the answer Zhang et al. (2023a);
Surís et al. (2023). However, these frameworks are often prone to
hallucinations of llms and are too restricted in terms of reasoning they can
perform and dependent on expensive external modules to execute the reasoning
Zhang et al. (2023b); Surís et al. (2023). To address these limitations and
elicit the reasoning capability of vlms, we propose Logic2Vision, a two-stage
vqa framework that (1) plans the necessary reasoning steps using the question
and (2) executes the plan with the help of an image leveraging the SOTA vlm
(Figure 4).

###  4.1 Stage 1: Pseudocode Generation

Given a natural language question, this module generates a consecutive set of
reasoning steps as pseudocodes. For training our pseudocode generation model,
we take advantage of the existing vqa dataset: gqa as it provides a semantic
string that decomposes the question into a sequence of reasoning steps. For
instance, the semantic string for the question ‘Is there a red apple on the
table?’ would be ‘select: table $\rightarrow$ relate: on, subject, apple
$\rightarrow$ exist: ?’. We build a custom parser (Figure 9) that converts
each line of gqa semantic string to pseudocode and extract the expected output
from the scene graph. The parsed (pseudocode, output) pairs serve as a
rationale to solve the question (Figure 8). For the pseudocode generation
model, we use an instruction finetuned vicuna-13B Chiang et al. (2023) model
which has shown good performance across various language tasks including code
generation. We finetune vicuna using LoRA Hu et al. (2022) to generate the
pseudocode for given a question. The finetuned model achieves 98.6% METEOR
Banerjee and Lavie (2005) score and 96.3% ROGUE-L Lin (2004) score against
ground-truth code parsed from gqa semantic strings.

###  4.2 Stage 2: Pseudocode-Guided Reasoning

Since the Pseudocode Generation module outlines the necessary steps to answer
the question, the remaining task is to perform pseudocode-guided sequential
reasoning on the image. For this stage, we choose state-of-the-art vlm,
LLaVA-1.5 Liu et al. (2023), due to its impressive performance in diverse
reasoning tasks. As LLaVA-1.5 was not trained to reason with pseudocode and
image, we fine-tuned it to generate an answer by executing sequential
reasoning with the pseudocode and the image. To adapt this framework in our
case, we rearrange the instruction as:

<Image> Given an image, the user asks a question and assistant executes the
code and logs the results step-by-step to provide an answer. Question:
{Question} Code: {Code} Outputs: {Outputs} Answer:

The essential training details of this stage can be found in subsection C.3.

##  5 Experiments and Analysis

In the subsequent sections, we conduct a comprehensive analysis of the VisReas
dataset and assess the performance of various benchmarks including
Logic2Vision, gpt-4v OpenAI (2023), and human participants, revealing a
notable disparity from human performance.

###  5.1 Baseline Experiments

To analyze the complexity and generalizability of our dataset and model, we
run experiments with models trained on both classification and generative
tasks. We cover two types of generative models: GEN (relies on pretrained
visual-language alingment module) and Code-GEN (generates a program and
utilizes external APIs to solve vqa tasks). We categorize Logic2Vision as
Logic-GEN as it produces intermediate logical reasoning steps before
answering. All model configurations can be found in Appendix C. To make the
training and inference consistent, we define our own prompt for all generative
models (as subsection C.4). Table 3 shows the results of different baselines
on both gqa and VisReas. Table 4 shows the finetuned results on VisReas and
further breaks down the performance of each model in each reasoning type along
with gpt-4v and human accuracy.

|  | Model | Accuracy (%)
---|---|---|---
gqa | VisReas
ZS | GEN | BLIP-2 Li et al. (2023) | 44.70 | 35.16
InstructBLIP Dai et al. (2023) | 49.50 | 36.84
LLaVA-1.5 Liu et al. (2023) | 63.3∗ | 38.98
Code-GEN | ViperGPT Surís et al. (2023) | 48.10 | 10.31
VisProg Gupta and Kembhavi (2023) | 50.50 | 20.82
FT | CLS | LXMERT Tan and Bansal (2019) | 60.05 | 50.15
ViLBERT Lu et al. (2020) | 60.65 | 53.05
CRF Nguyen et al. (2022) | 72.10 | 53.56
Logic-GEN | Logic2Vision | 60.32 | 66.20

Table 3: Performance comparison among baseline models on GQA and VisReas. (*)
GQA trainset images were used during training.

[CLS] For models trained with classification task, we finetune and evaluate on
both gqa and VisReas. From the fine-tuning results of the CLS models, it is
obvious that VisReas proposes a different task than gqa that can not be easily
solved by scaling the model size or changing the pretraining corpus.
Furthermore, the higher performance gap of the models between gqa and VisReas
tasks suggests the inefficacy of the existing CLS models on our proposed
spatial reasoning task.

Metric | CLS | GEN | Code-GEN | Logic-GEN | GPT-4V | Humans
---|---|---|---|---|---|---
LXMERT | ViLBERT | CRF | BLIP-2 | InstructBLIP | LLaVA-1.5 | ViperGPT | VisProg | Logic2Vision
Choose | 74.23 | 82.91 | 83.30 | 71.21 | 78.50 | 84.11 | 10.37 | 15.86 | 82.54 | 82.61 | 91.30
Compare | 65.62 | 69.86 | 71.87 | 28.72 | 53.29 | 67.75 |  05.97 | 26.09 | 59.25 | 68.33 | 86.12
Count | 45.32 | 47.80 | 49.59 | 25.88 | 49.86 | 43.08 |  07.85 |  06.02 | 39.47 | 39.52 | 85.78
Query | 44.05 | 47.65 | 48.11 | 41.55 | 47.77 | 50.31 |  04.35 | 19.30 | 63.79 | 58.78 | 81.78
Verify | 76.10 | 82.18 | 83.03 | 70.77 | 49.48 | 81.27 |  03.10 | 44.18 | 84.54 | 82.16 | 93.94
Problematic | 67.54 | 77.08 | 78.41 | 25.39 | 64.68 | 68.04 |  00.00 |  00.00 | 55.34 | 70.18 | 90.29
Non-Problematic | 56.11 | 59.16 | 61.60 | 51.41 | 52.25 | 60.31 | 11.97 | 24.17 | 67.94 | 55.47 | 84.89
Accuracy (%) | 50.15 | 53.05 | 53.56 | 47.81 | 53.97 | 61.38 | 10.31 | 20.82 | 66.20 | 62.83 | 87.21

Table 4: Accuracy breakdown of baseline models and humans on VisReas across
different reasoning types. Problematic type consists of questions that contain
certain relation, attribute, or object that is missing/ not consistent with
the image. In contrast, Non-Problematic questions have correct answers as the
question is consistent with the image. Except for the Code-GEN models, we
provide fine-tuned results on VisReas for all other models.

[GEN] From generative domain, we select three SOTA models, BLIP-2,
InstructBLIP, and LLaVA-1.5, that try to leverage the LLMs using two types of
vision-language alignment modules: Q-Former and MLP cross-modal connector. We
evaluate the models on zero-shot gqa and VisReas to probe the relevance of our
proposed task to their training domain. We notice that BLIP-2 performs poorly
on our task compared to gqa where InstructBLIP and LLaVA-1.5 shows higher
accuracy. Both LLaVA-1.5 and InstructBLIP are instruction tuned on diverse
downstream tasks which allows them to excel in vqa tasks compared to BLIP-2.
However, LLaVA-1.5 gains the highest zero-shot accuracy in this category due
to its training set images being overlapped with VisReas. Yet, it shows a
significant drop (-24.32%) in ZS accuracy compared to gqa, which proves that
VisReas highlights a novel reasoning task that can not be generalized using
gqa. Furthermore, the smaller performance gap among these models on VisReas
suggests the inefficacy of the current vlms on our proposed spatial reasoning
task.

[Code-GEN] From modular Code Generation models, we analyze recent works -
ViperGPT and VisProg. These models employ an llm to generate an executable
program that utilizes a pre-defined API, including functions such as
detect(image, obj_category) or segment(image, obj_category). VisProg also
utilizes the “in-context learning” abilities of llms, enabling the model to
respond to new queries with just a few examples of input and output behavior.
Zero-shot evaluations of Code-GEN models on gqa and VisReas reveal that
current models are struggling with our task more than gqa, where both corpora
use similar images. Specifically, their poor performance in Non-Problematic
questions denotes the inability of these models to cluster objects based on
commonalities.

###  5.2 Analysis

According to Table 4, all the models including gpt-4v struggle in Compare,
Count, and Query question-types which require grounding, clustering, and
verifying the existence of multiple objects, relations, and attributes.
Specifically in Query, the performance gap between humans and the models is
significantly higher which demonstrates the limitation of current models to
perform complex multi-hop reasoning. Logic2Vision, on the other hand, shows a
promising result in Query questions. We hypothesize that structured pseudocode
helps the model consider each object and its corresponding attributes and
relations before answering while the other models try to learn from the
surface-level word distribution. In addition, Query questions are in general
lengthier than other types of questions which makes it easier for the models
to lose attention to the details (7(b)).

In contrast, gpt-4v outperforms all generative models in Problematic
questions. After analyzing the predictions, we find that gpt-4v excels at
identifying problematic questions that involve an object not present in the
image or an object with a false attribute. However, when the question becomes
problematic due to an incorrect relation, gpt-4v consistently struggles to
recognize it which also holds for other models. This signifies the uniqueness
of our corpus that emphasizes understanding relations beyond simple object
detection.

To investigate the effect of llm’s scale on the vqa task, we test two versions
of llms (vicuna 7B and 13B) within VisReas architecture. Table 5 breaks down
the performance of Logic2Vision in the presence of different llms. We observe
that increasing llm’s size dramatically increases the accuracy of longer
questions (7(b)) such as Non-Problematic, Count, Query, and Compare instances
and marginally improves performance on question categories such as Choose and
Verify. This finding reassures the ability of larger llm to reason with longer
context. However, for problematic questions, increasing llm size has no
impact. As this category requires verification and grounding of information
with image, both llm and vision-language alignment need to be strong to excel
in this domain.

Model |  Choose |  Compare |  Count |  Query |  Verify |  Prob. |  Non-Prob. | All
---|---|---|---|---|---|---|---|---
7B | 81.20 | 54.90 | 35.13 | 59.24 | 82.75 | 55.38 | 63.92 | 62.74
13B | 82.54 | 59.25 | 39.47 | 63.79 | 84.45 | 55.34 | 67.94 | 66.20

Table 5: Breakdown of accuracies on VisReas for Logic2Vision’s vicuna model
size. We observe that vicuna’s model size improves performance in most
question-types except the problematic ones.

##  6 Conclusion

We introduce the VisReas dataset, for real-world complex and multihop visual
reasoning and compositional question answering. The dataset emphasizes object
commonalities, differences, and relational aspects, necessitating validation
of question-text relevance with the image before answering. We describe the
dataset curation process along with the performance of SOTA models from three
different domains in our task. Addressing the shortcomings in grounding and
clustering in recent models, we propose a novel Logic2Vision baseline that
deconstructs questions into pseudocodes and sequentially executes them using
images to generate answers. We anticipate that this dataset and model will
catalyze advancements in VQA research, pushing it toward complex semantic
comprehension, robust reasoning, and addressing unanswerability when the
provided context is not sufficient.

##  7 Discussion and Future Work

Solving vqa tasks via code generation and external APIs has gained attention
due to its capability to perform complex reasoning and planning in a modular
manner. However, code generation has limitations: a fixed set of operations
limits models to specific types of questions and heavy use of external modules
prevents end-to-end training. While modularity encourages specialization, in
practice it requires managing multiple environments and heavy GPU memory usage
as multiple large models are used to carry out visual and cognitive tasks like
detection and captioning. In addition, current code generation methods Surís
et al. (2023); Gupta and Kembhavi (2023) rely on OpenAI’s API to generate
executable code which hinders the accessibility of benchmarking due to its
high costs111Evaluation with VisProg requires approximately 2,500 tokens per
question including in-context examples, prompts, and outputs. Using original
text-davinci-003 model used in original code would cost $(0.0200/\text{1000
tokens})\cdot 2500~{}\text{tokens}\cdot 17171~{}\text{instances}\approx
858~{}\text{USD}$. and fluctuations of OpenAI models over time Chen et al.
(2023) which makes it hard to diagnose whether certain performance gains come
from OpenAI model or improvements in other components. In contrast, our model
and dataset suggest that one can use a single vlm model that combines both the
strength of structured reasoning and train it in a simple end-to-end manner.
VisReas requires many operations such as select, filter, relate, and query
which are limited to cognitive skills to standard vqa tasks and spatial
reasoning. Therefore, models trained on VisReas may not generalize well for
visual-language tasks such as visual storytelling and image captioning which
goes beyond the scope of our dataset. A natural future direction would be to
incorporate other visual-language tasks into the dataset as well.

## References

  * Agrawal et al. (2016) Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016.  Analyzing the behavior of visual question answering models.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1955–1960, Austin, Texas. Association for Computational Linguistics.
  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4971–4980.
  * Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016.  Neural module networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 39–48.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Chen et al. (2023) Lingjiao Chen, Matei Zaharia, and James Zou. 2023.  How is chatgpt’s behavior changing over time?
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Gupta and Kembhavi (2023) Tanmay Gupta and Aniruddha Kembhavi. 2023.  Visual programming: Compositional visual reasoning without training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14953–14962.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hu et al. (2017) Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017\.  Learning to reason: End-to-end module networks for visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Hudson and Manning (2019a) Drew Hudson and Christopher D Manning. 2019a.  Learning by abstraction: The neural state machine.  _Advances in Neural Information Processing Systems_ , 32.
  * Hudson and Manning (2018) Drew A Hudson and Christopher D Manning. 2018.  Compositional attention networks for machine reasoning.  In _International Conference on Learning Representations (ICLR)_.
  * Hudson and Manning (2019b) Drew A Hudson and Christopher D Manning. 2019b.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910.
  * Kafle and Kanan (2017) Kushal Kafle and Christopher Kanan. 2017.  Visual question answering: Datasets, algorithms, and future challenges.  _Computer Vision and Image Understanding_ , 163:3–20.
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023.  Improved baselines with visual instruction tuning.
  * Lu et al. (2020) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020.  12-in-1: Multi-task vision and language representation learning.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Miller (1994) George A. Miller. 1994.  WordNet: A lexical database for English.  In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_.
  * Nguyen et al. (2022) Binh X Nguyen, Tuong Do, Huy Tran, Erman Tjiputra, Quang D Tran, and Anh Nguyen. 2022.  Coarse-to-fine reasoning for visual question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4558–4566.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017.  Right for the right reasons: Training differentiable models by constraining their explanations.  In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ , IJCAI’17, page 2662–2670. AAAI Press.
  * Selvaraju et al. (2020) Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020.  Squinting at vqa models: Introspecting vqa models with sub-questions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10003–10011.
  * Subramanian et al. (2020) Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020.  Obtaining faithful interpretations from compositional neural networks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5594–5608, Online. Association for Computational Linguistics.
  * Subramanian et al. (2023) Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023.  Modular visual question answering via code generation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , Toronto, Canada. Association for Computational Linguistics.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019\.  A corpus for reasoning about natural language grounded in photographs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6418–6428, Florence, Italy. Association for Computational Linguistics.
  * Surís et al. (2023) Dídac Surís, Sachit Menon, and Carl Vondrick. 2023.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv:2303.08128_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_.
  * Tenenbaum (2018) Josh Tenenbaum. 2018.  Building machines that learn and think like people.  In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’18, page 5, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
  * Zhang et al. (2016a) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016a.  Yin and Yang: Balancing and answering binary visual questions.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2016b) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016b.  Yin and yang: Balancing and answering binary visual questions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5014–5022.
  * Zhang et al. (2023a) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023a.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_.
  * Zhang et al. (2023b) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023b.  Multimodal chain-of-thought reasoning in language models.  _arXiv preprint arXiv:2302.00923_.

##  Appendix A Data Balancing

A primary concern with current VQA datasets is the prevalence of question-
conditional biases, enabling models to make informed guesses without a genuine
grasp of the underlying images. Nevertheless, precise rendering of question
semantics could offer enhanced control over these biases, holding the
potential to significantly mitigate the issue Zhang et al. (2016b); Kafle and
Kanan (2017). Motivated by this observation, we perform a rigorous balancing
based on question categories, attribute/relation types, and answer
distribution.

Adopting the balancing approach outlined in previous research Hudson and
Manning (2019b), we employ a clustering strategy based on a fusion of two
labels: <attr/rel_type> and <res_type>. The former denotes attributes or
relation names (e.g., red or right), while the latter signifies reasoning
types (e.g., verify.rel). We refine the question set within each cluster,
filtering out questions that encompass overlapping sets of objects in their
texts or that contain subsets of objects already covered by other questions
with complete sets. We prioritize questions featuring larger sets of objects
and multihop relations, provided their length stays below 25. Finally, we
introduce an additional label <answer> and equilibrate the question sampling
through the answer distribution. After executing this balancing in an
iterative manner on 2.07M questions, we generate a balanced corpus of 72,244
questions with images.

##  Appendix B Overview and Analysis of the VisReas

This section provides an in-depth examination of the VisReas dataset, focusing
on various aspects of question types and their characteristics. It encompasses
an overview of question types, the distribution of semantic lengths, question
readability scores, average question lengths per reasoning type, the
relationship between question frequency and the number of attributes, and
human accuracy on attributed questions.

###  B.1 Questions Types and Templates

The VisReas dataset features a diverse array of question types that challenge
multimodal reasoning and compositional understanding. These question types
include query, count, compare, verify, and choose, each requiring a unique
approach to answer. Depending on how the clusters are made, each question type
can further be broken down into attr and rel subtypes. Therefore, in total,
there can be nine categories of questions. Figure 5 gathers all templates and
examples from the dataset to offer insights into the intricacies of these
question categories.

Figure 5: Overview of types of questions along with some templates and
examples from the VisReas corpus.

###  B.2 Distribution of Relation Hops and Readability

A comprehensive analysis of the distribution of relation hops in VisReas
questions reveals a predominant trend toward questions that involve about two
reasoning hops. These hops can entail tracking object relations, identifying
attributes, or executing logical operations. We conduct a readability test
using the workers from Amazon Mechanical Turk. Our analysis reveals that
questions with larger relation hops demonstrate a noticeable decline in
readability, emphasizing the complexity associated with extended reasoning
(Figure 6). To enhance the quality of the dataset so that it can reflect the
real-world day-to-day life questions, we choose to keep the relation hop
within two.

Figure 6: Distribution of VisReas questions semantic length (number of
computation steps to arrive at the answer) as well as the readability scores
for each semantic step type. We can see that most questions require at most
two reasoning steps, where each step may involve tracking a relation between
objects, an attribute identification, or a logical operation. At the same
time, questions with larger semantic steps are difficult to read.

###  B.3 Average Question Length per Reasoning Type

By dissecting question lengths across different reasoning categories in Figure
7(b), we observe a consistent trend: query questions tend to be longer than
other reasoning types. This phenomenon is particularly apparent due to the
inclusion of multiple objects sharing similar attributes and their
corresponding relations.

(a)

(b)

Figure 7: (a) Question distribution across the number of attributes in a
query. The question complexity increases with the number of attributes or
relations. (b) Average question length per reasoning type in VisReas corpus.
Query questions are lengthier than other reasoning categories as these
questions contain multiple objects of similar attributes with their relations.
Figure 8: Pseudocode format. Our method re-structures the format of gqa
semantic string to pseudocode to better leverage Code-LLMs without adding any
auxiliary information. Figure 9: Semantic string parser. For every line of
semantic string, we use regex and string manipulation to extract operator and
its arguments. We represent scene-graph in adjacency list format and run the
parsed operator to get formatted pseudocode and its expected output. Figure
10: Overview of pseudocodes for two different traversal types in the VisReas
corpus.

###  B.4 Question Frequency and Attribute Usage

The VisReas corpus has been generated using the clusters of objects that share
similar relation or attribute. However, clusters based on shared
attributes/relations can share objects that possess all of those
attributes/relations. For example, a table and a chair have the color brown
and material wood in an image. Initially, we have two clusters with brown and
wood. Now, if both clusters share some objects, we again create a new cluster
based on brown+wood adding the shared objects (i.e., table and chair). Using
this approach, we create clusters that share multiple attributes and relations
and generate questions that involve filtering multiple attributes/relations
along with the identification of objects of interest. Figure 7(a) shows the
distribution of questions in VisReas with respect to the number of
attributes/relations. As the number of attributes/relations goes higher, the
number of clusters also decreases resulting in decreasing number of questions.

###  B.5 Human Accuracy on Attributed Questions

The final facet of our exploration delves into human accuracy when answering
attributed questions from the VisReas dataset. By assessing the performance of
human subjects across different question types and attributes, we gain a
deeper understanding of the challenges inherent to this multimodal reasoning
task. Figure 11 breaks down the human accuracy across different attribute
types. It is noticeable that color and material questions have the lowest
accuracy, as they contain a higher amount of questions compared to other
attributes.

Figure 11: Human accuracy on different attributed questions

In summary, this section offers a comprehensive overview and analysis of the
VisReas dataset, encompassing question types, semantic lengths, question
readability, average lengths per reasoning type, attribute-based question
distribution, and human accuracy. These insights contribute to a holistic
understanding of the dataset’s intricacies and its potential to advance the
field of visual reasoning and question answering.

##  Appendix C Baseline Configuration

All baselines follow default settings provided by the original author
evaluation script. All configurations for model, optimizer, scheduler, and
training follow default parameters from Pytorch and Huggingface library. For
generative models, all inference is done using default settings without
temperature tuning, nucleus sampling, repetition penalty, etc. Specific
settings used for zeroshot and finetuning are presented below:

###  C.1 VisProg

The original VisProg script uses text-davinci-003 model which is around 10
times more expensive than gpt-3.5-turbo model. To cut evaluation costs, we use
the gpt-3.5-turbo model instead. All 20 examples are found in gqa evaluation
script for code generation.

###  C.2 ViperGPT

For similar reason as VisProg, we use gpt-3.5-turbo model for code generation
to reduce costs. Since generated code doesn’t always return functional Python
code, we return either “None” or “ERROR” in these cases. In cases where the
code throws an error, the answer defaults to “ERROR”. In cases where the code
didn’t have a return statement, the answer defaults to “None”.

###  C.3 Logic2Vision

The effective batch size is kept at 4 across experiments. LoRA modules are
only attached to query and value linear layers in attention layers. The batch
size and gradient accumulation steps are adjusted accordingly. Due to memory
requirements, we set batch size to 1 on each GPU and set gradient accumulation
steps to 4. We’ve have used 2-4 A6000 GPUs with distributed data parallel
(DDP) strategy for multi-GPU training. Training Logic2Vision on VisReas takes
around 13 hours using 2 A6000 with LLaVA-1.5 backbone.

Hyperparameters | Values
---|---
Effective batch size | 4
Learning rate | 5e-6 (gqa), 2e-5 (VisReas)
Precision | bfloat16
Optimiser | AdamW
Schedule | Linear warmup with cosine decay
Warmup steps | 128
Epoch | 1

Table 6: Hyperparameters for Logic2Vision model

Hyperparameters | Values
---|---
Rank | 8
Alpha | 16
Dropout | 0.05

Table 7: LoRA configurations

###  C.4 InstructBLIP / BLIP-2 / LLaVA-1.5

On gqa, we use identical configuration as Logic2Vision for LLaVA-1.5. For
InstructBLIP and BLIP-2, we observe that batch size of 4 causes the model to
output repetitive tokens during inference. For that reason, we increase the
effective batch size to 8. We use the same original prompt that the authors
have reported in their original papers.

On VisReas, we again use identical configuration as Logic2Vision for
LLaVA-1.5. For InstructBLIP and BLIP-2, we lower the learning rate to 5e-6 and
increase the effective batch size to 8 for the same reason above.

###  C.5 LXMERT / ViLBERT / CRF

For all three models trained with the classification task, we used the default
hyperparameters that have been used to finetune on GQA corpus for consistency.
As GQA and VisReas share the same image and scenegraphs, using the same model
with the same configuration should produce different results if the two tasks
are different. And the result section reflects the distinction between GQA and
VisReas.

Hyperparameters | LXMERT | ViLBERT | CRF
---|---|---|---
Learning rate | 1e-5 | 0.00004 | 1e-4
Optimizer | BertAdam | AdamW | BertAdam
Schedule | Linear Warmup | Linear Warmup | Linear Warmup
Epoch | 4 | 20 | 13

Table 8: Hyperparameters of all CLS baselines

##  Appendix D Effect of pseudocode finetuning

We study the effect of finetuning a VLM to perform VQA through pseudocode-
guided reasoning. Table 9 demonstrates that finetuning LLaVA-1.5 to follow
pseudocode consistently improves performance on VisReas for both 7B and 13B
models.

Model size | Without Pseudocode | With Pseudocode
---|---|---
7B | 57.36 | 62.74
13B | 61.38 | 66.20

Table 9: Effect of pseucode finetuning on LLaVA-1.5

##  Appendix E Examples from VisReas and GQA

In Figure 12, we show example questions from VisReas and GQA using the same
image. In general, VisReas tends to have longer questions compared to GQA.
Additionally, VisReas questions involve more than two objects, whereas GQA
primarily centers on one or two objects.

Figure 12: Example questions from the VisReas and the GQA corpuses.

##  Appendix F Mechanical Turk Details

To evaluate human performance, we used Amazon Mechanical Turk to collect human
responses for 5000 random questions, taking a majority vote among three
workers for each question. We limited our pool of crowdworkers to individuals
located in the US or Canada, requiring a minimum of 1,000 previously approved
HITs with a 95% approval rate. Additionally, participants had to achieve a
minimum score of 70% or higher on our qualification task before gaining access
to our main task. In the subsequent sections, we provide details of this
response collection process.

###  F.1 Qualification Test for Worker Selection

To secure accurate human assessments, we carefully designed a qualification
test using Amazon Mechanical Turk interfaces (Figure 13). This test aimed to
select proficient workers capable of accurately completing the VisReas task:
(1) The qualification test encompassed two distinct tasks. The initial task
focused on careful comprehension of instructions. Workers were required to
attentively read the instructions and subsequently answer a set of multiple-
choice questions to assess their grasp of the task’s nuances. (2) Upon
successful completion of the first task, the qualified workers proceeded to
the task proficiency evaluation stage. Here, a series of ten questions, each
accompanied by an image, were presented. The workers’ task was to select the
correct answer from a dropdown list of 2013 entries. The selection process for
the final evaluation cohort prioritized workers who achieved correct answers
for more than seven out of the ten questions.

Figure 13: Amazon Mechanical Turk interfaces used for Qualification Test to
choose the right workers for human accuracy assessment on VisReas task. We
study the workers by deploying two tasks. In the first task, we ask the
workers to read the instructions carefully (Top left) and answer some
multiple-choice questions (Top right). After passing this task, ten questions
with images will be presented and the final task would be to choose the right
answer from the answer dropdown list (Bottom right). We choose the workers for
the final evaluation who have correctly predicted more than seven answers out
of ten questions.

###  F.2 Human Accuracy Assessment Interfaces

After gathering qualified workers who are aware and proficient in our task, we
move to the final stage of the evaluation process (Figure 14). For each Human
Intelligence Task (HIT), an image and the corresponding question were
provided. Workers were tasked with selecting the correct answer from the same
dropdown list used for the worker selection stage. Furthermore, we requested
workers to rate the complexity and structural integrity of the presented
question, thereby acquiring insights into the inherent challenges posed by
various question types.

To facilitate a deeper understanding of the potential issues with the queries,
we encouraged workers to provide additional details about any perceived
problems. If a worker identified a problematic aspect within the question,
they were encouraged to rephrase or rewrite the query to address the issue.
This dynamic engagement aimed to uncover underlying complexities and refine
the evaluation process.

(a)

(b)

Figure 14: Amazon Mechanical Turk interfaces for human accuracy assessment on
VisReas task using the qualified workers. (a) For each HIT, we provide an
image and a question that needs to be answered from a dropdown list of 2013
entries. In addition, we ask for rating the complexity and structural
soundness of the query and further look for details if any Turker finds the
question problematic. (b) To investigate what type of problem the question
possesses, we ask for further details from the workers and even encourage them
to rewrite the query to remove the problem they faced while answering the
query.