MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
Under review as a conference paper at ICLR 2025

MLLMS KNOW WHERE TO LOOK:
TRAINING-FREE PERCEPTION OF SMALL VISUAL DE-
TAILS WITH MULTIMODAL LLMS

Anonymous authors
Paper under double-blind review

ABSTRACT

Multimodal Large Language Models (MLLMs) have recently achieved promising
performance on visual question answering (VQA)—a fundamental task affecting
various downstream applications and domains. Given MLLMs’ potential integra-
tion into many critical VQA applications, it is important to understand the limits of
their perception. In this work, we study whether MLLMs can perceive small details
as well as large details in images. In particular, we observe that their accuracy in
answering visual questions is very sensitive to the size of the visual subject of the
question. We further show that this effect is causal by observing that human visual
cropping can significantly mitigate this sensitivity. Next, we study the attention
patterns of MLLMs when answering visual questions, and intriguingly find that
they consistently know where to look, even when they provide the wrong answer.
Based on these findings, we then construct automatic visual cropping methods that
leverage the internal knowledge of any MLLM itself, in the form of attention and
gradient maps, to help it better perceive the small visual subject of any question.
We study our proposed methods on two MLLMs and seven visual question answer-
ing benchmarks, and show that they can significantly improve MLLMs’ accuracy
without requiring any training. Our findings suggest that MLLMs should be used
with caution in detail-sensitive applications, and that visual cropping is a promising
direction to improve their performance.

1

INTRODUCTION

Visual Question Answering (VQA) is a fundamental task with a broad range of downstream applica-
tions in many critical domains, from biomedicine (Seenivasan et al., 2022; Naseem et al., 2022) to
traffic monitoring (Xu et al., 2021; Zhang et al., 2023a) and remote sensing Sarkar and Rahnemoonfar
(2021); Lobry et al. (2020). Performing VQA without requiring data collection and fine-tuning
(i.e., zero-shot) is of particular interest since collecting reliable answers for an extensive number of
question-image pairs is impractical for many applications due to cost, limited access to experts, as well
as privacy and security concerns (Zhang et al., 2023b). Recently, Multimodal Large Language Models
(MLLMs) (Li et al., 2023a; OpenAI, 2023; Liu et al., 2023b;a; Bai et al., 2023; Dai et al., 2023) have
shown promising performance in VQA, and particularly in zero-shot settings, commonly attributed to
their pretraining on terabytes of image and language data with billion-parameter Transformer-based
neural networks. Given MLLMs’ potentially broad adoption in downstream applications, it is crucial
to study the limits of their performance in dealing with various properties of images and questions.

In Fig. 1, we provide three motivating examples to illustrate a limitation in MLLMs that we will
study in this paper in more detail. In these examples, we ask BLIP-2 (FlanT5XL) (Li et al., 2023a), a
popular MLLM, three questions about relatively small objects in the image, i.e., questions concerning
small visual details. In the absence of any prior empirical evidence, one might reasonably believe
that the accuracy is not significantly affected by the size of the question’s visual subject because
of the large representational capacity of MLLMs and their pretraining on a large variety of images
containing objects of various sizes. Contrary to this belief, in Fig. 1 (left), we observe that initially
the model does not recognize the existence of a small street sign and assigns a lower probability to
the correct answer; however, zooming into the image (via smaller visual cropping) towards the street

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: The effect of visual cropping on the probability of answers predicted by BLIP-2 FlanT5XL
zero-shot VQA model. The x-axis labels are indices to the respective cropped images displayed under
each plot that the model sees at each step. The model gradually finds the correct answer.

sign gradually increases the probability assigned to the correct answer, suggesting that the model
gradually perceives more and more relevant details of the street sign. In Fig. 1 (middle), we observe
further evidence of this difficulty to perceive small details: the model initially predicts white as the
type of the bird, but when we zoom into the image towards the bird, without changing the question in
any way, we observe that the model gradually assigns higher probability to the correct bird type of
egret. This suggests that the model was not making a semantic error of misunderstanding what type
means, rather it was unable to perceive sufficient details to discriminate egret from other white birds,
which is mitigated by visual cropping. Similarly, in Fig. 1 (right), we observe that the model’s initial
answer is not entirely irrelevant (“ama” compared to the correct answer “moma”), suggesting that the
model knows where to look based on the question but cannot perceive small visual details, which is
again mitigated by visual cropping.

The main goal of this paper is to investigate the extent of the limitation observed in Fig. 1, and
to explore potential solutions to mitigate its consequences. In Sec. 3, we quantitatively show that
there indeed exists a bias against small visual concepts in MLLMs. Our findings are consistent with
concurrent work on evaluating the text-image matching in vision-language joint embedding models,
which have observed a reverse correlation between visual object size in images and the text-image
matching score (Zhao et al., 2022), but we further provide an intervention study—manipulating
images directly through cropping—to illustrate the causal relationship between object size and
MLLM’s ability to perceive objects in images. In Sec. 4, we study whether the MLLMs’ difficulty in
perceiving small visual concepts stems from a difficulty to perceive visual details, or from a difficulty
to locate the concept due to its small size. We quantitatively show that MLLMs consistently know
where to look, even when they fail to answer the question correctly. In Sec. 5, we construct three
automatic cropping methods—leveraging the attention maps and gradients of the MLLM itself—as
scalable and training-free solutions to the perception limitation. Finally, in Sec. 6, we apply our
proposed methods to two SOTA MLLMs, and evaluate them on seven VQA benchmarks to study their
effectiveness. We will make the code for all proposed methods and experiments publicly available.

2 RELATED WORKS

Multimodal Large Language Models (MLLMs). MLLMs are designed as foundation models that
can perform various downstream language and image tasks. These models can be broadly grouped
into two categories: end-to-end pretrained models, and modular pretrained models. The former
group includes architectures that are explicitly designed for processing joint image and language
data, most notably, the dual-encoder (Radford et al., 2021), the fusion-encoder (Li et al., 2021),
the encoder-decoder (Cho et al., 2021), and the unified transformer (Wang et al., 2022), which are
trained with common pretraining objectives: image-text matching, contrastive and masked language
modeling. The latter group, which includes most recent SOTA MLLMs, aims to overcome the
expensive pretraining cost of the former group by learning to adapt existing pretrained models:
BLIP2 (Li et al., 2023a) and InstructBLIP (Dai et al., 2023) train a Transformer-based connector
between a frozen pretrained ViT (Dosovitskiy et al., 2021) image encoder and a frozen LLM, which
transforms ViT output tokens into a fixed set of image tokens in the input space of the LLM; Qwen-
VL (Bai et al., 2023), similarly uses a fixed token connector (a single cross-attention layer), but trains
both the connector and the LLM; LLaVA (Liu et al., 2023b) and LLaVA-1.5 (Liu et al., 2023a) instead
use a linear projection and a two-layer MLP as their connectors, respectively, and train both. Our

2

Q: Are there any street signs in the picture? Q: What kind of bird is this? 123456123456Smaller crop size123456Q: What brand of clock is this? Smaller crop sizeSmaller crop sizeQ: Are there any street signs in the picture?Q: What kind of bird is this?Q:Whatbrandofclockisthis?Smaller crop sizeSmaller crop sizeSmaller crop sizeUnder review as a conference paper at ICLR 2025

work will contribute to a better understanding of the perception limitations of MLLMs and improving
their perception scalably and without training.

Visual Localization Methods. Dedicated visual localization techniques, such as YOLO (Redmon
et al., 2016), SAM (Kirillov et al., 2023), GLIP (Li et al., 2022b), rely heavily on rich spatial
annotations to identify salient regions in images. In contrast, native visual localization techniques,
such as grad-cam (Selvaraju et al., 2017), try to localize the salient image region by tracking the
gradients of the convolutional classifier’s own decision, without requiring any need for spatial
annotation. Recent works, PNP-VQA (Tiong et al., 2022) and Img2LLM (Guo et al., 2023), have
successfully applied grad-cam to the Transformer structure, identifying the most relevant image
patches from BLIP (Li et al., 2022a) model’s vision transformer (ViT) by tracking the image-text
similarity. Recently, SEAL (Wu and Xie, 2023) trains specialized LLMs to enable iterative visual
search for localizing small objects on high resolution images. In addition, visual-based programming
techniques (Surís et al., 2023; Gupta and Kembhavi, 2023) inherit the code capability of LLMs
and use them as controllers to call different visual localization models, such as object detection.
In this work, we first show that MLLMs have a perception limitation rather than a localization
limitation, and then use an MLLM’s internal attention maps and gradients to construct training-free
and general-purpose visual cropping methods for enhancing the MLLM’s perception.

3 MLLMS’ SENSITIVITY TO THE SIZE OF VISUAL CONCEPTS

In this section, our goal is to quantitatively study our qualitative observations in Fig. 1 that MLLMs
struggle with describing small visual details in images. To that end, we consider the TextVQA dataset,
in which for each question we can find the image ground-truth bounding box that contains the correct
textual answer. We partition its validation set into three groups based on the relative size of the
ground-truth bounding box S = Abb
, where Abb denotes the area of the ground-truth bounding box,
Atotal
and Atotal the total area of the image: 1) S < 0.005 (small) consisting of 773 question-image pairs,
2) 0.005 ≤ S < 0.05 (medium) consisting of 2411 question-image pairs, and 3) S ≥ 0.05 (large)
consisting of 1186 question-image pairs. We choose TextVQA for this study because it contains
a significant number of questions about small visual concepts, and textual answers are harder for
MLLMs to guess from other side information in the image (compared to object types and attributes).

Sensitivity Study. If a model’s perception is not sensitive to the size of visual concepts, we expect it
to have similar accuracy in all three partitions. In Tab. 1, we observe that the accuracy of all MLLMs
declines as the ground-truth bounding box becomes relatively smaller (right to left on the no cropping
rows). BLIP-2 and InstructBLIP are not trained on TextVQA (i.e., are zero-shot models), and their

Table 1: Sensitivity of the accuracy of MLLMs to the size of visual concepts in TextVQA. As the
relative visual size of the answer decreases (right to left in each row), we observe a decline in the
accuracy of the original models (no cropping) in answering questions, whereas visual cropping
(human-CROP) significantly improves accuracy on smaller objects.

Model

Method

Answer Bbox Size (S)

small

medium

large

BLIP-2 (FlanT5XL)

InstructBLIP (Vicuna-7B)

LLaVA-1.5 (Vicuna-7B)

Qwen-VL (Qwen-7B)

GPT-4o

no cropping
human-CROP

no cropping
human-CROP

no cropping
human-CROP

no cropping
human-CROP

no cropping
human-CROP

12.13
55.76

21.79
69.60

39.38
69.95

56.42
70.35

65.76
75.63

19.57
52.02

30.58
61.56

47.74
65.36

65.09
75.49

72.81
81.32

36.32
45.73

45.30
53.39

50.65
56.96

68.60
71.05

69.17
71.72

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

accuracy declines by 24 and 23 absolute percentage points between the large and small partitions,
respectively. LLaVA-1.5 and Qwen-VL are trained on the training set of TextVQA, yet, their accuracy
also declines by 11 and 12 between the large and small partitions, respectively. Lastly, even the
most recent commercial GPT-4o, with an unknown training set that might include all of TextVQA,
is suffering from a 7 percentage points decline in accuracy between small and medium partitions.
These findings suggest that MLLMs have a bias against perceiving smaller visual concepts.

Intervention Study. The perceptual limitation we observed above might be merely correlated with
size. To study whether this limitation is causally related to size, we conduct an intervention study
where we provide the MLLMs with visually cropped images based on the ground-truth bounding
boxes, denoted as human-CROP. More specifically, for each image-question pair and each MLLM,
we crop the smallest square-shaped region containing the ground-truth bounding box from the image,
and resize it to the input image resolution of the MLLM (the square-shaped cropping prevents extreme
deformation of the cropped image when resizing). The cropped image is then provided to the MLLM
in addition to the original image-question pair (see more details in Fig. 4). We observe in Tab. 1
that human-CROP significantly improves the accuracy of all MLLMs on the small and medium
partitions, and to a lesser extent on the large partition. These findings show that the perception
limitation is indeed caused by the size of the visual concepts, and that visual cropping can be a
promising direction to mitigate this limitation.

4 DO MLLMS KNOW WHERE TO LOOK?

The limitation in perceiving small visual concepts can have two primary reasons: 1) they are hard
to locate in the larger image, and 2) their small details are hard to perceive correctly. In Fig. 1, we
observed that the MLLM’s incorrect answer may contain partially correct information, hinting that
it might know where to look in the image. In this section, we quantitatively study that observation
to answer whether MLLMs’ sensitivity to size is rooted in a perception limitation or a localization
limitation. To that end, we first utilize the attention maps computed inside the Transformer layers of
an MLLM to quantify its spatial attention over the image, and then compare the total amount of this
attention inside the ground-truth bounding box to other bounding boxes of the same size.

MLLMs’ Setup. The considered MLLMs process a given image-question pair (x, q) in four steps
(depicted in Fig. 4): 1) the image is divided into N × N non-overlapping patches and processed by
the ViT image encoder into N × N output tokens; 2) the ViT output tokens are transformed into
the input space of the backbone LLM—by either an MLP (LLaVA-1.5) or a Transformer connector
(BLIP-2, InstructBLIP and Qwen-VL)—which we refer to as image tokens; 3) the image tokens are
then prepended to the question tokens and a predefined starting answer token, and fed to the LLM; 4)
the LLM is sampled auto-regressively following the starting answer token (we use greedy sampling).

(cid:80)H

Quantifying MLLMs’ Spatial Attention over the Image. We first measure how important each
image token is to the MLLM’s decision (answer-to-token attention) by extracting the softmax cross-
attention of the starting answer token to all image tokens in all layers of the backbone LLM, resulting
in Ast(x, q) ∈ RL×H×1×T , where L, H are the number of layers and heads-per-layer in the LLM,
and T is the number of image tokens provided to the LLM. We then take the average over attention
heads to arrive at the answer-to-token attention ˆAst(x, q) = 1
h=1 Ast(x, q). Next, we measure
H
how important each image region is to each image token (token-to-image attention). For the MLLMs
that use a Transformer connector to resample ViT output tokens into a fixed number of image tokens
(BLIP-2, InstructBLIP and Qwen-VL), we extract the softmax cross-attention of each image token to
all ViT output tokens in all layers of the connector, resulting in Ati ∈ RLc×Hc×T ×N 2
, where Lc, Hc
are the number of layers and heads-per-layer in the connector, T the number of learnable query
tokens (that become input image tokens to the LLM), and N 2 the number of image patches of the ViT
image encoder. We then take the average over attention heads to arrive at the token-to-image attention
ˆAti(x) = 1
h=1 Ati(x). For LLaVA-1.5 that uses an MLP to transform ViT output tokens to
Hc
image tokens, we set ˆAti(x) to the identity tensor. Finally, we compute the answer-to-image attention
by computing the tensor product of the answer-to-token and token-to-image attentions, resulting in
Asi(x, q) ∈ RL×Lc×1×N 2
si (x, q) = ˆAm
ti(x) (superscripts m and k denote layer
indices on the LLM and the connector, respectively).

st(x, q) ˆAk

where Amk

(cid:80)Hc

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 2: Examples of MLLMs knowing where to look despite answering incorrectly. The right panel
in each example displays relative attention to image (defined in Sec. 4) of one layer in the MLLM.

Relative Attention. One issue with using the softmax cross-attention is that not all highly attended
tokens are semantically relevant to the input question. For example, recent work has observed that
Transformers may use several tokens as registers to aggregate global information (Darcet et al., 2023).
To emphasize semantically relevant attention, we propose to normalize the answer-to-image attention
of an image-question pair (x, q) by its value on a generic instruction q′. Specifically, we consider a
fixed instruction q′ =“Write a general description of the image.”, and compute relative attention as
Arel(x, q) = Asi(x,q)
Asi(x,q′) under element-wise division. Fig. 2 shows examples of relative attention for
LLaVA-1.5 and InstructBLIP (Amk

rel at layers m = 14, k = 0 and m = 15, k = 2, respectively).

Do MLLMs Know Where to Look? Equipped with relative attention, we now return to our question
of whether MLLMs have a localization limitation or perception limitation. To that end, we consider
the validation set of TextVQA again. For each image-question pair, we first compute the relative
attention. We then define attention ratio as the ratio of the total (sum) relative attention inside
the answer ground-truth bounding box to its average across all bounding boxes of the same size
as the ground-truth on the image. This ratio provides a measure of how significantly the MLLM
is attending to the ground-truth bounding box (in the sense of Markov’s inequality). In Fig. 3, we
plot the average (with 95% confidence interval) of the attention ratio, over the validation set of
TextVQA for all layers in the considered MLLMs. The horizontal axis shows the combined layer
index l = m + k × L for m ∈ {0 . . . L − 1} spanning the number of cross-attention layers in
the backbone LLM, and k ∈ {0 . . . Lc − 1} spanning the number of cross-attention layers in the

Figure 3: MLLMs’ attention ratio across all layers (average with 95% CI over TextVQA). The
attention ratio measures how significantly the MLLM is attending to the ground-truth bounding box
(defined in Sec. 4). We observe that it is greater than 1 in most layers, showing that the MLLMs know
where to look in the image even when they fail to answer correctly.

5

Q:Whatplayernumberisthisfootballplayer?A:21Q:What phone number can a person call?A:202-555-2000Q:What is the color of the bicycle? (A) blue (B) white (C) silver (D) red    A:CQ:Whatisthenumber?A:8Q:Whatnumberisnextexit?A:100Q:Isthereacarintheimage?A:No🌋LLaVA-1.5InstructBLIP024487296120144Ith Layer0.91.01.11.21.3Attention RatioBLIP-2 (FlanT5XL)0326496128160192Ith Layer1.01.52.02.53.0Attention RatioInstructBLIP (Vicuna-7B)048121620242832Ith Layer246Attention RatioLLaVA-1.5 (Vicuna-7B)048121620242832Ith Layer123Attention RatioQwen-VL (Qwen-7B)Correctly AnsweredIncorrectly AnsweredUnder review as a conference paper at ICLR 2025

connector (BLIP-2: L = 24, Lc = 6; InstructBLIP: L = 32, Lc = 6; Qwen-VL: L = 32, Lc = 1;
LLaVA-1.5: L = 32, Lc = 1). In all MLLMs, we observe a significantly larger than 1 attention ratio
in most layers, suggesting that the models are attending significantly to the ground-truth bounding
box region on the image. Intriguingly, the models show similarly strong attention to the correct region
regardless of whether they can answer the question correctly or incorrectly. These observations show
that the MLLMs tend to know where to look, even when they answer incorrectly.

5 AUTOMATIC VISUAL CROPPING (VICROP)

We observed in Sec. 4 that the sensitivity of MLLMs to visual concept size is primarily a perception
limitation (rather than a localization limitation). Therefore, one solution to mitigate this limitation is
to simply train MLLMs with a larger number of image patches while maintaining per-patch resolution
(hence increasing the image resolution of MLLMs). However, increasing the input image resolution
by a factor of α, increases the number of ViT input patches (and output tokens) from N 2 to α2N 2,
which in turn increases the softmax attention computation complexity on the order of α4N 4. Given
that this solution is not scalable for current Transformer-based MLLMs, we choose to explore an
alternative solution that does not require any training and is scalable to any image resolution.
We note that several concurrent works have explored the first direction of training MLLMs with
higher resolution image patches (Li et al., 2024a; Sun et al., 2024; Li et al., 2024b; McKinzie et al.,
2024; Xu et al., 2024; Luo et al., 2024), and notably LLaVA-Next (Liu et al., 2024) has achieved
the VQA state-of-the-art in several VQA benchmarks at the time of writing. We believe our work
is parallel to these efforts in the following sense: rather than training higher and higher resolution
MLLMs to enable them to see all resolutions (which is inevitably upper bounded), we explore how
to smartly adjust the input image towards what an MLLM already can see without any additional
training. We provide evidence showing that our training-free method can provide orthogonal benefits
to the training-based methods in Appendix B.

Inspired by our findings that MLLMs tend to know where to look (Sec. 4) and that visual cropping
can mitigate the perception limitation (Sec. 3), in this section we construct three automatic visual
cropping methods in order to mitigate the perception limitation of MLLMs. These methods seek to
use the internal information of an MLLM itself—in the form of attention maps and gradients—to find
the approximate region of interest in images (i.e., the region containing the subject of a question),
and then to zoom into that region via visual cropping. One potential drawback of visual cropping is
that some questions might need to have a global view of the image. To address this issue, we utilize
the fact that MLLMs typically convert the image into a series of tokens. This allows us to directly
extend the original image tokens by concatenating the visually cropped image tokens, as illustrated
in Fig. 4. We use this concatenation approach when applying all our methods to MLLMs.

Relative Attention ViCrop (rel-att). In this method, we directly compute the relative attention
Arel(x, q) defined in Sec. 4 for each image-question pair (x, q). We then select a target layer (in LLM
and connector) based on a small held-out set of samples in TextVQA, and use its relative attention as
the importance map for visual cropping (discussed below). We ablate on the choice of layer in Sec. 6.

Figure 4: Illustration of the proposed visual cropping approach applied to two MLLMs.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Image EncoderLLM…QuestionImage Encoder…Answer…Visual CroppingInstructBLIP𝑇SLLM…QuestionMLPAnswer…🌋LLaVA-1.5𝑁×𝑁STransformer𝑇𝑁×𝑁𝑁×𝑁Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Gradient-Weighted Attention ViCrop (grad-att). The relative attention runs an additional
generic instruction through the MLLM to normalize the answer-to-image attention and emphasize
semantically relevant attention. As an alternative that does not require a second forward pass, we
consider using the gradients to normalize attention, because the gradient of the model’s decision with
respect to an attention score shows how sensitive the decision is to changes in that attention, hence how
semantically relevant the attention is for answering the question. To get a differentiable representation
of the model’s decision, we consider the logarithm of the maximum output probability at the starting
answer token, v = log softmax(z(x, q))t∗ ∈ R, where z ∈ RD is the output logit of the LLM at the
starting answer position, D the vocabulary size, and t∗ = argmaxt zt. Then, following our notation
in Sec. 4, we can compute the gradient-weighted versions of answer-to-token attention ˜Ast(x, q) =
Ast(x, q) ⊙ σ(∇Astv(x, q)) and token-to-image attention ˜Ati(x, q) = Ati(x) ⊙ σ(∇Ativ(x, q)),
where ⊙ is element-wise product and σ(w) = max(0, w). We remove negative gradients because
they correspond to tokens that if attended to will reduce the model certainty, hence often distracting
locations Selvaraju et al. (2017). Finally, we compute the gradient-weighted answer-to-image
attention as the following tensor product ˜Asi(x, q) = ˜Ast(x, q) ⊗ ˜Ati(x, q) ∈ RL×Lc×1×N 2
. We
will select the same target layer identified in rel-att from ˜Asi(x, q) as the importance map for
visual cropping.

Input Gradient ViCrop (pure-grad). In this method, we seek to find the relevant regions on the
image directly using the gradient of the MLLM’s decision with respect to the input image. Compared
to the previous attention-based methods, pure-grad is more versatile because it does not rely on
the Transformer-based architecture. Specifically, for each image-question pair (x, q), we will compute
G(x, q) = ∥∇xv(x, q)∥2, where v(x, q) is the logarithm of the maximum output probability of the
LLM at the starting answer token (as defined in grad-att above), and the L2-norm is taken over the
image channel dimension. However, gradients sometimes show high values in entirely constant-color
regions (e.g., blue skies). Given that these non-edge regions do not contain any visual details, we
will explicitly diminish them in G. To that end, we first apply a 3 × 3-size Gaussian high-pass filter
to the image, followed by a median filter of the same size to reduce salt-and-pepper noise. The
resulting filtered image is then thresholded at its spatial median value to become a binary mask, and
is element-wise multiplied by G. Finally, the edge-emphasized G is spatially average-pooled into the
N × N patches of the MLLM to become an importance map for visual cropping.

Bounding Box Selection for Visual Cropping. To convert the importance map (from each of the
methods described above) to a bounding box, we use sliding windows of different sizes inspired by
object detection literature Redmon et al. (2016). Specifically, for each MLLM, we define a set of
windows with sizes equal to a multiple of the input image resolution of the MLLM. The multiples are
in {1, 1.2, . . . 2}. We slide each window over the image with a stride of 1, and find its best position
where the sum of importance map inside the window is maximized. After selecting the best position
per window, we choose the window whose internal sum has the largest difference from the average
internal sums of its adjacent positions. This latter step is a heuristic to avoid choosing too small
or too large windows (notice that in both cases, moving the window slightly left/right or up/down
will not change its internal sum significantly). The chosen window is then cropped from the image,
resized to the input image resolution of the MLLM, and provided to the MLLM in addition to the
image-question pair.
High-Resolution Visual Cropping. In one of the benchmarks we consider in this work, V∗ Wu
and Xie (2023), the images are of very high resolution (always more than 1K) and consequently the
resized input image provided to the MLLM might completely lose the visual concept of interest for
a question. To mitigate this, on this benchmark, we employ a two stage strategy. In the first stage,
we divide images into non-overlapping blocks of smaller than 1024 × 1024 with aspect ratio close
to 1, compute the importance map separately for each block according to the ViCrop methods, and
then re-attach the blocks back together. In the second stage, we find the bounding box for visual
cropping on this re-attached importance map exactly as describe before, and provide the original
image-question pair with the resized cropped image to the MLLM.

6 VICROP METHODS ANALYSIS

In this section, we apply our proposed visual cropping methods to two open-source SOTA MLLMs,
InstructBLIP (Vicuna-7B) (Dai et al., 2023) and LLaVA-1.5 (Vicuna-7B) (Liu et al., 2023a). We

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Figure 5: Examples of rel-att helping MLLMs correct their mistakes (cyan-colored bounding
box shows cropped region by rel-att; zoom-in insets are displayed for better readability).

Table 2: Accuracy of the proposed ViCrop methods on visual question answering benchmarks.

Model

LLaVA-1.5

InstructBLIP

Smaller Visual Concepts

Larger Visual Concepts

TextVQA†

V*

POPE DocVQA AOKVQA GQA VQAv2

no cropping
rel-att
grad-att
pure-grad

no cropping
rel-att
grad-att
pure-grad

47.80
55.17
56.06
51.67

33.48
45.44
45.71
42.23

42.41
62.30
57.07
46.07

35.60
42.41
37.70
37.17

85.27
87.25
87.03
86.06

84.89
86.64
86.99
86.84

15.97
19.63
19.84
17.70

9.20
9.95
10.81
8.99

59.01
60.66
59.94
59.92

60.06
61.28
61.77
61.60

60.48
60.97
60.98
60.54

49.41
49.75
50.33
50.08

75.57
76.51
76.06
75.94

76.25
76.84
76.08
76.71

evaluate their effectiveness in improving the perception of smaller visual concepts on 4 detail-
sensitive datasets (TextVQA 1 (Singh et al., 2019), V∗ (Wu and Xie, 2023), POPE (Li et al., 2023b),
DocVQA (Mathew et al., 2021)), and their ability to maintain performance on larger visual concepts
in 3 general-purpose datasets containing mostly large objects (GQA (Hudson and Manning, 2019),
AOKVQA (Schwenk et al., 2022), VQAv2 (Goyal et al., 2017)). InstructBLIP uses the hyper-
parameters N = 16, m = 15, k = 2 and input image resolution of 224 × 224. LLaVA-1.5 uses
N = 24, m = 14 and input image resolution of 336 × 336. When reporting accuracy, we compute
VQA-score2 for all benchmarks except GQA. For GQA, we compute accuracy using the official
code.3. See Appendix A for implementation details, and Appendix E for evaluation prompts.

ViCrop Improves VQA Accuracy. In Fig. 5, we show a few examples of the ViCrop helping
the MLLM correct itself (more examples in Appendix F), and in Tab. 2, we report the accuracy of

1†In TextVQA evaluation, we do not provide externally extracted OCR tokens to the MLLM since we want to
measure its true perception, this differs from the setup in the original paper. See more discussion in Appendix A.

2https://visualqa.org/evaluation.html
3https://cs.stanford.edu/people/dorarad/gqa/evaluate.html

8

Q: What is the last on the list the lady is pointing at?🌋 LLaVA 1.5: 10        🌋 LLaVA 1.5 (w/ ViCrop): Use numbersQ: What is the name of the player?     InstructBLIP: Rudolph            InstructBLIP (w/ ViCrop): HollandQ: What is the color of the clock? (A) black (B) yellow (C) green (D) red  🌋 LLaVA 1.5: A         🌋 LLaVA 1.5 (w/ ViCrop): CUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

the proposed ViCrop methods on the VQA benchmarks. We observe that all methods significantly
improve the accuracy of the original MLLMs (no cropping) on detail-sensitive benchmarks, without
requiring any training, while maintaining the MLLMs’ performance on benchmarks with larger
visual concepts. Thus, the accuracy gain on fine details (most notably in TextVQA and V∗) does not
seem to come at the cost of accuracy on larger visual details and relations. We also observe that the
accuracy gain for LLaVA-1.5 is more substantial than for InstructBLIP. This can be explained by the
fact that InstructBLIP only trains its connector and not its backbone LLM during tuning—the LLM
does not adapt to use the image tokens, rather the image tokens are adapted to optimally prompt the
LLM—and therefore the LLM cannot effectively use the additional (cropped) image tokens provided
through visual cropping. Nonetheless, the results show that ViCrop can be effectively applied to
different MLLMs, and is a promising inference-time solution for mitigating the perception limitation
observed in Sec. 3.

Ablation Study on the Choice of Layer. To understand the importance of the choice of an informative
layer for rel-att and grad-att (as discussed in Sec. 5), in Tab. 3 we compare the accuracy
of these methods when simply taking the average of all layers in Arel and ˜Asi, respectively, on
TextVQA. We observe that rel-att is robust to this choice and grad-att declines about 3.5
percentage points in accuracy. Importantly, both methods still improve the MLLMs’ accuracy even
when using the layer average, suggesting that average is a suitable choice in the absence of any data
for selecting a layer.

Ablation Study on the High-Resolution ViCrop. In Sec. 5, we proposed a two stage strategy for
processing the very high-resolution images in the V∗ benchmark. To see how effective this strategy
is, in Tab. 3 we compare the accuracy of ViCrop methods with and without this high-resolution
strategy on V∗. We observe that while this strategy is very beneficial to LLaVA-1.5, it declines the
performance of grad-att and pure-grad for InstructBLIP. However, all methods, with and
without this strategy, still improve the MLLMs’ accuracy.

ViCrop with External Tools. In addition to the internal ViCrop methods, we also considered the
use of external off-the-shelf models to find the region of interest in an image for visual cropping.
Specifically, we utilized SAM (Kirillov et al., 2023), YOLO (Redmon et al., 2016), and CLIP (Radford
et al., 2021) to find the most relevant part of an image to a given question (details of these external
ViCrop methods are provided in Appendix D). In Tab. 4, we compare the accuracy of external ViCrop
methods to the internal methods on TextVQA. While external models are also effective in improving
the accuracy of MLLMs, they are weaker than all the proposed internal ViCrop methods, thus we did
not explore them further.

Inference Time Overhead. In Tab. 4, we report the average inference-time overhead of the proposed
visual cropping methods on GPU (NVIDIA RTX A6000) and CPU (Intel(R) Gold 5317 CPU @
3.00GHz), and compare with the per-answer-token processing time of the MLLMs. We see that all
proposed methods (except SAM) are reasonably fast (1 to 2 seconds overhead on GPU). For example,
computing the visual cropping with rel-att takes the time of generating only 5 tokens by the
MLLM. Note that our methods’ time overhead will not scale with the number of answer tokens
and is constant regardless of how long the answer is because our external methods do not need
any answer token, and internal methods only need the starting answer token (see Sec. 5). In contrast,
MLLMs’ inference time scales approximately linearly with the number of answer tokens.

Table 3: Ablation study on the choice of layer and the use of high-resolution visual cropping.

Model

LLaVA-1.5

InstructBLIP

Choice of Layer

High-Resolution ViCrop

Selective Average

∆

w/ High-Res w/o High-Res

∆

no cropping
rel-att
grad-att
pure-grad

no cropping
rel-att
grad-att
pure-grad

47.80
55.17
56.06
51.67

33.48
45.44
45.71
42.23

–
55.45
56.26
–

–
44.40
44.98
–

9

–
+0.28
+0.20
–

–
-1.04
-0.73
–

42.41
62.30
57.07
46.07

35.60
42.41
37.70
37.17

42.41
47.64
49.74
45.03

35.60
38.74
42.41
42.41

–
-14.66
-7.33
-1.04

–
-3.67
+4.71
+5.24

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Table 4: Accuracy of ViCrop using external tools compared to attention/gradient (on TextVQA); and
the inference time overhead of ViCrop methods (in seconds). Original’s time is per answer token.

Model

Original

SAM YOLO CLIP

rel-att

grad-att

pure-grad

Accuracy
(TextVQA)

LLaVA-1.5
InstructBLIP

47.80
33.48

49.42
39.23

48.84
36.49

48.55
39.61

CPU Time

GPU Time

LLaVA-1.5
InstructBLIP

LLaVA-1.5
InstructBLIP

2.26
0.66

0.17
0.06

91.53

0.97

5.46

3.33

0.35

1.07

55.17
45.44

14.43
4.35

1.16
0.28

56.06
45.71

11.33
3.78

0.89
0.29

51.67
42.23

29.86
7.04

2.36
0.60

Limitations and Future Works. The proposed ViCrop methods do not enhance all types of questions
equally. We have observed that questions concerning relations and counting are particularly difficult
for ViCrop methods to help answer. This is expected as the proposed ViCrop can only focus on
one region in the image. We leave extending ViCrop to focus on multiple regions simultaneously to
future work. Another limitation of the proposed methods is their time overhead. While the overhead
is reasonable (a few seconds), we believe it can be significantly optimized as an inference-time
mechanism, for example by utilizing lower precision and weight quantization. We leave this time
optimization to future works. Lastly, we have observed that the proposed methods tend to have some
complementary benefits, and therefore exploring ways to combine them (for example based on the
prediction uncertainty) is also a very interesting direction for future research.

Conclusion. In this work, we qualitatively and quantitatively showed that there exists a perception
bias against small visual details in SOTA MLLMs, that it is rooted in a perception limitation (rather
than a localization limitation), and then proposed multiple automatic visual cropping methods as
scalable and training-free solutions to mitigate this limitation. Our findings suggest that MLLMs
should be used with caution in detail-sensitive VQA applications, and that visual cropping is a
promising direction to improve their performance.

REFERENCES

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou,
and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization,
text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text
generation. In International Conference on Machine Learning, pages 1931–1942. PMLR, 2021.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang,
Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language
models with instruction tuning, 2023.

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need

registers. arXiv preprint arXiv:2309.16588, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image
is worth 16x16 words: Transformers for image recognition at scale. In International Conference
on Learning Representations, 2021.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa
matter: Elevating the role of image understanding in visual question answering. In Proceedings of
the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and
Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen
large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 10867–10877, 2023.

Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning
without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 14953–14962, 2023.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning
and compositional question answering. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pages 6700–6709, 2019.

Glenn Jocher, Ayush Chaurasia, and Jing Qiu. YOLO by Ultralytics. January 2023. URL https:

//github.com/ultralytics/ultralytics.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete
Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint
arXiv:2304.02643, 2023.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum
distillation. Advances in neural information processing systems, 34:9694–9705, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In International Conference on
Machine Learning, pages 12888–12900. PMLR, 2022a.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-
training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597,
2023a.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong,
Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
10965–10975, 2022b.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng
Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.
arXiv preprint arXiv:2403.18814, 2024a.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object

hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023b.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and
Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal
models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 26763–26773, 2024b.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–
ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings,
Part V 13, pages 740–755. Springer, 2014.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction

tuning. arXiv preprint arXiv:2310.03744, 2023a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv

preprint arXiv:2304.08485, 2023b.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
llava-vl.github.io/blog/2024-01-30-llava-next/.

Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. Rsvqa: Visual question answering for
remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 58(12):8555–8566,
2020.

Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your
eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint
arXiv:2403.03003, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document
images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision,
pages 2200–2209, 2021.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter,
Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights
from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024.

Usman Naseem, Matloob Khushi, and Jinman Kim. Vision-language transformer for interpretable
pathology visual question answering. IEEE Journal of Biomedical and Health Informatics, 27(4):
1681–1690, 2022.

R OpenAI. Gpt-4 technical report. arXiv, pages 2303–08774, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International conference on machine learning, pages
8748–8763. PMLR, 2021.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified,
real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 779–788, 2016.

Argho Sarkar and Maryam Rahnemoonfar. Vqa-aid: Visual question answering for post-disaster
damage assessment and analysis. In 2021 IEEE International Geoscience and Remote Sensing
Symposium IGARSS, pages 8660–8663. IEEE, 2021.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.
A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision–
ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part
VIII, pages 146–162. Springer, 2022.

Lalithkumar Seenivasan, Mobarakol Islam, Adithya K Krishna, and Hongliang Ren. Surgical-vqa:
Visual question answering in surgical scenes using transformer. In International Conference on
Medical Image Computing and Computer-Assisted Intervention, pages 33–43. Springer, 2022.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh,
and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza-
tion. In Proceedings of the IEEE international conference on computer vision, pages 618–626,
2017.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and
Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pages 8317–8326, 2019.

Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo,
Kaifu Zhang, De-Chuan Zhan, et al. Parrot: Multilingual visual instruction tuning. arXiv preprint
arXiv:2406.02539, 2024.

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for

reasoning. arXiv preprint arXiv:2303.08128, 2023.

Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-and-
play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint
arXiv:2210.08773, 2022.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal,
Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language:
Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.

Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.

arXiv preprint arXiv:2312.14135, 2023.

12

Under review as a conference paper at ICLR 2025

Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient
network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 9878–9888, 2021.

Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu,
Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution
images. arXiv preprint arXiv:2403.11703, 2024.

Jiarui Zhang, Filip Ilievski, Kaixin Ma, Aravinda Kollaa, Jonathan Francis, and Alessandro Oltramari.

A study of situational reasoning for traffic understanding. KDD, 2023a.

Mingda Zhang, Rebecca Hwa, and Adriana Kovashka. How to practice vqa on a resource-limited
target domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer
Vision, pages 4451–4460, 2023b.

Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and
Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes
and relations. arXiv preprint arXiv:2207.00221, 2022.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

A IMPLEMENTATION DETAILS

We use python 3.10.6, transformers 4.29.1 and torch 2.1.2 for all the experiments. Our environment
consists of an Intel(R) Gold 5317 CPU @ 3.00GHz with 48 cores and 756 GB of RAM, and we
utilize NVIDIA RTX A6000 GPUs for our experiments. We use the huggingface implementations of
all studied MLLMs with the recommended hyper-parameters according to the respective papers. For
GPT-4o, we use the official public API as available at the time of submission.

Regarding the evaluation setting of the TextVQA dataset in Tab. 2, our setting is slightly different
from the one used by the LLaVA-1.5 original paper Liu et al. (2023a). They report accuracy
on TextVQA by using externally extracted OCR tokens to enrich its text prompt. This is a text-
specific trick that essentially out-sources the perception of text to an external OCR model. This
text-specific trick is not mentioned in their paper or supplementary material, but see their clarification
in response to a github issue here: https://github.com/haotian-liu/LLaVA/issues/
515#issuecomment-1763779341. In contrast, we treat TextVQA the same as any other vision
dataset in our experiments, that is, we do not provide any OCR extracted tokens to MLLMs when
applying them to TextVQA (only image and question, in the evaluation prompt format specified in
their respective papers). This results in a slightly lower accuracy compared to the one reported in the
originl paper, but instead, this number shows the true perception ability of LLaVA-1.5 on TextVQA,
not confounded by the ability of an external OCR model. For completeness, we also measured
TextVQA accuracy in presence of OCR tokens, which results in 59.8 for LLaVA-1.5 without any
visual cropping, and 63.95 with rel-att, showing that our proposed visual cropping can still be
beneficial even when OCR tokens are provided to the MLLM.

B ORTHOGONAL BENEFITS TO LLAVA-NEXT

We apply our proposed rel-att visual cropping method to an additional newer MLLM – LLaVA-
NeXT (Liu et al., 2024) current SOTA in several VQA benchmarks – that has support for higher-
resolution compared to LLaVA-1.5. In Tab. 5, we observe that our method can still boost the MLLM’s
performance, without requiring any training. This provides further evidence for the generalizability
of our proposed visual cropping, and its orthogonal benefits to training MLLMs with higher image
patch resolution.

Table 5: Orthogonal benefits of visual cropping when applied to LLaV-NeXT that is trained to adapt
to processing high resolution images.

Model

TextVQA

V∗

LLaVA-NeXT (Mistral-7B)
LLaVA-NeXT (Mistral-7B) + rel-att

65.17
68.65

58.11
61.78

C DATASET STATISTICS

In this section, we present the details of the datasets used for evaluation in this paper. We report the
average height and weight of the images in the dataset. We also report the number of images and
questions in each dataset.

Table 6: Average width ( ¯W ) and height ( ¯H) of images, number of images, and number of questions
on all datasets.

¯W
¯H
# Images
# Questions

V∗

2246
1582
191
191

DocVQA TextVQA POPE AOKVQA GQA VQAv2

1776
2084
1286
5349

954
818
3166
5000

584
478
500
8910

581
480
1122
1145

578
482
398
10781

577
485
14206
50000

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

For our analysis presented in Table 1 and Figure 3, we focused on TextVQA dataset, which includes
bounding box annotations for OCR-detected text within images. However, this dataset does not
specify which bounding boxes correspond to the regions where answers are located, necessitating
a manual annotation process. The TextVQA dataset comprises 5000 questions and 3166 images.
We manually annotated these question-image pairs, ensuring accurate bounding boxes over all the
regions of interest where the answers could be found. This manual annotation process was essential
for our analysis, allowing us to provide precise and reliable ground-truth data for the study. Given
that some questions were associated with multiple bounding boxes in their corresponding images, we
undertook a filtering process to isolate the question-image pairs. This effort resulted in a refined set
of 4370 question-image pairs, where there is only one instance of the subject of the question in the
image. For example, if the question is “what type of drink is sold here?” and there are two different
cans of drinks in the image, we remove this image-question pair.

D EXTERNAL TOOLS VICROP

In this section, we present three automatic question-guided localization methods based on popular
off-the-shelf vision-based models, namely CLIP Radford et al. (2021), YOLO Redmon et al. (2016),
and SAM Kirillov et al. (2023). These three methods utilize external vision-based knowledge for
the localization process through multimodal encoding, object detection, and semantic segmentation,
respectively. See Tab. 4 for their results compared to internal ViCrop methods.

CLIP ViCrop. The intuition of this method is to progressively refine the image towards the region of
highest relevance to a given question using CLIP Radford et al. (2021). CLIP consists of an image
encoder and a text encoder, which are trained on a large dataset of image-caption pairs to map each
image (caption) close to its caption (image) and far from all other captions (images). The result is
an aligned shared space where various images can be directly compared with various texts. To find
the region of interest, given an image-question pair, we first crop the image from the four sides (top,
bottom, left, and right) at a cropping ratio of 0.9 to produce four overlapping cropped images. We
then use CLIP to assess the semantic similarity between these cropped images and the question. The
highest-scoring crop is chosen as the input for the next iteration. This process is repeated for 20
iterations, and the cropped image with the highest CLIP similarity to the question is selected for
visual cropping.

YOLO ViCrop. Instead of a progressive approach to finding the region of interest, in this method we
select candidate regions based on a state-of-the-art object detection method: YOLOv8 (Jocher et al.,
2023) pretrained on COCO Lin et al. (2014). Using YOLO, we filter out regions that contain no
salient objects – i.e., regions for which CLIP could mistakenly assign high similarity. More concretely,
for each question-image pair, we first use YOLO to collect bounding boxes for all predicted objects
with confidence higher than 0.25 (the recommended default).4 Then, for each predicted bounding
box, we crop its corresponding image and compute its similarity to the question using CLIP. Finally,
the bounding box with the highest similarity score is selected as the region of interest for visual
cropping.

SAM ViCrop. A limitation of YOLO is that it only provides bounding boxes corresponding to a fixed
number of object classes. To overcome this issue, we use the segment anything model (SAM) Kirillov
et al. (2023), which has shown state-of-the-art zero-shot segmentation performance. SAM can provide
an extensive set of segmentation masks for each image, thus providing a more granular set of salient
candidate regions compared to YOLO. More concretely, for each image-question pair, we feed the
image into SAM, which provides an extensive set of segmentation masks corresponding to all objects
and object parts. Then, we translate these masks into bounding boxes by computing the smallest
bounding box that covers each segmentation mask. Finally, the bounding box with the highest CLIP
similarity to the question is selected as the region of interest for visual cropping.

Finally, for each method, we crop the smallest covering square (so that the cropped image is not
deformed when resized to the input resolution of the MLLM), and provide it to the MLLM in adition
to the original image-question pair (as depicted in Fig. 4).

4https://docs.ultralytics.com/modes/predict

15

Under review as a conference paper at ICLR 2025

E PROMPT FORMAT FOR ZERO-SHOT INFERENCE

In this section, we provide details about the prompt format used in models for zero-shot inference.
We use a different prompt format for LLaVA and InstructBLIP which we adapt from the original
papers, as shown below.

LLaVA-1.5

<image> USER:{question} Answer the question using a single
word or phrase. ASSISTANT:

InstructBLIP

<image> Question:{question} Short Answer:

F ADDITIONAL EXAMPLES ON MODEL’S PREDICTIONS

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 6: Success (first 3) and failure (last) examples of LLaVA-1.5 (rel-att) on the V∗ benchmark
(cyan-colored bounding box shows cropped region by rel-att; zoom-in insets are displayed for
better readability).

17

Q: What is the breed of the dog? (A)  Husky (B)  corgi (C)  Dalmatian (D)  golden retriever🌋 LLaVA 1.5:     C🌋 LLaVA 1.5 (w/ ViCrop):       DQ: Is the blue truck on the left or right side of the white vehicle? (A)  right (B)  left🌋 LLaVA 1.5:     B🌋 LLaVA 1.5 (w/ ViCrop):     AQ: What is the color of the handbag? (A)  white (B)  red (C)  black (D)  yellow                  🌋 LLaVA 1.5:     C🌋 LLaVA 1.5 (w/ ViCrop):         DQ: What is the color of the parachute?(A)  Blue   (B)  yellow(C)  Green (D)  red🌋 LLaVA 1.5:     B🌋 LLaVA 1.5 (w/ ViCrop):         AUnder review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 7: Success (first 9) and failure (last 6) examples of LLaVA-1.5 (rel-att) on the TextVQA
benchmark (cyan-colored bounding box shows cropped region by rel-att).

18

Q: what number is on the middle bike?🌋 LLaVA 1.5:     39🌋 LLaVA 1.5 (w/ ViCrop):     30Q: what number is on the man in whites jersey?🌋 LLaVA 1.5:     22🌋 LLaVA 1.5 (w/ ViCrop):     7Q: what does the sign at the crosswalk say?🌋 LLaVA 1.5:     Go🌋 LLaVA 1.5 (w/ ViCrop):     10avQ: what team is written on the baseball in the right hand corner?🌋 LLaVA 1.5:     Yankees🌋 LLaVA 1.5 (w/ ViCrop):     RiverdogsQ: what is the company that made this game / character?🌋 LLaVA 1.5:     Crazy brick🌋 LLaVA 1.5 (w/ ViCrop):     Steve Jackson gamesQ: what is the brand of the monitor?🌋 LLaVA 1.5:     Postugo🌋 LLaVA 1.5 (w/ ViCrop):     PositivoQ: what country is miss universe from?🌋 LLaVA 1.5:     Usa🌋 LLaVA 1.5 (w/ ViCrop):     CanadaQ: is city of ballard in europe?🌋 LLaVA 1.5:     no🌋 LLaVA 1.5 (w/ ViCrop):     yesQ: which brewery made this ale?🌋 LLaVA 1.5:     Perrys🌋 LLaVA 1.5 (w/ ViCrop):     CottrellQ: what does the bubble text say for the woman?🌋 LLaVA 1.5:     Hold it boys!🌋 LLaVA 1.5 (w/ ViCrop):     Hold itQ: what is styped in the search bar?🌋 LLaVA 1.5:     Sushi🌋 LLaVA 1.5 (w/ ViCrop):     Buscar dierccionQ: what is the number of the player in the middle?🌋 LLaVA 1.5:     3🌋 LLaVA 1.5 (w/ ViCrop):     22Q: what is the wine brand?🌋 LLaVA 1.5:     Dog house🌋 LLaVA 1.5 (w/ ViCrop):     MassetoQ: what kind of exchange shows on the banner?🌋 LLaVA 1.5:     Policy🌋 LLaVA 1.5 (w/ ViCrop):     Sab millerQ: who is the picture of?🌋 LLaVA 1.5:     Charles grayson🌋 LLaVA 1.5 (w/ ViCrop):     ManUnder review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Figure 8: Success (first 9) and failure (last 6) examples of InstructBLIP (rel-att) on the TextVQA
benchmark (cyan-colored bounding box shows cropped region by rel-att).

19

Q: what does the sign say above the wheelchair symbol?     InstructBLIP:     Men     InstructBLIP (w/ ViCrop):     RestroomQ: what is the number at the bottom right corner?     InstructBLIP:     100     InstructBLIP (w/ ViCrop):     16Q: what is the name in the very top right hand corner of this page?     InstructBLIP:     Shakespeare     InstructBLIP (w/ ViCrop):     NewcomeQ: what is the name of the group on the bottom right of the poster?     InstructBLIP:     Public     InstructBLIP (w/ ViCrop):     Les socialistesQ: more sharing what?     InstructBLIP:     Photo     InstructBLIP (w/ ViCrop):     OptionsQ: what brand of bike wheel is this?     InstructBLIP:     Pace     InstructBLIP (w/ ViCrop):     PacentiQ: what is written on the milk carton?     InstructBLIP:     Milk     InstructBLIP (w/ ViCrop):     U.s. forced europeQ: what letter is on the mans ball cap?     InstructBLIP:     F     InstructBLIP (w/ ViCrop):     TQ: what degree angle has been drawn?     InstructBLIP:     45     InstructBLIP (w/ ViCrop):     90Q: what brand are the guy's shorts?     InstructBLIP:     Gators     InstructBLIP (w/ ViCrop):     NikeQ: who is the service provider?     InstructBLIP:     Mophie     InstructBLIP (w/ ViCrop):     AppleQ: what is the middle book title?     InstructBLIP:     Mess     InstructBLIP (w/ ViCrop):     MessilyQ: what is the word under the state name?     InstructBLIP:     Karazy     InstructBLIP (w/ ViCrop):     CaliforniaQ: what is the manufacturer of these bullets?     InstructBLIP:     Sears     InstructBLIP (w/ ViCrop):     RemingtonQ: what is the word written in the bottom of the box?     InstructBLIP:     Hardcast     InstructBLIP (w/ ViCrop):     Flexible