# Detecting Multimodal Situations with Insufficient Context and Abstaining
from Baseless Predictions

Junzhang Liu  Columbia UniversityNew YorkUSA<EMAIL_ADDRESS>,  Zhecan
Wang  Columbia UniversityNew YorkUSA<EMAIL_ADDRESS>,  Hammad Ayyubi
Columbia UniversityNew YorkUSA<EMAIL_ADDRESS>,  Haoxuan You  Columbia
UniversityNew YorkUSA<EMAIL_ADDRESS>,  Chris Thomas  Virginia
TechVirginiaUSA<EMAIL_ADDRESS>,  Rui Sun  Columbia UniversityNew YorkUSA
<EMAIL_ADDRESS>,  Shih-Fu Chang  Columbia UniversityNew YorkUSA
<EMAIL_ADDRESS>and  Kai-Wei Chang  University of California, Los
AngelesLos AngelesUSA<EMAIL_ADDRESS>
###### Abstract.

Despite the widespread adoption of Vision-Language Understanding (VLU)
benchmarks such as VQA v2, OKVQA, A-OKVQA, GQA, VCR, SWAG, and VisualCOMET,
our analysis reveals a pervasive issue affecting their integrity: these
benchmarks contain samples where answers rely on assumptions unsupported by
the provided context. Training models on such data foster biased learning and
hallucinations as models tend to make similar unwarranted assumptions. To
address this issue, we collect contextual data for each sample whenever
available and train a context selection module to facilitate evidence-based
model predictions. Strong improvements across multiple benchmarks demonstrate
the effectiveness of our approach. Further, we develop a general-purpose
Context-AwaRe Abstention (CARA) detector to identify samples lacking
sufficient context and enhance model accuracy by abstaining from responding if
the required context is absent. CARA exhibits generalization to new benchmarks
it wasn’t trained on, underscoring its utility for future VLU benchmarks in
detecting or cleaning samples with inadequate context. Finally, we curate a
Context Ambiguity and Sufficiency Evaluation (CASE) set to benchmark the
performance of insufficient context detectors. Overall, our work represents a
significant advancement in ensuring that vision-language models generate
trustworthy and evidence-based outputs in complex real-world scenarios.

Context, Multimodal, Vision, Language

††ccs: Computing methodologies Natural language generation††ccs: Computing
methodologies Image representations††ccs: Computing methodologies Natural
language generation††ccs: Computing methodologies Scene understanding Figure
1.  Examples of samples with insufficient context to answer the question
across several representative Vision Language Understanding (VLU) benchmarks.
“Q” represents the question, “A” stands for the answer, and “P” denotes a
typical Vision Language Model (VLM; here BLIP-2) prediction. We find that
samples with insufficient context are common across several VLU benchmarks,
causing VLMs to hallucinate predictions. Using (wearing) our proposed CARA
(hat), VLMs are able to abstain from responding instead of making baseless
predictions in such cases.

##  1\. Introduction

A number of Vision Language Understanding (VLU) benchmarks have been proposed
to evaluate the capability of models to interpret complex multimodal scenarios
and events (Marino et al., 2019; Schwenk et al., 2022; Zellers et al., 2019a;
Antol et al., 2015; Zellers et al., 2018; Park et al., 2020). However, these
benchmarks often include samples with insufficient event-specific context to
answer the given questions. For instance, in the first example of Figure 1, it
is impossible to answer why the boat is messy without knowing what had
happened before. Similarly, in the second example of Figure 1, knowledge of
[Person 1]’s prior interaction is required to determine how the person feels.
Answering the questions for these examples requires more contextual
information about the events depicted in the images than is available from the
image alone.

Our analysis reveals that this issue of insufficient event-specific context is
pervasive in many VLU datasets. Figure 1 illustrates examples from some
representative benchmarks – VQA v2 (Antol et al., 2015), Visual Commonsense
Reasoning (VCR) (Zellers et al., 2019b), OKVQA (Marino et al., 2019), A-OKVQA
(Schwenk et al., 2022), and VisualCOMET (Park et al., 2020). The lack of
sufficient and specific context in the provided samples forces models trained
on such data to guess possible answers, leading to models that confidently
predict answers without evidential support. Models that tend to hallucinate
assumptions in this way undermine their trustworthiness and limit their real-
world applicability in settings where accuracy is critical e.g., assistive
technologies for the visually impaired (Puzis et al., 2013; Gurari et al.,
2018; Zorec et al., 2020), autonomous vehicles and robotics (Hochgeschwender
et al., 2019; Jahan et al., 2019), healthcare applications (Lin et al., 2023)
or security and surveillance (Toor et al., 2019).

Our findings of the ubiquity of this problem lead us to two critical
questions: 1) If the context can be retrieved, e.g., we can obtain the
corresponding video as context when the sample has an image from that video,
how to identify the most necessary context and effectively incorporate it into
models? 2) If there is no available context, e.g., the sample has an in-the-
wild image, can we develop a generalizable method to identify samples with
insufficient context and abstain from making baseless predictions?

Regarding the first question, numerous methods (Lee and Kim, 2021; Song et
al., 2020) have been proposed to enhance image-text understanding with
external knowledge. Yet, these approaches fail to address the absence of
event-specific context, which is not available in external sources. The
challenges presented in Figure 1, for example, cannot be overcome simply
through the application of general knowledge, as they require insights
directly related to the depicted events.

As for the second question, no prior work has focused on abstaining from
speculative responses by identifying insufficient event-specific context
across existing VLU benchmarks. Existing works refrain from answering either
due to low model confidence (Whitehead et al., 2022) or due to out-of-
distribution samples (Dancette et al., 2023). Consequently, they would still
make unfounded predictions for samples with high model confidence or in-domain
samples but insufficient context.

We address these limitations by 1) Collecting contextual data where available
(VCR, SWAG, and VisualCOMET) and building a novel model-agnostic plug-and-play
context selection module to incorporate context into model prediction (see
Figure 2); 2) Reusing the aforementioned module to collect pseudo-labels to
train a Context-AwaRe Abstention (CARA) module, capable of identifying samples
with insufficient context. Both our context selection module and CARA are
model- and task-agnostic.

Our experiments demonstrate that our context selection module consistently
improves performance across models and tasks. In the process, we also
investigate several important questions: 1) In which modality (visual,
textual, or both) does context benefit the most? 2) How long is context useful
before it becomes noise? Further, we show that CARA boosts state-of-the-art
model performance in full-shot and zero-shot settings by reducing inaccurate
predictions through abstention from baseless or hallucinated responses. CARA,
trained on one benchmark, can effortlessly generalizes to other benchmarks as
well. This provides evidence that CARA could be useful for future benchmarks
as well without any re-training. It could even be used to clean future
benchmarks of samples with insufficient context. Moreover, as CARA prevents
models from making predictions that are not grounded in contextual evidence,
we believe it will significantly improve model trustworthiness.

Lastly, to evaluate CARA’s quality and benchmark its performance, we also
curate an evaluation set manually annotated with the labels – sufficient or
insufficient context. This data is valuable for the future development of
insufficient context detectors.

In summary, our work makes several key contributions:

  * •

Highlighting a Systemic Issue: We identify a pervasive problem in common VLU
benchmarks, i.e., the presence of samples with insufficient context. This
issue has been largely overlooked in prior studies, despite its impact on the
performance and reliability of VLU models. We conduct an extensive analysis
across benchmarks to reveal the extent of this problem.

  * •

Incoporating Context Effectively: We address the issue of insufficient event-
specific context in VCR, SWAG, and VisualCOMET benchmarks by introducing a
novel context selection method. This enhances model performance by accurately
identifying and integrating relevant context into task resolution.

  * •

Multimodal Abstention Detector: We develop CARA, a method for abstaining on
samples lacking necessary context, and demonstrate its generalization across
new benchmarks.

  * •

Data Contribution: We collect contextual data for VCR, SWAG, and VisualCOMET,
which is valuable for further exploration of context-aware model prediction.
Moreover, we create a Context Ambiguity and Sufficiency Evaluation (CASE) set
for insufficient context detection.

Figure 2.  Illustration of how we obtain contextual data for VCR, Visual SWAG,
and VisualCOMET. The video from which the image sample is sourced is
identified to obtain temporal context in the form of frames and captions
around the image sample in question. The context provides the necessary
evidence required to answer these highly semantic questions.

[Illustration of our context dataset and how context can be helpful.]Examples
of data lacking context and how context helps answer them.

##  2\. Related Work

####  2.0.1. Unanswerable Visual Questions

The challenge of determining the answerability of visual questions has been
explored primarily from two main directions: 1) relevance of the question or
2) quality of the image. The former direction focuses on creating datasets and
methods that test models’ ability to flag irrelevant questions (Kafle and
Kanan, 2017; Toor et al., 2017; Li et al., 2020; Mahendru et al., 2017; Ray et
al., 2016) or questions inquiring about objects absent in the image (Lovenia
et al., 2023; Li et al., 2023a; Wu et al., 2023). On the other hand, the
latter direction requires models to flag unanswerable samples due to low image
quality (Gurari et al., 2018; Bhattacharya et al., 2019). Both directions
overlook the nuanced complexity of unanswerability in the case of insufficient
context for high-quality images paired with relevant questions. It is this gap
that our work aims to bridge.

####  2.0.2. VLU with External Resources

When information in the image is insufficient to answer the question (Sanket
Shah and Talukdar, 2019; Marino et al., 2019; Schwenk et al., 2022), several
methods have been proposed to augment the provided information with external
knowledge from Wikipedia (Lin and Byrne, 2022), the internet (Hu et al.,
2023), and knowledge graphs (Song et al., 2020; Lee and Kim, 2021). Our
approach is similar in retrieving extra information to complement the provided
visual information. However, we retrieve contextual information directly
related to events and entities depicted in the image, while prior approaches
search for general factual (Vrandečić and Krötzsch, 2014; Auer et al., 2007)
or commonsense knowledge (Speer et al., 2018; Sap et al., 2019). The
contextual information we seek, for example, the reason for [Person 1]’s
injury in Figure 2, is unavailable in those external sources. Limited works
have explored specific sample-related contextual information. Naik et al.
(2023) utilize image source metadata while Biten et al. (2019); Tran et al.
(2020) leverage paired news article. Both cases bypass context retrieval by
exploiting image metadata as is, unlike our work. Furthermore, they do not
focus on integrating temporal or event-specific context, which is crucial for
reasoning in semantically complex VLU tasks.

####  2.0.3. Abstention in Multimodal Systems

Abstaining from responding instead of making incorrect predictions was
originally explored in the unimodal language domain to address out-of-
distribution or adversarial inputs (Chow, 1957; De Stefano et al., 2000; El-
Yaniv and Wiener, 2010; Jiang et al., 2018; Kamath et al., 2020; Varshney et
al., 2022). In the multimodal domain, recent works have been proposed that
abstain similarly in the case of out-of-distribution samples (Dancette et al.,
2023) or low model confidence (Whitehead et al., 2022). In contrast, our
proposed approach avoids making predictions when sufficient context to answer
the question is unavailable. Unlike prior works, our abstention mechanism
works from a data-centric view and applies to new benchmarks without any re-
training.

##  3\. Problem Space

We investigate several benchmarks to study the problem of insufficient context
in VLU domain: VQA v2, OKVQA, A-OKVQA, GQA, VCR, VisualCOMET, and Visual SWAG.
These datasets cover a range of VLU tasks, such as visual question answering,
image-based text generation, and image-text matching. Notably, SWAG is a text-
only entailed event inference dataset. To facilitate the study on multimodal
event entailment inference, we replace the text premise in SWAG with the
corresponding image frame retrieved from the samples’ source video. We call
this multimodal dataset, Visual SWAG, where given an image premise, the
required task is to infer the entailed event in textual form.

For the datasets with contextual data available – VCR, VisualCOMET, and Visual
SWAG – we first collect that contextual data, as in Section 4. Then, we
utilize it to facilitate evidence-based VLM prediction via a context selection
module, as in Section 5.1. Further, we leverage a combination of vanilla VLM
and VLM trained with context to pseudo-label samples with insufficient
context. The pseudo labels are then used to train an insufficient context
detector, CARA. We demonstrate that CARA generalizes to VQA v2, OKVQA,
A-OKVQA, and GQA without having ever been trained on them.

The input information for the above reasoning benchmarks can be generally
denoted as $x=(x_{T},x_{I})$ where $x_{T}$ is the textual input and $x_{I}$ is
the image input. In our first study, we explore whether adding another input
Context, $x=(x_{T},x_{I},C)$, can help and explore how to obtain the most
beneficial context. For our second study, we develop functions to detect
samples with insufficient context in the input, $x=(x_{T},x_{I})$, and abstain
from making baseless predictions.

##  4\. Contextual Data Collection

We begin by collecting contextual data for the three VLU benchmarks described
above. These benchmarks evaluate models’ understanding of events using images
sourced from existing video datasets. To ensure comprehensive context is
provided for each sample across the tasks, we collected multimodal contextual
data, including preceding and subsequent visual frames along with paired text
scripts. We first discuss how we retrieved the source video data and then how
context was retrieved and filtered. Finally, we present statistics about our
assembled dataset.

###  4.1. Data Fetching

The images from VCR, VisualCOMET, and Visual SWAG are derived from video
sources like LSMDC (Rohrbach et al., 2015), ActivityNet(Fabian Caba Heilbron
and Niebles, 2015), or YouTube. These video datasets consist of sequences of
video clips, where each clip is paired with a sentence describing the event in
the clip. Since annotations in VCR, VisualCOMET, and Visual SWAG include
specific frame IDs and clip IDs for most samples, we can locate and retrieve
the source clip of the corresponding sample. We removed all samples for which
we could not find the corresponding source clip or paired video scripts.

###  4.2. Context Retrieval and Filtering

####  4.2.1. Context Retrieval

We retrieve the clips before and after the corresponding source clip as visual
context. These video clips are also paired with video scripts. We retrieve
these scripts as text context for data points. However, using video frames as
visual context can be highly redundant due to their repetitive nature (i.e.
adjacent frames are generally very similar), thus we find the most descriptive
frame from each of these clips by finding the best match with the script using
a pre-trained CLIP (Radford et al., 2021) model. More formally, we denote
context from preceding clips with negative indices ($c_{-3},c_{-2}...)$, while
context from succeeding clips has positive indices ($c_{1},c_{2}...)$, where
each $c_{i}$ consists of both vision and language contexts.

####  4.2.2. Context Filtering

Given that a substantial portion of our datasets comprises temporal questions,
specifically those inquiring about states before and after, we take
precautions to avoid inadvertently providing context that may disclose the
answer to the model. We achieve this by identifying such cases using keywords
and then filtering out contexts that could potentially lead to cheating. For
instance, samples featuring questions about the past will be devoid of
negatively indexed contexts.

###  4.3. Data Statistics

####  4.3.1. Training Data

Our training dataset split includes 41,008 image-text pairs from the train
split of Visual SWAG, with an additional 94,404 distinct image-text pairs as
multimodal context; 119,994 image-question pairs from the train split of VCR
with 79,913 distinct image-text context pairs; and 190,457 image-text pairs
(from 63,499 unique situations) from the train split of VisualCOMET, with
79,109 distinct image-text context pairs.

####  4.3.2. Evaluation Data

Our validation set includes 10,645 image-text pairs from the train split of
Visual SWAG with 57,880 distinct image-text pairs as multimodal context;
15,092 image-question pairs from the train split of VCR with 10,014 distinct
image-text context pairs; and 23,930 image-text pairs (from 7,978 unique
situations) from the train split of VisualCOMET, with 9,933 distinct image-
text context pairs.

Figure 3. A high-level demonstration of the probabilistic context selection
method. For the VLM’s input, in addition to the question and image, a context
sentence selected by the Context Selection Module is appended to the original
input.

[Pipeline of the probabilistic selection method.]The context selection method
uses a selection module to generate a score for each context to select the
most relevant context for the VLM to answer the question.

##  5\. Method

Using our collected contextual data, we first develop a model-agnostic smart
context selection module to add relevant context to samples to improve the
model’s understanding of the sample and, hence, its performance. We then
create a multimodal abstention model to identify samples lacking sufficient
event-specific context and prevent models from making baseless predictions on
such samples.

###  5.1. Context Selection Module

Consider a Vision-Language Model (VLM) $M$, with image input $x_{I}$ and text
input $x_{T}$. The most straightforward way to incorporate context,
$C=[c_{i}]_{i\in[-n,n]}$, is to append it to the model input. That is,
$y_{pred}=M(x,C)$, where $x=(x_{I},x_{T})$. This results in a brute-force
context injection approach, which is both heavily computationally expensive
and potentially noisy. Instead, we aim to build a method to intelligently
select the most relevant context according to the given target image and text
premises.

We thus propose a “probabilistic context selection” method (see Figure 3).
This end-to-end, model-agnostic technique aims to streamline the selection of
event-specific context. Our method features a context selection module $M_{c}$
designed to identify the most relevant context $c^{*}$ for the given input
$x$. As a result, the model’s output is given by $y_{pred}^{*}=M(x,c^{*})$.
The core idea behind this approach is that it can dynamically select the
context that is most aligned with the input. This allows it to integrate only
the most relevant context into the downstream reasoning process while
filtering out noisy context. We demonstrate that this significantly improves
the model’s ability to handle complex reasoning tasks requiring contextual
information.

Specifically, for a given input $x$ and a set of context $c_{i}$, the
selection module $M_{c}$ computes a score vector
$S=[s_{i}]_{i\in[-n,n]}=M_{c}(x,c_{i})$. Each score $s_{i}$ within this vector
denotes the relevance of $c_{i}$ to $x$. The $c_{i}$ with the highest $s_{i}$
is chosen as the selected context for inference. During training, each $s_{i}$
is used to (softly) select the $c_{i}$ as the context in the VLM, $M$. This
encourages the context selection module $M_{c}$ to assign a low weight to
context $c_{i}$, which leads to a high loss in $M$ and vice versa. Thus,
$M_{c}$ is trained to assign a high weight to the most relevant context. The
resulting loss function is:

(1) $\small\mathcal{L}=\sum_{i=-n}^{n}s_{i}\cdot l(M(x,c_{i}),y)$

where, $l$ represents the cross-entropy loss.

This probabilistic sampling procedure, where $c_{i}$ is sampled using $s_{i}$,
is differentiable end-to-end. We illustrate how the context selection module
interacts with the backbone VLM in Figure 3.

For a given input $x$ and a specific context $c_{i}$, we append the context
with the input to create $x_{i}=[x\parallel c_{i}]$. More specifically, text
context $c_{i_{T}}$ is appended to text input $x_{T}$ and image context
$c_{i_{I}}$ with image input $x_{I}$, creating $x_{i_{T}}=[x_{T}\parallel
c_{i_{T}}]$ and $x_{i_{I}}=[x_{I}\parallel c_{i_{I}}]$ respectively.
$x_{i_{T}}$ and $x_{i_{I}}$ are then processed by $M$ as it would normally
process $x_{T}$ and $x_{I}$.

###  5.2. Multimodal Abstention Detector

The above section assumes additional context is available to be recovered
through retrieval. However, in many real-world scenarios, additional context
may not be available. Thus, we propose a generalized multimodal abstention
detector that aims to identify if a sample is unanswerable due to a lack of
context thereby preventing baseless predictions.

Developing a mechanism to detect samples with insufficient event-specific
context is an extremely challenging problem because the model must first
hypothesize what the sufficient context would be to answer the question before
determining if that context is lacking. In this work, we present a
straightforward yet effective solution to address this issue. We leverage our
previously trained model with context and compare its response with a vanilla
model trained without context to pseudo-label if the sample contains or lacks
sufficient context. Our key insight is that if a sample already has sufficient
context, the model’s response should remain relatively consistent when
additional context is added. Conversely, if the sample lacks sufficient
context, the model’s response should improve on adding additional context. The
pseudo labels are then used to train our insufficient context detector. We
illustrate this process in Figure 4 and detail it below.

####  5.2.1. Confidence-Driven Pseudo-Labelling

We train two models: a Context-VLM (C-VLM), which incorporates context into
its decision-making process (as detailed in Section 5.1, and a vanilla VLM,
which operates without context. We compare the responses from both models to
pseudo-label samples as follows:

  * •

Positive: Instances correctly answered by the C-VLM model with high confidence
above a designated threshold, $\gamma$, but incorrectly answered by the VLM
with low confidence below a designated threshold $\mu$. The significant
difference in accuracy and confidence suggests that these instances previously
lacked sufficient context for unambiguous understanding.

  * •

Negative: Instances correctly recognized by both models with confidence above
threshold $\gamma$, implying that context does not play a critical role in
their identification.

  * •

Excluded: Instances not fitting into the above categories are excluded. The
impact of context on these instances is uncertain, and including them could
introduce noise into the model training process.

When pseudo-labeling the training set, we divide the dataset into two equal
parts. Each part is labeled based on the inference results obtained from the
two models trained on the other half. This strategy ensures a robust pseudo-
labeling process that mitigates overfitting risks and data leakage from the
validation set.

####  5.2.2. Training.

After pseudo-labeling the data points, we train CARA, a 24-layer cross-modal
attention module as our insufficient context detector via cross-entropy loss.
The input to CARA is a sample image and the corresponding question or
statement, and the output is a binary label denoting whether the sample lacks
sufficient context or not.

####  5.2.3. Inferencing.

When inferencing using CARA, we predict if a data point lacks sufficient
context based on whether CARA’s prediction score exceeds a dataset-specific
threshold $\theta$.

Note that during both training and inference, the detector operates without
access to context. Our goal is to develop a generalized detector that
maintains high performance and generalizability across datasets, regardless of
whether context is available.

Figure 4. Top: We use models with/without context to pseudo-label whether
instances need context. The labeled data is then used to train CARA. Bottom:
CARA decides whether to abstain based on whether the input contains sufficient
context.

$0$$1$$2$$3$$4$$5$$6$$7$$69$$71$$73$$75$$77$$79$Metric ValueVL-BERT Acc.Visual
SWAGVCR (a) Context Window Size Ablation.

$0$$1$$2$$3$$74$$76$$78$$80$$82$$84$Visual SWAG Acc.

$0$$1$$2$$3$$70$$72$$74$$76$$78$$80$$82$VCR Acc.BLIP2MiniGPT4VL-BERT

$0$$1$$2$$3$$0.112$$0.115$$0.118$$0.121$$0.124$VisualCOMET BLEU

$0$$1$$2$$3$$0.445$$0.450$$0.455$$0.460$$0.465$VisualCOMET CIDER

(b) Context Selection Number Ablations.

Figure 5. (a) Ablation study for context window size. The performance of VL-
BERT in accuracy is plotted on y-axis against context window size on x-axis
for Visual SWAG (dotted) and VCR (solid). The peak accuracy at window size 3
indicates the optimal size. (b) Ablation study for context selection number.
The performance of multiple models (in different colors) is plotted on y-axis
against context selection number on x-axis for multiple datasets. We allow the
VLM to observe multiple contexts within the window size. The peak in the curve
indicates selecting 2 out of 3 contexts results in the best performance.

##  6\. Experiments

In this section, we present experimental results and analysis to illustrate
the effectiveness of our context selection methodology and CARA. We first
present implementation details, followed by context selection results and
abstention detection results.

###  6.1. Implementation Details

####  6.1.1. Base Models

We demonstrate the efficacy of our approach on two different classes of VLMs:
discriminative (VL-BERT(Su et al., 2020)) and generative (BLIP (Li et al.,
2022), BLIP2 (Li et al., 2023b), MCAN(Yu et al., 2019), MiniGPT-4 (Zhu et al.,
2023), OFA (Wang et al., 2022), PNP(Tiong et al., 2023), Prophet(Shao et al.,
2023), and PromptCap(Hu et al., 2022) ). Generalization across models shows
that our approach is model-agnostic. We adhered strictly to the implementation
described in the original papers and repositories of all models.

####  6.1.2. Training

Fine-tuning of VL-BERT, BLIP, and OFA is done with 2 NVIDIA-RTX 24 GB GPUs
with batch size 32, BLIP2 and MiniGPT-4 are trained with 2 NVIDIA A100 GPUS
with the same batch size. The initial learning rates for VL-BERT, BLIP, BLIP2,
MiniGPT-4, and OFA are 7e-5, 1e-5, 2e-5, 3e-5, and 3e-5, respectively. VL-BERT
is trained for 20 epochs, and BLIP is for 10 epochs, while BLIP2 and MiniGPT-4
are trained for 5 epochs. For OFA, we follow the original implementation and
train a total of 40K steps. Training of the models takes $\sim$48 hours. The
abstention detector is trained for 10 epochs with a learning rate of 7e-5.

####  6.1.3. Context Selection

BLIP, BLIP2, MiniGPT-4, and OFA lack native RoI functions as in VL-BERT. Thus,
to process datasets requiring RoI handling such as VCR and VisualCOMET we
adopt Merlot’s approach (Zellers et al., 2021) of drawing colored highlights
around referenced entities in pixel space, as shown in Figure 2. In our
experiments, we employ a Sentence-BERT (Reimers and Gurevych, 2019) as the
text encoder for $M_{C}$, and a ViT (Dosovitskiy et al., 2021) as the vision
encoder. We fuse the global embeddings from those two encoders’ output via
concatenation and apply an MLP with sigmoid to map the fused feature into a
score ranging from 0 to 1.

####  6.1.4. Abstention Detector

We use a 24-layer cross-modal attention model as the multimodal abstention
detector, following (Su et al., 2020) to initialize and train it on datasets
labeled with the pseudo-labeling method described in Section 5.2. However,
this results in an unbalanced training set with significantly more negative
data points. To address this, we apply loss weighting during training. Our
experiments show increasing the weight of positive data points in the loss by
six results in the highest evaluation performance.

The CARA tailored for the Visual SWAG and VCR tasks are trained on their
respective datasets. However, since VisualCOMET is a generative dataset
without a binary correctness measure, we utilize the CARA trained on VCR for
it instead. In practice, we utilize heuristics rules 111Please refer to the
Supplementary Materials for the detailed implementation steps. integrating the
confidence scores of the VLM and CARA’s prediction to obtain the best model
performance over downstream tasks.

When conducting confidence-driven pseudo-labeling, the hyperparameters for
filtering thresholds, $\gamma$, is 0.7 and $\mu$ is 0.5.

###  6.2. Context Selection Results

To determine the best way to integrate context into VLU tasks, we first
perform extensive ablation experiments. We analyze various components of our
context selection method, including modalities, window size, number of
selected frames or scripts, and selection strategies. Finally, we apply our
method to benchmark approaches.

Table 1. (a) Ablation of context and selection modality on Visual SWAG. Best
performance is achieved when both context and selection modality is textual.
(b) Ablation of context selection methods. The top half selects context using
language models while The bottom half selects using context index.

(a) Context and Selection Modality Context | Selection | Acc.
---|---|---
No context | N/A | 74.20
Text | Text | 78.01
Text | Image | 77.87
Text | Text+Image | 77.78
Image | Text | 75.28
Image | Image | 74.97
Image+Text | Text | 77.57
Image+Text | Image | 76.43

(b) Selection Strategy Method | V. SWAG | VCR
---|---|---
Embedding Sim. | 76.91 | 72.44
Prob Selection | 78.4 | 73.78
Random | 76.79 | 71.43
Index -1 | 77.86 | 71.76
Index -2 | 77.25 | 71.01
Index -3 | 76.15 | 70.23

Table 2. Experiment results of VLMs on Visual SWAG, VCR, and VisualCOMET with/without context. Models with Prob Selection (Prob.) show significant improvement over the baselines. VL-BERT cannot be trained for generative tasks, so results on VisualCOMET are not shown. Model | V.SWAG | VCR | VisualCOMET
---|---|---|---
Acc. | Acc. | BLEU4 | CIDER | METEOR
VL-BERT | 74.20 | 70.18 | - | - | -
VL-BERT+Prob. | 78.40 | 73.78 | - | - | -
BLIP | 62.65 | 69.03 | 0.1098 | 0.4468 | 0.1656
BLIP+Prob. | 63.22 | 70.74 | 0.1147 | 0.4595 | 0.1674
BLIP2 | 81.30 | 78.20 | 0.1120 | 0.4492 | 0.1648
BLIP2+Prob. | 84.36 | 81.32 | 0.1163 | 0.4612 | 0.1672
OFA | 54.07 | 69.35 | 0.1329 | 0.4446 | 0.1527
OFA+Prob. | 59.44 | 73.20 | 0.1354 | 0.4642 | 0.1558
MiniGPT-4 | 80.82 | 76.66 | 0.1189 | 0.4503 | 0.1653
MiniGPT-4+Prob. | 83.88 | 79.28 | 0.1217 | 0.4634 | 0.1679
Table 3. Performance Analysis of CARA on CASE. While performing better than baselines, CARA also generalizes to new benchmarks without ever been trained on them.  Method |  | Pseudolabelled
---
Data Source
CASE-VCR | CASE-V.SWAG
CARA | VCR | 75.69 | 64.55
V.SWAG | 54.09 | 73.05
Selector-MaxProb | - | 51.03 | 50.1
Selector-MLP | - | 54.82 | 53.84
Table 4. VLM performance enhancement by using CARA. CARA consistently improves performance across VLMs and benchmarks. * indicates the performance obtained via applying CARA trained on VCR. Model | V.SWAG | VCR | VisualCOMET
---|---|---|---
Acc. | Acc. | BLEU4 | CIDER | METEOR
VL-BERT | 73.72 | 70.13 | - | - | -
VL-BERT+CARA | 77.04 (74.76*) | 73.40 | - | - | -
BLIP2 | 81.30 | 78.20 | 0.1120 | 0.4492 | 0.1648
BLIP2+CARA | 82.93 | 79.77 | 0.1179* | 0.4642* | 0.1674*
Table 5. CARA Generalization. CARA improves VLM performance across several benchmarks despite never being trained on them, indicating its utility even for future benchmarks.  Model | VQA v2 | GQA | OKVQA | A-OKVQA
---|---|---|---|---
| +CARA |  | +CARA |  | +CARA |  | +CARA
Zero-shot |  |  |  |  |  |  |  |
BLIP2 | 62.5 | 64.9 | 46.33 | 47.57 | 34.68 | 36.55 | 43.94 | 45.00
PNP | 57.52 | 60.42 | 35.68 | 36.96 | 26.98 | 28.54 | 27.78 | 28.43
Prophet | - | - | - | - | 61.1 | 62.28 | 58.20 | 58.37
PromptCap | - | - | - | - | 60.44 | 61.54 | 60.43 | 60.59
MCAN | - | - | - | - | 53.05 | 53.63 | 51.97 | 52.09
Table 6.  Analysis of CARA abstained samples by humans, with percentages indicating ”Abstained” samples where the model refrained from predicting, and ”Ambiguous” and ”Insufficient” denoting the proportions of abstained samples judged as such. Samples lacking context are considered ambiguous, but not vice versa. Majority of CARA-abstained samples are ambiguous, proving CARA works by removing ambiguous samples, not hard samples.  | Abstention | VQA v2 | GQA | OKVQA | A-OKVQA
---|---|---|---|---|---
Abstained | CARA | 10.90 | 5.78 | 28.77 | 5.12
Ambiguous | 69.00 | 70.00 | 64.00 | 69.00
Insufficient Context | 47.00 | 42.00 | 46.00 | 53.00
Abstained | Selector-MLP | 21.07 | 17.05 | 34.08 | 25.08
Ambiguous | 23.00 | 16.00 | 25.00 | 17.00
Insufficient Context | 18.00 | 14.00 | 20.00 | 16.00
Table 7. Abstention analysis on VQA v2. We compare CARA’s system risk $\mathcal{R}$, effective reliability $\Phi_{1}$, and coverage $\mathcal{C}$ against baselines. The arrows following the metrics indicate the direction of improvement (for example $(\downarrow)$ indicates lower the better). Risks in red are higher than tolerance, and metrics are highlighted in blue when CARA outperforms the baseline methods. VLM | Method |  Risk Tolerance $r=10\%$ |  Risk Tolerance $r=30\%$ |  Risk Tolerance $r=50\%$
---|---|---|---|---
$\mathcal{R}(\downarrow)$ | $\Phi_{1}(\uparrow)$ | $\mathcal{C}(\uparrow)$ | $\mathcal{R}(\downarrow)$ | $\Phi_{1}(\uparrow)$ | $\mathcal{C}(\uparrow)$ | $\mathcal{R}(\downarrow)$ | $\Phi_{1}(\uparrow)$ | $\mathcal{C}(\uparrow)$
BLIP2 | Selector-MaxProb | 1.4 | 5.0 | 5.2 | 4.9 | 18.7 | 21.2 | 8.5 | 29.3 | 36.8
Selector-MLP | 2.3 | 7.9 | 8.7 | 7.4 | 25.1 | 30.2 | 11.7 | 36.0 | 46.2
CARA | 2.6 | 10.0 | 10.7 | 8.8 | 31.1 | 38.8 | 13.1 | 39.0 | 56.3
PNP | Selector-MaxProb | 29.5 | 2.3 | 7.3 | 35.5 | 12.1 | 45.8 | 41.6 | 23.2 | 79.5
Selector-MLP | 29.0 | 9.8 | 11.6 | 35.4 | 23.5 | 48.9 | 41.4 | 29.8 | 78.5
CARA | 28.9 | 13.6 | 15.4 | 36.6 | 28.6 | 51.3 | 41.8 | 33.5 | 80.1

####  6.2.1. Data Modality Ablation

To ensure the best context utilization, we examine which modalities are most
effective for both selecting relevant context and integrating it into VLMs.
Table 1(a) shows results from experiments over VL-BERT with different input
and output modalities for our context selection module. The “Selection
modality” column denotes the modality used to select context and “Context
Modality” refers to the modality of selected context 222Please refer to the
Supplementary Materials for details. . We find that using text as both the
selection and context modality is the most effective approach. This trend
holds across different context modalities (image, text, or both).

Regardless of the selection modality, we find adding visual context typically
leads to a performance drop. Using text alone for context consistently yields
the best results. While the visual context may offer rich information, our
findings suggest it often introduces noise, which hurts performance. This
highlights an opportunity for future research in integrating multimodal
context in VLU tasks. However, one critical finding is that our approach never
decreases the performance of the VLM, even in the absence of text for
selection or context. To perform this ablation, we use a window size of 3,
select one context unit, and use our probabilistic selection method.

####  6.2.2. Window Size Ablation

We next experiment with different context window sizes for VL-BERT on the
Visual SWAG and VCR datasets. In this experiment, we limit the number of
selected context units to 1, and the window size can range from 0 (no context)
to 7. Figure 5(a) shows how VL-BERT’s performance varies with different window
sizes. Our results indicate that models with nonzero window sizes outperform
the baseline (window size of 0). However, performance plateaus and eventually
decreases with excessively large window sizes. The peak accuracies on both VCR
and Visual SWAG datasets suggest that their optimal window sizes are 3.

####  6.2.3. Selection Number Ablation.

Next, we analyze the impact of the amount of context on model performance.
Figure 5(b) presents the results of training VL-BERT, BLIP2, and MiniGPT-4 on
VCR and Visual SWAG datasets with different numbers of selected contexts over
a window size of 3. In this setup, $M_{c}$ considers all possible combinations
of concatenating $r$ context from $n$ available options arranged temporally
rather than being limited to a single optimal context. The models achieved
their best performance across all three datasets with a selection number of 2.
Notice the drop at the right end of each graph, where the selection number
equals the window size. This extreme scenario inputs all the contexts inside
the window without a context selection module and shows the importance of our
selection module for improving context utilization.

####  6.2.4. Selection Strategy Ablation.

In Table 1(b), we compare context selection strategies with VL-BERT to
determine the most effective one. The bottom of the table presents results
from heuristic methods based on context indices, while the top part explores
dynamic selection strategies leveraging the embedding similarity. More
specifically, we can rely on sentence similarity between the question and text
context using Sentence-BERT (Reimers and Gurevych, 2019) after determining
textual modality as the optimal selection modality. Both the embedding
similarity method and heuristic methods are notably outperformed by our
jointly trained model, the probabilistic context selection method. In this
comparison, selection methods are limited to a window size of 3 and 2 selected
contexts.

####  6.2.5. Benchmark Comparison.

We apply our probabilistic context selection approach to various base models
and report the results in Table 2. With our probabilistic selection method (+
Prob.), all five base models can generally improve their performance across
three tasks. Furthermore, the base models can achieve SOTA scores on
VisualCOMET with our selection method. These results verify the benefits of
incorporating contextual information into VLU tasks and the effectiveness of
our method.

###  6.3. Abstention Detector Results

In this section, we discuss the effect of our abstention detector by comparing
the performance of VLMs with and without CARA.

####  6.3.1. Evaluation of Detection Accuracy for Samples with Insufficient
Context.

Building on the confidence-driven pseudo-labeling method outlined in Section
5.2.1, we assembled a small data pool of 500 positive and 500 negative image-
question pairs from the VCR validation set and a similar one from Visual SWAG.
These datasets were evaluated by Amazon Mechanical Turk workers to ascertain
their ambiguity 2. With this curated data, we created the Context Ambiguity
and Sufficiency Evaluation (CASE) Set, spanning both benchmarks to evaluate
the efficacy of abstention methods in detecting samples with insufficient
context.

We compare CARA to two established methods (Whitehead et al., 2022): Selector-
MaxProb, which abstains based on a predefined confidence threshold, and
Selector-MLP, which predicts the likelihood of correct predictions using a
Multilayer Perceptron module from an image and question. As demonstrated in
Table 3, CARA exhibits high detection superior accuracy across these
evaluation sets. Notably, when trained with the pseudo-labeled data from VCR,
CARA also demonstrates strong performance on the CASE set of Visual SWAG,
underscoring its generalizability. Moreover, the gap between CARA’s
performance and human judgment (oracle accuracy) underscores the ongoing
challenges in detecting samples with insufficient context and highlights the
value of the CASE set for future research.

####  6.3.2. Performance Enhancement with CARA

We compare the performance of baseline VLMs with and without CARA across three
VLU tasks in Table 4. When using CARA (+CARA), we calculate the accuracy of
the baseline VLM only in instances where CARA indicates sufficient context for
accurate prediction. The results show that CARA results in a significant
improvement in performance across all three tasks. Surprisingly, adding CARA
can approach or even exceed the benchmarks set by context-aware models
(referenced in Table 2), showing that CARA adds substantial value for
multimodal abstention.

####  6.3.3. Generalization Across VLU Benchmarks

To test CARA’s generalizability, we trained it on VCR and evaluated on VQA v2,
GQA, OKVQA, and A-OKVQA in Table 5. Our findings demonstrate CARA’s robust
generalizability, emphasizing its significance for future VLU benchmarks for
performance gains or insufficient samples cleaning.

If we further investigate the data points filtered out by CARA and examine
them by humans333Please refer to the Supplementary Materials for the detailed
implementation steps., as in table 6, we can observe that the majority of data
points filtered out by CARA consist of ambiguous questions and most of them
lack sufficient context for a determined answer. These results demonstrate
CARA’s effectiveness in enhancing model performance across benchmarks and
underscore the key problem of instances lacking sufficient context within
these benchmarks 444The verification results of abstained samples by human
review over VCR, VisualCOMET, and Visual SWAG can be found in the
Supplementary Materials..

####  6.3.4. Benchmarking Risk and Coverage

Previous abstention strategies or selective prediction systems (De Stefano et
al., 2000; El-Yaniv and Wiener, 2010) were designed to optimize the balance
between risk and coverage, where risk refers to the error rate for the
predictions made, and coverage quantifies the total number of predictions
issued. An optimal abstention strategy aims to minimize risk while maximizing
coverage to the greatest extent possible. We assess the risk ($\mathcal{R}$)
and coverage ($\mathcal{C}$) performance metrics of CARA compared to previous
abstention strategies. Our goal is to minimize risk while maximizing coverage.
We also evaluate the effective reliability ($\Phi_{c}$) of CARA, rewarding
accurate predictions and penalizing incorrect responses.

Table 7 presents the evaluation results with varying risk tolerance levels
(i.e. how much risk a model accepts before abstention). We observe that in
most cases CARA’s $\mathcal{R}$’s system risk is roughly on par with existing
methods while achieving significantly higher reliability and effective
coverage for both BLIP2 (Li et al., 2023b) and PNP (Tiong et al., 2023).

##  7\. Qualitative Examples

Figure 6. Qualitative Examples. Correct answers are highlighted in green.
Incorrect answers are highlighted in red.

Figure 6 shows qualitative examples of effective context in VLU (top) and
context-aware abstention (bottom).

##  8\. Conclusion

In this paper, we discussed the issue of insufficient context grappling
existing VLU benchmarks and proposed strategies to effectively integrate
context, when available, or abstain from speculative prediction in case of
samples with insufficient context. We also contributed datasets to enable
further exploration of this problem.

##  9\. Limitation

Please refer to the LIMITATION section in the Supplementary Materials for a
detailed discussion.

## References

  * (1)
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2425–2433.
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.  DBpedia: A Nucleus for a Web of Open Data.  _Lecture Notes in Computer Science_ 6, 722–735.  https://doi.org/10.1007/978-3-540-76298-0_52
  * Bhattacharya et al. (2019) Nilavra Bhattacharya, Qing Li, and Danna Gurari. 2019.  Why does a visual question have different answers?. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4271–4280.
  * Biten et al. (2019) Ali Furkan Biten, Lluis Gomez, Marçal Rusiñol, and Dimosthenis Karatzas. 2019.  Good News, Everyone! Context driven entity-aware captioning for news images.  arXiv:1904.01475 [cs.CV]
  * Chow (1957) Chi-Keung Chow. 1957.  An optimum character recognition system using decision functions.  _IRE Transactions on Electronic Computers_ 4 (1957), 247–254.
  * Dancette et al. (2023) Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. 2023.  Improving Selective Visual Question Answering by Learning From Your Peers. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 24049–24059.
  * De Stefano et al. (2000) C. De Stefano, C. Sansone, and M. Vento. 2000.  To reject or not to reject: that is the question-an answer in case of neural classifiers.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 30, 1 (2000), 84–94.  https://doi.org/10.1109/5326.827457
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  arXiv:2010.11929 [cs.CV]
  * El-Yaniv and Wiener (2010) Ran El-Yaniv and Yair Wiener. 2010.  On the Foundations of Noise-free Selective Classification.  _Journal of Machine Learning Research_ 11 (05 2010), 1605–1641.
  * Fabian Caba Heilbron and Niebles (2015) Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. 2015.  ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 961–970.
  * Falotico and Quatto (2015) Rosa Falotico and Piero Quatto. 2015.  Fleiss’ kappa statistic without paradoxes.  _Quality & Quantity_ 49 (2015), 463–470.  https://api.semanticscholar.org/CorpusID:121849847
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018.  Vizwiz grand challenge: Answering visual questions from blind people. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3608–3617.
  * Hochgeschwender et al. (2019) Nico Hochgeschwender, Gary Cornelius, and Holger Voos. 2019.  Arguing Security of Autonomous Robots. In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. 7791–7797.  https://doi.org/10.1109/IROS40897.2019.8967670
  * Hu et al. (2022) Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022.  PromptCap: Prompt-Guided Task-Aware Image Captioning.  _arXiv preprint arXiv:2211.09699_ (2022).
  * Hu et al. (2023) Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023.  Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 23369–23379.
  * Jahan et al. (2019) Farha Jahan, Weiqing Sun, Quamar Niyaz, and Mansoor Alam. 2019.  Security Modeling of Autonomous Systems: A Survey.  _ACM Comput. Surv._ 52, 5, Article 91 (sep 2019), 34 pages.  https://doi.org/10.1145/3337791
  * Jiang et al. (2018) Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018.  Pythia v0. 1: the winning entry to the vqa challenge 2018.  _arXiv preprint arXiv:1807.09956_ (2018).
  * Kafle and Kanan (2017) Kushal Kafle and Christopher Kanan. 2017.  An analysis of visual question answering algorithms. In _Proceedings of the IEEE international conference on computer vision_. 1965–1973.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective Question Answering under Domain Shift. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 5684–5696.  https://doi.org/10.18653/v1/2020.acl-main.503
  * Lee and Kim (2021) Jaeyun Lee and Incheol Kim. 2021.  Vision–Language–Knowledge Co-Embedding for Visual Commonsense Reasoning.  _Sensors (Basel, Switzerland)_ 21 (2021).  https://api.semanticscholar.org/CorpusID:233462391
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  arXiv:2301.12597 [cs.CV]
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.  arXiv:2201.12086 [cs.CV]
  * Li et al. (2020) Mengdi Li, Cornelius Weber, and Stefan Wermter. 2020.  Neural networks for detecting irrelevant questions during visual question answering. In _Artificial Neural Networks and Machine Learning–ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15–18, 2020, Proceedings, Part II 29_. Springer, 786–797.
  * Li et al. (2023a) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023a.  Evaluating object hallucination in large vision-language models.  _arXiv preprint arXiv:2305.10355_ (2023).
  * Lin and Byrne (2022) Weizhe Lin and Bill Byrne. 2022.  Retrieval augmented visual question answering with outside knowledge.  _arXiv preprint arXiv:2210.03809_ (2022).
  * Lin et al. (2023) Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, and Zongyuan Ge. 2023.  Medical visual question answering: A survey.  _Artificial Intelligence in Medicine_ (2023), 102611.
  * Lovenia et al. (2023) Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023.  Negative object presence evaluation (nope) to measure object hallucination in vision-language models.  _arXiv preprint arXiv:2310.05338_ (2023).
  * Mahendru et al. (2017) Aroma Mahendru, Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, and Stefan Lee. 2017.  The Promise of Premise: Harnessing Question Premises in Visual Question Answering. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, Copenhagen, Denmark, 926–935.  https://doi.org/10.18653/v1/D17-1097
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Naik et al. (2023) Nandita Naik, Christopher Potts, and Elisa Kreiss. 2023.  Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_. 2821–2825.
  * Park et al. (2020) Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. 2020.  VisualCOMET: Reasoning about the Dynamic Context of a Still Image.  arXiv:2004.10796 [cs.CV]
  * Puzis et al. (2013) Yury Puzis, Yevgen Borodin, Rami Puzis, and Iv Ramakrishnan. 2013.  Predictive web automation assistant for people with vision impairments. 1031–1040.  https://doi.org/10.1145/2488388.2488478
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  arXiv:2103.00020 [cs.CV]
  * Ray et al. (2016) Arijit Ray, Gordon Christie, Mohit Bansal, Dhruv Batra, and Devi Parikh. 2016.  Question relevance in VQA: identifying non-visual and false-premise questions.  _arXiv preprint arXiv:1606.06622_ (2016).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  https://arxiv.org/abs/1908.10084
  * Rohrbach et al. (2015) Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015.  A dataset for movie description. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3202–3212.
  * Rohrbach et al. (2016) Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2016.  Movie Description.  arXiv:1605.03705 [cs.CV]
  * Sanket Shah and Talukdar (2019) Naganand Yadati Sanket Shah, Anand Mishra and Partha Pratim Talukdar. 2019.  KVQA: Knowledge-Aware Visual Question Answering. In _AAAI_.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 3027–3035.
  * Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022.  A-okvqa: A benchmark for visual question answering using world knowledge. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII_. Springer, 146–162.
  * Shao et al. (2023) Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023.  Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering. In _Computer Vision and Pattern Recognition (CVPR)_. 14974–14983.
  * Song et al. (2020) Dandan Song, Siyi Ma, Zhanchen Sun, Sicheng Yang, and Lejian Liao. 2020.  KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning.  arXiv:2012.07000 [cs.AI]
  * Speer et al. (2018) Robyn Speer, Joshua Chin, and Catherine Havasi. 2018.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  arXiv:1612.03975 [cs.CL]
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  arXiv:1908.08530 [cs.CV]
  * Tiong et al. (2023) Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C. H. Hoi. 2023.  Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training.  arXiv:2210.08773 [cs.CV]
  * Toor et al. (2017) Andeep Toor, Harry Wechsler, and Michele Nappi. 2017.  Question Part Relevance and Editing for Cooperative and Context-Aware VQA (C2VQA). 1–6.  https://doi.org/10.1145/3095713.3095718
  * Toor et al. (2019) Andeep S Toor, Harry Wechsler, and Michele Nappi. 2019.  Biometric surveillance using visual question answering.  _Pattern Recognition Letters_ 126 (2019), 111–118.
  * Tran et al. (2020) Alasdair Tran, Alexander Mathews, and Lexing Xie. 2020.  Transform and Tell: Entity-Aware News Image Captioning.  arXiv:2004.08070 [cs.CV]
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings. In _Findings of the Association for Computational Linguistics: ACL 2022_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 1995–2002.  https://doi.org/10.18653/v1/2022.findings-acl.158
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A Free Collaborative Knowledgebase.  _Commun. ACM_ 57 (09 2014), 78–85.  https://doi.org/10.1145/2629489
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.  arXiv:2202.03052 [cs.CV]
  * Whitehead et al. (2022) Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. 2022.  Reliable visual question answering: Abstain rather than answer incorrectly. In _European Conference on Computer Vision_. Springer, 148–166.
  * Wu et al. (2023) Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E Gonzalez, and Trevor Darrell. 2023.  See, say, and segment: Teaching lmms to overcome false premises.  _arXiv preprint arXiv:2312.08366_ (2023).
  * Yu et al. (2019) Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019.  Deep Modular Co-Attention Networks for Visual Question Answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 6281–6290.
  * Zellers et al. (2019a) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a.  From recognition to cognition: Visual commonsense reasoning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 6720–6731.
  * Zellers et al. (2019b) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b.  From Recognition to Cognition: Visual Commonsense Reasoning.  arXiv:1811.10830 [cs.CV]
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference.  arXiv:1808.05326 [cs.CL]
  * Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021.  MERLOT: Multimodal Neural Script Knowledge Models.  arXiv:2106.02636 [cs.CV]
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.  arXiv:2304.10592 [cs.CV]
  * Zorec et al. (2020) Matej Zorec, Tim Carrington, and Matija Marolt. 2020.  Visual Assistant-TV for the visually impaired. In _ICCHP_. 43.

Figure 7. Three scenarios of how context can help understanding in an image-
language reasoning task. In the first row, The fighting scene in the context
suggests he is down because of the injury, but not what he seems to be doing
in the image. In the second row, the context mentions the presence of a corpse
invisible in the image, so the woman is more likely to stay away because of
fear instead of distaste. In the third row, The appearance of a vodka bottle
and his stumbling indicate he is drunk, which makes the correct answer more
plausible.

##  Appendix A Importance of Context

Context provides critical information to explain situations, avoid
misinterpretations, and leverage fine-grained knowledge for prediction. It is
particularly important in visual language understanding. For example, the
ambiguities in Figure 7 cannot be clarified without context. Lack of
sufficient context can harm model learning and performance evaluation.
However, ensuring adequate context exists in multimodal inputs with images and
text is challenging and impractical for real-world scenarios, where additional
context might not be available. Thus, the ability to abstain when needed
context is missing is equally crucial.

Figure 8. Interface layout for annotators in verifying the correctness of
CARA’s detection results. We implemented this interface over the Amazon Turker
platform to facilitate turkers to effectively understand the assignment and
annotate the data. In practice, we also include plenty of annotated examples
beforehand as the instruction or reference.

##  Appendix B Additional Implementation Details

###  B.1. Heuristics with Context-Aware Abstention

Since our method is data-centric and does not base its predictions on the
output of the Vision Language Model (VLM), when deciding whether to abstain
from an answer generated by a VLM, to account for the VLMs’ variance, we
combine the VLM’s confidence with the prediction of the Context-AwaRe
Abstention (CARA) detector according to a heuristic rule:

(2) $H=w(1-C)+(1-w)V$

where $V$ is the VLM’s confidence, $C$ is CARA’s confidence, and $0<w\leq 1$
is the weighting of CARA’s score. A high $C$ indicates CARA predicts a need
for context, so $1-C$ represents CARA’s confidence in that the data point’s
has sufficient context. We use the heuristic score $H$ and a risk tolerance
threshold to decide on abstaining or answering. This heuristic incorporates
both CARA’s and VLM’s confidence scores via a weighted sum.

##  Appendix C Additional Ablation Details

###  C.1. Context Modality

As introduced in Section 6.1.3 of the main paper, for the context selection
module, we encode image context and text context using ViT (Dosovitskiy et
al., 2021) and Sentence-BERT(Reimers and Gurevych, 2019), respectively. The
two embeddings are combined and passed through a Multilayer Perceptron to
obtain the final score. The context is inputted to VLM by appending the
image/text context to the input sequence. Thus, we can control the modality of
context the VLM can observe by appending the corresponding contexts to the
inputs. Similarly, the modality the context selection module uses to select
context can also vary by adding/removing the vision or language encoder. For
instance, when using text to select text-only context, we append only the text
context to the input sequence for the VLM, and we only use the embeddings from
Sentence-BERT for the context selection module.

Figure 9. Dataset Construction Process: 1. Remove non-LSMDC data points. 2.
Find the source clip for each image by matching the file names and save the
corresponding captions as context. 3. Filter out context that can potentially
reveal the correct answer.

##  Appendix D Data Collection

###  D.1. Context retrieval

The data points in VCR, VisualCOMET, and Visual SWAG are sourced from either
ActivityNet (Fabian Caba Heilbron and Niebles, 2015), LSMDC (Rohrbach et al.,
2016), or YouTube. Since only LSMDC data points have consistent and ordered
context information available, we initially remove all non-LSMDC sourced data
points in the Data Filtering stage, as depicted in Figure 9,

In the Context Retrieval stage, we first sort the clips temporally. Then, we
locate the source LSMDC clip for each QA data point. The script of the source
clip serves as the text context of $c_{0}$. The corresponding vision context
is collected by finding the most relevant frame using a pre-trained CLIP
(Radford et al., 2021) model, as mentioned in the main paper.

The contexts at positive and negative indices are acquired with a similar
procedure. For the context $c_{\pm n}$, we traverse $n$ clips forward or
backward and apply the procedure mentioned above. Collecting all the $c_{\pm
n}$ will result in a context window size of $2n+1$.

We set the maximum $n$ to be 20. This means each data point will include a
range from $c_{-20}$ to $c_{20}$, totaling 41 context data sourced from LSMDC.
We believe this adequately encompasses the necessary context for each
question. Given that the average duration of LSMDC clips is 4.16 seconds,
these 41 contexts collectively span approximately 2 minutes and 56 seconds of
content.

Finally, in the Context Filtering stage, we remove the potentially cheating
contexts for temporal questions by matching the keywords in the question. For
example, Figure 9 shows contexts with positive indices are removed for
questions asking about “After” to prevent the answer from leaking.

Table 8.  Analysis of CARA abstained samples by humans, with percentages indicating ”Abstained” samples where the model refrained from predicting, and ”Ambiguous” and ”Insufficient” denoting the proportions of abstained samples judged as such. Samples lacking context are considered ambiguous, but not vice versa. Majority of CARA-abstained samples are ambiguous, proving CARA works by removing ambiguous samples, not hard samples.  | Abstention | VCR | VisualCOMET | Visual SWAG | VQA v2 | GQA | OKVQA | A-OKVQA
---|---|---|---|---|---|---|---|---
Abstained | CARA | 13.66 | 18.14 | 18.73 | 10.90 | 5.78 | 28.77 | 5.12
Ambiguous | 88.00 | 98.00 | 78.00 | 69.00 | 70.00 | 64.00 | 69.00
Insufficient Context | 82.00 | 98.00 | 74.00 | 47.00 | 42.00 | 46.00 | 53.00
Abstained | Selector MLP | 24.83 | 25.90 | 24.08 | 21.07 | 17.05 | 34.08 | 25.08
Ambiguous | 58.00 | 72.00 | 65.00 | 23.00 | 16.00 | 25.00 | 17.00
Insufficient Context | 32.00 | 70.00 | 58.00 | 18.00 | 14.00 | 20.00 | 16.00

###  D.2. DATA QUALITY CONTROL FOR CASE

Building on the confidence-driven pseudo-labeling method (Section 5.2.1), we
assembled a small data pool of 500 positive and 500 negative image-question
pairs from the VCR validation set and Visual SWAG. With this curated data, we
created the Context Ambiguity and Sufficiency Evaluation (CASE) Set, spanning
both benchmarks to evaluate the efficacy of abstention methods in detecting
samples with insufficient context. We evaluated these samples by Amazon
Mechanical Turk workers to assess their ambiguity. We implemented the
interface layout shown in Figure 8 and hired experienced annotators to
manually verify the filtered samples. For each sample detected as positive
(lacking sufficient context) by CARA, four experienced annotators re-verified
it. The annotators were not informed of CARA’s prediction and answered two
curated questions independently. Based on the annotation results, we
calculated the voting percentage to determine if each question was considered
ambiguous and lacking sufficient context. To ensure annotation consistency, we
used Fleiss’ Kappa ($\kappa$) (Falotico and Quatto, 2015) to assess inter-
annotator agreement. For determining if the question is ambiguous, $\kappa$ is
0.81, and for determining if the question lacks sufficient context, $\kappa$
is 0.84.

##  Appendix E Addtional Experiments and Results

###  E.1. Abstention Results Verification

In Tables 4 and 5 of the main paper, we can observe that adding CARA on top of
base VLMs can generally improve the performance across benchmarks. To further
verify CARA’s effectiveness and ensure that CARA focuses on removing
problematic ambiguous samples (including samples with insufficient context)
instead of challenging but answerable ones, we conduct manual human
verification to examine the filtered-out data by CARA. Specifically, we let
human annotators verify 100 randomly sampled instances for each dataset where
CARA predicts positive (i.e., need context). In Table 8, we show human
verification results on different datasets. We label “ambiguous” for data
points that have no obvious correct answer, as shown in the examples in Figure
12 of the supplementary materials. The ambiguity of these questions may vary.
For example, the first question’s reference to laptops is ambiguous since
there is more than one brand in the image, and some cannot be determined due
to poor image quality. Among these, a significant portion of ambiguity is
caused by insufficient context, which happens when the question is ambiguous.
Still, such ambiguity can be alleviated when additional information about the
scene (i.e., context) is provided. Examples of this type are shown in Figures
1, 2, and 6 of the main paper, as well as highlighted in Figure 12. We are
surprised to find that CARA is able to identify other types of samples with
ambiguities as well, such as those with ambiguous questions or poor image
quality.

Figure 10. Qualitative examples of Visual SWAG (example 1-3) and VCR (4-6)
with/without context. Predictions made by context model are highlighted in
Green. Predictions made by no context models are highlighted in Red. The
selected context is highlighted in Blue. Correct choices are in Bold font for
Visual SWAG and VCR examples. Figure 11. Qualitative examples of VisualCOMET
with/without context. Predictions made by context model are highlighted in
Green. Predictions made by no context models are highlighted in Red. The
selected context is highlighted in Blue. Figure 12. Additional qualitative
examples answered by BLIP2. The labels “Ambiguous” and “Insufficient Context”
under samples abstained by CARA are determined by human annotators.

###  E.2. Qualitative Examples

####  E.2.1. Context Selection

In Figure 10 and Figure 11, our contextual model demonstrates superior
performance over the non-contextual model across numerous instances. Take, for
instance, the third example from the Visual SWAG dataset. Without context, the
correct choice, A, appears arbitrary, leading to the model incorrectly
selecting choice D. However, our contextual model effectively identifies and
leverages the relevant context—“someone gets up and goes over to the cool
box”—to correctly associate it with the answer “returns with four cans”.

####  E.2.2. Abstention

Figure 12 shows the prediction of CARA, with the abstained samples labeled
with “Ambiguous” or “Insufficient Context” by humans. We also provide BLIP2’s
response to these questions. Compared to the non-abstained questions (bottom
two), the abstained ones have significantly diverse answer references,
indicating disagreement among annotators.

##  Appendix F Limitation

Although CARA can be adapted to different problems and VLMs without needing to
be retrained, the decision threshold and parameters for the heuristic rule in
Equation 2 may require additional tuning to achieve optimal performances.

The context selection method defined in Section 5 of the main paper works only
for segmented contexts, which in our case consists of short sentences and
videos. However, when applying it in other scenarios, for example, when
context is in the form of paragraphs, context needs to be broken into pieces
to adapt our method. In addition, the loss function mentioned in Section 5.1
of the main paper requires the model to recompute the input $m$ times given
the context window size of $m$. This raises scalability issues for large
context window sizes.