# MMRel: A Relation Understanding Dataset and Benchmark in the MLLM Era

Jiahao Nie1,2 Gongjie Zhang311footnotemark: 1 Wenbin An4 Yap-Peng Tan2 Alex C.
Kot2 Shijian Lu2
1IGP, Nanyang Technological University 2Nanyang Technological University
3Alibaba DAMO Academy 4Xi’an Jiaotong University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Project page: https://github.com/niejiahao1998/MMRel
Equal contributionCorresponding author

###### Abstract

Despite the recent advancements in Multi-modal Large Language Models (MLLMs),
understanding inter-object relations, i.e., interactions or associations
between distinct objects, remains a major challenge for such models. This
issue significantly hinders their advanced reasoning capabilities and is
primarily due to the lack of large-scale, high-quality, and diverse multi-
modal data essential for training and evaluating MLLMs. In this paper, we
provide a taxonomy of inter-object relations and introduce Multi-Modal
Relation Understanding (MMRel), a comprehensive dataset designed to bridge
this gap by providing large-scale, high-quality and diverse data for studying
inter-object relations with MLLMs. MMRel features three distinctive
attributes: (i) It includes over 15K question-answer pairs, which are sourced
from three distinct domains, ensuring large scale and high diversity; (ii) It
contains a subset featuring highly unusual relations, on which MLLMs often
fail due to hallucinations, thus are very challenging; (iii) It provides
manually verified high-quality labels for inter-object relations. Thanks to
these features, MMRel is ideal for evaluating MLLMs on relation understanding,
as well as being used to fine-tune MLLMs to enhance relation understanding and
even benefit overall performance in various vision-language tasks. Extensive
experiments on various popular MLLMs validate the effectiveness of MMRel. Both
MMRel dataset and the complete labeling scripts have been made publicly
available.

##  1 Introduction

Multi-Modal Large Language Models (MLLMs) [1, 2, 3, 4, 5, 6, 7] have
demonstrated impressive capabilities in various vision-language tasks [8, 9,
10, 11]. One key step in these tasks is to comprehend inter-object relations,
which are defined by the interactions or associations between distinct objects
in images [12]. Understanding these relations is crucial for MLLMs to perform
advanced reasoning, provide accurate responses in visual question answering
[8, 9], and generate detailed image captions [10, 11]. However, recent studies
[13, 14, 15, 16] have shown that most existing MLLMs struggle with handling
inter-object relations, as illustrated in Fig. 1. The primary reasons for this
challenge include the lack of a formal definition of inter-object relations
and the absence of benchmarks that feature large-scale, diverse, and high-
quality vision-language data.

To fill up the gap by building a comprehensive dataset for MLLMs’ relation
understanding, we first partition inter-object relations into three distinct
categories, namely, spatial relations (e.g., dog is left to cat), action
relations (e.g., boy eats pizza), and comparative relations (e.g., apple is
smaller than watermelon). Furthermore, we define the term relation
hallucinations by scenarios where the MLLM-generated inter-object relations
align with common sense but deviate from the factual contents in images. For
example, MLLMs responds that “a man drives a car” while the ground truth is “a
man pushes a car”. In addition, we examine pioneer benchmarks that were not
intended for but did allow assessing MLLMs’ relation understanding
capabilities [17, 18, 19, 20, 21, 22, 23, 24, 25]. As Tab. 1 shows, most
existing benchmarks are typically small-scale with homogeneous data, capture
one or two categories of inter-object relations only, and lack diversity with
images from a single domain.

Table 1: A comparison of MMRel with existing relation understanding
benchmarks. MMRel is featured with large-scale, diverse, and high-quality data
on inter-object relations.

Dataset | Scale | Taxonomy | Domain | MLLM$+$ | Adv- | Evaluation
---|---|---|---|---|---|---
Spatial | Action | Comparative | Real | SD | Dall-E | Human | subset
MMHAL [18] | 12 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | LLM
RAH [21] | 500 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | Yes/No
MERLIM [22] | - | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | Yes/No
FAITHSCORE [23] | - | - | - | - | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | LLM
M-HalDetect [20] | - | - | - | - | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | Human
AMBER [17] | 1.7K | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | Yes/No
SPEC [19] | 3.5K | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | Choice
Hallusion [25] | 150 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | Yes/No
MME [19] | 60 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | Yes/No
MMRel | 15K | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | Yes/No

Figure 1: MLLMs often fail to understand inter-object relations.

With the definition of inter-object relations and relation hallucinations, we
create Multi-Modal Relation Understanding (MMRel), a large-scale benchmark
that comprises multi-modal data of three categories of relations (i.e.
spatial, action, and comparative) that are sourced from three distinctive
domains (i.e., real images, synthetic images from SDXL [26], and synthetic
images from Dall-E [27]). We present sample images from MMRel in Fig. 2. To
ensure the quality of the collected data, we adopt a semi-automatic pipeline
that leverages MLLMs [27, 28] to generate images and annotations based on
textual prompts and then verifies and corrects the generated images and
annotations by human reviewers. As MLLMs are known to have the issue of
hallucination (especially on relation understanding tasks), to evaluate and
prevent hallucinations, we create a challenging adversarial subset in MMRel
which utilizes relations that deviate from common sense to assess the relation
understanding capabilities of MLLMs rigorously. All these lead to a benchmark
that comprises over 15,000 question-answer pairs, each of which comes with
clearly defined annotations and relation categories. To make the evaluation
straightforward and easy to understand, we adopt a Yes/No framework, which has
been widely used in previous datasets [17, 21, 22, 24, 25, 29].

Figure 2: Illustration of MMRel samples. MMRel covers three categories of
inter-object relations, namely, spatial (orange), action (blue), and
comparative (green), and its images are sourced from three domains of
different styles: (a) natural images, (b) synthetic images by SDXL [26], and
(c) synthetic images by Dall-E [27]. It is also featured by a challenging
subset that comprise images with unusual relations as in (c). More Dall-E
samples are provided in the Appendix.

MMRel can serve different purposes thanks to its large-scale, diverse, and
high-quality vision-language data. One typical scenario is evaluating the
relation understanding capabilities of existing MLLMs. We conduct
comprehensive evaluations on this aspect, revealing that both existing MLLMs
[2, 5, 7] and hallucination mitigation techniques [30, 31, 32] exhibit sub-
optimal performance in understanding inter-object relations. Another typical
scenario is instruction tuning of MLLMs for understanding inter-object
relations [1, 2]. Extensive experiments verify the effectiveness of MMRel as
both evaluation set and training data, and also demonstrate that MMRel data is
beneficial not only to relation understanding but also to other perceptual
tasks. All these studies affirm MMRel’s values in exploring more powerful
MLLMs in terms of improving advanced perception.

The contributions of this work can be summarized in three aspects. First, it
systematically defines inter-object relations as well as relation
hallucinations under the context of MLLMs. This lays a solid foundation for
future study of inter-object relations and hallucination mitigation. Second,
we create MMRel, a large-scale benchmark that comprises over 15K high-quality
vision-language data that are sourced from diverse domains with clearly
defined inter-object relations. To the best of our knowledge, this is the
first large-scale benchmark on inter-object relations that comes with diverse
and high-quality vision-language data. Third, extensive experiments
demonstrate the value of MMRel for both evaluating existing MLLMs on relation
understanding and fine-tuning for the enhanced power of relation
understanding.

##  2 Reviews on Existing Datasets for Relation Understanding

Research on inter-object relations boasts a rich history, with Scene Graph
Generation (SGG) [33, 34] serving as a foundational area of focus. SGG is
specifically dedicated to identifying objects and their interrelations within
images. Leveraging specialized object detection methods [35, 36] that excel in
detecting both tiny and inconspicuous objects, SGG datasets such as Visual
Genome (VG) [12] are distinguished by their extremely detailed and
comprehensive annotations. Following the recent successes [37], Multi-Modal
Large Language Models (MLLMs) have received significant attention. In
response, new datasets [14, 38] have been developed to assess the capabilities
of MLLMs in relation understanding.

Figure 3: Comparing MMRel with existing datasets. Existing datasets exhibit
the following limitations: (a)-(b) negative choices for action relations are
implausible, thus can be easily ruled out; (c) evaluation metrics are complex
and subjective; (d) definitions and taxonomies of relations are ambiguous.
MMRel mitigates the limitations of existing datasets by enhancing the
precision and utility of inter-object relation data.

VL-Checklist [13] and ARO [38] are representative datasets that evaluate the
capabilities of MLLMs to understand fine-grained information, including inter-
object relations. However, these datasets inherit images and labels directly
from VG [12] and adopt a binary Positive/Negative evaluation framework, as
illustrated in Fig. 3(a)-(b). This evaluation approach has significant
limitations, as it generates negative choices randomly without human
verification. Specifically, the negative choices are generated by: (i)
randomly altering the relations (e.g. man riding shirt); (ii) simply reversing
the order of objects (e.g. grass is eating the horse), making such negative
options semantically implausible or even impossible. Given this limitation, we
hypothesize that MLLMs can easily deduce the correct answers even without
seeing images, as the advanced capabilities of LLM enable MLLMs to effectively
dismiss negative choices that starkly deviate from common sense.

Moreover, although recent MLLMs [1, 2, 3, 4, 5, 6, 7] are versatile, their
proficiency in object detection is limited due to pre-training on coarse
granularity image-text pairs [39, 40]. Therefore, the overly dense and
occasionally incorrect annotations from existing datasets, such as VG [12],
are unsuitable for assessing MLLMs. Consequently, re-labeling some existing
dataset is essential for developing new benchmarks that are more suitable for
the evaluation of MLLMs’ capabilities in relation understanding. This has
triggered the recent initiatives such as M-HalDetect [20] and AMBER [17] that
concentrate on generating new labels. Specifically, M-HalDetect [20]
incorporates manually annotated relations within lengthy descriptive
sentences, with evaluations performed by human assessors (refer to Fig. 3(c)).
However, this metric does not straightforwardly reflect the capabilities in
relation understanding and introduces potential risks of subjective
comparisons. On the other hand, AMBER [17] simplifies the classification of
relationships to contact or not and employs a Yes/No evaluation framework
(refer to Fig. 3(d)). While this approach is straightforward, it leads to an
imprecise evaluation due to the restricted definition of relations.

The limitations of existing benchmarks can thus be summarized: (i) They are
small-scale and lack diversity; (ii) Their annotations are of low quality,
with negative choices for action relations often being meaningless; (iii) They
lack clear definitions and taxonomies of inter-object relations.

##  3 Multi-Modal Relation Understanding (MMRel) Dataset

As analyzed in Sec. 1 and Sec. 2, existing datasets for relation understanding
are generally constrained by their small-scale and homogeneous nature. This
motivates us to develop a new benchmark, Multi-Modal Relation Understanding
(MMRel), characterized by large-scale, diverse, and high-quality data on
inter-object relations. In this section, we first describe the image sources
and relation taxonomies in Sec. 3.1. Subsequently, we introduce a Semi-
automatic Data Collection pipeline in Sec. 3.2, designed to amass large-scale,
diverse, and high-quality inter-object relation data. Finally, we present the
statistics and additional details of the MMRel dataset in Sec. 3.3.

###  3.1 Data Source and Relation Taxonomy

Following previous works [13, 38], we source our real image subset from the
Visual Genome (VG) [12] and select synthetic images from the SPEC [19], which
is built based on SDXL [26]. Specifically, we use the relative spatial subset
from SPEC [19] for spatial relations and the relative size subset for
comparative relations. To further diversify the MMRel, we also generate
synthetic images via Dall-E [27]. Given the good clarity of annotations in
SPEC [19], we adapt their labels to enhance our dataset’s utility for relation
understanding. Details regarding annotation and image generation across other
domains are outlined in Sec. 3.2.

Moreover, we employ distinct annotation strategies for each category of
relations within our taxonomy. For spatial relations, we limit the categories
to left, right, on/over, and under, as other descriptors such as around or
next to do not accurately capture relative inter-object positions. For action
relations, our annotations are based on the capabilities of Multi-Modal Large
Language Models (MLLMs), as detailed in Sec. 3.2. Comparative relations are
annotated manually.

###  3.2 Semi-Automatic Data Collection

In this section, we introduce our Semi-automatic Data Collection pipeline
(SemiDC), as illustrated in Fig. 4. This pipeline is capable of annotating
large-scale existing images and generating a substantial amount of high-
quality synthetic images. As discussed in Sec. 2, re-labeling existing images
is essential since their original labels are incompatible with MLLMs. To this
end, we design SemiDC to generate high-quality relation annotations via GPT-4V
[28] (refer to Fig. 4(a)) for large-scale VG dataset [12]. This process is
divided into three stages: (i) Pre-processing: We selectively exclude images
featuring complex scenes that pose challenges for GPT-4V [28] in generating
accurate annotations. Specifically, we utilize the object labels and
corresponding bounding boxes from the VG dataset [12] to identify and exclude
images that contain more than 10 tiny objects, each occupying less than 1/20
of the total image area; (ii) Re-labeling via GPT-4V [28]: We employ the in-
context learning paradigm [4, 41] to use GPT-4V [28] to generate relation
annotations. For example, we provide an image and its accurate relation
annotations as a reference, together with a query image accompanied by the
textual prompt “Please generate the action relationships between salient
objects in this image”. GPT-4V [28] will then produce annotations for the
query image based on the text prompt. More details can be found in the
Appendix; (iii) Human verification: We manually assess and correct the
annotations that are generated by GPT-4V [28], to ensure the quality of the
collected inter-object relation data.

Figure 4: Pipeline of Semi-automatic Data Collection (SemiDC). (a) Re-labeling
images from Visual Genome [12] datasets. (b) Generating synthetic images based
on textual prompts.

As MLLMs are known to have a bias towards positive responses [29, 42], we
design MMRel to maintain a balance between “Yes” and “No” answers to mitigate
potential biases in evaluation metrics [29]. Specifically, negative for
spatial and comparative relations can be simply derived by reversing the order
of two objects. For instance, “dog is left to cat” is the inverse of “cat is
left to dog”. Nevertheless, as discussed in Sec. 2, randomly generated
negative choices for action relations are implausible, which can be easily
ruled out even without referring to the images. To address this issue, we
first adopt GPT-3.5 [43] to generate plausible negative relations that are
well aligned with common sense for given objects, and then verify the
generated negative relations by manual checking. For instance, a suitable
negative relation for “man eats cake” might be generated as “man bakes cake”.
This design is a significant improvement over prior works [13, 38], in terms
of both training and evaluation.

Furthermore, we introduce generative models for generating more diverse inter-
object relation data. The SemiDC pipeline for generating images (refer to Fig.
2(b)) is also divided into three stages: (i) Preparing textual prompts: We
prepare textual prompts describing inter-object relations; (ii) Image
generation via Dall-E-3 [27]: We utilize the prepared text prompts as inputs
for DALL-E-3 [27] to generate corresponding images. For instance, if we want
Dall-E [27] to generate an image depicting a crocodile on the dessert, then we
should employ the prompt “a photo-realistic picture depicting an object-object
relation: crocodile on dessert” as input. These textual prompts define our
targeted relations and the visual style of the generated images. We consider
four styles, photo-realistic, watercolor, abstract, and oil painting (refer to
Fig. 2(c)), to enrich our dataset; (iii) Human verification of generated
images: If the generated images do not match the input relations, we will
modify the relations instead of re-generating new images. The same as
annotating negative choices for VG [12], we utilize GPT-3.5 [43] to generate
negative action relations. More details of SemiDC for both relabeling and
generation of inter-object relation data are described in the Appendix.

###  3.3 Statistics of MMRel

Table 2: Statistics of each subset in MMRel.

Domain | Real | SDXL | Dall-E | Total
---|---|---|---|---
Taxonomy | Spatial | Action | Comparative | Total | Spatial | Comparative | Total | Spatial | Action | Total
Scale |  $\sim$5.22K |  $\sim$4.63K |  $\sim$0.66K |  $\sim$10.51K |  $\sim$1.75K |  $\sim$0.65K |  $\sim$2.40K |  $\sim$1.20K |  $\sim$1.18K |  $\sim$2.38K |  $\sim$15.29K

Table 3: Number of objects and categories of relations in MMRel.

Object | Spatial | Action | Comparative
---|---|---|---
855 | 5 | 378 | 9

Tab. 2 shows the statistics of inter-object relation data in MMRel.
Specifically, MMRel comprises over 15,000 question-answer pairs across 7
subsets, spanning 3 domains and 3 categories of relations. Concretely,
approximately 68% of the data consists of real images, while the remaining 32%
comprises synthetic images (i.e., SDXL [26] and Dall-E [27]). In terms of
relation categories, MMRel includes a larger scale of spatial and action
relations compared to comparative relations. Thanks to the open-vocabulary
capability of GPT-4V [28], MMRel guarantees a diverse range of objects and
action relations, as shown in Tab. 3. Regarding comparative relations, we
manually re-label VG images, resulting in a subset that, though relatively
small in scale with limited diversity, ensures accurate annotations. It is
important to note that our annotations strictly describe the objects and their
relations, without including any attributes, so that the evaluation can
accurately reflect the relation understanding capabilities. For example, if we
introduce attributes into the question “Is blue bird right to yellow bird?”,
and the MLLMs answers differ from the ground-truth, we cannot ascertain
whether the discrepancy stems from a misunderstanding of attributes (i.e.,
blue and yellow) or inter-object relation (i.e., right). Inspired by [44], we
recognize that unusual data are crucial for evaluating and enhancing MLLM
capabilities. We therefore create an adversarial relation subset comprising
710 challenging question-answer pairs. This subset is designed to rigorously
assess the relation understanding capabilities of MLLMs.

In summary, MMRel is featured with four distinctive characteristics as
illustrated in Fig. 3(e): (i) It comprises large-scale and diverse relation
data that are collected from different sources; (ii) It comprises high-quality
relation annotations and negative-choice data for action relations; (iii) It
adopts an effective and efficient evaluation framework [29] that employs
straightforward Yes/No instead of other complicated discriminative or
generative evaluations; (iv) It includes an adversarial subset designed to
evaluate the MLLMs’ ability to understand challenging relations.

##  4 Experiments

In this section, we systematically analyze the effectiveness of Multi-Modal
Relation Understanding (MMRel). We first describe the experiment setups in
Sec. 4.1. Subsequently, we employ MMRel to evaluate the relation understanding
capabilities of multiple representative Multi-Modal Large Language Models
(MLLMs) and hallucination mitigation techniques in Sec. 4.2. Finally, we
employ MMRel to fine-tune LLaVA-1.5 [2] to examine how much the fine-tuning
enhances their relation understanding capabilities, which is further discussed
in Sec. 4.3.

Table 4: Evaluation results of existing MLLMs on our MMRel benchmark. Existing
MLLMs can not handle the task of relation understanding well.

Domain | Taxonomy | Model | Accuracy(%) | Precision(%) | Recall(%) | F1-Score(%)
---|---|---|---|---|---|---
Real | Spatial | InstructBLIP [5] | 50.58 | 50.36 | 81.38 | 62.22
LLaVA-1.5 [2] | 51.48 | 51.01 | 74.88 | 60.68
Qwen-VL [7] | 54.10 | 52.65 | 81.34 | 63.93
VCD [32] | 50.76 | 50.42 | 92.01 | 65.14
DOLA [30] | 51.54 | 51.00 | 78.46 | 61.82
OPERA [31] | 51.21 | 50.68 | 90.24 | 64.91
Action | InstructBLIP [5] | 65.14 | 60.59 | 89.92 | 72.40
LLaVA-1.5 [2] | 67.67 | 62.98 | 88.35 | 73.54
Qwen-VL [7] | 69.87 | 64.04 | 92.94 | 75.83
VCD [32] | 65.33 | 59.86 | 96.60 | 73.91
DOLA [30] | 69.10 | 63.53 | 92.09 | 75.19
OPERA [31] | 70.35 | 63.74 | 96.68 | 76.83
Comparative | InstructBLIP [5] | 56.23 | 54.96 | 69.00 | 61.19
LLaVA-1.5 [2] | 61.70 | 61.56 | 62.31 | 61.93
Qwen-VL [7] | 72.19 | 69.95 | 77.81 | 73.67
VCD [32] | 62.16 | 60.64 | 69.30 | 64.68
DOLA [30] | 60.64 | 60.48 | 61.40 | 60.94
OPERA [31] | 74.32 | 75.64 | 71.73 | 73.63
SDXL | Spatial | InstructBLIP [5] | 51.37 | 50.92 | 75.74 | 60.90
LLaVA-1.5 [2] | 52.46 | 51.50 | 84.55 | 64.01
Qwen-VL [7] | 55.84 | 54.59 | 69.45 | 61.13
VCD [32] | 50.63 | 50.33 | 96.68 | 66.20
DOLA [30] | 52.12 | 51.21 | 89.70 | 65.20
OPERA [31] | 50.92 | 50.48 | 97.25 | 66.46
Comparative | InstructBLIP [5] | 51.23 | 50.95 | 65.85 | 57.45
LLaVA-1.5 [2] | 59.08 | 58.75 | 60.92 | 59.82
Qwen-VL [7] | 70.15 | 77.64 | 56.62 | 65.48
VCD [32] | 64.31 | 63.17 | 68.62 | 65.78
DOLA [30] | 58.46 | 57.49 | 64.92 | 60.98
OPERA [31] | 67.85 | 65.34 | 76.00 | 70.27
Dall-E | Spatial | InstructBLIP [5] | 55.29 | 53.06 | 91.74 | 67.23
LLaVA-1.5 [2] | 53.22 | 51.88 | 88.93 | 65.53
Qwen-VL [7] | 57.44 | 54.98 | 82.15 | 65.87
VCD [32] | 51.65 | 50.86 | 97.85 | 66.93
DOLA [30] | 62.98 | 61.39 | 77.56 | 68.53
OPERA [31] | 69.23 | 64.68 | 89.92 | 74.24
Action | InstructBLIP [5] | 63.14 | 61.29 | 79.02 | 69.03
LLaVA-1.5 [2] | 61.20 | 60.80 | 71.38 | 65.67
Qwen-VL [7] | 62.47 | 62.90 | 67.80 | 65.26
VCD [32] | 61.62 | 58.65 | 88.78 | 70.63
DOLA [30] | 53.06 | 51.68 | 94.05 | 66.71
OPERA [31] | 50.41 | 50.21 | 97.85 | 66.37

###  4.1 Experiment Setups

MLLM Baselines. We conduct experiments with six representative MLLMs [2, 5, 7,
30, 31, 32]. Three of them, namely, LLaVA-1.5 [2], InstructBLIP [5], and Qwen-
VL [7], are versatile and can work on a variety of vision-language tasks.
However, they are trained with image-text pairs with coarse granularity [45,
46] and thus prone to generating hallucinations [47, 48]. We therefore select
another three MLLMs that were designed for hallucination mitigation. As MMRel
involves inter-object relations only, we select DOLA [30], OPERA [31], and VCD
[32] that focus on mitigating object hallucinations and so are more suitable
for evaluating their relation understanding capabilities. Notably, the
techniques selected for managing object hallucinations do not involve
additional training, ensuring fair comparisons.

Evaluation. For fair benchmarking, we evaluate all six MLLMs using their 7B
versions. Specifically, LLaVA-1.5 [2] and InstructBLIP [5] incorporate Vicuna
7B as their language decoder [49], while Qwen-VL adopts Qwen-7B backbone [7].
Given the impressive performance of LLaVA-1.5 [2], we also employ DOLA [30],
OPERA [31], and VCD [32] in our evaluation, all based on LLaVA-1.5 [2]. As
MMRel adopts a Yes/No evaluation framework, we adopt the same metrics,
accuracy, precision, recall, and F1-Score, as in POPE [29]. We follow all
details of evaluation as VCD [32].

###  4.2 Evaluations with MMRel

We first employ the 15K question-answer pairs in MMRel to evaluate how MLLMs
perform when facing three categories of relations from three data domains. As
Tab. 4 shows, all six MLLMs face various challenges in relation understanding.
Specifically, we observe that the performance on action relations is superior
to that on spatial and comparative relations in real images. This disparity
arises because the coarsely-aligned training data (all are real images)
predominantly features action descriptions, whereas spatial and comparative
terms are often represented by more vague expressions, such as near, next to,
in front of, etc. Furthermore, because the synthetic images generated by SDXL
[26] and DALL-E [27] have distributions that differ significantly from the
pre-training data, their performance on action relations is lower than that
observed with real images. Moreover, even techniques designed to address
object hallucinations, such as those reported in [30, 31, 32], continue to
struggle with relation understanding. Their limited performances underscore
that the primary challenges in MMRel arise from understanding relations rather
than recognizing objects. The performances in Tab. 4 could serve as baselines
for further training-free methods.

Table 5: Statistics of training, test and adversarial sets in MMRel.

Domain | Real | SDXL | Dall-E | Total
---|---|---|---|---
Taxonomy | Spatial | Action | Comparative | Total | Spatial | Comparative | Total | Spatial | Action | Total
Training |  $\sim$3.70K |  $\sim$3.62K |  $\sim$0.59K |  $\sim$7.91K |  $\sim$1.43K |  $\sim$0.45K |  $\sim$1.88K |  $\sim$0.80K |  $\sim$0.79K |  $\sim$1.59K |  $\sim$11.38K
Test |  $\sim$1.52K |  $\sim$1.01K |  $\sim$0.07K |  $\sim$2.60K |  $\sim$0.32K |  $\sim$0.20K |  $\sim$0.52K |  $\sim$0.40K |  $\sim$0.39K |  $\sim$0.79K |  $\sim$3.91K
Adversarial |  $\sim$0.43K |  $\sim$0.18K |  $\sim$0.10K |  $\sim$0.71K

###  4.3 Fine-Tuning with MMRel

The large scale and high diversity of MMRel also enable it to fine-tune MLLMs
to enhance their relation understanding capabilities. To verify this, we split
the 15K data into training and test sets, as shown in Tab. 5. Here, we only
adopt LLaVA-1.5 [2] as a strong baseline due to its popularity, outstanding
performance, and versatility. To maintain the generality of the MLLMs and
prevent overfitting to our training data, we have integrated the MMRel
training set with 260K instruction data points from LLaVA [1]. Furthermore,
adhering to the instruction template used in LLaVA [1], we have expanded our
Yes/No answers into various formats to enhance the diversity of the dialogues
(with more details in the Appendix). We strictly follow the steps for
instruction tuning from LLaVA-1.5 [2]. The statistics of our adversarial
subset are shown in Tab. 5.

Table 6: Fine-tuning LLaVA-1.5 with MMRel improves the performance
significantly.

Domain | Taxonomy | Model | Accuracy(%) | Precision(%) | Recall(%) | F1-Score(%)
---|---|---|---|---|---|---
Real | Spatial | LLaVA-1.5 [2] | 51.29 | 50.88 | 74.83 | 60.57
Finetune with MMRel | 79.66 | 82.45 | 75.34 | 78.74
Action | LLaVA-1.5 [2] | 65.08 | 61.68 | 84.97 | 71.47
Finetune with MMRel | 81.15 | 81.21 | 82.47 | 81.84
Comparative | LLaVA-1.5 [2] | 64.04 | 83.33 | 66.18 | 73.77
Finetune with MMRel | 76.40 | 79.75 | 92.65 | 85.71
SDXL | Spatial | LLaVA-1.5 [2] | 50.63 | 50.39 | 80.63 | 62.02
Finetune with MMRel | 95.63 | 96.20 | 95.00 | 95.60
Comparative | LLaVA-1.5 [2] | 61.24 | 82.89 | 63.00 | 71.59
Finetune with MMRel | 77.13 | 77.43 | 99.50 | 87.09
Dall-E | Spatial | LLaVA-1.5 [2] | 65.46 | 51.57 | 89.60 | 65.46
Finetune with MMRel | 94.06 | 94.95 | 93.07 | 94.00
Action | LLaVA-1.5 [2] | 57.69 | 57.94 | 66.83 | 62.07
Finetune with MMRel | 71.79 | 67.04 | 89.60 | 76.69
Adversarial | Mix | LLaVA-1.5 [2] | 50.35 | 47.10 | 70.50 | 56.47
Finetune with MMRel | 67.52 | 60.79 | 81.37 | 69.59

Following instruction tuning with the MMRel training set, we observe
significant and consistent improvements in performance across all three
domains and relation categories, as detailed in Tab. 6. Additionally, such
fine-tuning enhances the performance on the adversarial subset as well. To
validate our hypothesis that integrating our MMRel with LLaVA instruction data
[1] is beneficial to other perceptual tasks, we conduct comparative
experiments using the MME benchmark [24] against the baseline LLaVA-1.5 model
[2]. Experiments in Tab. 7 show that instruction tuning with MMRel achieves
clear improvements for most of the MME tasks, verifying the scalability and
applicability of such fine-tuning in various multi-modal tasks. The evaluation
metrics of MME [24] are described in the Appendix.

Table 7: Fine-tuning LLaVA-1.5 with MMRel improves performances clearly on
MME.

Method | Position | Existence | Count | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR | Total Score
---|---|---|---|---|---|---|---|---|---|---|---
LLaVA-1.5 [2] | 114.0 | 175.7 | 124.7 | 151.0 | 127.8 | 113.6 | 148.3 | 130.0 | 102.2 | 92.0 | 1279.3
FT | 126.7 | 180.0 | 128.3 | 153.3 | 129.6 | 122.1 | 149.3 | 131.9 | 117.0 | 117.5 | 1355.7

##  5 Conclusion

We notice that Multi-Modal Large Language Models (MLLMs) encounter
difficulties in understanding inter-object relations, a research area that
remains under-explored. To address this gap, in this paper, we first provide a
clear taxonomy and define relation hallucinations for inter-object relations,
and further introduce the Multi-Modal Relation Understanding (MMRel) dataset.
MMRel is specifically designed to probe inter-object relations in MLLMs and
includes a challenging adversarial subset featuring highly unusual relations.
Thanks to its large-scale, high-quality and diverse data, MMRel serves as a
versatile tool for evaluating and fine-tuning MLLMs, enhancing their
capabilities in relation understanding and other vision-language tasks.
Extensive experiments across various popular MLLMs validate MMRel’s
effectiveness.

## Acknowledgments and Disclosure of Funding

This study is supported by the Interdisciplinary Graduate Programme, Nanyang
Technological University. We thank students from Nanyang Technological
University: Zhixiao Jin, Siwei Huang, Guanqiao Fu, Yu Wei, and Taorui Wang for
establishing the MMRel dataset.

## References

  * [1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * [3] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [4] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * [5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  arXiv preprint arXiv:2306.04387, 2023.
  * [6] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.  Multimodal-gpt: A vision and language model for dialogue with humans.  arXiv preprint arXiv:2305.04790, 2023.
  * [7] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966, 2023.
  * [8] Jusung Lee, Sungguk Cha, Younghyun Lee, and Cheoljong Yang.  Visual question answering instruction: Unlocking multimodal large language model to domain-specific visual multitasks.  arXiv preprint arXiv:2402.08360, 2024.
  * [9] Haibo Wang, Chenghang Lai, Yixuan Sun, and Weifeng Ge.  Weakly supervised gaussian contrastive grounding with large multimodal models for video question answering.  arXiv preprint arXiv:2401.10711, 2024.
  * [10] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [11] Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al.  Caption anything: Interactive image description with diverse multimodal controls.  arXiv preprint arXiv:2305.02677, 2023.
  * [12] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123:32–73, 2017.
  * [13] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin.  Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.  arXiv preprint arXiv:2207.00221, 2022.
  * [14] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky.  Teaching structured vision & language concepts to vision & language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657–2668, 2023.
  * [15] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng.  A survey on hallucination in large vision-language models.  arXiv preprint arXiv:2402.00253, 2024.
  * [16] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou.  Hallucination of multimodal large language models: A survey.  arXiv preprint arXiv:2404.18930, 2024.
  * [17] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang.  An llm-free multi-dimensional benchmark for mllms hallucination evaluation.  arXiv preprint arXiv:2311.07397, 2023.
  * [18] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented rlhf.  arXiv preprint arXiv:2309.14525, 2023.
  * [19] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu.  Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding.  arXiv preprint arXiv:2312.00081, 2023.
  * [20] Anisha Gunjal, Jihan Yin, and Erhan Bas.  Detecting and preventing hallucinations in large vision language models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143, 2024.
  * [21] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang.  Mitigating hallucination in visual language models with visual supervision.  arXiv preprint arXiv:2311.16479, 2023.
  * [22] Andrés Villa, Juan Carlos Leon Alcazar, Alvaro Soto, and Bernard Ghanem.  Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models.  arXiv preprint arXiv:2312.02219, 2023.
  * [23] Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du.  Faithscore: Evaluating hallucinations in large vision-language models.  arXiv preprint arXiv:2311.01477, 2023.
  * [24] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394, 2023.
  * [25] Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou.  Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.  arXiv preprint arXiv:2310.14566, 2023.
  * [26] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [27] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
  * [28] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [29] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355, 2023.
  * [30] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He.  Dola: Decoding by contrasting layers improves factuality in large language models.  arXiv preprint arXiv:2309.03883, 2023.
  * [31] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu.  Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation.  arXiv preprint arXiv:2311.17911, 2023.
  * [32] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing.  Mitigating object hallucinations in large vision-language models through visual contrastive decoding.  arXiv preprint arXiv:2311.16922, 2023.
  * [33] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei.  Image retrieval using scene graphs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678, 2015.
  * [34] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3716–3725, 2020.
  * [35] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [36] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [38] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In The Eleventh International Conference on Learning Representations, 2022.
  * [39] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [40] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021.
  * [41] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang.  Mmicl: Empowering vision-language model with multi-modal in-context learning.  arXiv preprint arXiv:2309.07915, 2023.
  * [42] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  In The Twelfth International Conference on Learning Representations, 2023.
  * [43] O Blog.  Introducing chatgpt.  Internet: https://openai. com/blog/chatgpt, 2022.
  * [44] Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al.  Coig-cqia: Quality is all you need for chinese instruction fine-tuning.  arXiv preprint arXiv:2403.18058, 2024.
  * [45] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [46] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [47] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  arXiv preprint arXiv:1809.02156, 2018.
  * [48] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al.  Evaluation and analysis of hallucination in large vision-language models.  arXiv preprint arXiv:2308.15126, 2023.
  * [49] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
  * [50] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [51] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [52] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.

## Appendix

##  Appendix A Limitations and Future Work

Despite conducting extensive experiments to verify the effectiveness of MMRel,
our fine-tuning efforts are currently limited to using LLaVA-1.5 [2] as the
base model. Besides, we only adopt Yes/No evaluation just like POPE [29].
Future work should consider incorporating generative evaluation metrics, such
as those used in CHAIR [47], to assess object hallucinations more
comprehensively. Furthermore, the dataset for comparative relations remains
relatively limited, thus, there is a need to collect and annotate more data
within this subset. Lastly, the taxonomy of inter-object relations could be
expanded to include additional categories.

##  Appendix B MMRel Dataset

Sample Images. We specifically aim to generate images featuring unusual inter-
object relations in multiple styles using DALL-E-3 [27]. Additional examples
of synthetic images from Dall-E-3 [27] are shown in Fig. A1.

Figure A1: Illustration of sample images generated by Dall-E. These images
contain unusual relations with different styles.

SemiDC via GPT-4V [28]. We employ an in-context learning paradigm to label
images in the Visual Genome (VG) dataset [12] using GPT-4V [28]. The specific
prompts used are illustrated in Fig. A2

Figure A2: In-context learning prompts for labeling images from VG [12].

SemiDC via Dall-E-3 [27]. We use the following prompts as inputs for DALL-E 3
[27] to generate images: a style photo depicting a strange or unusual object-
object relation: inter-object relations. Here, style specifies the image
style, selected from photo-realistic, watercolor, abstract, and oil painting,
and inter-object relations describes the specific relation we aim to depict.
For example, the prompt for generating the first image in Fig. A1 is: an
abstract photo depicting a strange or unusual object-object relation: cat on
toilet.

##  Appendix C Experiments

Data for Fine-tuning LLaVA-1.5 [2]. We integrate the training set of MMRel
with 260K original instances from the total 665K dataset utilized for LLaVA
[2] instruction tuning to maintain the versatility of MLLMs and prevent
overfitting to our specialized training data. Specifically, this 260K subset
includes question-answer instructions comprising 73K data points from COCO
[50], 53K from VG [12], 72K from GQA [51], 22K from TextVQA [52], and 41K
text-only data.

Question-answer Instructions. The original annotations are in the form of
triplets, such as “cat on car”. We expand these into both discriminative and
generative question-answer pairs.
The discriminative instructions are as follows:
Positive:
Q: Is there a cat on a car in the image? Please answer with one word.
A: Yes.
Negative:
Q: Is there a car on a cat in the image? Please answer with one word.
A: No.
The generative instructions are as follows:
Q: What is the spatial relation between a cat and a car in the image?
A: A cat is on a car.

Evaluation on MME [24]. MME [24] employs a Yes/No evaluation framework.
Adhering to the original settings, we calculate the final score using the sum
of accuracy and accuracy+, where accuracy is determined for each individual
question and accuracy+ is assessed for each image, requiring correct answers
to both associated questions. Accuracy+ serves as a stricter metric, offering
a more comprehensive reflection of the MLLMs’ capabilities.