capbtabboxtable[][] 00footnotetext:  $\dagger$ Correspondence:
<EMAIL_ADDRESS>
# Comparison Visual Instruction Tuning

Wei Lin†1 Muhammad Jehanzeb Mirza2 Sivan Doveh3,4 Rogerio Feris7 Raja Giryes5
Sepp Hochreiter1,6 Leonid Karlinsky7
1ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz, Austria
2TU Graz ICG, Austria 3IBM Research, Israel 4Weizmann Institute of Science,
Israel
5Tel-Aviv University, Israel 6NXAI GmbH, Austria 7MIT-IBM Watson AI Lab, USA

Project Page: https://wlin-at.github.io/cad_vi
Dataset Repo: https://huggingface.co/datasets/wlin21at/CaD-Inst

###### Abstract

Comparing two images in terms of Commonalities and Differences (CaD ) is a
fundamental human capability that forms the basis of advanced visual reasoning
and interpretation. It is essential for the generation of detailed and
contextually relevant descriptions, performing comparative analysis, novelty
detection, and making informed decisions based on visual data. However,
surprisingly, little attention has been given to these fundamental concepts in
the best current mimic of human visual intelligence - Large Multimodal Models
(LMMs). We develop and contribute a new two-phase approach CaD-VI for
collecting synthetic visual instructions, together with an instruction-
following dataset CaD-Inst containing 349K image pairs with CaD instructions
collected using CaD-VI . Our approach significantly improves the CaD spotting
capabilities in LMMs, advancing the SOTA on a diverse set of related tasks by
up to 17.5%. It is also complementary to existing difference-only instruction
datasets, allowing automatic targeted refinement of those resources increasing
their effectiveness for CaD tuning by up to 10%. Additionally, we propose an
evaluation benchmark with 7.5K open-ended QAs to assess the CaD understanding
abilities of LMMs.

##  1 Introduction

Figure 1:  CaD-VI concept. We collect and pair densely captioned source images
to form synthetic CaD instructions using an LLM. The resulting synthetic CaD
Visual Instruction dataset is used to train the first CaD enabled LMM that is
in turn used in iterative self-refinement by annotating new paired images from
additional sources using the CaD LMM, and re-training the model with a growing
and more comprehensive CaD-Inst dataset (contributed in this work).  Figure 2:
Pipeline of our two-phase CaD-VI : In Phase-1, we leverage captions for image
pairs and an LLM to generate CaD VI data - CaD-InstV1 (278K), and perform
visual instruction tuning on it to arrive at the Phase-1 model CaD-LLaVAV1 .
In Phase-2, we leverage CaD-LLaVAV1 to generate CaD VI data on additional
image pairs and collect CaD-InstV2 (71K). Visual instruction tuning with CaD-
InstV1 and CaD-InstV2 leads to our final model CaD-LLaVAV2 .

Understanding the Commonalities and Differences (CaD ) between two signals
(e.g., images) is a basic capability innate to humans [1]. Spotting change and
difference alerts us to interesting events happening in our surroundings,
warns us of hazard, and drives us toward learning new concepts exposed after
the change or relative movement. Understanding what is common helps structure
visual information and allows differences to emerge by elimination. Together,
these form powerful tools for human learning and acquiring world knowledge.

The forefront of modern AI shifted with the recent emergence of foundation
Large Language Models (LLMs) [2], where the top-performing ones [3, 4, 5, 6]
closely align to human reasoning and world-knowledge capabilities. LLMs’ great
performance and wide applicability quickly led to their wide adoption into
most of the current ML pipelines. In the Vision community, this impacted the
development of Large Multi-modal Models (LMMs) [7, 8, 4, 9, 10, 11, 12]
largely considered the best available mimic of human visual intelligence to
date. While multiple methods for adding multi-modal support to LLMs have been
proposed, currently the more popular and better performing open LMMs largely
rely on tuning using Visual Instructions (VI) [7, 13]. These methods align
image tokens produced by visual encoders to be ‘understandable’ by an LLM
decoder, allowing images to be seamlessly integrated into the LLM decoder
input context stream together with the query text during inference. In most
recent methods [7, 9, 10, 11], VI takes the form of a multi-turn conversation:
with ‘human’ turns providing image context and asking the questions, and LMM
turns answering them [7]. However, the majority of VI data focused on
providing merely a single image in the VI conversations [7], while only a few
works included multi-image VI samples [12, 14], and surprisingly, very few
included some form of CaD VI data [9, 10, 15] to enable CaD support in the
resulting LMM.

Due to the fundamental importance of endowing LMMs with CaD capabilities, thus
getting them closer to achieving human visual intelligence in all its
diversity, we propose CaD-VI - a multi-phase CaD generation approach, for
progressive dense and structured CaD VI data collection (concept shown in Fig.
1), which we employ to build CaD-Inst training curriculum and associated CaD-
QA benchmark comprised of CaD-related open-ended questions, both contributed
in this work. In essence, the final CaD-Inst curriculum associates diverse and
large-scale (349K) image pair collection with highly detailed and structured
CaD summaries. CaD summaries computed for an additional set of 7.6K image
pairs, are used for extracting open CaD-related QA resulting in CaD-QA .

As shown in Fig. 2, the Phase-1 of CaD-VI is a ‘cold start’ where, in the
absence of LMMs with substantial CaD capabilities, we leverage image captions
and an LLM to hallucinate (coarse) CaD VI data - CaD-InstV1 (278K), where we
collect structured and detailed CaD summaries for our paired images sourced
from a dense & large-scale image collection [16]. Training on the first phase
CaD-InstV1 data we arrive at CaD-LLaVAV1 \- an LMM that has strong CaD
capabilities compared to a large variety of leading LMMs including the very
few trained with some CaD data (see Sec. 5). Next, leveraging our CaD-LLaVAV1
model to produce non-hallucinated, image-informed CaD data, we generate
additional CaD instructions into the collection CaD-InstV2 (71K). Combining
CaD-InstV1 and CaD-InstV2 we form CaD-Inst and train our final CaD-LLaVAV2 7B
and 13B LMMs to achieve (1) significant (up to 17.5%) absolute improvement
over a large variety of recent SOTA LMMs over a variety of 5 CaD-related
existing closed-QA evaluation benchmarks (namely BISON[17], SVO Probes[18],
NLVR2[19], EQBEN[20], and COLA[21]), and (2) strong (up to over 20%) relative
improvements on our contributed open-QA CaD benchmark - CaD-QA . Additionally,
as CaD-Inst can be safely mixed with the LLaVA VI data [22], we show in Tab. 3
that our CaD-LLaVAV2 models effectively avoid forgetting the general
capabilities of the corresponding LLaVA LMMs.

Our contributions are as follows: (i) we contribute CaD-Inst - a large-scale
visual instruction tuning dataset for enhancing CaD reasoning capabilities of
LMMs; (ii) we contribute CaD-QA - an open QA evaluation benchmark for
assessing CaD capabilities; (iii) we contribute and open source a CaD-VI
methodology for collecting and enhancing CaD instruction tuning data; (iv) we
demonstrate significant (up to 17.5%) improvements in CaD reasoning for LMMs
trained using CaD-Inst as well as potential to scale CaD-Inst via self-
improvement by CaD-Inst -trained models.

##  2 Related Work

Large Multimodal Models. LMMs have shown significant advancements in
integrating visual and textual data, enhancing the ability of deep neural
networks to understand and generate multimodal content. BLIP-2 employs a
bootstrapping approach that leverages frozen image encoders and large language
models through a querying transformer, achieving remarkable results on various
vision-language tasks with fewer parameters compared to previous models [23].
Similarly, MiniGPT-4 [24] and LLaMA-Adapters [25] utilize pretrained visual
and language models, with adapters aligning image tokens to language tokens,
improving the efficiency and performance of multimodal understanding and
generation. In addition to these early models, the LLaVA series [7], including
LLaVA 1.5 [22] and LLaVA 1.6 [26], have enhanced visual instruction tuning,
enabling better handling of single-image inputs and more accurate multimodal
outputs. The InternLM XComposer 2.0 VL [27], EMU2 [28], Otter [10],
SparklesChat [9], and MMICL [29] extend these capabilities by incorporating
multiple images as input, thereby enriching the models’ understanding and
generation of text based on complex visual scenes. These models showcase the
evolution from single-image to multi-image inputs, highlighting the progress
in multimodal learning architectures and applications.

Visual Instruction Tuning Datasets. The success of LMMs builds on the
collection of high-quality visual instruction tuning data, either constructed
from existing VQA datasets [30, 31, 32, 33, 34], curated image-text pairs [24]
and LLM-generated instruction-following data with input of rich human
annotations [7, 22, 35, 36, 15]. However, the collection of multimodal data
for learning commonalities and differences between two images is still under-
explored.

Image Commonalities and Differences. Only a few datasets contain difference-
only related annotation [37, 15]. Spot-the-diff [38] collects human-annotated
short change descriptions for surveillance video frames. Our CaD-InstV1 data
collection is partially inspired by the differences-only data collection done
by [15] as a small part of their VI strategy. However, different from [15] we:
(i) collect both differences and commonalities (compared to only differences
in [15]); (ii) we leverage a significantly more dense caption-source of [16]
compared to [39] used in [15]; (iii) we are structuring our differences in CaD
according to 6 axes (whichever applicable on case basis) - object types,
attributes, counting, actions, locations, and relative positioning, also
explicitly asking the LLM to extract (from the dense captions) information
along these axes, while [15] produced unstructured difference description
text; (iv) unlike [15] we are not relying on the existence of manually
collected object bounding boxes; (v) the scale of our data is approx. 4 times
larger than of [15]. Due to these differences, as evident from the direct
comparison in Tab. 5, training the same model on CaD-InstV1 has significant
performance advantages over training on CaD instructions of [15]. To
summarize, our work focuses on CaD understanding, largely neglected by the
visual instruction tuning community. We propose a new CaD-VI approach for
collecting synthetic visual instructions and enhancing the CaD analysis
capabilities in LMMs. CaD-VI not only advances the state-of-the-art in related
tasks by significant margins but also complements existing datasets [37, 15]
by enabling their automatic targeted refinement, thereby improving their
effectiveness for CaD tuning.

##  3 CaD-VI - Two-Phase CaD Visual Instruction Tuning

As illustrated in Fig. 2, our CaD-VI consists of two phases: in Phase-1, we
employ an LLM to generate summary of CaD for image pairs (Sec. 3.1) and
perform visual instruction tuning on the collected data (Sec. 3.2); in
Phase-2, we leverage the Phase-1 model to generate CaD on additional image
pairs and perform training with combined instruction data from both phases
(Sec. 3.3).

###  3.1 Phase-1a: LLM Instruction Data Collection - CaD-InstV1

In our first phase, we leverage an LLM to generate a summary of commonalities
and differences for a pair of two images, as shown in Fig. 2 (top row).
Specifically, we construct image pairs and prompt an LLM, supplying it with
two image captions (one per image) and an instruction prompt asking it to
summarize all the commonalities and differences according to the provided
captions, contributing to our first phase CaD instruction data collection
denoted as CaD-InstV1 .

Image Source. We select the Localized Narratives dataset [16] which consists
of 873K image-caption pairs with diverse samples sourced from COCO [40, 39],
Flickr30K [41], ADE20K [42] and Open Images [43]. The captions are generated
by transcription from spoken descriptions of the image content, which are
quite dense, detailed, and descriptive with an average length of 36.5 words.
To cover comprehensive visual contents and increase the diversity in terms of
commonalities and differences, we collect 278K image pairs with different
levels of similarity between their captions. We compute similarity by counting
the number of overlapping nouns in the corresponding captions.

Figure 3:  (a) Distribution of characteristics (first two words) in the CaD
summary collected in CaD-InstV1 ; (b) Distribution of question types (first
five words) in the evaluation benchmark CaD-QA ; (c) Axis counts in CaD
summaries; (d) Two-turn conversation template.

LLM Data Generation. Inspired by LLaVA [7] who used an LLM for single images
visual instruction collection, we leverage the Mixtral 8$\times$7B open LLM
[44] for generating detailed and structured summaries of commonalities and
differences for pairs of images. As the LLM can only accept text as input, in
Phase 1 we use image captions to represent visual content of images. This is a
rather crude approximation, which is alleviated in Phase 2 of our CaD-VI
approach. To encourage the diverse and creative generation of commonalities
and differences, we do not provide in-context examples of expected output in
the prompt to the LLM. Furthermore, we specifically prompt the LLM to
structure the commonalities and differences summaries according to the
following 6 visual aspects: (i) object types; (ii) attributes; (iii) counts;
(iv) actions; (v) locations; and (vi) relative positions; as illustrated in
Fig. 2. We provide detailed prompts in the Supplementary. Importantly, LLM is
not forced to produce all 6 aspects in every summary; they are generated
adaptively according to the available content.

Generated Data Statistics. In CaD-InstV1 we collected structured summaries of
CaD for 278K image pairs, with average length of 157 words (40 for
commonalities and 117 for differences). The summaries are structured according
to 6 axes, appearing unevenly on a case-to-case basis based on the LLM
decision. We illustrate the distribution of data characteristics in Fig. 3(a),
and the total observed axis counts in Fig. 3(c). More statistics and details
are provided in the Supplementary.

CaD visual instructions data. We construct a two-turn conversation for each
image pair. In the first turn, we define the task of summarizing CaD by
providing the encoded visual tokens of the two images and instructing the
model to summarize the CaD , where the response part of the turn is the LLM-
generated structured summary collected above. In this instruction, we do not
provide the image captions, forcing the model to rely only on image tokens to
complete the task. In the second turn, we reinforce the image-text alignment
by employing a simple task of text-to-image retrieval to avoid forgetting the
model’s general capabilities. We randomly sample one of the two captions and
request the model to select the image (from the current pair) to which the
caption belongs. The template for the two-turn conversation is illustrated in
Fig. 3(d).

###  3.2 Phase-1b: CaD Visual Instruction Tuning

Architecture. As illustrated in Fig. 2, we use our collected CaD-InstV1 data
to perform visual instruction tuning using the open-sourced code of LLaVA-1.5
[22] LMM. The LLaVA-1.5 model consists of $\phi_{L}(\cdot;\theta_{L})$ \- a
pretrained Vicuna 1.5 [45] LLM (finetuned from LLama 2 [46]);
$\phi_{V}(\cdot;\theta_{V})$ \- a pretrained visual encoder CLIP
ViT-L/14@336px [47]; and $\phi_{M}(\cdot;\theta_{M})$ \- a two-layer MLP
projector converting the visual encoder tokens to post-embedding layer LLM
tokens.

Given a pair of two images $x_{V_{1}}$, $x_{V_{2}}$ and the instruction
$x_{I}$, the MLP projects the visual features computed by the visual encoder
into embedded language tokens, _i.e_.
$v_{k}=\phi_{M}(\phi_{V}(x_{V_{k}};\theta_{V});\theta_{M}),k\in\\{1,2\\}$.
Then the projected visual features and instruction text tokens are
concatenated and fed into the LLM, where the response text tokens are
generated in an autoregressive manner, _i.e_.

$\hat{x}^{i}_{R}=\phi_{L}([v_{1},v_{2},x_{I},\hat{x}^{<i}_{R}];\theta_{L}),$
(1)

where $\hat{x}^{i}_{R}$ denotes the $i$-th token in the generated response.

Training. We finetune the LLaVA-1.5 model using the LLaVA [7] pipeline.
Specifically, following LLaVA pre-training, we finetune only the pretrained
projection MLP and the (frozen) LLM with LoRA adapters [48]. We minimize the
CLM loss of the next token prediction in the responses:

$\mathcal{L}_{CLM}=\sum_{i}-\log
p({\hat{x}}^{i}_{R}|{V_{1}},{V_{2}},{x_{I}},{x^{<i}_{R}})$ (2)

To preserve the general VL capabilities of the LMM, we merge our CaD-InstV1
with the finetuning data of LLaVA-1.5 (665K samples). In Tab. 3 we show that
CaD-VI indeed preserves the general LMM capabilities compared to LLaVA-1.5 as
evaluated on the popular SEED benchmark [49]. The Phase-1 CaD visual
instruction tuning results in our cold-start model CaD-LLaVAV1 which is an LMM
that can be leveraged for annotating visual commonalities and differences.

###  3.3 Phase-2: Data Collection and Visual Instruction Tuning

Phase-2a: LMM-based CaD Instruction Collection. While in Phase 1 we used an
LLM to extract a CaD summary based on human-generated captions, for Phase 2
data collection we leverage our Phase 1 model CaD-LLaVAV1 and additional image
pairs to extract the CaD summaries informed by the images directly. Here we
select the Scene-Difference [15] collection as an additional image source. It
contains 71K pairs of similar images from COCO [40] and provides annotation of
unstructured difference-only summaries (see Fig. 2 bottom left for an
example). We feed both the image pairs and the original annotations into our
CaD-LLaVAV1 model, and generate a structured summary of both commonalities and
differences. The exact prompt is provided in the Supplementary. This leads to
our phase-2 CaD instruction data - CaD-InstV2 . As shown in Tab. 5, our
collected CaD instructions significantly improve over the utility of the
original [15] annotations. As part of our analysis in Tab. 5 and 6, and
additional experiments provided in Supplementary, we also show that similarly
out-of-distribution image pair collections or even unlabeled image pair
collections can be effectively leveraged for our Phase-2.

Phase-2b CaD Visual Instruction Tuning We follow the Phase-1b introduced in
Sec. 3.2 for CaD visual instruction tuning. Here we finetune on a combination
of LLaVA 1.5 [22] finetune data (665K), CaD-InstV1 data (278K) and CaD-InstV2
data (71K). This phase of CaD visual instruction tuning leads to the Phase 2
model, denoted as CaD-LLaVAV2 .

##  4 CaD-QA - Benchmark of Open-Ended CaD QA

In order to evaluate the capability of LMMs on answering open-ended questions
regarding commonalities and differences of a pair of two images, we construct
and contribute the CaD-QA benchmark.

Data Collection. Similar to the data collection pipeline introduced in Sec.
3.1, we employ Visual Genome [50] and the detailed image captions from SVIT
[36] as image & caption source. We collect 7.5K image pairs with 8 or more
overlapping nouns in their captions. For each pair, we employ the Mixtral
8$\times$7B LLM to produce the structured CaD summaries from the captions.
Next, we prompt Mixtral with both the image captions and the CaD summary,
instructing it to generate a multi-turn conversation with several rounds of
Q&A, providing some in-context examples of the desired layout (see
Supplementary for the prompt). Finally, we randomly select one Q&A per
conversation.

Benchmark Statistics. There are 7520 QA pairs with an average answer length of
26 words. Among these, we also include 2916 questions asking about the content
of only one of the two images. It requires the precise attention of the LMM on
the corresponding image to correctly answer these questions. Our CaD-QA covers
diverse question types as illustrated in Fig. 3(b).

LLM-assisted Evaluation. Motivated by LLMs’ ability to judge response quality
consistently with human assessment [45], we employ the Mixtral 8$\times$7B LLM
to compare the generated responses to the collected open-ended QA responses.
We feed the question, correct answer, and the predicted answer into the LLM
and instruct it to provide a rating between 0 and 5 for the predicted answer
quality (where higher score indicates a better prediction). We provide the
prompt in the Supplementary.

Dataset |  # Instruction Data |  BISON |  SVO |  NLVR2 |  EQBEN |  COLA
---|---|---|---|---|---|---
Random chance |  |  50% |  50% |  50% |  25% |  25%
SparklesChat |  6.5K |  56.70% |  43.93% |  58.00% |  19.17% |  20.00%
Otter |  2.8M |  40.67% |  47.33% |  52.00% |  08.33% |  08.10%
MMICL |  5.8M |  80.00% |  88.13% |  56.67% |  20.83% |  25.71%
EMU2-Chat |  1.3M |  46.00% |  47.93% |  60.00% |  07.50% |  13.33%
InternLM-XComposer2-VL |  >600K |  80.67% |  82.07% |  66.67% |  25.00% |  32.38%
LLaVA 1.6 7B |  <1M |  66.00% |  70.40% |  58.67% |  20.83% |  11.90%
LLaVA 1.6 13B |  <1M |  81.33% |  82.13% |  60.00% |  17.50% |  24.76%
LLaVA 1.5 7B |  665K |  54.00% |  46.80% |  61.33% |  17.50% |  07.62%
LLaVA 1.5 13B |  665K |  59.33% |  56.27% |  66.00% |  16.67% |  12.38%
CaD-VI 7B |  1M |  95.33% |  92.73% |  66.67% |  39.17% |  40.95%
CaD-VI 13B |  1M |  96.67% |  93.00% |  69.33% |  42.50% |  43.33%
Table 1: Performance on closed-ended VQA tasks with image pairs in accuracy.
Here the method CaD-VI denotes our Phase-2 model CaD-LLaVAV2 .

##  5 Experiments

Evaluation Datasets We evaluate on several VQA benchmarks of closed-ended and
open-ended questions. For closed-ended VQA on image pairs, we include BISON
[17] and SVO Probes [18] both consisting of samples with an image pair and a
text query that needs to be matched with one of the images in the pair (chance
is 50%). EQBEN [20] and COLA [21] contain samples composed of a pair of two
images together with the two textual descriptions. The goal is to correctly
match images with corresponding texts (chance is 25%). Furthermore, we
evaluate on NLVR2 [19] which comprises samples of a pair of two images and a
reasoning sentence. The task is to assess the correctness of the reasoning and
has a random chance of 50%. We also evaluate SEED-Bench Video [49] with two
frames sampled from the video to explore the generalization value of our CaD
tuning for video understanding. SEED-Bench Video contains three partitions
from SEED-Bench and has multi-choice questions on action
recognition/prediction or procedure understanding with four answer options per
question. For open-ended tasks, use the LLM-as-a-judge metric (Sec. 4). We
evaluate open-ended QAs on our CaD-QA . Furthermore, we also directly evaluate
the quality of LMM predicted CaD summaries for 210 image pairs in COLA with
shorter summaries generated from brief captions, and for the 7.5K lengthy
summaries from CaD-QA generated from detailed VG captions. More details and
statistics of the datasets are provided in the Supplementary.

Implementation Details We leverage the Mixtral 8$\times 7$B Instruct v0.1 and
set the maximum token size to 750 data collection and 20 for open-ended task
evaluation. For visual instruction tuning, we use the official implementation
of LLaVA and tune the LLaVA 1.5 7B model with LoRA. We set the batch size to
128 and LoRA learning rate for LLM and the projector is set to $1\times
10^{-4}$ and $2\times 10^{-5}$ correspondingly. All experiments are run on
4$\times$A100 80G GPUs. More details are in Supplementary.

# Input Frames | 1 | 2
---|---|---
SparklesChat | 21.81% | 19.09% ($\blacktriangledown$-2.72%)
Otter | 18.19% | 23.00% ($\blacktriangle$+4.81%)
EMU2-Chat | 43.43% | 41.09% ($\blacktriangledown$-2.34%)
InternLM-XComposer2-VL | 41.07% | 40.16% ($\blacktriangledown$-0.91%)
LLaVA 1.6 7B | 41.95% |  42.03% ($\blacktriangle$+0.08%)
LLaVA 1.6 13B | 41.85% | 41.35% ($\blacktriangledown$-0.50%)
LLaVA 1.5 7B | 37.43% | 36.68% ($\blacktriangledown$-0.75%)
LLaVA 1.5 13B | 40.12% | 38.78% ($\blacktriangledown$-1.34%)
CaD-VI 7B | 38.40% | 40.44% ($\blacktriangle$+2.04%)
CaD-VI 13B | 40.16% |  43.09% ($\blacktriangle$+2.93%)

Table 2: Performance on SEED-Bench video partitions by feeding one or two
frames into the LMMs.

Model | SEED-Image
---|---
LLaVA 1.5 7B | 67.34%
CaD-VI 7B | 67.48%
LLaVA 1.5 13B | 68.83%
CaD-VI 13B | 69.11%

Table 3: Performance on SEED-Bench image partitions for evaluation of general
VL capabilities with single-image input.

Comparison to State-of-the-Art LMMs

Dataset |  CaD-QA |  VG comm. |  VG diff. | COLA comm. | COLA diff.
---|---|---|---|---|---
SparklesChat |  3.01 |  2.41 |  3.12 | 1.52 | 1.22
Otter |  2.20 |  1.88 |  1.97 | 1.37 | 0.81
MMICL |  2.01 |  1.79 |  1.94 | 1.73 | 0.59
EMU2-Chat |  1.20 |  1.04 |  1.08 | 1.22 | 0.41
InternLM-XComposer2-VL |  2.90 |  2.08 |  2.69 | 1.72 | 1.36
LLaVA 1.6 7B |  3.10 |  2.23 |  2.73 | 1.71 | 1.22
LLaVA 1.6 13B |  3.19 |  2.19 |  2.69 | 1.93 | 1.01
LLaVA 1.5 7B |  2.54 |  1.79 |  1.75 | 1.44 | 1.02
LLaVA 1.5 13B |  2.65 |  2.16 |  2.41 | 1.57 | 1.10
CaD-VI 7B |  3.29 |  2.32 |  3.85 | 2.14 | 1.25
CaD-VI 13B |  3.34 |  2.58 |  3.68 | 2.13 | 1.31
Table 4:  Performance on CaD-QA and tasks of CaD summary prediction evaluated
using LLM-as-a-judge ratings (range 0 to 5). Here the method CaD-VI denotes
our Phase-2 model CaD-LLaVAV2 .

We first compare our final model CaD-LLaVAV2 (denoted by CaD-VI in Table) to
state-of-the-art LMMs on closed-ended VQA in Table 1. SparklesChat [9], Otter
[10], MMICL [29], EMU2-Chat [12], InternLM-Xcomposer2-VL [27] all include
samples with multi-image inputs in the visual instruction tuning while LLaVA
1.5 [22] and LLaVA 1.6 [26] are tuned with only single image instructions. The
evaluated benchmarks are challenging due to the visually very similar image
pairs with subtle compositional differences where the LMMs could easily make
an incorrect decision leading to performance below random chance. Our CaD-VI
7B model already outperforms all the other baselines on the five benchmarks
and our 13B finetuned model further boosts the performance.

Table 4 demonstrates the comparison to the baseline LMMs on open-ended tasks
of CaD-QA and of CaD summary prediction on image pairs. Our CaD-VI models
outperform the baselines on four of the five open-ended tasks, with the
exception of COLA difference summary where our 13B model achieves a rating
(1.31) close to the best performing InternLM-XComposer2 model (1.36).

Furthermore, we explore whether our CaD instruction tuning improves video
understanding evaluated using SEED-Bench Video in Table 2. In the evaluation
setting of LLaVA, only one frame per SEED-Bench video is passed to the LMM. To
explore the impact of our CaD tuning, we compare this to evaluating using two
frames as input. As shown in Table 2, although multiple baseline LMMs achieve
better performance in single-frame setting, our CaD-VI 13B model performs the
best in the two-frame setting with a significant performance improvement of
2.93% on top of the single-frame performance. The only higher improvement is
achieved by Otter, which however struggles below the 25% chance level
performance. This underlines that our CaD tuning improves the temporal
understanding between video frames.

Additionally, to verify that introducing multi-image CaD data into the tuning
does not lead to catastrophic forgetting of general single-image input LMM
capabilities, we also evaluate the SEED-Bench Image partitions and report the
results in Table 3. Here we directly compare to same architecture baseline of
LLaVA 1.5 fine-tuned using its single-image LLaVA mix 665K data. Table 3
demonstrates that our CaD tuning indeed preserves the competence in single-
image understanding.

|  Training Data |  BISON |  SVO |  EQBEN |  COLA |  CaD-QA
---|---|---|---|---|---|---
A: |  LLaVA mix (L) |  54.00% |  46.80% |  17.50% |  07.62% |  2.54
B: |  L + ScDiff orig. annot. |  92.67% |  90.07% |  22.50% |  33.81% |  2.90
C: |  L + ScDiff our annot. (from scratch) |  88.67% |  90.80% |  38.33% |  36.67% |  3.17
D: |  L + ScDiff our annot. (refined from orig. annot.) |  94.67% |  91.80% |  32.50% |  34.76% |  3.17
E: |  L + CaD-InstV1 |  92.00% |  92.27% |  34.17% |  36.67% |  3.27
F: |  L + CaD-InstV1 \+ ScDiff our annot. (refined from orig. annot.) |  95.33% |  92.73% |  39.17% |  40.95% |  3.29
Table 5:  Ablation of phase-2 data collection from 71K image pairs in Scene-Difference (ScDiff). We use CaD-LLaVAV1 to generate CaD on ScDiff either from scratch or by refining from the original annotation of unstructured difference-only summaries. Training settings in E and F lead to our CaD-LLaVAV1 and CaD-LLaVAV2 models correspondingly.  |  Training Data |  BISON |  SVO |  EQBEN |  COLA |  CaD-QA
---|---|---|---|---|---|---
A: |  LLaVA mix (L) |  54.00% |  46.80% |  17.50% |  07.62% |  2.54
B: |  L + A/G orig. captions only |  55.33% |  55.67% |  03.33% |  02.86% |  2.78
C: |  L + A/G our annot. (from scratch) |  90.00% |  88.53% |  40.83% |  42.86% |  3.21
D: |  L + A/G our annot. (given orig. captions) |  88.00% |  86.87% |  43.33% |  30.48% |  3.06
Table 6:  Ablation of phase-2 data collection from 66K pairs of video frames
in Action Genome and GEBC (A/G). We use CaD-LLaVAV1 to generate CaD on A/G
either from scratch or with the prior information from the original frame
captions.

##  6 Ablations

Phase-2 Data Collection analysis. Our Phase-2 data collection introduced in
Sec. 3.3 can be used to leverage image pairs from various sources for
producing effective CaD instructions. We first ablate the data collection from
the 71K image pairs in Scene-Difference [15] (ScDiff) which contains
annotation of unstructured difference-only summaries. As shown in Table 5,
training with original annotation of difference-only summaries (row B)
significantly improves on the baseline of training with LLaVA data only (row
A). Then we show that using CaD-LLaVAV1 to generate CaD instructions on ScDiff
remarkably improves further, either if used from scratch (row C) or by
refining from the original annotation (row D, also illustrated in Fig. 2
bottom row). Training with our re-annotation from scratch outperforms the
original annotation on all datasets except for BISON. Our re-annotation by
refining the original annotation leads to a more balanced performance
improvement and is used as the phase-2 instruction data CaD-InstV2 . We
combine this with our phase-1 data CaD-InstV1 and demonstrate the further
performance boost in row F of Table 5.

In order to show the robustness of CaD data collection capability using our
CaD-LLaVAV1 model, we also explore applying our phase-2 data collection to
visually similar frames from user videos in Action Genome and GEBC (A/G). In
Table 6, we first train a baseline using original frame captions only and a
simple instruction task of image description (row B), which leads to a
significant performance drop on EQBEN and COLA, and minimal improvement on
other datasets. Then we use our CaD-LLaVAV1 to generate CaD instructions on
the frame pairs either from scratch (row C) or conditioned on the frame
captions (row D). Interestingly, on most datasets CaD instructions generated
by our CaD-LLaVAV1 from scratch are found to be more effective than ones
generated using original captions conditioning, likely due to lack of detail
in these captions. This once again demonstrates that our model is effective in
generating CaD instructions on unlabeled data.

|  Training Data | BISON | SVO | CaD-QA | VG comm. | VG diff.
---|---|---|---|---|---|---
A: |  LLaVA mix (L) | 54.00% | 46.80% | 2.54 | 1.79 | 1.75
B: |  L + t2i retriev. | 58.00% | 51.33% | 2.47 | 1.58 | 1.46
C: |  L + comm. | 64.67% | 79.73% | 3.23 | 2.67 | 2.52
D: |  L + diff. | 55.33% | 72.13% | 3.24 | 1.97 | 2.89
E: |  L + comm. + diff. | 72.00% | 82.60% | 3.24 | 2.13 | 3.42
F: |  L + comm. + diff. + t2i retriev. | 92.00% | 92.27% | 3.27 | 2.21 | 3.69
G: |  F + CaD-InstV2 | 95.33% | 92.73% | 3.29 | 2.32 | 3.85
Table 7: Ablation on components in the instruction data. Training settings in
F and G lead to our CaD-LLaVAV1 and CaD-LLaVAV2 models correspondingly. Here
t2i retriev. refers to the text-to-image retrieval task (see Sec. 3.1).
Training settings in F and G lead to our CaD-LLaVAV1 and CaD-LLaVAV2 models
correspondingly.

Analysis of CaD Instruction Data Components We verify the effectiveness of the
components in our instruction data by ablating on the different combinations
of our tuning tasks, including: (i) commonality summary (comm.); (2)
difference summary (diff.); and (iii) text-to-image retrieval (t2i retriev.)
in Table 7. Training solely on the t2i retrieval task (row B) leads to minimum
performance improvement on BISON and SVO Probes, and performance degradation
on the three benchmarks of the open-ended tasks due to lacking of any CaD
learning. Training with the commonality (row C) and difference summary (row D)
tasks separately lead to a significant boost on the VG comm (2.67) and VG diff
(2.89) tasks correspondingly. Training with combinations of the three tasks
(F) boosts the performance in comparison to the case of each single component,
except for VG comm where the commonality training (row C) leads to better
results on this task. Finally, combining phase-1 and phase-2 data (row G)
leads to further performance boosts on most of the benchmarks.

##  7 Conclusions, Limitations, and Broader Impact

We are contributing CaD-VI - an effective, two-phase strategy for collecting
Commonalities and Differences (CaD ) Visual Instruction (VI) data, resulting
in the also contributed large scale CaD-Inst with 349K samples for verified
improvement of CaD and related image and text comparative capabilities of
LMMs. Additionally, we contribute CaD-QA - a benchmark of 7.6K open-ended QA
to directly evaluate CaD capabilities between pairs of images. We extensively
evaluate and validate our CaD-VI approach, showing it leads to substantial
improvements in CaD abilities and related tasks. We further show how the very
few existing CaD resources are complementary to our approach and can be
further refined automatically using our CaD-VI . We believe that our work
contributes to the important investigation and improvement of (currently
somewhat missing) CaD abilities of modern LMMs and leads to exciting future
work of CaD VI tuning.
Limitations Currently, our CaD-VI only focuses on the CaD between two images,
and we leave the extension of understanding CaD and group relations on three
or more images to future work.
Broader Impact Our CaD-VI , CaD-Inst , and CaD-QA significantly contribute to
the understanding and improvement of CaD capabilities in LMMs, and are
intended to enhance the applicability and utility of AI across various fields,
from robotics to industrial applications. However, this LMM improvement could
also lead to job displacement, as these models could increasingly automate
complex tasks traditionally performed by humans.

##  8 Acknowledgments

We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at
IT4Innovations, Czech Republic, Leonardo at CINECA, Italy, MeluXina at
LuxProvide, Luxembourg and LUMI at CSC, Finland.

## Appendix

In the appendix, we first introduce our dataset release (Sec. A) and the list
of assets (Sec. B) used in this project.

For further insights into our approach CaD-VI , we report more statistics on
our generated data (Sec. C.1), and statistics on the external evaluation
datasets (Sec. C.2). We provide more implementation details (Sec. D) including
the specifics of baseline methods, data generation, training and evaluation
details.

As additional results, we report the error bars (Sec.E.1), analyze the Phase-2
data collection on Out-Of-Distribution data (Sec. E.2). Finally, we show
qualitative results of the collected CaD summaries (Sec. E.3), and compare LMM
predictions on our CaD-QA benchmark (Sec. E.4), and LMM predictions on the
BISON dataset (Sec. E.5).

##  Appendix A Dataset Release

HuggingFace dataset repository: https://huggingface.co/datasets/wlin21at/CaD-
Inst

We upload both the instruction-following dataset CaD-Inst and the evaluation
benchmark CaD-QA to the HuggingFace dataset repository at
https://huggingface.co/datasets/wlin21at/CaD-Inst in the repository.

The dataset card with the dataset viewer, dataset description with intended
use, and instructions for loading the dataset and downloading the image
sources are available.

The dataset repository will be hosted in a long term with data access and
necessary maintenance ensured.

##  Appendix B List of Assets

Our image sources and annotations are obtained from public datasets. We
release our data in accordance to the source data licenses.

Here is a list of image sources:

  * •

Open Images v6 [43]
(https://storage.googleapis.com/openimages/web/download_v6.html): The images
are under Creative Commons Attribution (CC BY) 2.0 license.

  * •

COCO 2017 [39, 40] (https://cocodataset.org/#download): The images are under a
Creative Commons Attribution 4.0 license.

  * •

Flicker30K [41] (https://shannon.cs.illinois.edu/DenotationGraph/): The images
are the property of SmugMug or its third party licensors and are protected by
United States and international intellectual property laws. The images are
provided for researchers and educators who wish to use the dataset for non-
commercial research and/or educational purposes.

  * •

ADE20K [42]
(https://groups.csail.mit.edu/vision/datasets/ADE20K/index.html#Download): The
images belong to MIT CSAIL and are licensed under a Creative Common BSD-3
License.

  * •

Visual Genome [50]
(https://homes.cs.washington.edu/~ranjay/visualgenome/api.html): The images
are under a Creative Commons Attribution 4.0 license.

Here is a list of image annotation sources:

  * •

Localized narratives [16] (https://google.github.io/localized-narratives/):
The annotations are released under a Creative Common Attribution (CC BY) 4.0
license.

  * •

MIMIC-IT [15] (https://huggingface.co/datasets/pufanyi/MIMICIT): The
annotations are released under an MIT license.

  * •

SVIT [36] (https://huggingface.co/datasets/BAAI/SVIT): The annotations are
licensed under a Creative Commons Attribution 4.0 license. It should abide by
the policy of OpenAI (https://openai.com/policies/terms-of-use). The use of
original images and annotations from Visual Genome and MS-COCO should comply
with the original licenses.

Here is a list of implementation sources or model weights:

  * •

LLaVA [7, 22] (https://github.com/haotian-liu/LLaVA): The code is released
under an Apache-2.0 license. The project utilizes certain datasets and
checkpoints that are subject to their respective original licenses, including
but not limited to the OpenAI Terms of Use111https://openai.com/policies/eu-
terms-of-use/ for the dataset and the specific licenses for base language
models for checkpoints trained using the dataset (_e.g_. LLaMA community
license222https://ai.meta.com/llama/license/ for LLaMA-2 and Vicuna-v1.5).

  * •

Mixtral 8$\times$7B model [44]
(https://huggingface.co/mistralai/Mixtral-8x7B-v0.1): The model is released
under an Apache-2.0 license. Usage is subject to the term of use for Mistral
products and services333https://mistral.ai/terms/#terms-of-use.

##  Appendix C Dataset Statistics

###  C.1 Generated Data Statistics

Figure 4:  Distribution of length of CaD summaries (in terms of number of
words) in (a) CaD-InstV1 and (b) CaD-InstV2

CaD-InstV1 and CaD-InstV2 . In CaD-InstV1 , we collected structured summaries
of CaD for 278K image pairs, with an average length of 157 words (40 for
commonalities and 117 for differences). In CaD-InstV2 , we collected summaries
of CaD for 71K images pairs used in Scene-Difference [15], with an average
length of 156 words (28 for commonalities and 128 for differences). We
demonstrate the distribution of CaD summary length (number of words) in CaD-
InstV1 (Fig. 4(a)) and in CaD-InstV2 (Fig. 4(b)).

Figure 5:  Word clouds of CaD summaries in (a) CaD-InstV1 and (b) CaD-InstV2

In Fig. 5, we also illustrate the cloud of words covered in the CaD summaries
in CaD-InstV1 (Fig. 5(a)) and in CaD-InstV2 (Fig. 5(b)).

Figure 6:  Word cloud of sample-specific characteristics in CaD summaries in
CaD-InstV1 . The distribution of these sample-specific characteristics is also
shown in a Sunburst chart in Fig. 3(a)(main paper).

In the main paper, we mentioned that the collected summaries are structured
according to approximate 6 axes of characteristics: object types, attributes,
counting, actions, locations and relative positions. Note that the
characteristics appear unevenly on a case-to-case basis based on the LLM
decision on individual samples. In Fig. 3(a)(main paper), we illustrate the
distribution of these sample-specific characteristics in a Sunburst chart.
Here in Fig. 6, we also illustrate the cloud of words in these characteristics
in CaD summaries in the Phase-1 data collection CaD-InstV1 .

Figure 7:  Distribution of (a) number of overlapping nouns between captions in
an image pair and (b) image-image similarities in the 278K image pairs in CaD-
InstV1

In the main paper, we introduced that we collect 278K image pairs with
different levels of similarity between their captions. We measure the
similarity between two captions by counting the number of overlapping nouns in
the corresponding captions. Here we show the distribution of the number of
overlapping nouns in Fig. 7(a). We see that we cover image pairs with
different levels of caption-caption similarity. Furthermore, we use the CLIP
ViT-B/32 model [47] to compute the similarity scores between the two images in
each pair and report the distribution in Fig. 7(b). We verify that image pairs
of diverse similarity levels are covered in our Phase-1 data collection CaD-
InstV1 .

Figure 8:  Distribution of (a) questions (first 5 words) and (b) answers
(first 3 words) in the evaluation benchmark CaD-QA .

CaD-QA . Our CaD-QA benchmark contains 7.5K open-ended questions with answers.
Here we show the distribution of questions types (first 5 words) and answer
types (first 3 words) in Sunburst charts in Fig. 8. There are diverse question
categories covered such as Yes/No questions, What questions on scene
characteristics such as objects, attributes and setting, and also requests to
describe specific characteristics in details.

###  C.2 Statistics of External Evaluation Datasets

We evaluate on several external VQA benchmarks of closed-ended and open-ended
questions. Here we give a brief introduction on the contents and statistics.

BISON is a dataset for the binary image selection task [17]. There are 150
samples in the evaluation benchmark, each sample consisting of a pair of two
visually similar images and a query caption. Only one image correctly matches
with the query caption. It measures the ability of the LMMs to relate fine-
grained text content in the caption to visual content in the images.

SVO Probes is a benchmark designed to probe for subject, verb and object
understanding in vision-language models [18]. In the benchmark, each sample
consists of a pair of two images and a query sentence, where only one image
correctly matches with the query sentence. The negative image differs from the
positive image with regard to either the subject, the verb or the object.
There are 36.8K samples in the dataset. For efficient evaluation, we randomly
select 1500 samples that can be divided into 3 partitions subject, verb and
object where each partition has 500 samples with the image pair contradiction
in either subject, verb or object.

EQBEN is a benchmark that focuses on visual minimal change between two images
[20]. Each sample in the benchmark consists of a pair of two images with
subtle visual changes and two corresponding captions. The dataset is comprised
of frames from natural video datasets such as YouCook2 [51], Action Genome
[52] and GEBC [53], as well as sythetic image pairs with subtle differences
generated by the photo-realistic scene generator Kubric [54] and the diffusion
model Stable-Diffusion [55]. We employ an EQBEN subset444https://entuedu-
my.sharepoint.com/:u:/g/personal/tan317_e_ntu_edu_sg/ETkpKSsmun1MpBw7FqfUUS8BwTX2gKkTQkDFsfOGCw-9yA?e=KGtpg0
which is released by the authors in [20] for evaluating the performance of
LMMs specifically. The subset consists of 120 samples, comprised of frame
pairs from Action Genome and GEBC, image pairs with changes in attributes,
count and location generated by Kubric, and image pairs with style change
generated by Stable-Diffusion. For each sample, we perform the binary image
selection task twice, feeding one of the descriptions for image selection at a
time. The sample is considered positively answered only when both selection
tasks are correctly solved.

COLA is a benchmark for evaluating the capabilities of vision-language models
on representing simple compositions by combing objects with their attributes
[21]. Each sample in the benchmark consists of two images with two
corresponding captions. The two images have attributes and objects that are
swapped in the captions, _e.g_. large tree to the right of little short green
tree, and tall green tree to the right of large tall green tree. We employ the
partition of multi-object setting in the benchmark which consists of 210 image
pairs and captions. Similar to evaluation on EQBEN, we perform the binary
image selection task twice for each sample.

NLVR2 is a benchmark for evaluation of the visual reasoning with natural
language task which aesses the ability of LMMs to predict whether a sentence
is true about a pair of images [19]. The task focuses on understanding of
compositionalities in terms of relations, comparisons and counting. We use the
subset of 150 samples provided in SparklesChat [9] for a fair comparison.

SEED-Bench is an evaluation benchmark on comprehensive vision-language
understanding, consisting of 19K multiple choice questions [49]. The are two
major categories in the benchmark: SEED-Image with 14K samples and SEED-Video
with 5K samples. SEED-Image consists of 9 dimensions: scene understanding,
instance identity, instance attributes, instance location, instance counting,
spatial relation, visual reasoning and text understanding. All samples contain
only a single input image. SEED-Video consists of 3 dimensions: action
recognition, action prediction and procedure understanding. The videos are
from Something-Something-v2 [56], EPIC-Kitchen [57] and Breakfast [58].

##  Appendix D Implementation Details

###  D.1 Baselines

SparklesChat [9] is finetuned from the first-stage pretrained model of
MiniGPT4 [24]. The model is finetuned with their collected multi-image
dialogue data. SparklesChat follows the architecture of MiniGPT4 and uses
Vicuna 7B [59], EVA-CLIP ViT-G/14 [60] with a Q-Former from BLIP-2 [23]. We
use the model weights and instruction templates available at
https://github.com/HYPJUDY/Sparkles.

Otter [10] is finetuned from the OpenFlamingo model [14] with the collected
multimodal in-context instruction-response data in MIMIC-IT [15]. We use their
most recent open-sourced version Otter-Image-LLaMA7B-LA-InContext available at
https://huggingface.co/luodian/OTTER-Image-LLaMA7B-LA-InContext.

MMICL [29] is based on the InstructBLIP model [33]. The model is finetuned
their own collected multimodal in-context learning datast consisting of
interleaved text-image inputs, inter-related multiple image inputs and
multimodal in-context learning inputs. We evaluate with their model of the
largest scale MMICL-InstructBLIP-T5-XXL, available at
https://huggingface.co/BleachNick/MMICL-Instructblip-T5-xxl.

EMU2-Chat [28] is a generative multimodal model trained on large-scale
multimodal sequences. The model consists of pretrained EVA-02-CLIP-E-plus [61]
and LLaMA-33B [62]. The model weights and inference code are available at
https://huggingface.co/BAAI/Emu2-Chat.

InternLM-XComposer2-VL [27] consists of CLIP ViT-L [47] and InternLM2-7B [63].
The model weights of the InternLM-XComposer2-VL-7B and inference code are
available at https://huggingface.co/internlm/internlm-xcomposer2-vl-7b.

LLaVA 1.5 [22] is an improved version from LLaVA [7] with CLIP-ViT-L-336px
[47] as the visual backbone and Vicuna 1.5 [45] as the LLM. Our visual
instruction tuning is performed using the open-sourced code of LLaVA 1.5. We
train on the first-stage pretrained weights of LLaVA 1.5 via LoRA finetuning.
We evaluate both LLaVA 1.5 7B lora and LLaVA 1.5 13B lora as baselines. The
models are available at https://huggingface.co/liuhaotian/llava-v1.5-7b-lora
and https://huggingface.co/liuhaotian/llava-v1.5-13b-lora.

LLaVA 1.6 [26] is an improved version from LLaVA 1.5 with increased input
image resolution and improved mixture of instruction tuning data. The 7B and
13B versions are avaible on Huggingface at
https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b and
https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b. However, the training
code is not yet available.

###  D.2 Implementation Details

Figure 9:  Prompt for the task of Phase-1 LLM-based CaD summary.

Data Collection. In Phase-1, we leverage the Mixtral 8x7B Instruct v0.1
model555Huggingface source:
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 with 8-bit
inference for data generation. We set the batch size to 16 and max new token
to 750. The prompt for the task of LLM-based CaD summary is given in Fig. 9.
The generation with batch 16 fits to an A100 80G GPU.

Figure 10:  Prompt for the task of Phase-2 LMM-based CaD summary.

In Phase-2, we leverage the Phase-1 model CaD-LLaVAV1 13B model to generate
CaD summary on additional image pairs. The temporature, max new tokens and
number of beams are set to 0, 256 and 1. The prompt for the task of LMM-based
CaD summary is given in Fig. 10.

Figure 11:  Prompt for the task of generating Q&A pairs based on both image
captions and the CaD summary.

For collecting open-ended QAs in CaD-QA , we first use the LMM to generate the
CaD summaries based on the image captions (see Fig. 9). Then we prompt the LLM
with both the image captions and the CaD summary, instructing it to generate a
multi-turn conversation with several rounds of Q&A. We also provide some in-
context samples to demonstrate the desired layout. The prompt for the task of
generating Q&A pairs based on both image captions and the CaD summary is
illustrated in Fig. 11.

Training. We perform visual instruction tuning following the configuration in
LLaVA 1.5. We set the batch size to 128 and train for one epoch. The learning
rate for LLM with LoRA and for the projector are set to $1\times 10^{-4}$ and
$2\times 10^{-5}$ correspondingly. The LoRA rank and alpha values are set to
128 and 256. The training experiments are run on 4$\times$A100 80G GPUs.

Inference. For VQA inference, the temperature, max new tokens and number of
beams are set to 0, 256 and 1.

Figure 12:  Prompt for the LLM-assisted evaluation.

LLM-assisted Evaluation We leverage the Mixtral 8$\times$7B model for LLM-
assisted evaluation on open-ended questions. We feed the question, correct
answer and the predicted answer into the LLM and instruct it to provide a
rating between 0 and 5. The prompt for generating the evaluation rating is
given in Fig. 12.

##  Appendix E Additional Results

###  E.1 Error Bars

Training Data | BISON | SVO | EQBEN | COLA | CaD-QA
---|---|---|---|---|---
LLaVA mix + CaD-LLaVAV1 | 91.78% $\pm$ 1.02% | 92.33% $\pm$ 0.57% | 33.06% $\pm$ 0.96% | 34.64% $\pm$ 2.09% | 3.270 $\pm$ 0.002
Table 8:  Average performance of the Phase-1 model CaD-LLaVAV1 on multiple
runs of training.

We run the training of the Phase-1 model CaD-LLaVAV1 multiple times and report
the average performance with standard deviation in Table 8. In most evaluation
cases, the standard deviation is within around 1%.

###  E.2 Ablation on Phase-2 Data Collection - OOD CaD refinement

| Training Data |  BISON |  SVO |  Difference Spotting |  CaD-QA
---|---|---|---|---|---
A: | LLaVA mix (L) |  54.00% |  46.80% |  49.50% |  2.54
B: | L + SpotDiff orig. annot. |  51.33% |  52.27% |  60.48% |  2.51
C: | L + SpotDiff our annot. (refined from orig. annot.) |  54.00% |  54.87% |  66.67% |  2.86
Table 9:  Ablation of phase-2 data collection from 15K pairs of video frames
in Spot-the-diff (SpotDiff). We use CaD-LLaVAV1 to generate CaD on SpotDiff by
refining from the original human-annotated difference descriptions.

In Section 6 (main paper), we perform ablation the Phase-2 data collection.
Here we further explore applying our phase-2 data collection on out-of-
distribution (OOD) data of Spot-the-diff (SpotDiff) dataset. The dataset
contains distant-view frame pairs with very subtle changes from video-
surveillance footage, which are OOD from most LMM training data.

In Table 9, we train with SpotDiff original human-annotated difference
description (row B) and with our CaD-LLaVAV1 generated CaD summaries which is
refined from the original annotation (row C). We also evaluate on the
Difference-Spotting partition on SEED-Bench 2 [64] which contains multi-choice
questions based on frame pairs from SpotDiff. In data collection and training
for this experiment, we only used the 15K training image pairs from SpotDiff
which are not included in the Difference-Spotting SEED partition. The results
in Table 9 verify that our phase-2 data collection using CaD-LLaVAV1 is also
effective on OOD data.

###  E.3 Qualitative Results of CaD Summaries

Figure 13:  Examples of (a) Phase-1 LLM-collected CaD summary and (b) Phase-2
LMM-collected CaD summary

In Fig. 2 (main paper), we illustrate the pipeline of our two-phase CaD-VI
together with two examples of Phase-1 LLM-collected CaD summary and Phase-2
LMM-collected CaD summary. Here in Fig. 13, we provide two additional
examples. Note that in Fig. 13(a), we only pass the captions with the
instruction prompt (in Fig. 9) into the LLM. In Fig. 13(b), we pass the
original annotation and both images with the instruction prompt (in Fig. 10)
into the Phase-1 model. In the main paper (Table 5), we demonstrate the
generated CaD summary without using the original annotation also leads to
effective results.

Figure 14:  Examples of Q&A pairs in CaD-QA together with LMM predicted
answers and the corresponding LLM evaluation rating for the prediction (Red
and green texts denote incorrect and correct description).  Figure 15:
Examples of Q&A pairs in CaD-QA together with LMM predicted answers and the
corresponding LLM evaluation rating for the prediction (Red and green texts
denote incorrect and correct description).  Figure 16:  Examples of Q&A pairs
in CaD-QA together with LMM predicted answers and the corresponding LLM
evaluation rating for the prediction (Red and green texts denote incorrect and
correct description).

###  E.4 Qualitative Results on CaD-QA

In Fig. 14, Fig. 15 and Fig. 16, we show examples of Q&A pairs in our CaD-QA ,
together with the predicted answers from CaD-LLaVAV2 model and the vanilla
LLaVA 1.5 model. We also report the LLM ratings for the predicted answers. The
vanilla LLaVA model has incorrect answers by either mistakenly combining the
contents in two images (Fig. 14(b), the man is standing in front of the toilet
while holding an umbrella), omitting one of the images (Fig. 15(a), Fig.
16(a)), attending to the incorrect image (Fig. 15(c)) or hallucinating non-
existent contents (Fig. 16(b)). The failure demonstrates the lacking of
capability of properly comparing two images. At the same time, our CaD-InstV2
manages to correctly differentiate between the two images, attend to the
corresponding content asked in the question and draw a summary of comparison.

###  E.5 Qualitative Results on BISON

Figure 17:  Examples of predictions of the binary image selection task on
BISON (red and green texts denote incorrect and correct predictions). We
instruct the LMMs to, besides the selection, also give a reasoning for the
answer.

In Fig. 17, we illustrate some examples of the binary image selection task on
BISON. We instruct the LMMs to give both the selection answer and also the
reasoning for the selection. Here we compare the vanilla LLaVA 1.5 and our
CaD-LLaVAV2 . The LLaVA model, even if it captures the relevant content in
some cases, has confusion differentiating the two images (Fig. 17(a)(b)). For
our CaD-LLaVAV2 , the key reasoning that leads to the correct answer is always
covered in the structured difference summary.

## References

  * [1] Interaction Design Foundation - IxDF  “What are the Gestalt Principles?”, 2016  URL: https://www.interaction-design.org/literature/topics/gestalt-principles
  * [2] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Percy Liang and al.  “On the Opportunities and Risks of Foundation Models”, 2022  arXiv:2108.07258 [cs.LG]
  * [3] OpenAI al.  “GPT-4 Technical Report”, 2024  arXiv:2303.08774 [cs.CL]
  * [4] Gemini Team al.  “Gemini: A Family of Highly Capable Multimodal Models”, 2024  arXiv:2312.11805 [cs.CL]
  * [5] Anthropic  “The Claude 3 Model Family: Opus, Sonnet, Haiku”, 2024  arXiv:2312.11805 [cs.CL]
  * [6] AI@Meta  “Llama 3 Model Card”, 2024  URL: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * [7] Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee  “Visual instruction tuning”  In _Advances in neural information processing systems_ 36, 2023
  * [8] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu and Lijuan Wang  “The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)”, 2023  arXiv:2309.17421 [cs.CV]
  * [9] Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Su, Nigel Collier and Yutong Lu  “Sparkles: Unlocking chats across multiple images for multimodal instruction-following models”  In _arXiv preprint arXiv:2308.16463_ , 2023
  * [10] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang and Ziwei Liu  “Otter: A Multi-Modal Model with In-Context Instruction Tuning”  In _arXiv preprint arXiv:2305.03726_ , 2023
  * [11] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin and Jiaqi Wang  “InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model”  In _arXiv preprint arXiv:2401.16420_ , 2024
  * [12] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang and Xinlong Wang  “Generative Multimodal Models are In-Context Learners”  arXiv preprint arXiv:2312.13286, 2023
  * [13] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li and Mohamed Elhoseiny  “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models”  In _arXiv preprint arXiv:2304.10592_ , 2023
  * [14] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre and Shiori Sagawa  “Openflamingo: An open-source framework for training large autoregressive vision-language models”  In _arXiv preprint arXiv:2308.01390_ , 2023
  * [15] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li and Ziwei Liu  “Mimic-it: Multi-modal in-context instruction tuning”  In _arXiv preprint arXiv:2306.05425_ , 2023
  * [16] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut and Vittorio Ferrari  “Connecting vision and language with localized narratives”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , 2020, pp. 647–664  Springer
  * [17] Hexiang Hu, Ishan Misra and Laurens Van Der Maaten  “Evaluating text-to-image matching using binary image selection (bison)”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0
  * [18] Lisa Anne Hendricks and Aida Nematzadeh  “Probing Image-Language Transformers for Verb Understanding”  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , 2021, pp. 3635–3644
  * [19] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai and Yoav Artzi  “A Corpus for Reasoning about Natural Language Grounded in Photographs”  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 6418–6428
  * [20] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu and Lijuan Wang  “Equivariant similarity for vision-language foundation models”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 11998–12008
  * [21] Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna and Kate Saenko  “cola: A Benchmark for Compositional Text-to-image Retrieval”  In _Advances in Neural Information Processing Systems_ 36, 2023
  * [22] Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee  “Improved baselines with visual instruction tuning”  In _arXiv preprint arXiv:2310.03744_ , 2023
  * [23] Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi  “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models”  In _International conference on machine learning_ , 2023, pp. 19730–19742  PMLR
  * [24] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li and Mohamed Elhoseiny  “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models”  In _The Twelfth International Conference on Learning Representations_ , 2023
  * [25] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li and Yu Qiao  “Llama-adapter: Efficient fine-tuning of language models with zero-init attention”  In _arXiv preprint arXiv:2303.16199_ , 2023
  * [26] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen and Yong Jae Lee  “LLaVA-NeXT: Improved reasoning, OCR, and world knowledge”, 2024  URL: https://llava-vl.github.io/blog/2024-01-30-llava-next/
  * [27] Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan and Hang Yan  “Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition”  In _arXiv preprint arXiv:2309.15112_ , 2023
  * [28] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu and Tiejun Huang  “Generative multimodal models are in-context learners”  In _CVPR_ , 2024
  * [29] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han and Baobao Chang  “MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning”  In _The Twelfth International Conference on Learning Representations_ , 2024
  * [30] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo and Kai Chen  “Multimodal-gpt: A vision and language model for dialogue with humans”  In _arXiv preprint arXiv:2305.04790_ , 2023
  * [31] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra and Devi Parikh  “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6904–6913
  * [32] Drew A Hudson and Christopher D Manning  “Gqa: A new dataset for real-world visual reasoning and compositional question answering”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6700–6709
  * [33] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung and Steven Hoi  “Instructblip: Towards general-purpose vision-language models with instruction tuning”  In _Advances in Neural Information Processing Systems_ 36, 2023
  * [34] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu and Xu Sun  “M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning”  In _arXiv preprint arXiv:2306.04387_ , 2023
  * [35] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang and Tong Sun  “Llavar: Enhanced visual instruction tuning for text-rich image understanding”  In _arXiv preprint arXiv:2306.17107_ , 2023
  * [36] Bo Zhao, Boya Wu and Tiejun Huang  “Svit: Scaling up visual instruction tuning”  In _arXiv preprint arXiv:2307.04087_ , 2023
  * [37] Harsh Jhamtani and Taylor Berg-Kirkpatrick  “Learning to Describe Differences Between Pairs of Similar Images”  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 4024–4034
  * [38] Harsh Jhamtani and Taylor Berg-Kirkpatrick  “Learning to Describe Differences Between Pairs of Similar Images”  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 4024–4034
  * [39] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár and C Lawrence Zitnick  “Microsoft coco captions: Data collection and evaluation server”  In _arXiv preprint arXiv:1504.00325_ , 2015
  * [40] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C Lawrence Zitnick  “Microsoft coco: Common objects in context”  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 2014, pp. 740–755  Springer
  * [41] Peter Young, Alice Lai, Micah Hodosh and Julia Hockenmaier  “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions”  In _Transactions of the Association for Computational Linguistics_ 2 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2014, pp. 67–78
  * [42] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso and Antonio Torralba  “Semantic understanding of scenes through the ade20k dataset”  In _International Journal of Computer Vision_ 127 Springer, 2019, pp. 302–321
  * [43] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci and Alexander Kolesnikov  “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale”  In _International journal of computer vision_ 128.7 Springer, 2020, pp. 1956–1981
  * [44] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna and Florian Bressand  “Mixtral of experts”  In _arXiv preprint arXiv:2401.04088_ , 2024
  * [45] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li and Eric Xing  “Judging llm-as-a-judge with mt-bench and chatbot arena”  In _Advances in Neural Information Processing Systems_ 36, 2023
  * [46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava and Shruti Bhosale  “Llama 2: Open foundation and fine-tuned chat models”  In _arXiv preprint arXiv:2307.09288_ , 2023
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin and Jack Clark  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [48] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen  “Lora: Low-rank adaptation of large language models”  In _arXiv preprint arXiv:2106.09685_ , 2021
  * [49] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge and Ying Shan  “Seed-bench: Benchmarking multimodal llms with generative comprehension”  In _arXiv preprint arXiv:2307.16125_ , 2023
  * [50] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li and David A Shamma  “Visual genome: Connecting language and vision using crowdsourced dense image annotations”  In _International journal of computer vision_ 123 Springer, 2017, pp. 32–73
  * [51] Luowei Zhou, Chenliang Xu and Jason Corso  “Towards automatic learning of procedures from web instructional videos”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018
  * [52] Jingwei Ji, Ranjay Krishna, Li Fei-Fei and Juan Carlos Niebles  “Action genome: Actions as compositions of spatio-temporal scene graphs”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10236–10247
  * [53] Yuxuan Wang, Difei Gao, Licheng Yu, Weixian Lei, Matt Feiszli and Mike Zheng Shou  “Geb+: A benchmark for generic event boundary captioning, grounding and retrieval”  In _European Conference on Computer Vision_ , 2022, pp. 709–725  Springer
  * [54] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo and Charles Herrmann  “Kubric: A scalable dataset generator”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 3749–3761
  * [55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer  “High-resolution image synthesis with latent diffusion models”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10684–10695
  * [56] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos and Moritz Mueller-Freitag  “The" something something" video database for learning and evaluating visual common sense”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5842–5850
  * [57] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett and Will Price  “Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100”  In _International Journal of Computer Vision_ Springer, 2022, pp. 1–23
  * [58] Hilde Kuehne, Ali Arslan and Thomas Serre  “The language of actions: Recovering the syntax and semantics of goal-directed human activities”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 780–787
  * [59] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang and Joseph E Gonzalez  “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality”  In _See https://vicuna. lmsys. org (accessed 14 April 2023)_ 2.3, 2023, pp. 6
  * [60] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang and Yue Cao  “Eva: Exploring the limits of masked visual representation learning at scale”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 19358–19369
  * [61] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao  “Eva-clip: Improved training techniques for clip at scale”  In _arXiv preprint arXiv:2303.15389_ , 2023
  * [62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro and Faisal Azhar  “Llama: Open and efficient foundation language models”  In _arXiv preprint arXiv:2302.13971_ , 2023
  * [63] InternLM Team  “Internlm: A multilingual language model with progressively enhanced capabilities”  In _2023-01-06)[2023-09-27]. https://github. com/InternLM/InternLM_ , 2023
  * [64] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang and Ying Shan  “Seed-bench-2: Benchmarking multimodal large language models”  In _arXiv preprint arXiv:2311.17092_ , 2023


§ MODEL FREE ALGORITHM

In this section, we consider problem (<ref>) in the model free setting. That is, we assume that $(A, B, C, \mathcal D)$ in system (<ref>) is unknown. First, we introduce a gradient estimation algorithm
based on the derivative-free optimization and show that the estimated gradient is close to the exact gradient with high probability.
Then, we provide Policy Gradient Projection Algorithm in Algorithm <ref>.
Despite the error in the estimated gradient, we can prove the global convergence to $\varepsilon$-stationary points.
Finally, we propose a variance reduction method using the baseline technique and prove its optimality.

§.§ Gradient estimation

In the model free setting, the exact gradient of the objective function $f(K)$
in (<ref>) cannot be accessed directly.
Then, based on the zeroth-order or derivative-free optimization approach <cit.>,
we propose Algorithm <ref>
to calculate the stochastic estimate $\hat \grad f(K)$ of the gradient $\grad f(K)$.

Gradient Estimation

[1]
$K\in \Omega, N>0, r > 0, \tau > 0$
$i=1$ to $N$
Sample $U_i$ from the uniform distribution $\mathcal{S}$ over matrices with $\normF{U_i} = \sqrt{mp}$.
Simulate the system

\begin{align}
\dot x_i(t) & = Ax_i(t)+Bu_i(t), \\
y_i(t)      & = Cx_i(t),
\end{align}

where $u_i(t) = -(K+rU_i)y_i(t)$, $x_i(0) \sim \mathcal D$
until time $\tau$ and
calculate the empirical cost $c_i$ by

\begin{align}
c_i = \int_0^\tau \qty[y_i^\top(t) Q y_i(t) + u_i(t)^\top R u_i(t)]dt.
\end{align}

Define the estimated gradient by

\begin{align}
\hat \grad f(K) = \frac{1}{rN}\sum_{i=1}^N c_iU_i. \label{eq:hatgrad}
\end{align}

$\hat \grad f(K)$.

In this section, we assume the following in addition to Assumption <ref>.

* ${U}_i$ and $x_i(0)$ are independent.

* $K\in S(a)$ with a given constant $a$.

* The distribution $\mathcal D$ of initial state $x(0) \sim \mathcal{D}$
satisfies$\norm{x(0)}\leq P\quad \text{a.s.}$
for a constant $P > 0$.

The second assumption is justified in Section <ref>.

The following lemma ensures $K+rU \in S(2a)$ for sufficiently small $r$.

There exists $r_0  > 0$ such that
for any $r \leq r_0$, $K\in S(a)$ and $U$ such that $\norm{U} = \sqrt{mp}$, we have $K+rU\in S(2a)$.

Using the same argument as Lemma 4 in <cit.>, we obtain the result.

The following theorem is an extension of Lemma 27 in Supplementary material of <cit.>
to output feedback control.
To prove this, we derive some inequalities in Appendix <ref>.

Let $\hat{\nabla} f(K)$ be defined as (<ref>).
For any $\varepsilon' > 0$ and $\delta > 0$, set $r = O(\varepsilon'), \tau = O(\log 1/\varepsilon'), N = O((\log 1/\delta)/\varepsilon'^4)$. Then, we have

\begin{align}
&\normF{\hat \grad f(K)-\grad f(K)} \leq \varepsilon' \label{eq:high-probability},
\end{align}

for any $K\in S(a)$,
with probability greater than $1-\delta$.

See Appendix <ref>.

§.§ Convergence properties

In this section, we prove the global convergence of the policy gradient method in the model free setting.
We show a model free control algorithm, Policy Gradient Projection, in Algorithm <ref>.
The positive integer $T$ is the iteration number, $\alpha$ is the step size, $K_0$ is the initial point of the feedback gain $K$,
and $\proj$ is the projection onto $\Omega$ with respect to the Frobenius norm.
The termination condition $\normF{K_{i+1}-K_i}\leq \varepsilon\alpha$ is added for technical reasons.

Policy Gradient Projection

[1]
$T > 0, \alpha > 0, \varepsilon > 0, K_0\in \Omega, N > 0, r>0, \tau>0$
$i=0$ to $T-1$
Calculate $\hat \grad f(K_i)$ using Algorithm <ref>.
$K_{i+1} := \proj\qty(K_i-\alpha \hat \grad f(K_{i}))$.
$\normF{K_{i+1}-K_i}\leq \varepsilon\alpha$

Lemma <ref> describes the property of orthogonal projections,
which is essential to our convergence analysis.

Let $\proj:\R^{m\times p}\to \Omega$ be an orthogonal projection onto $\Omega$.
For any $x\in \Omega$ and $y\in \R^{m\times p}$, we have

\begin{align}
\q<x-\proj(y), y-\proj(y)> & \leq 0.
\end{align}

See Theorem 6.41 of <cit.>.

The following definition is required to show our main result.

For positive constants $\alpha$ and $\varepsilon$,
$K$ is called an $\varepsilon$-stationary point if $\normF{G_\alpha(K)} \leq \varepsilon$, where $K^+ := \proj(K-\alpha\grad f(K))$ and $G_\alpha(K) := \frac{1}{\alpha}(K^+-K)$.

The following theorem is a main result, which is an extension of Theorem 4.2 in <cit.> to the model-free and constrained problems.
The proof is based on the proof for projected gradient method without gradient error for $L$-smooth functions <cit.>. However, in the presence of gradient errors, the termination condition and some extra arguments to bound the effect of the difference between the true and estimated gradients are required.

Assume that the constants $N, r, \tau$ satisfies the condition in Theorem <ref> with $\varepsilon' = \lambda \varepsilon$
for the given constants $0<\lambda<1, \varepsilon > 0, \delta > 0$.
Let $\{K_i\}_{i=0}^{T'}$ be the sequence generated by the Algorithm <ref>,
where $T'$ is the total number of iterations, which can be different from $T$ due to the terminate condition.
For a step size $\alpha \in \qty(0, \frac{2(1-\lambda)}{L})$,
where $L$ is the Lipschitz constant of $\grad f$,
we have the following result with probability greater than $1-T\delta$.

* The sequence $\{K_i\}_{i=0}^{T'}$ remains in $S$ and $\{f(K_i)\}_{i=0}^{T'}$ is strictly decreasing. That is, for any $0\leq i\leq T'-1$,

\begin{align}
f(K_{i+1}) < f(K_i).\label{eq:tmp4}
\end{align}

* If $T > \frac{f(K_0)}{\varepsilon^2\alpha^2\qty(\frac{1-\lambda}{\alpha}-\frac{L}{2})}$,
$K_{T'}$ is a $(1+\lambda)\varepsilon$-stationary point.

We define $G'_\alpha(K) = \frac{1}{\alpha}{v}$, where $v:=\hat{K}^+-K$ and $\hat{K}^+ := \proj(K-\alpha\hat \grad f(K))$.
First, we show that if $\normF{G'_\alpha(K)} \leq \varepsilon$ and (<ref>) holds,
$K \in S_0$ is a $(1+\lambda)\varepsilon$-stationary point.

\begin{align}
\normF{G_\alpha(K)} & \leq \normF{G'_\alpha(K)}+\frac{1}{\alpha}\normF{K^+-\hat{K}^+} \\
& \leq \varepsilon+\normF{\grad f(K)-\hat\grad f(K)}                \leq (1+\lambda)\varepsilon.
\end{align}

The second inequality holds, because projections onto convex sets are contractive,
and the last inequality follows from (<ref>).

Next, we show $\normF{G'_\alpha(K_{T'})} \leq \varepsilon$ with high probability.
The termination condition ensures $\normF{G'_\alpha(K_i)} > \varepsilon$ for $i = 0, \dots, T'-1$.
Assume that $\normF{G'_\alpha(K)} > \varepsilon$ and Eq. (<ref>) with $\varepsilon' = \lambda \varepsilon$ holds for $K\in S_0$.
We define $K_t:=K+tv$ and $t^*:=\max\qty{t\geq 0\mid f(K_t)\leq f(K)}$.
Then, Lemma <ref> yields

\begin{align}
\q<\hat \grad f(K), v> & \leq -\frac{1}{\alpha}\normF{v}^2.\label{eq:mf-proj}
\end{align}

and $L$-smoothness of $f(K)$ on $S_0$ implies

\begin{align}
f(K_t)-f(K) & \leq \q<\grad f(K), K_t-K>+\frac{L}{2}\normF{K_t-K}^2\\
&=t\q<\grad f(K), v>+\frac{Lt^2}{2}\normF{v}^2.\label{eq:mf-L-smooth}
\end{align}

By adding both sides of the equations (<ref>) multiplied by $t$ and (<ref>), we obtain

\begin{align}
f(K_t)&-f(K) \\
& \leq  \qty(\frac{Lt^2}{2}-\frac{t}{\alpha}+t\frac{\normF{\grad f(K)-\hat \grad f(K)}}{\normF{v}})\normF{v}^2 \\
& < \qty(\frac{Lt^2}{2}-t\frac{1-\lambda}{\alpha})\normF{v}^2.\label{eq:mf-sufficient-decrease}
\end{align}

The last inequality follows from (<ref>) and $G'_\alpha(K)=\normF{v}/\alpha > \varepsilon$.
For $t = t^*$, we have

\begin{align}
0 &= f(K_{t^*})-f(K)\leq \qty(\frac{L{t^*}^2}{2}-{t^*}\frac{1-\lambda}{\alpha})\normF{v}^2
\end{align}

Since $\norm{v} > 0$, we have $t^* \geq \frac{2(1-\lambda)}{L\alpha}\geq 0$.
Therefore, (<ref>) holds for $t = 1$.
Eq. (<ref>) with $K=K_i$ and $t=1$ leads us to

\begin{align}
f(K_{i+1})-f(K_i) & < \qty(\frac{L}{2}-\frac{1-\lambda}{\alpha})\normF{K_{i+1}-K_i}^2 <0,
\end{align}

because $\alpha \in (0, \frac{2(1-\lambda)}{L})$.
Thus, (<ref>) holds for any $0\leq i\leq T'-1$.

If $T' < T$, the termination condition ensures $\normF{G'_\alpha(K_{T'})} \leq \varepsilon$.
Therefore, it suffices to show $T' < T$.
Since $T' \leq T$ by definition, suppose that $T' = T$.
Then, we obtain

\begin{align}
f(K_0) & \geq f(K_0)-f(K_{T})=\sum_{i = 0}^{T-1} f(K_i)-f(K_{i+1})   \\
& \geq\qty(\frac{1-\lambda}{\alpha}-\frac{L}{2}) \sum_{i = 0}^{T-1} \normF{K_{i+1}-K_i}^2  \\
&\geq \qty(\frac{1-\lambda}{\alpha}-\frac{L}{2})T\varepsilon^2\alpha^2.
\end{align}

The assumption $T > \frac{f(K_0)}{\varepsilon^2\alpha^2\qty(\frac{1-\lambda}{\alpha}-\frac{L}{2})}$
yields $f(K_0)>f(K_0)$, which is a contradiction. Thus, $T'<T$, that is, $\normF{G'_\alpha(K_{T'})}\leq \varepsilon$.
From Theorem <ref>, the probability that Eq. (<ref>) holds for $K = K_i(i=0, \dots, T')$ is greater than
$(1-\delta)^{T'+1}\geq 1-T\delta$.
This completes the proof.

The convergence rate $T = O(1/\varepsilon^2)$ is essentially the same as the rate of projected gradient method without gradient error for $L$-smooth functions <cit.>. The same rate was obtained for discrete-time state feedback LQR problems in the model free setting <cit.> but not known for the model free and output feedback setting.

§.§ Variance reduction

Policy gradient methods tend to suffer from a large variance, which leads to slow learning <cit.>.
The use of baseline is one of the variance reduction techniques for policy gradient methods <cit.>.
State-depending functions are often used as a baseline, because it does not add any bias to the estimated gradient <cit.>.
In this section, we propose to use the cost function $\tilde f$ defined in (<ref>) as a baseline and show its optimality.

For a baseline function $b(x)$, the estimated gradient $\bar \grad f$ is defined as

\begin{align}
\bar \grad f(K)  &:= \frac{1}{rN}\sum_{i=1}^N (\tilde f_\tau(K+rU_i;x_i(0))-b(x_i(0)))U_i,\label{eq:grad-baseline-b}
\end{align}

\begin{align}
\tilde f_\tau(K;x(0)) := \int_0^\tau y^\top(t) ( Q + K^\top RK)y(t)dt\label{eq:ftau}.
\end{align}

Note that $\tilde f_\tau$ satisfies $\lim_{\tau \to \infty} \tilde f_\tau(K;x(0)) = \tilde f(K;x(0))$.

\begin{align}
\hat \grad f(K) = \frac{1}{rN}\sum_{i=1}^N \tilde f_{\tau}(K+r U_i;x_i(0))U_i,
\end{align}

the estimated gradient $\bar \grad f(K)$ satisfies

\begin{align}
E\qty[\bar \grad f(K)]
& = E\qty[\hat \grad f(K)] - \frac{1}{rN}\sum_{i=1}^N E\qty[b(x_i(0))U_i] \\
& = E\qty[\hat \grad f(K)] - \frac{1}{rN}\sum_{i=1}^N E\qty[b(x_i(0))]E[U_i] \\
& = E\qty[\hat \grad f(K)].
\end{align}

The second equality holds from the assumption that $U_i$ and $x_i(0)$ are independent, and the last equality follows from $E[U_i] = 0$.
Thus, the bias in $\bar \grad f$ is the same as the one in $\hat \grad f$.

In terms of the variance, the baseline $b(x(0)) = \tilde f(K;x(0))$ is almost optimal for small $r$ and large $\tau$.

For $a > 0$, $r \leq r_0$, $\tau > 0$, and $K\in S(a)$,
the optimal baseline $b^*(x(0))$ which minimizes the variance of the estimated gradient (<ref>) is given by

\begin{align}
b^*(x(0)) = E_{U\sim \mathcal{S}}\qty[\tilde f_\tau(K+rU;x(0))],\label{eq:b-star}
\end{align}

where $\mathcal{S}$ is defined in Algorithm <ref>
and it satisfies

\begin{align}
\lim_{r\to 0, \tau \to \infty}b^*(x(0)) = \tilde f(K;x(0)).\label{eq:limit}
\end{align}

Since $\qty{x_i(0)}_{i=1}^N, \qty{U_i}_{i=1}^N$ are assumed to be independent, $\qty{f_\tau(K;x_i(0))-b(x_i(0)))U_i}_{i=1}^N$ are independent and we have

\begin{align}
V\qty[\bar \grad f(K)]
& = \frac{1}{r^2N}\left[E\qty[\normF{(\tilde f_\tau(K+rU;x(0))-b(x(0)))U}^2]\right.        \\
& \quad\quad-\left.\normF{E[(\tilde f_\tau(K+rU;x(0))-b(x(0)))U]}^2\right].\label{eq:baseline-variance}
\end{align}

for any baseline $b(x)$.
Because we have assumed that $x(0)$ and $U$ are independent and $E[U] = 0$,

\begin{align}
E\qty[b(x(0))U] = E\qty[b(x(0))]E[U] = 0.
\end{align}

Thus, the expectation $E[(\tilde{f}_\tau(K+rU;x(0))-b(x(0)))U]$ is independent of the choice of a baseline $b(x)$,
that is,

\begin{align}
& E\qty[(\tilde f_\tau(K+rU;x(0))-b(x(0)))U] \\ =& E\qty[\tilde f_\tau(K+rU;x(0))U].
\end{align}

Then, the second term in (<ref>) is independent of the choice of $b(x)$ and we just need to minimize the first term in (<ref>),

\begin{align}
& E\qty[\norm{\qty(\tilde f_\tau(K+rU;x(0))-b(x(0)))U}^2_F] \\
=& mp E_{x(0)\sim \mathcal{D}}\qty[E_{U\sim S}\qty[\qty(\tilde f_\tau(K+rU;x(0))-b(x(0)))^2]].
\end{align}

Since the expectation $E_{U\sim \mathcal{S}}\qty[\tilde f_\tau(K+rU;x(0))]$ minimizes the mean squared error $E_{U\sim S}\qty[\qty(\tilde f_\tau(K+rU;x(0))-b(x(0)))^2]$ for any $x(0)$,
the optimal baseline is given by (<ref>).
Eq. (<ref>) follows from the continuity of $\tilde f_\tau(K+rU;x(0))$.

Based on Theorem <ref>, we propose to use $\tilde f(K;x(0))$ as a baseline.
Since $\tilde f(K;x(0))$ cannot be computed directly for a given $x(0)$ in the model free setting, we provide the estimation procedure for $\tilde f(K;x(0))$ in Algorithm <ref>. In the following, we define $\bar{y}(t;x(0)) = [y(t)^\top, y(t+h_1)^\top, \dots, y(t+h_{D-1})^\top]^\top \in \R^{pD}(0\leq h_1<h_2<\dots<h_{D-1}=T)$.

Estimate $\tilde f(K;x(0))$

[1]
$K\in S, x(0) \in \R^n, s > 0$.
$i=1$ to $n(n+1)/2$
Simulate system (<ref>) for $x(0) = x_i(0) \sim \mathcal D$ until time $s$.
Find a solution to the following equations.

\begin{align}
& \bar y(0;x_i(0))^\top \hat P(K)\bar y(0;x_i(0)) \\
& = \tilde f_s(K;x_i(0)) + \bar y(s;x_i(0))^\top \hat P(K)\bar y(s;x_i(0)),\label{eq:bellmaneq}
\end{align}

where $\tilde f_s$ is defined as (<ref>).
Define $\hat f(K;x(0)) = \bar y(0;x(0))^\top \hat P(K)\bar y(0;x(0))$.
$\hat f(K;x(0))$.

The following theorem ensures that the estimated cost $\hat f(K;x(0))$ is equal to $\tilde f(K;x(0))$.

For any $T > 0$, $D > 2(n-1) + \frac{T}{2\pi} \beta$, where $\beta = 2(\norm{A}_2 + \norm{B}_2\norm{C}_2\kappa(a))$,
if $x_j(0)x_j(0)^\top - x_j(s)x_j(s)^\top(j=1, \dots, \frac{n(n+1)}{2})$
are linearly independent on $\mathbb{S}^n$ for $s$ in Algorithm <ref>, then $\hat f(K;x(0)) = \tilde f(K;x(0))$ for any $K$ and $x(0)$.

See appendix <ref>.