#  *○Sparkles: Unlocking Chats Across Multiple Images for Multimodal
Instruction-Following Models

Yupan Huang12 Zaiqiao Meng23, Fangyu Liu2, Yixuan Su2, Nigel Collier2, Yutong
Lu1
1Sun Yat-sen University 2University of Cambridge 3University of Glasgow
{huangyp28<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large language models exhibit enhanced zero-shot performance on various tasks
when fine-tuned with instruction-following data. Multimodal instruction-
following models extend these capabilities by integrating both text and
images. However, existing models such as MiniGPT-4 face challenges in
maintaining dialogue coherence in scenarios involving multiple images. A
primary reason is the lack of a specialized dataset for this critical
application. To bridge these gaps, we present SparklesChat, a multimodal
instruction-following model for open-ended dialogues across multiple images.
To support the training, we introduce SparklesDialogue, the first machine-
generated dialogue dataset tailored for word-level interleaved multi-image and
text interactions. Furthermore, we construct SparklesEval, a GPT-assisted
benchmark for quantitatively assessing a model’s conversational competence
across multiple images and dialogue turns. Our experiments validate the
effectiveness of SparklesChat in understanding and reasoning across multiple
images and dialogue turns. Specifically, SparklesChat outperformed MiniGPT-4
on established vision-and-language benchmarks, including the BISON binary
image selection task and the NLVR2 visual reasoning task. Moreover,
SparklesChat scored 8.56 out of 10 on SparklesEval, substantially exceeding
MiniGPT-4’s score of 3.91 and nearing GPT-4’s score of 9.26. Qualitative
evaluations further demonstrate SparklesChat’s generality in handling real-
world applications. All resources will be available at
https://github.com/HYPJUDY/Sparkles.

Figure 1: The architecture of SparklesChat. SparklesChat integrates multiple
images at the word level within the dialogue, facilitating a fine-grained and
human-like multimodal interaction. Figure 2: Comparison between our
SparklesChat (left) and MiniGPT-4 [61] (right) on an example from
SparklesEval. We adapt MiniGPT-4 to accept multiple images as input.
SparklesChat shows conversational competence in open dialogues for image
understanding and reasoning, maintaining cross-image and cross-turn coherence,
and generating relevant and complete responses. In contrast, MiniGPT-4 faces
challenges in these aspects, leading to difficulty following user instructions
across various images and dialogue turns.

##  1 Introduction

Large language models (LLMs) have shown remarkable progress in zero-shot
performance across a variety of tasks when fine-tuned using instruction-
following data [37, 35, 47, 9, 51, 50]. In the multimodal domain, multimodal
instruction-following models such as MiniGPT-4 extend these capabilities by
integrating pretrained vision encoders with instruction-following LLMs using
projection layers [61]. MiniGPT-4 adapts the projection layer to align vision
and language domains by training on concatenated embeddings of images and
their descriptions. The training occurs in two stages: first, on a large-scale
collection of image-text pairs and then on a smaller dataset of detailed,
human-like image descriptions [61]. With this training method, MiniGPT-4
learns alignments between individual images and sentences and performs single-
image understanding and reasoning. However, models such as MiniGPT-4 struggle
to capture interactions between diverse images and text. This capability is
crucial for user-assistant conversations, where users often refer to multiple
images with text snippets to convey their instructions in detail. As shown in
Figure 2, MiniGPT-4 mixes up the content of multiple images, fails to
establish coherence between images, and consequently falls short in following
user instructions during open dialogues.

One key limitation hindering progress in this area is the lack of specialized
datasets designed for multimodal dialogues that involve multiple images and
fine-grained, word-level text interactions. Existing models such as Flamingo
can adapt to various image understanding tasks when prompted with a few
relevant examples due to their training on image-text interleaved web data
[2]. However, these models often fall short in following intricate human
instructions because they are trained to predict the next word on a large web
dataset rather than perform the task the user wants [37].

To address these gaps, we present SparklesChat, a multimodal instruction-
following model for open-ended dialogues across multiple images. Unlike
previous approaches such as MiniGPT-4 that takes the concatenation of a single
image with sentence-level text as input (e.g., "*○Can you describe this image
as detailed as possible?" - where *○denotes a single image), SparklesChat, as
shown in Figure 1, integrates multiple images at the word level (e.g., "Can
you link the celebration occurring in IMAGE#2331159*○and the dirt bike race in
IMAGE#2330601*○?"). This innovation enables fine-grained integration of images
and text, mimicking natural human communication more closely.

To support the training of SparklesChat, we introduce SparklesDialogue, the
first machine-generated dialogue dataset designed for word-level interleaved
multi-image and text interactions. We use OpenAI’s GPT-4 [35] to simulate
user-assistant conversations with visual capabilities by leveraging detailed
image descriptions. Our dataset achieves greater robustness and diversity by
incorporating two subsets, namely SparklesDialogueCC and SparklesDialogueVG,
constructed from different image and description sources.

Furthermore, we introduce SparklesEval, a GPT-assisted benchmark to
quantitatively evaluate a model’s conversational competence in multimodal,
open-ended dialogues across multiple images and dialogue turns. SparklesEval
features a comprehensive and interpretable scoring system based on three
distinct criteria: Image Understanding and Reasoning, Cross-Image and Cross-
Turn Coherence, and Relevance and Completeness of Responses.

For quantitative evaluation, we validate the effectiveness of SparklesChat
through extensive experiments. We conduct zero-shot evaluations on two
standard vision-and-language tasks, including binary image selection on the
BISON dataset [16] and visual reasoning on the NLVR2 dataset [45]. On the
BISON dataset, SparklesChat achieved an accuracy of 56.7%, surpassing
MiniGPT-4’s 46.0%. On the NLVR2 dataset, SparklesChat reached an accuracy of
58.0%, outperforming MiniGPT-4’s 51.3%. In our SparklesEval benchmark,
SparklesChat scores 8.56 out of 10, significantly exceeds MiniGPT-4’s score of
3.91, and closely approaches GPT-4’s score of 9.26. Qualitative evaluations
further demonstrate SparklesChat’s applicability in real-world scenarios. All
resources related to this study will be made publicly available.

##  2 Related works

###  2.1 Image-text alignment

Various datasets have been proposed to facilitate image-text alignment by
associating images with corresponding descriptions. These datasets include but
are not limited to SBU Captions [36], MSCOCO [31], YFCC100M [46], Visual
Genome [25], Conceptual Captions [42], Conceptual 12M [5], ALIGN [22], LAION
[41], LAION-2B [40], COYO700M [4], and WebLI [8]. These datasets have
significantly contributed to the development of multimodal models for image-
and-text generation [19, 20, 39, 30, 7] and understanding [22, 21, 38, 53].

Emerging trends in this research area include datasets featuring interleaved
sequences of images and text sourced from web corpora, such as M3W [2], web
and Wikipedia articles [1], Common Crawl Interleaved data [17], and the
Multimodal C4 dataset [62]. These datasets extend conventional image-text
alignment training by incorporating multiple images and sentences in an
interleaved manner. When trained on these enriched datasets, models such as
Flamingo [2], OpenFlamingo [3], and Kosmos-1 [17] can be adapted to various
image understanding tasks by being prompted with a few task-relevant examples.

###  2.2 Image-text instruction tuning

Image-text instruction tuning has grown substantially with the advent of
multimodal instruction datasets. For instance, MultiInstruct [52] offers a
benchmark comprising 62 diverse multimodal tasks unified in a seq-to-seq
format. InstructBLIP [10] extended the scope by transforming 26 existing
datasets into the instruction-tuning form. More recently, Otter [29] is
trained on MIMIC-IT [28], a multi-modal in-context instruction tuning dataset
constructed by grouping multiple similar instructions into a contextual
example.

In addition to training on large-scale image-text pairs, MiniGPT-4 is further
fine-tuned on a smaller dataset of detailed image descriptions to better align
with user intentions [61]. In the same vein, PF-1M [6] is a collection of 37
vision-language datasets that rewrite image annotations in a human-like style.
Furthermore, techniques such as LLaVA [32], SVIT [58], LRV-Instruction [57],
and LAMM [55] have emerged. These methods leverage language-only APIs such as
OpenAI’s GPT-4 [35] and self-instruction methods [50] to generate instructions
and responses. Using images represented by their associated annotations, such
as image captions, region descriptions, object bounding boxes, attributes, and
relationships, language models can generate responses in various forms, such
as short conversations, image captioning, and visual reasoning. Models such as
mPLUG-Owl [54], PandaGPT [44], LLaMAAdapter V2 [14], and Multimodal-GPT [15]
further extended this area by being jointly trained with language-only and
vision-and-language instruction data.

###  2.3 Multimodal dialogue datasets

Existing multimodal dialogue datasets broadly fall into two categories. The
first group comprises datasets where conversations are heavily rooted in and
driven by images. Traditional datasets of this type are primarily generated by
inviting crowd workers to engage in dialogues about a common image. Notable
examples include Visual Dialog [11], which emphasizes question-answering tasks
within AI-human chat about visual content, and IGC [34], a compilation of
dialogues featuring an image, a corresponding textual description, and a
conversation centered on the image. Image-Chat [43] presents a corpus of
image-grounded dialogues crafted around provided images. Recently, dialogue
datasets entirely generated by LLMs in conjunction with image annotations have
surfaced, e.g., LLaVA [32], SVIT [58], and LAMM [55]. Each dialogue in these
datasets begins with an inquiry about image attributes or factual knowledge,
with responses expected to be brief within 50 words.

The second category features datasets derived from daily human conversations,
with images interspersed within multi-turn conversations sparsely. For
example, OpenViDial 1.0 [33] and OpenViDial 2.0 [49] are sourced from
dialogues in movies and TV series, whereas PhotoChat [56] is a human-human
dialogue dataset developed through crowdsourcing and features photo-sharing.
Other datasets, such as DialogCC [27], MultiModalDialogue [26], and IMAD [48]
enhance text-only dialogues by incorporating semantically relevant images. In
addition, MMChat [60] and MMDialog [13] encompass image-grounded dialogues
derived from social media interactions.

In the present work, we exploit image-text alignment data to construct an
instruction-tuning dataset in the form of dialogues. Our dataset,
SparklesDialogue, is the first dataset explicitly crafted to explore the
interactions between multiple images and word-level textual content.
SparklesChat trained on it unlocks the hidden capability of aligned multimodal
models that serve as helpful AI assistants capable of interpreting complex
prompts involving image interactions.

Table 1: Prompt and response sequence formats used to train SparklesChat. The
first and the second conversation turns are illustrated here. The model is
trained to predict the assistant answers, and thus only  green sequence are
used to compute the loss in the auto-regressive model.

Dialogue Turn One |
---|---
$\mathbf{X}_{\texttt{prompt}}^{i,1}$ = $\mathbf{X}_{\texttt{system}}$<SEP>Human: $\mathbf{X}_{\texttt{q}}^{i,1}$<SEP>Assistant: |
$\mathbf{X}_{\texttt{response}}^{i,1}$ = $\mathbf{X}_{\texttt{a}}^{i,1}$<SEP> |
Dialogue Turn Two |
$\mathbf{X}_{\texttt{prompt}}^{i,2}$ = $\mathbf{X}_{\texttt{system}}$ <SEP> Human: $\mathbf{X}_{\texttt{q}}^{i,1}$<SEP> Assistant: $\mathbf{X}_{\texttt{a}}^{i,1}$<SEP> Human: $\mathbf{X}_{\texttt{q}}^{i,2}$<SEP> Assistant: |
$\mathbf{X}_{\texttt{response}}^{i,2}$ = $\mathbf{X}_{\texttt{a}}^{i,2}$<SEP> |

##  3 SparklesChat

We present a multimodal instruction-following model SparklesChat to foster
interactions between users and AI assistants across multiple images and
illustrate the framework in Figure 1. More implementation details can be found
in Appendix B.

###  3.1 Model

The foundation for SparklesChat is the MiniGPT-4 architecture, which connects
a pretrained vision encoder and a pretrained LLM with a projection layer [61].
The language decoder, Vicuna [9], is based on the LLaMA framework [47], which
can handle diverse language tasks. For image processing, we use the visual
encoder from BLIP-2 [30], combining a Vision Transformer (ViT) backbone [12]
with a pretrained Q-Former [30]. In the MiniGPT-4, the input to the language
model is a single image representation followed by a sentence embedding of the
image description. In SparklesChat, image representations of different images
are embedded between text according to their positions in dialogues. Only the
projection layer is trainable in the model while other vision and language
components are frozen.

###  3.2 Instruction-tuning

We represent an $i$-th $T$-turn dialogue as
$\mathbf{X}^{i}=(\mathbf{X}_{\texttt{q}}^{i,1},\mathbf{X}_{\texttt{a}}^{i,1},\cdots,\mathbf{X}_{\texttt{q}}^{i,T},\mathbf{X}_{\texttt{a}}^{i,T})$,
where each pair of
$(\mathbf{X}_{\texttt{q}}^{i,t},\mathbf{X}_{\texttt{a}}^{i,t})$ includes a
question from the user and an answer from the assistant in turn-$t$. For each
$\mathbf{X}^{i}$, we construct $T$ training samples by organizing each pair of
questions and answers as a sequence. The prompt
$\mathbf{X}_{\texttt{prompt}}^{i,t}$ and response
$\mathbf{X}_{\texttt{response}}^{i,t}$ at the $t$-th turn are:

$\mathbf{X}_{\texttt{prompt}}^{i,t}=\begin{cases}\mathbf{X}_{\texttt{system}}\texttt{<SEP>}\texttt{Human}:\mathbf{X}_{\texttt{q}}^{i,1}\texttt{<SEP>}\texttt{Assistant}:,&\text{if
}t=1,\\\
\mathbf{X}_{\texttt{prompt}}^{i,t-1}\texttt{<SEP>}\texttt{Human}:\mathbf{X}_{\texttt{q}}^{i,t}\texttt{<SEP>}\texttt{Assistant}:,&\text{if
}t>1.\end{cases}$ (1)
$\mathbf{X}_{\texttt{response}}^{i,t}=\mathbf{X}_{\texttt{a}}^{i,t}\texttt{<SEP>}$
(2)

Table 1 illustrates the unified format for two-turn dialogue training
sequences. We perform instruction-tuning of the LLM on the prediction tokens
using the auto-regressive training objective. Specifically, for a sequence of
length $L$, we compute the probability of generating target responses
$\mathbf{X}_{\texttt{response}}$ by:

$p(\mathbf{X}_{\texttt{response}}|\mathbf{X}_{\texttt{prompt}})=\prod_{l=1}^{L}p_{\boldsymbol{\theta}}({\color[rgb]{0.13,0.55,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{0.13,0.55,0.13}\boldsymbol{x}_{l}}|\mathbf{X}_{\texttt{prompt},<l},\mathbf{X}_{\texttt{response},<l}),$
(3)

where $\boldsymbol{\theta}$ is the trainable parameters,
$\mathbf{X}_{\texttt{prompt},<l}$ and $\mathbf{X}_{\texttt{response},<l}$ are
prompt and response tokens in all turns before the current prediction token
${\color[rgb]{0.13,0.55,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{0.13,0.55,0.13}\boldsymbol{x}_{l}}$,
respectively. We do not compute the regression loss for the prompt
$\mathbf{X}_{\texttt{prompt}}$ since the prompt is provided by users in real-
world applications, making it unnecessary for the model to make predictions in
this context.

Figure 3: The GPT-assisted data construction process of SparklesDialogue based
on Dialogue Demonstrations and Candidate Image Descriptions. The process
involves instructing GPT-4 to simulate dialogues between a user and an
assistant, focusing on multiple images. Dialogue Demonstrations act as in-
context learning examples, enabling GPT-4 to generate well-formatted and
diverse dialogues. Meanwhile, Candidate Image Descriptions serve as a pool
from which GPT-4 selects relevant images for discussion. Visual images are not
sent to GPT-4 for data generation.

##  4 SparklesDialogue and SparklesEval

To augment the conversational competence of instruction-following LLMs across
multiple images and dialogue turns, we introduce two novel resources:
SparklesDialogue and SparklesEval, which are multimodal dialogue datasets
designed for instruction-following fine-tuning and evaluation, respectively.
Our evaluation data, SparklesEval, is constructed using a method similar to
that of SparklesDialogue but is only used to benchmark models for multiple
images dialogue competence.

###  4.1 GPT-assisted data construction

We aim to construct a multimodal dialogue dataset that offers fine-grained
interactions between multiple images and words, mimicking user-assistant
conversations. The dialogues cover real-world concepts, objects, and entities,
spanning scenarios that involve generating text materials, seeking advice,
guidance, assistance, and much more. Given GPT-4’s capabilities in following
complex instructions and extensive world knowledge, it is the primary tool in
our dialogue data collection. The data collection process is visualized in
Figure 3. We instruct GPT-4 to simulate realistic and diverse conversations
between a user and an assistant with advanced image understanding and
reasoning capabilities. Each conversation has two turns. In the first turn,
the user sends the assistant a reasonable and creative message regarding some
images. In response, the assistant offers detailed answers that provide
comprehensive reasoning regarding the visual content. In the second turn, the
user introduces a new image for further discussion, referencing both the newly
introduced and previously discussed images. Again, we prompt the assistant to
respond with highly helpful and exceptionally detailed answers that provide
comprehensive reasoning to better align with human preference.

We provided GPT-4 with two crucial components to generate the dialogues:
Dialogue Demonstration and Candidate Image Descriptions. The Dialogue
Demonstrations serve as in-context learning examples, steering GPT-4 towards
generating well-formatted and diverse responses. We initiated the creation of
hundreds of demonstration dialogues with GPT-4’s assistance, using similar
prompts and checking their quality. A small subset of dialogues is randomly
chosen each time for demonstration purposes. The Candidate Image Descriptions
serves as a candidate pool for the model to select relevant images for
discussion. From the pool of image-text paired dataset, we randomly select a
small subset as candidates each time. We include the IDs of images in
dialogues to avoid reference ambiguity. Given that the publicly accessible
GPT-4 API only accepts text input, we represent images with detailed
descriptions. These descriptions, sourced from various image annotations such
as image captions, bounding boxes, and region descriptions, comprehensively
portray the content of images [61, 58, 32]. We will elaborate on the sources
of SparklesDialogue in subsection 4.2. Finally, the generated responses are
parsed, and only well-structured results adhering to our desired format are
retained for the final dataset. More details, such as prompt templates and
visualized examples, can be found in Appendix F and Appendix G.

Table 2: Statistics of SparklesDialogue and SparklesEval. Dataset Name |  | Image
---
Source
| Caption
---
Source
#Dialogue |  | #Image
---
Turn 1
| #Image
---
Turn 2
#Unique/Total Image
SparklesDialogueCC | CC [42] | MiniGPT-4 [61] | 1,653 | 1 | 1 | 2,067/3,306 | 3,373/12,979
1,799 | 2 | 1 | 2,642/5,397
1,069 | 3 | 1 | 2,408/4,276
SparklesDialogueVG | VG [25] | SVIT [58] | 1,000 | 2 | 1 | 3,000/3,000 | 7,000/7,000
1,000 | 3 | 1 | 4,000/4,000
SparklesEval | VG [25] | SVIT [58] | 50 | 2 | 1 | 150/150 | 550/550
50 | 2 | 2 | 200/200
50 | 3 | 1 | 200/200
|
---|---
(a) Root verb-noun pairs in user messages. | (b) Word cloud of assistant messages.
|
(c) Distribution of word lengths in user | (d) Distribution of word lengths in assistant
messages with an average of 31.5 words. | messages with an average of 210.5 words.
Figure 4: Characteristics of SparklesDialogueVG.

###  4.2 Statistics and characteristics

We collect two subsets to construct a robust and diverse dataset:
SparklesDialogueCC and SparklesDialogueVG. The respective detailed
descriptions, provided in MiniGPT-4 [61] and SVIT [58], correspond to image
sources from Conceptual Captions (CC) [42] and Visual Genome (VG) [25].
SparklesDialogueVG is of high quality as the VG image descriptions generated
by GPT-4 benefit from human-annotated captions, objects, and regions [58]. On
the other hand, SparklesDialogueCC enriches SparklesDialogue by drawing from a
more extensive set of images - 3.3 million in CC compared to 0.1 million in
VG. However, the CC image descriptions are generated by a multimodal model
with image features but not human annotations and are more prone to object
hallucination issues [61]. Our ablation study elaborated in subsection 5.3
demonstrates that combining these two subsets improves SparklesChat’s capacity
for understanding and reasoning across images and text. SparklesEval
emphasizes more on accuracy and is thus constructed using the same source as
SparklesDialogueVG.

Table 2 provides the data statistics for SparklesDialogue and SparklesEval.
SparklesDialogueCC comprises 4.5K dialogues, each consisting of at least two
images spanning two conversational turns. On the other hand,
SparklesDialogueVG includes 2K dialogues, each with at least three distinct
images across two turns. SparklesEval includes 150 dialogues, with one-third
containing two images in both the first and second conversational turns.

Figure 4 shows the characteristics of our dataset using SparklesDialogueVG as
a representative subset. We explore key elements such as the root verb-noun
pairs in user messages, a word cloud of assistant messages, and the length
distributions. The questions from users are diverse, ranging from generating
text materials to seeking advice or discussing the relationships between
images, such as comparison and connection. The dialogues span various real-
world topics, including the environment, nature, life, cities, etc. The high
average word count in assistant messages suggests that the responses in
SparklesDialogue are thorough and detailed. For details on extracting root
verb-noun pairs and its visualization based on image count in each turn,
please refer to Appendix D.

###  4.3 GPT-assisted evaluation: SparklesEval

While previous research, such as visual storytelling, has leaned toward human
evaluations as superior to quantitative measures, these evaluations are often
subjective, costly, and time-consuming [18]. Inspired by recent advancements
in LLMs, which have shown consistency with human assessment to assess the
quality of outputs [59], we devised SparklesEval, an evaluation benchmark
assisted by GPT-4 in data construction and evaluation. This approach enables a
quantitative assessment of a model’s conversational competence across multiple
images and dialogue turns. The evaluation prompt for SparklesEval is given in
Appendix E.

In our evaluation, we provide the GPT models with the complete dialogue and
corresponding visual information formatted as captions. Our judge models are
designed to assess a single dialogue per prompt, a strategy that eliminates
position bias and improves evaluation efficiency. Position bias refers to
potential favor given to certain positions when multiple dialogues are
assessed within a single prompt [59]. This approach is more efficient because
it avoids recalculating combined scores when evaluating multiple dialogues.

Our approach differs from prior GPT-assisted evaluations, which typically
prompt the GPT models to produce a final score along with a rationale [32,
59]. Instead, we prompt the GPT models to assess the dialogue based on three
distinct criteria across two turns, providing reasons and rating on a scale of
1 to 10 for each assessment. The three criteria are as follows:

(C1) Image understanding and reasoning: Assess the assistant’s proficiency in
accurately identifying and describing objects, contexts, and relationships
within and across the images.
(C2) Cross-image and cross-turn coherence: Evaluate the assistant’s ability to
maintain consistent understanding across multiple images and dialogue turns.
(C3) Relevance and completeness of responses: Determine the extent to which
the assistant’s responses are directly related to the user’s inquiries and the
images’ content, and whether the responses provide comprehensive and detailed
answers.

Following this, we ask the GPT models to assign a combined score for each
turn. For each model’s evaluation results, we gather scores for three criteria
across two turns. First, we compute the mean scores for all criteria over
evaluation samples. Next, we calculate the combined scores A1 and A2 by
averaging their respective criteria scores, namely $A1=mean(C1,C2,C3)$ for the
first turn and $A2=mean(C1,C2,C3)$ for the second turn. We refrain from using
the A1 and A2 scores provided by the judge models, as their calculations may
be inaccurate. Ultimately, we derive a final overall score by averaging A1 and
A2. Through this methodology, our evaluation is more holistic and
interpretable. To encourage diversity in evaluation, SparklesEval was curated
by analyzing the verb-noun distribution in user questions and selecting those
that appear only once.

Figure 5: Comparison between SparklesChat (left) and MiniGPT-4 (right) on
examples of NLVR2 and BISON.

##  5 Experiments

###  5.1 Zero-shot evaluation on vision-and-language tasks

We chose two vision-and-language tasks, binary image selection and visual
reasoning, to evaluate zero-shot understanding and reasoning capabilities over
multiple images.

#### Binary image selection on BISON

The Binary Image Selection task measures a model’s ability to select the
correct image from a pair given a text query that describes one of them [16].
The model’s performance is assessed in terms of binary classification
accuracy. For this task, 150 examples were randomly sampled from the COCO-
BISON dataset111https://github.com/facebookresearch/binary-image-
selection/blob/main/annotations/bison_annotations.cocoval2014.json.

#### Visual reasoning with natural language on NLVR2

The evaluation of the Visual Reasoning with Natural Language task assesses the
model’s ability to predict whether a sentence is true about a pair of images
[45]. This task addresses the challenge of compositional visual reasoning on
relations, comparisons, and quantities. The NLVR2 dataset [45] was used for
this evaluation, with 150 examples randomly sampled from the public balanced
test set222https://github.com/lil-
lab/nlvr/blob/master/nlvr2/data/balanced/balanced_test1.json.

#### Evaluation protocol and prompt design

Models are evaluated on these tasks without any additional training. Inspired
by [24], we used a simple prompt, "Let’s think step by step", to facilitate
step-by-step reasoning before answering each question. We used the phrase
"Therefore, the answer is" to prompt the answer. Instead of using a two-stage
prompting as in [24], we combined the reasoning extraction and answer
extraction stages into a single prompt: "Please start your response with
’Let’s think step by step.’ and end with ’Therefore, the answer is’". For the
full evaluation prompt, please refer to Appendix E. We regenerated the
response if the model failed to follow the instructions to output responses in
the specified format. This approach ensures an unambiguous response and allows
us to extract a potential answer from the text following the last occurrence
of "Therefore".

Table 3: Model comparison on BISON, NLVR2 and SparklesEval. For the BISON and NLVR2 benchmarks, the evaluation metric is accuracy. For SparklesEval, scores are rated from 1 to 10. MiniGPT-4* represents our re-implement of MiniGPT-4 following the same experimental setup as SparklesChat. We investigate training models on different data sources, including detailed descriptions, complex reasoning, and dialogue data. Notably, GPT-4, a text-based reference LLM, achieves high scores on SparklesEval largely due to its use of detailed ground-truth annotations. Model | Data | BISON | NLVR2 | SparklesEval
---|---|---|---|---
Score | Turn1 | Turn2
A1 | C1 | C2 | C3 | A1 | C1 | C2 | C3
GPT-4 | - | - | - | 9.26 | 9.26 | 9.23 | 9.18 | 9.38 | 9.26 | 9.25 | 9.15 | 9.38
MiniGPT-4 |  description [61] | 46.0% | 51.3% | 3.91 | 3.55 | 3.67 | 3.53 | 3.44 | 4.28 | 4.38 | 4.21 | 4.23
MiniGPT-4* |  description [61] | 51.3% | 46.7% | 3.50 | 3.14 | 3.27 | 3.16 | 3.00 | 3.86 | 3.97 | 3.81 | 3.81
SparklesChat |  description [32] | 52.0% | 48.0% | 3.06 | 2.64 | 2.79 | 2.67 | 2.46 | 3.48 | 3.76 | 3.40 | 3.29
SparklesChat |  reasoning [32] | 52.7% | 54.0% | 6.71 | 6.55 | 6.63 | 6.42 | 6.59 | 6.87 | 6.89 | 6.73 | 6.98
SparklesChat | SparklesDialogue | 56.7% | 58.0% | 8.56 | 8.76 | 8.81 | 8.67 | 8.81 | 8.35 | 8.37 | 8.28 | 8.41

###  5.2 Comparison of model performance

Table 3 compares the performance of different models on BISON, NLVR2, and
SparklesEval evaluation datasets. We adapt all models to accept multiple
images as input and evaluate them under the same settings for fair
comparisons. We compare our SparklesChat with MiniGPT-4. Additionally, we
provide the results of our re-implemented version of MiniGPT-4, denoted as
MiniGPT-4*, under the same experimental settings as SparklesChat. We
investigate training these models on different data sources, including
detailed description data from MiniGPT-4 [61] and LLaVA [32], complex
reasoning data from LLaVA [32], and dialogue data from our SparklesDialogue.
Side-by-side comparisons of example outputs for SparklesChat and MiniGPT-4 on
SparklesEval, and on BISON and NLVR2 can be found in Figure 2 and Figure 5
respectively. What’s more, we provide a detailed evaluation of the performance
of GPT-4, MiniGPT-4, and SparklesChat on SparklesEval with three different
versions of judge models in Appendix C.

We compare our dialogue dataset SparklesDialogue with related description and
reasoning datasets from LLaVA [32] using data formats similar to
SparklesDialogue with interleaved images and text. We have eliminated samples
from train sets that overlapped with evaluation sets. For the results in Table
3, we can see that when SparklesChat is trained on description data [32], it
exhibits lower performance compared to when trained on dialogue data. However,
it still outperforms MiniGPT-4* on the BISON and NLVR2 tasks. When
SparklesChat is trained on reasoning data [32], it achieves improved
performance over models trained on description data on all metrics. This
suggests that incorporating reasoning data during training enhances the
model’s performance.

SparklesChat, trained on our SparklesDialogue, outperforms all other models on
BISON and NLVR2, achieving an accuracy of 56.7% and 58.0%, respectively. This
demonstrates the effectiveness of SparklesChat in handling tasks that require
fine-grained visual grounding and compositional visual reasoning over two
images. Moreover, SparklesChat outperforms multimodal models significantly on
the SparklesEval benchmark, with an overall score of 8.56 out of 10. In
comparison, models trained on description data have an approximate score of 3,
and models trained on reasoning data achieve a score of 6.71. SparklesChat
attains the highest scores among multimodal models in both the first and
second turns across all criteria. This indicates its superior ability in image
understanding and reasoning, maintaining cross-image and cross-turn coherence,
and generating relevant and complete responses. GPT-4 achieves the highest
score of 9.26, mainly due to its utilization of detailed ground-truth
annotations. SparklesChat’s score is about 92% of the GPT-4 score,
underscoring SparklesChat’s conversational competence across various images
and dialogue turns.

Table 4: Ablation studies on BISON, NLVR2 and SparklesEval. We study the effects of training SparklesChat using variants of SparklesDialogue on different ratios of dialogue turns and using different subsets. For the BISON and NLVR2 benchmarks, the evaluation metric is accuracy. For SparklesEval, scores are rated from 1 to 10. Data |  Turns Ratio | BISON | NLVR2 | SparklesEval
---|---|---|---|---
Score | Turn1 | Turn2
A1 | C1 | C2 | C3 | A2 | C1 | C2 | C3
SparklesDialogue (CC+VG) | 1:0 | 57.3% | 55.3% | 8.50 | 8.65 | 8.70 | 8.52 | 8.73 | 8.35 | 8.38 | 8.24 | 8.44
0:1 | 50.7% | 46.7% | 8.24 | 8.24 | 8.23 | 8.18 | 8.32 | 8.24 | 8.24 | 8.15 | 8.32
1:1 | 59.3% | 51.3% | 8.73 | 8.80 | 8.83 | 8.66 | 8.91 | 8.65 | 8.62 | 8.55 | 8.79
1:2 | 49.3% | 51.3% | 8.43 | 8.54 | 8.57 | 8.43 | 8.63 | 8.31 | 8.28 | 8.21 | 8.43
2:1 | 56.7% | 58.0% | 8.56 | 8.76 | 8.81 | 8.67 | 8.81 | 8.35 | 8.37 | 8.28 | 8.41
3:1 | 50.7% | 48.7% | 8.45 | 8.69 | 8.74 | 8.52 | 8.83 | 8.20 | 8.18 | 8.08 | 8.33
SparklesDialogueCC | 2:1 | 44.7% | 53.3% | 8.18 | 8.26 | 8.29 | 8.16 | 8.33 | 8.10 | 8.10 | 8.00 | 8.20
SparklesDialogueVG | 2:1 | 54.7% | 52.0% | 8.59 | 8.71 | 8.76 | 8.60 | 8.78 | 8.47 | 8.47 | 8.35 | 8.60

###  5.3 Ablation studies

We study the effect of training SparklesChat using data variants while keeping
other training parameters constant. We assess performance on BISON, NLVR2, and
SparklesEval benchmarks in Table 4, which mainly evaluate models’ ability to
describe, reason, and converse across images.

#### Effect of dialogue turns in SparklesDialogue.

We first train models with individual dialogue turns. The results in Table 4
show that the model trained solely on the first turn performs better than
solely on the second across all metrics. Furthermore, this model outperforms
those trained on the baseline datasets, demonstrating that our dataset boosts
reasoning and conversational abilities even when used in isolation with just
the first turn. Conversely, training only with the second dialogue turns
reduces scores on BISON and NLVR2. This could stem from the extended prompts
in the second turn, which includes the content of the first turn, making them
less aligned with the short prompt format favored by BISON and NLVR2. Then, we
train models with SparklesDialogue blending with the same ratios of samples
constructed from two dialogue turns. The results are better than only training
with the second turn in all metrics, while worse than only training from the
first turn in the task of NLVR2. An increase in the sampling ratios of the
second turn data results in a performance drop as expected. Thus, we increase
the sampling ratio of the first-turn data until we cannot observe performance
boosting. We finally settled on a 2:1 ratio for the first turn to the second
turn as our default setting as it achieves balanced good performance across
all benchmarks.

#### Effect of subsets of SparklesDialogue.

Our model has been trained on two subsets of SparklesDialogue:
SparklesDialogueCC and SparklesDialogueVG. We observe from Table 4 that the
model trained on SparklesDialogueVG outperforms that trained on
SparklesDialogueCC in both the BISON and SparklesEval evaluations, scoring
54.7% and 8.59, respectively, compared to 44.7% and 8.18. This enhanced
performance is partly due to the higher quality of SparklesDialogueVG, which
benefits from human-annotated data as discussed in subsection 4.2. It’s worth
noting that SparklesDialogueVG and SparklesEval use the same sources of images
and captions, which could partially account for the higher score achieved by
SparklesDialogueVG on SparklesEval. Both subsets demonstrate similar efficacy
on the NLVR2 test. Combining both subsets yields higher performance on the
BISON and NLVR2 tests, scoring 56.7% and 58.0% respectively. This surpasses
the scores achieved by using either subset alone. In addition, the model
trained on the combined dataset performs comparably to SparklesDialogueVG in
the SparklesEval test, scoring 8.56 versus 8.59. This suggests that combining
SparklesDialogueVG’s high-quality data and SparklesDialogueCC’s diverse data
results in a more robust and versatile dataset for enhancing models’
capabilities in understanding and reasoning across images and text.

Figure 6: Demonstration of SparklesChat to create a story and a dialogue that
connects places and characters. Figure 7: Demonstration of SparklesChat to
compose a song containing two scenes and generate a song title inspired by
another image. Figure 8: Demonstration of SparklesChat to describe and reason
about different groups of images.

###  5.4 Demonstrations and applications

We conducted qualitative demonstrations to showcase SparklesChat’s broad
applications in free-form scenarios by asking questions such as: “Create a
story that takes place in *○for the characters depicted in *○.”, “Imagine a
dialogue between Harry Potter and *○that takes place in the scene of *○.”,
“Create a song where the scene twists from *○to *○.”, “Create a title for this
song that takes inspiration from *○.”. The visualization of results is shown
in Appendix A.

##  6 Limitations, future works, and conclusion

We discuss some limitations of this work to inspire future research in this
field. First, SparklesChat shares common drawbacks with large language models,
such as being out-of-date in its knowledge, sometimes providing inaccurate
information, and having limited context length and inference speed. Potential
solutions may include regular updates to the model’s knowledge base and fine-
tuning with more reliable data sources. Second, SparklesChat inherits
weaknesses from vision models, such as inaccurate object recognition,
people/places identification, or visual relationships reasoning. This calls
for a more powerful visual perception model, and training on more well-aligned
image-text datasets. Third, SparklesChat occasionally encounters difficulties
maintaining multi-image and multi-turn consistency. Specifically, the model
may lose the context of prior images after several dialogue turns or mix up
the contents of different images. Potential solutions involve advanced model
designs in position encoding and attention mechanisms to enhance the model’s
consistency in recalling historical images and dialogues. Fourth,
SparklesDialogue primarily concentrates on natural images, which limits its
versatility in handling text-rich images such as charts, tables, and receipts,
as well as domain-specific images such as medical scans, math illustrations,
and satellite photos. Moreover, the dialogues in SparklesDialogue do not cover
all possible user scenarios. Therefore, broadening the dataset to cover more
diverse image types and user cases is a direction for future work. Fifth, the
reliability of SparklesEval is tied to the capabilities of current GPT models.
This limitation can be mitigated by incorporating more robust judge models and
the assistance of human evaluators. Lastly, further safety considerations are
needed to mitigate potential misuse of the model. Future works addressing
these issues should make for a more reliable and robust system.

In conclusion, this work unlocks multimodal instruction-following models’
capabilities in open-ended dialogues involving multiple images. We introduced
SparklesChat, a model designed to handle word-level text interactions in a
multimodal context, offering natural conversational flow and direct context
awareness. To facilitate the training of SparklesChat, we also presented
SparklesDialogue, the first machine-generated dialogue dataset tailored for
multi-image and word-level text interactions. Furthermore, we proposed
SparklesEval, a specialized benchmark for quantitatively assessing a model’s
multimodal conversational competence. Experimental results demonstrated
SparklesChat’s superiority over existing models in both standard vision-and-
language tasks and the newly-introduced SparklesEval benchmark. We also
conducted qualitative demonstrations to showcase the model’s broad
applications in free-form scenarios.

## References

  * [1] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al.  Cm3: A causal masked multimodal model of the internet.  arXiv preprint arXiv:2201.07520, 2022.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736, 2022.
  * [3] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  arXiv preprint arXiv:2308.01390, 2023.
  * [4] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset, 2022.
  * [5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021.
  * [6] Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang.  Visual instruction tuning with polite flamingo.  arXiv preprint arXiv:2307.01003, 2023.
  * [7] Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei.  Textdiffuser: Diffusion models as text painters.  arXiv preprint arXiv:2305.10855, 2023.
  * [8] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  arXiv preprint arXiv:2209.06794, 2022.
  * [9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [10] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  arXiv preprint arXiv:2305.06500, 2023.
  * [11] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra.  Visual dialog.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326–335, 2017.
  * [12] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  arXiv preprint arXiv:2211.07636, 2022.
  * [13] Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin.  MMDialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7348–7363, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * [14] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010, 2023.
  * [15] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.  Multimodal-gpt: A vision and language model for dialogue with humans.  arXiv preprint arXiv:2305.04790, 2023.
  * [16] Hexiang Hu, Ishan Misra, and Laurens Van Der Maaten.  Evaluating text-to-image matching using binary image selection (bison).  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [17] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.  Language is not all you need: Aligning perception with language models.  arXiv preprint arXiv:2302.14045, 2023.
  * [18] Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al.  Visual storytelling.  In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233–1239, 2016.
  * [19] Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu.  Unifying multimodal transformer for bi-directional image and text generation.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1138–1147, 2021.
  * [20] Yupan Huang, Zhaoyang Zeng, and Yutong Lu.  Be specific, be clear: Bridging machine and human captions by scene-guided transformer.  In Proceedings of the 2021 Workshop on Multi-Modal Pre-Training for Multimedia Understanding, pages 4–13, 2021.
  * [21] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976–12985, 2021.
  * [22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  * [23] Nikita Kitaev and Dan Klein.  Constituency parsing with a self-attentive encoder.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [24] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  Advances in neural information processing systems, 35:22199–22213, 2022.
  * [25] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * [26] Nyoungwoo Lee, Suwon Shin, Jaegul Choo, Ho-Jin Choi, and Sung-Hyon Myaeng.  Constructing multi-modal dialogue dataset by replacing text with semantically relevant images.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 897–906, 2021.
  * [27] Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, and Ho-Jin Choi.  Dialogcc: Large-scale multi-modal dialogue dataset.  arXiv preprint arXiv:2212.04119, 2022.
  * [28] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu.  Mimic-it: Multi-modal in-context instruction tuning.  arXiv preprint arXiv:2306.05425, 2023.
  * [29] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * [30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * [31] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [33] Yuxian Meng, Shuhe Wang, Qinghong Han, Xiaofei Sun, Fei Wu, Rui Yan, and Jiwei Li.  Openvidial: A large-scale, open-domain dialogue dataset with visual contexts.  arXiv preprint arXiv:2012.15015, 2020.
  * [34] Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende.  Image-grounded conversations: Multimodal context for natural question and response generation.  In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, November 2017.
  * [35] Openai.  GPT-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [36] Vicente Ordonez, Girish Kulkarni, and Tamara L Berg.  Im2text: Describing images using 1 million captioned photographs.  In NeurIPS, 2011.
  * [37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.
  * [40] Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [41] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [43] Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston.  Image-chat: Engaging grounded conversations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414–2429, Online, July 2020. Association for Computational Linguistics.
  * [44] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355, 2023.
  * [45] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  In ACL, pages 6418–6428, 2019.
  * [46] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 2016.
  * [47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [48] Moskvoretskii Viktor, Frolov Anton, and Kuznetsov Denis.  Imad: Image-augmented multi-modal dialogue, 2023.
  * [49] Shuhe Wang, Yuxian Meng, Xiaoya Li, Xiaofei Sun, Rongbin Ouyang, and Jiwei Li.  Openvidial 2.0: A larger-scale, open-domain dialogue generation dataset with visual contexts.  arXiv preprint arXiv:2109.12761, 2021.
  * [50] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language model with self generated instructions, 2022.
  * [51] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  arXiv preprint arXiv:2109.01652, 2021.
  * [52] Zhiyang Xu, Ying Shen, and Lifu Huang.  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  arXiv preprint arXiv:2212.10773, 2022.
  * [53] Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo.  Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training.  Advances in Neural Information Processing Systems, 34:4514–4528, 2021.
  * [54] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [55] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al.  Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark.  arXiv preprint arXiv:2306.06687, 2023.
  * [56] Xiaoxue Zang, Lijuan Liu, Maria Wang, Yang Song, Hao Zhang, and Jindong Chen.  PhotoChat: A human-human dialogue dataset with photo sharing behavior for joint image-text modeling.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6142–6152, Online, August 2021. Association for Computational Linguistics.
  * [57] Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong.  What matters in training a gpt4-style language model with multimodal inputs?  arXiv preprint arXiv:2307.02469, 2023.
  * [58] Bo Zhao, Boya Wu, and Tiejun Huang.  Svit: Scaling up visual instruction tuning.  arXiv preprint arXiv:2307.04087, 2023.
  * [59] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685, 2023.
  * [60] Yinhe Zheng, Guanyi Chen, Xin Liu, and Jian Sun.  MMChat: Multi-modal chat dataset on social media.  In Proceedings of The 13th Language Resources and Evaluation Conference. European Language Resources Association, 2022.
  * [61] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
  * [62] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal C4: An open, billion-scale corpus of images interleaved with text.  arXiv preprint arXiv:2304.06939, 2023.

Appendix

##  Appendix A Demonstrations and applications

We conducted qualitative demonstrations to showcase the model’s wide
applications in free-form scenarios in Figure 6, Figure 7, and Figure 8.

##  Appendix B Implementation details

We implemented SparklesChat on the MiniGPT-4 codebase
[61]333https://github.com/Vision-CAIR/MiniGPT-4, which is derived from Vicuna
[9]. We refer to MiniGPT-4’s efficient fine-tuning process and tune
SparklesChat using 1,500 training steps with a batch size of 8, based on
MiniGPT-4’s first-stage pretrained model. Our training data of
SparklesDialogue is sampled with the same ratio from SparklesDialogueCC and
SparklesDialogueVG, and with sampling ratios of 2 and 1 from the first and
second turns of dialogues, respectively.

During instruction-tuning, we follow MiniGPT-4 to use <Img><ImageHere></Img>
to represent images [61]. In practice, all tags of <ImageHere> are replaced by
the ViT visual features produced by a linear projection layer. Tags of <Img>
and </Img> are language tokens that serve as signals for the start and end of
images. A system message $\mathbf{X}_{\texttt{system}}$ is appended to the
beginning of each prompt. We also append Human: and Assistant: before each
user and assistant messages to equip the model with conversation capability.
System, user, and assistant messages are separated by a separator <SEP>. The
system message $\mathbf{X}_{\texttt{system}}$ = Give the following image:
<Img>ImageContent</Img>. You will be able to see the image once I provide it
to you. Please answer my questions. The separator <SEP> = ###.

We tailored the OpenAI’s GPT-4 API (gpt-4-0613) parameters to balance
diversity and quality for constructing SparklesDialogue and SparklesEval. We
set the temperature and top_p parameters to 1.0, the max_tokens parameter to
2048, and both the frequency_penalty and presence_penalty parameters to 0.0.
In each query to the GPT-4 API, the “system” role was allocated the default
instruction You are a helpful assistant. As of July 2023, the cost for
generating 1,000 tokens was $0.06 for outputs and $0.03 for inputs within an
8K context444https://openai.com/pricing, leading to a total dataset generation
cost of approximately $500. The cost of evaluating a model on SparklesEval is
approximately $1.4 and $14 using gpt-3.5-turbo-0613 and gpt-4-0613,
respectively.

##  Appendix C Judging with different versions of GPT models

As of July 2023, while it is widely recognized that employing gpt-4 as a judge
model outperforms alternatives such as gpt-3.5-turbo, the cost of using gpt-4
is significantly higher. Therefore, we also provide scores generated by
gpt-3.5-turbo as a reader reference, although we strongly recommend utilizing
gpt-4 or more advanced future models as reliable judges. We adopt the latest
version gpt-4-0701* as our default judge model.

We evaluate GPT-4, MiniGPT-4, and SparklesChat using SparklesEval, leveraging
three versions of judge models, as presented in Table 5. Both MiniGPT-4 and
SparklesChat generate responses based on the question and accompanying visual
image. At the same time, GPT-4 is a reference LLM that only uses textual
information, including the question, the ground-truth bounding boxes, and
captions. From the table, we observe that the more advanced judge models -
gpt-4-0613 and gpt-4-0701* \- provide higher scores compared to the older
gpt-3.5-turbo-0613 when assessing both GPT-4 and OurModel (approximately nine
versus eight). However, these advanced judge models yield considerably lower
scores for MiniGPT-4 (about three versus five). GPT-4 achieves the highest
score of 9.26 out of 10 when evaluated by the default gpt-4-0701* mainly due
to its use of detailed ground-truth annotations. Nevertheless, it’s worth
noting LLM judge models may display a self-enhancement bias, favoring the
responses they generate [59]. In contrast, MiniGPT-4 performs behind with a
score of just 3.91. SparklesChat achieves a score of 8.56 - about 92% of the
GPT-4 score - demonstrating SparklesChat’s efficacy in generating responses
that are not only relevant and complete but also exhibit cross-image and
cross-turn coherence.

Table 5: Evaluation results on SparklesEval with different judge models. The version of gpt-4-0701* refers to API version 2023-07-01-preview for the GPT-4 model.  Model |  Judge Model Version | Score | Turn1 | Turn2
---|---|---|---|---
A1 | C1 | C2 | C3 | A2 | C1 | C2 | C3
GPT-4 | gpt-3.5-turbo-0613 | 8.48 | 8.61 | 8.69 | 8.18 | 8.95 | 8.35 | 8.27 | 8.08 | 8.68
gpt-4-0613 | 9.51 | 9.50 | 9.53 | 9.37 | 9.60 | 9.53 | 9.49 | 9.46 | 9.64
gpt-4-0701* | 9.26 | 9.26 | 9.23 | 9.18 | 9.38 | 9.26 | 9.25 | 9.15 | 9.38
MiniGPT-4 | gpt-3.5-turbo-0613 | 5.51 | 5.46 | 6.11 | 4.78 | 5.48 | 5.55 | 5.92 | 5.23 | 5.51
gpt-4-0613 | 3.31 | 3.11 | 3.12 | 3.09 | 3.10 | 3.51 | 3.57 | 3.40 | 3.56
gpt-4-0701* | 3.91 | 3.55 | 3.67 | 3.53 | 3.44 | 4.28 | 4.38 | 4.21 | 4.23
SparklesChat | gpt-3.5-turbo-0613 | 8.37 | 8.51 | 8.59 | 8.04 | 8.89 | 8.24 | 8.16 | 7.92 | 8.64
gpt-4-0613 | 8.82 | 8.75 | 8.78 | 8.59 | 8.89 | 8.88 | 8.89 | 8.79 | 8.97
gpt-4-0701* | 8.56 | 8.76 | 8.81 | 8.67 | 8.81 | 8.35 | 8.37 | 8.28 | 8.41

##  Appendix D Verb-noun distribution analysis

For verb-noun distribution, we follow Self-instruct [50] to extract the verb
closest to the root and its first direct noun object and plot the top 20 most
common root verbs and their top 4 direct noun objects. We use the Berkeley
Neural Parser555https://parser.kitaev.io/ [23] to parse user messages. We
mainly focus on the last sentence of each message because it usually contains
the question. If we can’t extract the verb-noun pair from it, we look at the
first sentence instead. For SparklesDialogueVG, we visualize the verb-noun
distributions regarding different numbers of images in each turn in Figure 9.

|
---|---
(a) The first turn | (b) The second turn
|
(c) The first turn, two images | (d) The first turn, three images
Figure 9: Root verb-noun distributions of SparklesDialogueVG.

##  Appendix E Evaluation Details

The prompt formats to evaluate NLVR2 and BISON datasets are presented in Table
6. The prompt format of GPT-assisted evaluation on SparklesEval is presented
in Table 7.

The image source of COCO-BISON is COCO images. The image source of
SparklesDialogueCC is Conceptual Captions, which should have no overlap with
COCO. However, our SparklesDialogueVG originates from the Visual Genome, which
includes a subset of COCO images. We carefully eliminate any overlapping
images to ensure no overlap between the training and evaluation data. The
images in the NLVR2 dataset are sourced from Google Images, distinct from our
SparklesDialogueVG’s image source of the Visual Genome [25] and primarily
feature images from Flickr.

Table 6: Prompt Formats to evaluate NLVR2 and BISON datasets.  NLVR2 |
---|---
Carefully examine a pair of images: the left IMAGE#1<Img><ImageHere></Img> and the right IMAGE#2<Img><ImageHere></Img>. |
Determine whether the following statement is true about the pair of images: |
’{Statement}’ |
Jointly reasoning about the statement grounded in IMAGE#1 and IMAGE#2. |
The task requires compositional joint reasoning, including quantities, comparisons, and relations. Let’s think step by step. |
Please start your response with "Let’s think step by step." and end with "Therefore, the answer (TRUE or FALSE) is". |
BISON |
Carefully examine the two similar images of IMAGE#1<Img><ImageHere></Img> and IMAGE#2<Img><ImageHere></Img>. |
Given the following caption, you must select which of two images best matches the caption. |
The caption is: ’{Caption}’. |
This task requires fine-grained visual reasoning between the caption and each image. Let’s think step by step. |
Please start your response with "Let’s think step by step." and end with "Therefore, the answer (IMAGE#1 or IMAGE#2) is". |
Table 7: Prompt Format for SparklesEval Evaluation.

Users will interact with a conversational assistant. The assistant is designed to understand, analyze, and reason about multiple images across two turns of conversation. The assistant is expected to provide highly helpful and exceptionally detailed answers providing comprehensive reasoning regarding the visual content of the images. |
---|---
Below are images represented by their image ids and captions (delimited by triple quotes):  ⬇ ```json {Target Image Descriptions} ⬇ ``` |
Next is a dialogue between a user and the assistant regarding the images above:  ⬇ ``` ###User Q1: |
{Q1} |
###Assistant A1: |
{A1} |
###User Q2: |
{Q2} |
###Assistant A2: |
{A2} ⬇ ``` |
Your task as an impartial judge is to evaluate the responses (A1 and A2) provided by the assistant to the user’s questions. |
Please rate the following three criteria C1, C2, and C3 on a scale of 1-10 for A1 and A2 separately, where a higher score indicates better overall performance: |
(C1) Image Understanding and Reasoning: This measures the assistant’s ability to accurately identify and describe objects, context, and relationships within and between the images. |
(C2) Cross-Image and Cross-Turn Coherence: This evaluates the assistant’s ability to maintain a consistent understanding across multiple images and dialogue turns. |
(C3) Relevance and Completeness of Responses: This assesses whether the assistant’s responses are directly related to the user’s inquiries and the images’ content, and whether the responses provide thorough, detailed answers. |
Begin your evaluation by providing a short explanation for each criterion. Be as objective as possible. After providing your explanation, rate the response on a scale of 1 to 10 by strictly following the format below (note that "5" and "…" are placeholders):  ⬇ ``` * Evaluating A1 |
\- (C1) Explanation: "…" Rating: [[5]] |
\- (C2) Explanation: "…" Rating: [[5]] |
\- (C3) Explanation: "…" Rating: [[5]] |
Therefore, the overall rating of A1 is [[5]] |
Evaluating A2 |
\- (C1) Explanation: "…" Rating: [[5]] |
\- (C2) Explanation: "…" Rating: [[5]] |
\- (C3) Explanation: "…" Rating: [[5]] |
Therefore, the overall rating of A2 is [[5]]  ⬇ ``` |

##  Appendix F LLM-instructed single dialogue generation for
SparklesDialogueVG

For SparklesDialogueVG, we generate one two-turn dialogue at a time, with the
first turn incorporating two or three images. We derive the demonstration
dialogues from SparklesDialogueCC to encourage diversity. However, to minimize
redundancy, we retain only those dialogues with unique verb-noun combinations
in the user questions. This results in pools of 661 and 441 demonstration
dialogues for conversations incorporating two or three images in the first
turn, respectively. We pull from an expansive collection of roughly 100,000
image-text pairs for this dataset. We randomly select four candidates each
time, and they are not reused by excluding them from future selections.

We first present our designed prompt for LLM-instructed Single Dialogue
Generation to generate SparklesDialogueCC in Table 8. Then, we show a case of
the Dialogue Demonstration and Candidate Image Descriptions to construct the
prompt. Finally, we show the corresponding generated dialogue using the
example prompt.

Table 8: Prompt for LLM-instructed Single Dialogue Generation.

System: You are a helpful assistant. |
---|---
Users will interact with a conversational assistant that has advanced capabilities of understanding, analyzing, and reasoning about images. This includes discussing a variety of real-world concepts, objects, and entities, generating a range of text materials, seeking advice, guidance, or assistance, and much more. |
Below are an illustrative dialogue presented in a JSON format. The dialogue represents a meaningful conversation between a "user" and the "assistant" regarding multiple images. Each "user" message contains an "image_ids" field recording the IDs of newly selected images. The images are referred to in the "content" field as IMAGE#image_id. ⬇ ```json {Dialogue Demonstration} ⬇ ``` Please note that the user contents in the JSON above may be a counterexample that reveals the content of images and can be answered without looking at the images. Please make sure not to reveal the content of the images or describe the images in the user messages in the conversation that follows. |
Please note that the specific "image_ids" and "content" in the JSON above are for illustrative purposes only. The actual candidate images are shown below delimited by triple quotes, each accompanied by an image ID and a caption. Avoid using phrases similar to ’caption’ and ’description’ in your dialogue as if the user and the assistant have visual capabilities. ⬇ ```json {Candidate Image Descriptions} ⬇ ``` Each dialogue consists of four messages: |
1\. A user examines all candidate images, selects {Number of Images} highly relevant images, and sends a reasonable and creative message to the assistant. |
2\. Once the images are provided, the assistant thoroughly perceives and comprehends them, responding with highly helpful and exceptionally detailed answers that provide comprehensive reasoning regarding the visual content of the images. |
3\. Considering the past dialogue, the user chooses other candidate images for further inquiry. The user should refer to both the newly selected images and those mentioned earlier in the same dialogue. |
4\. The assistant provides a highly helpful and exceptionally detailed answer providing comprehensive reasoning regarding the visual content of the images. |
The following is a dialogue between the user and the assistant, adhering to the given JSON format. |
Make sure to formulate accurate and diverse "content" that does not follow the illustrative dialogues. And remember to develop the last "content" even though it is shown as "…" in the JSON format provided above. |
Dialogue Demonstration ⬇ [[{'role': 'user', 'image_ids': $\mathbf{I}^{1,1}$, 'content': $\mathbf{X}_{\texttt{q}}^{1,1}$}, {'role': 'assistant', 'content': $\mathbf{X}_{\texttt{a}}^{1,1}$}, {'role': 'user', 'image_ids': $\mathbf{I}^{1,2}$, 'content': $\mathbf{X}_{\texttt{q}}^{1,2}$}, {'role': 'assistant', 'content': '…'}]] Candidate Image Descriptions ⬇ [{'image_id': $\mathbf{J}^{1}$, 'caption': $\mathbf{C}^{1}$}, …… {'image_id': $\mathbf{J}^{4}$, 'caption': $\mathbf{C}^{4}$}] Number of Images is "two" or "three". |

#### Example of dialogue demonstration

We visualize the images corresponding to image IDs in the dialogues in Figure
10 for reference, while these visual images were not sent to GPT-4 for data
generation. Note that we abbreviate the message content of the assistant in
the second turn as “…” to save space, considering that the previous message
contents have provided enough demonstrations.

|  |
---|---|---
2675 | 1081 | 2679
Figure 10: Reference images corresponding to the image IDs in the
demonstration dialogues in Appendix F. These images were not sent to GPT-4 for
data generation.

⬇

1```json

2[

3[

4 {’role’: ’user’, ’image_ids’: [2675, 1081], ’content’: ’How would you
connect the woman in IMAGE#2675 with the clothing items in IMAGE#1081?’},

5 {’role’: ’assistant’, ’content’: ’In IMAGE#2675, we see a woman dressed in
fashionable attire consisting of a grey turtleneck sweater, blue jeans, a
black leather belt, a black leather jacket with fur detailing and black ankle
boots. Her overall outfit looks stylish, suggesting she has good fashion
sense.\\\\\\\n\\\\\\\nIMAGE#1081 represents a group of clothing items hanging
on a rack. It shows an outfit comprising a striped shirt, a brown skirt, and a
pair of black shoes. The clothes and shoes have a sophisticated touch to
them.\\\\\\\n\\\\\\\nConnecting these images, one could infer that the
clothing items could belong to the same woman. Maybe she’s someone who likes
keeping her wardrobe organized or could be in the fashion industry focusing on
outfit planning. Her style in IMAGE#2675 is modern and chic, while the
ensemble in IMAGE#1081 is equally trendy, with classic combinations of stripes
and brown, which many consider a style staple, and the elegant black shoes.
Therefore, we could imagine that the outfit from IMAGE#1081 may be another
stylish option that she might opt for. The clothing items could be a part of
her personal style or they might signal her work attire, given the formal vibe
the striped shirt and brown skirt ensemble radiate.’},

6 {’role’: ’user’, ’image_ids’: [2679], ’content’: ’Looking at IMAGE#2679, how
would you weave it into the narrative of this stylish woman from IMAGE#2675
and the clothes in IMAGE#1081?’},

7 {’role’: ’assistant’, ’content’: ’...’}

8]

9]

10```

#### Example of Candidate Images Descriptions

The image IDs in the dialogues refer to the images in Figure 11.

|  |  |
---|---|---|---
2315390 | 2408549 | 2330601 | 2331159
Figure 11: Candidate images corresponding to the image IDs in the dialogues
generation process in Appendix F. These images were not sent to GPT-4 for data
generation.

⬇

1```json

2[

3 {’image_id’: 2315390, ’caption’: ’The image captures a dynamic scene at a
skate park, specifically a boy mid-air, performing a trick on his skateboard.
The boy appears to be a bit on the dark side, silhouetted against the lighter
sky behind him. He’s adorned in casual skate attire: a sweater, blue jeans,
and athletic shoes. His arms are raised in the air, which adds to the
intensity of the trick he’s executing. Interestingly, he is not wearing a
helmet.\\\nThe skateboard itself is flipped, and it appears to be on its side.
This unusual positioning gives the impression that the skater is performing an
intricate and complex trick. The skateboard has multiple wheels, described as
black, and is noticeably detailed in the image.\\\nThere’s another person
present in the scene, presumably a photographer or a spectator, positioned
toward the right corner of the frame. However, this individual is located
quite close to the edge, suggesting that they are not the main focus of the
photograph.\\\nThe skate park consists of a grey concrete ramp that the boy is
using for his tricks. It stretches across the majority of the bottom part of
the picture, a hard, flat contrast to the dynamic motion taking place above
it. There’s also a metal gate visible in the scene, possibly part of the
boundary or safety measures at the park.\\\nThe backdrop is a vibrant blue sky
with clusters of white clouds scattered across it. It seems to be a bright,
clear day, perfect for outdoor activities. Lastly, a safety net in the
distance lends an additional element of safety to the environment.\\\nOverall,
the photograph encapsulates an exhilarating moment of skill, action, and
athleticism at a bustling skate park, set against a serene, blue-skied day.’},

4 {’image_id’: 2408549, ’caption’: "This image captures a dynamic scene of a
large blue train moving rapidly on railroad tracks. The train’s hue stands in
beautiful contrast with the clear, blue sky overhead. The train is quite long,
stretching almost the entire width of the image, and it appears to be well
maintained, with grey stripes highlighting its design. The train’s lower half
is primarily filled with windows and double doors. Three windows are clearly
visible, each reflecting the bright sunlight.\\\nWithin the train, passengers
can be seen through the windows. Notably, one person dressed in a white
t-shirt is looking out of the window, taking in the scenery or perhaps
observing the vehicle whose side mirror is captured in the frame. The double
doors, one on the left and the other on the right, stand out on the body of
the train. Each door has a number ’2’ inscribed on it.\\\nInterestingly, in
the right section of the image, the side mirror of a car is in the frame,
reflecting a blurry image of another vehicle, further contributing to the
sense of movement in this scene.\\\nThe foreground of the image is filled with
a wide expanse of green grass that contrasts nicely with the railroad tracks
and a nearby road. To the right of the train, there’s a tall pole that rises
high into the image, likely used for mounting signs. In this case, the pole
hosts a railroad crossing sign with lights and a large X on top. There is also
a triangular sign with three lights underneath the X sign, providing important
safety information for approaching vehicles.\\\nBehind the pole, a red metal
barrier is barely visible. It appears to be part of the infrastructure that
surrounds the tracks. With the beautiful sunny sky overhead, this picture
seems to represent a typical day with normal hustle and bustle at this
railroad crossing. The sunlight reflecting off the train windows adds a
stunning glow to the scene.\\\nDespite the fast motion of the train, details
such as the wheels and even the driver’s side view mirror are captured in the
image, emphasizing the skill of the photographer in capturing this dynamic and
detailed snapshot of a moment in time."},

5 {’image_id’: 2330601, ’caption’: ’This image depicts an exciting scene of a
man dressed in a blue and black racing suit, riding a dirt bike on a muddy
track. The man is prominently positioned in the image, seeming to occupy a
considerable portion of it from left to right. His blue helmet, matching his
attire, is clearly visible.\\\nThe motorcycle he’s racing is intricately
detailed. Its prominent front and back black wheels kick up wet mud as they
tear through the track, while the metallic shimmer of the exhaust and the
sturdy grey frame suggest its rugged durability. A number, black in color,
stands out on the side of the bike, and there’s a patch of blue at the bike’s
back that contributes to the cohesive color scheme.\\\nThe rider’s attire
stands out as well. Apart from the matching helmet, he’s wearing a blue and
orange shirt, black pants, and blue and yellow shoes. A black visor on his
helmet and black gloves further accessorize his ensemble. His coat, in shades
of blue and grey, fits snugly, outlining his physique.\\\nThe scene around the
bike is as dynamic as the rider. The track underneath is a dark brown, most
likely a mix of dirt and water, suggestive of recent rain or the challenging
conditions of a dirt bike race. Patches of water and water spots can be seen
at various locations, indicating the wetness of the track and the splashing
caused by the bike speeding through.\\\nMoreover, there’s an evident sense of
motion in the image with water splashing up from the bike and wet sand
scattering in its wake. The ground can be seen in patches, displaying its dark
brown color. Amidst all this action, the bike stands as a striking subject in
the image, catching the eye with its blue frame and detailings, while the
rider, dressed in coordinating colors, charges forward.\\\nAll of these
elements combined create an image that is full of life and action, capturing a
thrilling moment of a dirt bike race in progress.’},

6 {’image_id’: 2331159, ’caption’: ’The image is lively, filled with people
gathered possibly for a party or a social event. In the center of the image, a
woman dressed elegantly draws attention. She stands prominently, making a
distinct statement with her long, dark hair. Her face, sharply defined,
features a noticeably distinct nose. She is holding a white plastic spoon in
one hand, which also showcases a black wristwatch. As she raises the spoon,
it’s clear that she is indulging in a delicious treat, a piece of cake resting
on a small plate.\\\nThis cake is a stunning creation, white with red
frosting. It’s adorned with a delightful mix of red strawberries and an array
of white candles. On the cake, there seems to be a flag as well, perhaps
signifying a special occasion. An unused serving knife rests in the cake, and
it appears that the woman has just served herself a piece.\\\nAround her,
numerous other faces peer out, all engaged in their individual conversations.
Most of them appear to be men, some notable for their long hair and glasses.
There’s an interesting mix of attire in the scene, from yellow and white-
striped shirts to red and black plaid ones. One Asian man to the right seems
to be focused on the woman with the cake, adding to the collective sense of
attention directed towards her.\\\nIn the background, several intriguing
details pop out. For instance, the twinkling lights to the right catch the
eye, likely part of the party’s decoration. Additionally, there is a mirror
behind the woman, reflecting the attendees and amplifying the sense of a
bustling crowd. A lamp stands beside a wall, casting a warm glow, while the
silhouettes of patrons in the dark restaurant form an atmospheric backdrop. A
window lets in some additional light, illuminating a vacant chair.\\\nThe
overall atmosphere conveys the joyous, friendly nature of the gathering. You
can almost hear the buzz of conversation and feel the warmth of shared
laughter. It’s clear that this is an occasion of happiness and togetherness.’}

7]

8```

Figure 12: Dialogue Example from SparklesDialogueVG. Visual images were not
provided to GPT-4 during data generation but will be incorporated during
SparklesChat training.

#### Dialogue Example from SparklesDialogueVG.

The generated dialogue is visualized in Figure 12. The raw text is shown as
follows. The image IDs in the dialogue refer to the images in Figure 11.

⬇

1```json

2[

3[

4 {’role’: ’user’, ’image_ids’: [2331159, 2330601], ’content’: ’Can you link
the celebration occurring in IMAGE#2331159 and the dirt bike race in
IMAGE#2330601?’},

5 {’role’: ’assistant’, ’content’: ’Certainly, in IMAGE#2331159, we see a
lively gathering with a woman at the centre of the attention. She is seen
grasping a slice of intricately decorated cake from a vibrant party of sorts.
Her jovial demeanor and elegant apparel suggest she may be the guest of honor
at this event. The ambiance in the background features warm lighting,
laughter, conversation, and a sense of togetherness and happiness. This scene
could possibly be a celebration of a personal achievement or
milestone.\\\n\\\nOn the other hand, in IMAGE#2330601, a different form of
excitement and thrill is captured. This showcases a man in a blue and black
racing suit, performing at a dirt bike race, implying a high level of
commitment, skill, and passion. He is decisively navigating the muddy track,
indicating a race in progress.\\\n\\\nLinking these images, one possible
narrative could be that the woman in the first image might be celebrating a
successful day at the dirt bike race. The man in the second image who is
skilfully navigating the muddy race track could be related to her in some way.
Perhaps he is her son, husband, or even a team member, and they are
celebrating his win or a significant accomplishment in his racing career at
the party. The sense of joy and exhilarity is a common thread that connects
these two images.’},

6 {’role’: ’user’, ’image_ids’: [2408549], ’content’: ’How would you
incorporate the train scene from IMAGE#2408549 with the earlier party and
racing narratives?’},

7 {’role’: ’assistant’, ’content’: ’In IMAGE#2408549, we witness the rush of a
large, striking blue train speeding through a railroad crossing. A passenger,
perhaps contemplating the journey, watches the world outside through a window.
This train scene can signify a journey or transition.\\\n\\\nBringing these
images together into a coherent story, it might be speculated that after the
celebration in IMAGE#2331159 for the successful dirt bike race from
IMAGE#2330601, the next chapter of their journey is starting. It could be
speculated that the woman or the dirt bike racer or both are now on this train
in IMAGE#2408549, heading towards a new location for perhaps the next race or
a well-deserved vacation post celebration. This train ride could represent
their movement towards the next chapter in their journey, literally as well as
metaphorically.\\\n\\\nThe passenger looking out the window could be
contemplating the recent event of the dirt bike race victory and the warm
celebration that followed. The train’s vigorous motion and the reflected
sunlight might be symbolising their sense of achievement and their eager
anticipation for upcoming adventures. Thus, these images together weave a tale
of triumph, celebration, and anticipation for what lies ahead.’}]

8]

9```

##  Appendix G LLM-instructed multiple dialogues generation for
SparklesDialogueCC

For SparklesDialogueCC, we prompt GPT-4 to generate three dialogues in a
single response. These dialogues incorporate one, two, and three images in the
first turn and a single image in the second. Each prompt includes three
demonstration dialogues and nine candidate image descriptions to facilitate
this. We curated 150 demonstration dialogues, evenly split with 50 dialogues
for each type. The complete image-text dataset comprises about 3,500 pairs.

We first present our designed prompt for LLM-instructed Multiple Dialogues
Generation to generate SparklesDialogueCC in Table 9. Then, we show a case of
the Dialogue Demonstrations and Candidate Image Descriptions to construct the
prompt. Finally, we show the corresponding generated dialogues using the
example prompt.

Table 9: Prompt for LLM-instructed Multiple Dialogues Generation.

Users will interact with a conversational assistant that has advanced capabilities of understanding, analyzing, and reasoning about images. This includes discussing a variety of real-world concepts, objects, and entities, generating a range of text materials, seeking advice, guidance, or assistance, and much more. |
---|---
Below are three illustrative dialogues presented in a JSON format. Each one represents a self-contained conversation between a "user" and the "assistant" regarding multiple images. Each "user" message contains an "image_ids" field recording the IDs of newly selected images. The images are referred to in the "content" field as IMAGE#image_id. ⬇ ```json {Dialogue Demonstrations} ⬇ ``` Please note that the specific "image_ids" and "content" in the JSON above are for illustrative purposes only. The actual candidate images are shown below delimited by triple quotes, each accompanied by an image ID and a caption. Avoid using phrases similar to ’caption’ and ’description’ in your dialogue as if the user and the assistant have visual capabilities. ⬇ ```json {Candidate Image Descriptions} ⬇ ``` Each dialogue consists of four messages: |
1\. A user examines all candidate images, selects highly relevant ones, and sends a reasonable and creative message to the assistant. |
2\. Once the images are provided, the assistant thoroughly perceives and comprehends them, responding with highly helpful and exceptionally detailed answers that provide comprehensive reasoning. |
3\. Considering the past dialogue, the user chooses another candidate image for further inquiry. The user should refer to both the newly selected image and those mentioned earlier in the same dialogue. |
4\. The assistant provides a highly helpful and exceptionally detailed answer providing comprehensive reasoning regarding the visual content of the images. |
The following are three independent dialogues between the user and the assistant, adhering to the given JSON format. In this format, the first message in the three dialogues includes 1, 2, and 3 image ids respectively. |
Make sure to formulate accurate and diverse "content" that does not strictly follow the illustrative dialogues. And remember to develop the last "content" even though it is shown as "…" in the JSON format provided above. |
Dialogue Demonstrations ⬇ [[{’role’: ’user’, ’image_ids’: $\mathbf{I}^{1,1}$, ’content’: $\mathbf{X}_{\texttt{q}}^{1,1}$}, {’role’: ’assistant’, ’content’: $\mathbf{X}_{\texttt{a}}^{1,1}$}, {’role’: ’user’, ’image_ids’: $\mathbf{I}^{1,2}$, ’content’: $\mathbf{X}_{\texttt{q}}^{1,2}$}, {’role’: ’assistant’, ’content’: ’…’}], …… [{’role’: ’user’, ’image_ids’: $\mathbf{I}^{3,1}$, ’content’: $\mathbf{X}_{\texttt{q}}^{3,1}$}, {’role’: ’assistant’, ’content’: $\mathbf{X}_{\texttt{a}}^{3,1}$}, {’role’: ’user’, ’image_ids’: $\mathbf{I}^{3,2}$, ’content’: $\mathbf{X}_{\texttt{q}}^{3,2}$}, {’role’: ’assistant’, ’content’: ’…’}]] Candidate Image Descriptions ⬇ [{’image_id’: $\mathbf{J}^{1}$, ’caption’: $\mathbf{C}^{1}$}, …… {’image_id’: $\mathbf{J}^{9}$, ’caption’: $\mathbf{C}^{9}$}] |

#### Example of dialogue demonstrations

We visualize the images corresponding to image IDs in the dialogues in Figure
13 for reference, while these visual images were not sent to GPT-4 for data
generation. Note that we abbreviate the message content of the assistant in
the second turn as “…” to save space, considering that the previous message
contents have provided enough demonstrations.

|  |
---|---|---
3775 | 4799 | 301
|  |
3122 | 204 | 3045
|  |
1863 | 1336 | 3302
Figure 13: Reference images corresponding to the image IDs in the
demonstration dialogues in Appendix G. These images were not sent to GPT-4 for
data generation.

⬇

1```json

2[

3[

4 {’role’: ’user’, ’image_ids’: [’3775’], ’content’: ’What kind of ink is
usually used for such tattoos that we see in IMAGE#3775, and how long can we
expect it to last?’},

5 {’role’: ’assistant’, ’content’: ’Tattoo artists typically use a specific
type of ink known as tattoo ink for their designs, like the one shown in
IMAGE#3775. Tattoo ink is usually made from metal salts, plastic-based
pigments, or heavy metals for creating different colors, although black ink,
like the one used in the "trust yourself" tattoo in the image, is generally
carbon-based.\n\nHow long the tattoo lasts in its crisp form depends on a
series of factors such as the depth of the ink placement, the type of ink
used, how well it\’s cared for during the healing process, and how it\’s
protected from the sun and aging. \n\nGenerally, tattoos are permanent because
the human body\’s immune system isn\’t capable of dealing with the large
molecules of tattoo ink. Over several years, a tattoo can fade slightly due to
the natural regeneration cycle of skin cells, but it won\’t disappear
completely. The good thing about black tattoos, as seen in IMAGE#3775, is that
black pigment is absorbed well by all skin types and tends to last longer
without fading. However, the specific longevity of a tattoo will differ from
person to person. If one wants their tattoo\’s health to be maintained, they
should keep it moisturized, use sunscreen when out in harsh sun, and consider
touch-ups from a professional tattoo artist when necessary.’},

6 {’role’: ’user’, ’image_ids’: [’4799’], ’content’: ’Can the ink used in the
tattoo from IMAGE#3775 potentially cause any skin irritations or allergies,
similar to how some people have reactions to certain foods like in
IMAGE#4799?’},

7 {’role’: ’assistant’, ’content’: ’...’}

8],

9[

10 {’role’: ’user’, ’image_ids’: [’301’, ’3122’], ’content’: ’What kind of
environment is depicted in IMAGE#301 and how does it contrast with the one in
IMAGE#3122?’},

11 {’role’: ’assistant’, ’content’: "IMAGE#301 depicts a tropical, beach
environment. In the image, there are two kayaks resting on a sandy beach, and
this detail hints at a location that’s likely in a warmer climate, suitable
for water sports and recreational activities. The presence of palm trees
further supports this, as they are commonly found in regions that are hot and
sunny most of the year. The blue sky and the small building that looks like a
beach hut or kiosk suggest a relaxed, leisure-oriented environment.\n\nOn the
contrary, IMAGE#3122 portrays a snowy, mountainous environment. The presence
of snow suggests a colder climate, likely during winter. The pair of black
rubber snowshoes with metal spikes on the bottom implies activities related to
snow, such as hiking or mountain climbing. The person wearing a hooded jacket
and grey pants with their hands in their pockets further indicates the cold
nature of this environment. The trees and mountains visible through the snow
complete the impression of a wilderness setting, possibly remote and less
populated.\n\nIn summary, the two images contrast significantly - one depicts
a warm, tropical beach setting that is likely oriented towards leisure and
relaxation, while the other depicts a colder, mountainous setting associated
with more strenuous outdoor activities. This contrast goes beyond just the
climatic conditions; it also extends to the kind of activities and lifestyle
one might associate with each setting."},

12 {’role’: ’user’, ’image_ids’: [’204’], ’content’: ’Does the scene in
IMAGE#204 remind you of the environment depicted in IMAGE#301, or are they
entirely different?’},

13 {’role’: ’assistant’, ’content’: ’...’}

14],

15[

16 {’role’: ’user’, ’image_ids’: [’3045’, ’1863’, ’1336’], ’content’:
’Generate a promotional text that could fit for a sports event based on
IMAGE#3045. Also, please include elements from IMAGE#1863 and IMAGE#1336 to
give it some unique angles.’},

17 {’role’: ’assistant’, ’content’: "Join us for an electrifying display of
sportsmanship like no other! Just as the fans in IMAGE#3045, cheer on your
favorite team in an outdoor stadium filled with a roaring crowd, all decked
out in their team’s gear, waving pink flags in unity and excitement.
Experience an atmosphere buzzing with energy beneath the open sky, surrounded
by the vibrant greenery.\n\nDrawing inspiration from IMAGE#1863, watch as the
team’s performance graph soars high, much like the increasing deposits in a
bank account. Witness the thrilling trajectory of action unfold, as each
second, each decision, contributes to an upward trend in their game. Be a part
of this journey, as they race not just to maintain the pace but to set a new
norm, akin to the decreasing withdrawals mark, showcasing improvement and
growth over time.\n\nIn the spirit of IMAGE#1336, let your hearts beat as one,
with the passion of the red and the harmony of the black and white. Let the
dual celebrations of vibrant love for the game and steadfast unity resound in
the air. Let the duality of a heartbeat stir a symphony of balance - just like
the yin and yang of competitiveness and sportsmanship.\n\nSo, let’s delve into
this exciting world, where not just the players, but even the spectators
embark on a journey of passion, unity, growth, and harmony. Let the games
begin!"},

18 {’role’: ’user’, ’image_ids’: [’3302’], ’content’: ’Could the atmosphere
and energy captured in IMAGE#3045 somehow be invoked in the office space
depicted in IMAGE#3302 to make it more lively? Any suggestions?’},

19 {’role’: ’assistant’, ’content’: ’...’}

20]

21]

22```

#### Example of candidate images descriptions

An example of Candidate Image Descriptions is shown below, and their
corresponding source images are shown in Figure 14 for reference (they are not
sent to GPT-4).

|  |
---|---|---
2439 | 3065 | 1093
|  |
4704 | 1438 | 3120
|  |
2630 | 2071 | 2966
Figure 14: Candidate images corresponding to the image IDs in the dialogues
generation process in Appendix G. These images were not sent to GPT-4 for data
generation.

⬇

1```json

2[

3 {’image_id’: ’2439’, ’caption’: ’This image shows a kitchen with wooden
cabinets, black countertops, and white appliances. The floor is made of tiles
and the walls are painted white. There is a large window above the sink that
lets in plenty of natural light. The room is spacious and well lit.’},

4 {’image_id’: ’3065’, ’caption’: "This is an image of an airplane flying in
the sky at sunset. The plane is a large, commercial jet with a white body and
red and blue stripes on the tail. It is flying low in the sky, with the sun
setting behind it, casting a warm orange glow on the left side of the image
and a blue glow on the right. The plane’s engines are visible at the bottom of
the image, with smoke coming from them. The sky is a deep blue, with clouds in
the distance that are tinged with pink from the sunset."},

5 {’image_id’: ’1093’, ’caption’: ’The image shows a small room with a wooden
shelf on the wall, several rolls of wrapping paper stacked on it, a door on
the right side, and a window on the left side. The walls are painted white and
there is a wooden floor.’},

6 {’image_id’: ’4704’, ’caption’: ’The image shows a view of a golf course
with a red flag on the green. In the background, there is a city skyline with
buildings and a church steeple. The grass on the course is lush and green, and
there are trees on either side of the fairway. The sky is clear and blue, and
there are a few clouds in the distance. The flag on the green is a small, red
flag with a white pole. It is standing upright in the middle of the green, and
it looks like it is blowing in the wind. The city skyline in the background is
quite impressive, with several tall buildings and a church steeple visible.
The church steeple is quite tall and has a pointed top.’},

7 {’image_id’: ’1438’, ’caption’: ’The Eiffel Tower is a famous landmark in
Paris, France. It is a wrought iron lattice tower that was built in 1889 to
commemorate the centenary of the French Revolution. The tower stands 324
meters tall and is located on the Champ de Mars in the heart of Paris. It is
one of the most visited tourist attractions in the world, with millions of
people visiting it every year. The tower has become an iconic symbol of Paris
and France. The tower is painted in blue, white, and red, the colors of the
French flag. The lights on the tower create a beautiful and magical
atmosphere, making it a popular spot for romantic walks and photos. The tower
is surrounded by water, with a river running underneath it.’},

8 {’image_id’: ’3120’, ’caption’: ’This image is a logo for a company or
brand. The logo consists of the letters "z" and "l" in a red and black color
scheme. The letters are connected by a diagonal line through the center of the
image, creating a simple and modern design suitable for various businesses and
industries.’},

9 {’image_id’: ’2630’, ’caption’: "The image shows a black and tan dog lying
on a large, gray dog bed. The bed has a geometric pattern on it and is placed
in front of a brick fireplace. The dog’s head is resting on its paws, and it
appears to be sleeping. The image is well lit and the details of the dog and
the bed are clear."},

10 {’image_id’: ’2071’, ’caption’: "The image shows a man in a black suit and
tie standing in front of a blue and yellow background. The man is smiling and
has his hands in his pockets.\n\nThe man is wearing a black suit and tie, with
a white shirt underneath. He has short, dark hair and a well-groomed beard. He
is standing in front of a large blue and yellow background that looks like a
cartoon character’s face. The character’s eyes and mouth are wide open, and it
looks like the man is standing in front of the character’s face.\n\nThe
background looks like a cartoon character’s face, with blue and yellow colors.
The character’s eyes and mouth are wide open, and it looks like the man is
standing in front of the character’s face. The character’s eyes and mouth are
wide open, and it looks like the man is standing in front of the character’s
face."},

11 {’image_id’: ’2966’, ’caption’: ’The image shows a garden with a stone
statue of a person sitting under a tree. The statue has a serene expression on
its face, and the sun is setting behind the trees, casting a warm orange light
on the scene. The foreground grass is green and well-maintained, and there are
no other objects or people in the image.’}

12]

13```

Figure 15: Dialogue Example from SparklesDialogueCC. Visual images were not
provided to GPT-4 during data generation but will be incorporated during
SparklesChat training.

#### Dialogue examples from SparklesDialogueCC.

The generated dialogue is visualized in Figure 15. The raw text is shown as
follows. The image IDs in the dialogues refer to the images in Figure 14.

⬇

1```json

2[

3[

4 {’role’: ’user’, ’image_ids’: [’3120’], ’content’: ’Could you tell me what
the design of the logo in IMAGE#3120 might symbolize or represent?’},

5 {’role’: ’assistant’, ’content’: ’The logo in IMAGE#3120, with the combined
letters "z" and "l" connected by a diagonal line, seems to convey simplicity,
modernity, and approachability. The use of "z" and "l" could represent the
initials of a company or individual\’s name, while the diagonal line may
symbolize a link or connection, emphasizing unity or collaboration. The choice
of red and black colors could be conveying a sense of power and sleek
sophistication, respectively. Black often symbolizes solidity and formality,
while red can represent passion and energy. However, the specific meaning
could vary as logo interpretations often depend on the organization\’s
mission, brand values, and the industry it\’s part of.’},

6 {’role’: ’user’, ’image_ids’: [’1438’], ’content’: ’Considering the design
and message one might infer from the logo in IMAGE#3120, would it be a good
fit for the place depicted in IMAGE#1438? How would they complement each
other?’},

7 {’role’: ’assistant’, ’content’: ’The logo from IMAGE#3120 can bring a
modern, energetic, and connected vibe to the traditional and iconic setting of
the Eiffel Tower in IMAGE#1438. This can create an interesting juxtaposition
between historic and contemporary themes.\n\nThe simplicity and modernity of
the logo design would sharply contrast yet harmonize with the complexity and
intricacy of the Eiffel Tower. This could appeal to audiences who appreciate
both the historical and futuristic aspects of a brand. Similarly, the vibrancy
and energy indicated by the red color in the logo may align well with the
romantic and lively atmosphere associated with Paris, the city where Eiffel
Tower is situated.\n\nIf this was for a brand that intended to evoke a sense
of modern connection in a place steeped with history, or perhaps a company
that combines technology with tourism in Paris, this logo could indeed
complement the place effectively.’}],

8[

9 {’role’: ’user’, ’image_ids’: [’2630’, ’3065’], ’content’: ’Can you imagine
a story that might link the peaceful dog lounging in IMAGE#2630 and the
airplane soaring in the sunset sky of IMAGE#3065?’},

10 {’role’: ’assistant’, ’content’: ’Certainly! Here\’s a story possibility:
The dog in IMAGE#2630, named Boomer, belongs to a dedicated pilot who often
takes on long haul flights, like the airplane depicted in IMAGE#3065. Whenever
the pilot sets off for a lengthy trip, Boomer retreats to his comfortable dog
bed in front of the cozy fireplace, waiting for his owner\’s return.\n\nThe
pilot, while often away, never fails to think about Boomer. From high above
the earth, amidst beautiful sunsets, the canine\’s owner counts the hours
until they can be reunited. Just the thought of Boomer’s serene expression on
the tastefully patterned dog bed provides a sense of peace and comfort even
thousands of miles away.\n\nSo despite their geographical separation depicted
in these images, there’s an underlying connection of loyalty and companionship
between Boomer and his pilot owner, which is a heartwarming staple in their
world filled with frequent departures and heartfelt reunions.’},

11 {’role’: ’user’, ’image_ids’: [’4704’], ’content’: ’How would the
atmosphere in IMAGE#4704 contribute to the story of our pilot-owner and Boomer
from IMAGE#2630 and IMAGE#3065?’},

12 {’role’: ’assistant’, ’content’: ’The setting in IMAGE#4704 could introduce
another layer to our narrative. Imagine this: When off-duty, our pilot-owner
is also an avid golfer who cherishes the peacefulness and serenity found on
the golf course, like the one seen in IMAGE#4704. After returning from a long
trip, the pilot usually heads to the fairway, and Boomer eagerly accompanies
him.\n\nThe manicured greens serve as a perfect place for Boomer to roam
freely, chase golf balls, and feel the wind in his fur. The distant city
skyline seen from the golf course acts as a reminder of the urban life our
pilot leaves behind when they find solace in the peaceful embrace of the game
and their cherished canine companion.\n\nSubsequently, the lush green golf
course fills in a missing piece of the puzzle, acting as a mediating space
that connects the frequent travels of the pilot and Boomer\’s waiting. It
becomes a place of joy, reconnection, and shared experiences between the duo,
marking the happy ending to their waiting and the start of their time together
until the next flight.’}],

13[

14 {’role’: ’user’, ’image_ids’: [’2439’, ’1093’, ’2071’], ’content’: ’Could
you help me design a scenario the place in IMAGE#1093 is being used as a
temporary office for the man in IMAGE#2071, who usually works in the kitchen
in IMAGE#2439?’},

15 {’role’: ’assistant’, ’content’: ’Sure, here\’s a possible scenario: In
IMAGE#2071, we see a man who typically works from his spacious kitchen shown
in IMAGE#2439. The kitchen, with its wooden cabinets and black countertops,
doubles as his office. Its large windows provide ample natural light, creating
a conducive and productive work environment. It is here that he plans his
strategies, makes important business decisions, and leads virtual
meetings.\n\nHowever, for a change of scenario - perhaps due to a home
renovation or family visiting - the man decides to shift his workspace to the
small room shown in IMAGE#1093 for a while. The room is likely a storage space
or craft room with wrapping paper stacked on a wooden shelf, indicating
creativity and resourcefulness.\n\nHe sets up his temporary office here,
positioning his desk near the window to replicate the natural light he enjoys
in his kitchen-office. The surrounding array of vibrant wrapping paper
inspires creativity and brings a touch of liveliness to his work routine.
Also, the closed door ensures that he has quiet and privacy for his business
calls.\n\nSo in this unusual but charming workspace, our businessman continues
his work, exhibiting that all you need to be productive is a bit of
adaptability, irrespective of where you are.’},

16 {’role’: ’user’, ’image_ids’: [’2966’], ’content’: ’Taking into account all
the previous images, how could we incorporate the scene from IMAGE#2966 into
the story of the man changing his workspace?’},

17 {’role’: ’assistant’, ’content’: ’The serene garden scene in IMAGE#2966
could be an excellent addition to the narrative. Let\’s say, towards the back
of the house where both the kitchen-office and the temporary workspace reside,
there’s this quiet garden depicted in the image, featuring a stone statue
sitting under a tree.\n\nAt the end of each workday, the man enjoys taking a
stroll in the garden to unwind. The setting sun casts a warm orange light on
the scene creating a peaceful atmosphere, and he often finds himself sitting
next to the stone statue, reflecting on his day. The tranquility of the garden
effectively helps him transition from work mode, and it provides him with a
sense of balance that complements his adaptable work life.\n\nTherefore,
despite the change in his workspace, the man maintains his evening ritual in
the garden, capturing the idea that while our surroundings may change, there’s
comfort in keeping certain routines and finding peace in nature.’}

18]

19]

20```