# StoryGPT-V: Large Language Models as Consistent Story Visualizers

Xiaoqian Shen
KAUST
<EMAIL_ADDRESS>Mohamed Elhoseiny
KAUST
<EMAIL_ADDRESS>
###### Abstract

Recent generative models have demonstrated impressive capabilities in
generating realistic and visually pleasing images grounded on textual prompts.
Nevertheless, a significant challenge remains in applying these models for the
more intricate task of story visualization. Since it requires resolving
pronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution,
and ensuring consistent characters and background synthesis across frames.
Yet, the emerging Large Language Model (LLM) showcases robust reasoning
abilities to navigate through ambiguous references and process extensive
sequences. Therefore, we introduce StoryGPT-V, which leverages the merits of
the latent diffusion (LDM) and LLM to produce images with consistent and high-
quality characters grounded on given story descriptions. First, we train a
character-aware LDM, which takes character-augmented semantic embedding as
input and includes the supervision of the cross-attention map using character
segmentation masks, aiming to enhance character generation accuracy and
faithfulness. In the second stage, we enable an alignment between the output
of LLM and the character-augmented embedding residing in the input space of
the first-stage model. This harnesses the reasoning ability of LLM to address
ambiguous references and the comprehension capability to memorize the context.
We conduct comprehensive experiments on two visual story visualization
benchmarks. Our model reports superior quantitative results and consistently
generates accurate characters of remarkable quality with low memory
consumption. Our code will be made publicly available111Please refer to the
webpage for qualitative results..

Figure 1: We present StoryGPT-V, which empowers a large language model for
interleaved image-text comprehension and aligns its output with character-
aware Latent Diffusion (Char-LDM) for autoregressive story visualization
grounded on co-referential text descriptions.

##  1 Introduction

Image generation algorithms have made significant strides and are on the verge
of matching human-level proficiency. Despite this progress, even a powerful
image generator suffers from story visualization task, which involves
generating a series of frames that maintain semantic coherence based on
narrative descriptions [22, 56, 27, 28]. This challenge arises from the fact
that captions for a single image are typically self-sufficient, lacking the
continuity needed to capture the narrative of object interactions that unfold
through multiple sentences over a sequence of frames. This poses a promising
avenue for further research and exploration of story visualization. Such a
task demands a model capable of producing high-quality characters and detailed
environmental objects grounded on given text descriptions. Moreover, it
requires the ability to disambiguate referential pronouns in the subsequent
frame descriptions, e.g., “she, he, they”.

Prior studies [26, 22, 28, 47, 4] explore the realm of story visualization but
do not take reference resolution [44] (i.e., anaphora resolution in the
context of natural language processing [2, 29]) into consideration. Story-LDM
[38] first extended story visualization benchmarks with referential text and
devises an attention memory module that retains visual context throughout the
series of generated frames. However, it still struggles to generate precise
characters for referential text since the interaction between current
descriptions and contextual information occurs within the CLIP [36] semantic
space, causing a loss in fine-grained language understanding and hindering
referencing capabilities. Furthermore, the attention memory module requires
maintaining all previous images in latent pixel space for attention
calculations, significantly increasing memory demands with each additional
frame in autoregressive generation.

The limitation of previous works leads us to rethink how to achieve accurate
and efficient reference resolution toward consistent story visualization.
Large Language Models (LLMs) [34, 37, 3, 58], trained on extensive text
corpora, have exhibited impressive capabilities in deciphering contextual
references in natural language descriptions. Prior works [18, 11] have
demonstrated the effectiveness of harnessing LLMs for tasks involving image
comprehension and generation, where the visual features are adapted within
LLM’s token space rather than the pixel space. Hence, such a model could be
utilized to efficiently address ambiguous references for story visualization
tasks.

In this work, we aim at story visualization grounded on given co-referential
frame descriptions, focusing on delivering high-quality and coherent
portrayals of characters. To achieve this, we leverage a powerful text-to-
image model [41] to generate high-quality characters and environmental objects
grounded on given frame descriptions, coupled with the reasoning ability of
Large Language Models (LLMs) to resolve ambiguous references and improve the
cohesiveness of the context. To improve the generation of highly faithful
characters, we enhance the pre-trained Latent Diffusion (LDM) towards
character-aware training in the first stage. We first augment the token
feature by incorporating the visual representation of the corresponding
character. Additionally, we regulate the cross-attention map of the character
token to highlight the interaction between the conditional token and specific
latent pixels.

In addressing the challenge of ambiguous reference, which cannot be
effectively handled by a robust text-to-image model alone, we leverage an LLM
that takes interleaved images and co-referential frame descriptions as input,
and aligns its visual output with the character-augmented embedding encoded by
first-stage model. Such semantic guidance, along with LLM’s casual modeling,
enables effective reference resolution and consistent generation. Furthermore,
our approach efficiently preserves context by processing images as sequences
of tokens in the LLM input space with low memory consumption.

Contributions. Our contributions are as follows:

  * •

We first enhance the text representation by integrating the visual features of
the corresponding characters, then refine a character-aware LDM for better
character generation by directing cross-attention maps with character
segmentation mask guidance.

  * •

We adapt LLM by interlacing text and image inputs, empowering it to implicitly
deduce references from previous contexts and produce visual responses that
align with the input space of the first-stage Char-LDM. This leverages the
LLM’s reasoning capacity for reference resolution and the synthesis of
coherent characters and scenes.

  * •

Our model is capable of visualizing stories featuring precise and coherent
characters and backgrounds on story visualization benchmarks. Furthermore, we
showcase the model’s proficiency in producing extensive (longer than 40
frames) visual stories with low memory consumption.

##  2 Related Work

Text-to-image synthesis. Numerous works [8, 10, 9] have demonstrated
unprecedented performance on semantic generation. Recently, diffusion-based
text-to-image models [40, 39, 41, 43] have shown significant advancements in
enhancing image quality and diversity through the utilization of diffusion
models. However, these text-to-image approaches primarily concentrate on
aligning individual-generated images grounded on text descriptions and do not
take into account the crucial aspects of character and scene consistency
across multiple frames in the story visualization task. Additionally, they
lack the capability to effectively resolve co-reference issues within a
narrative description.

Multi-modal Large Language Models. Large Language Models (LLMs) wield an
extensive repository of human knowledge and exhibit impressive reasoning
capabilities. Recent studies [49, 5, 1, 21] utilize pre-trained language
models to tackle vision-language tasks, and subsequent studies [60, 59, 52,
20, 16, 6] further enhance multi-modal abilities by aligning vision models
with LLM input space. In addition to multi-modal comprehension, several works
are dedicated to more challenging multi-modal generation tasks. FROMAGe [19]
appends a special retrieval token to LLM and maps the hidden representation of
this token into a vector space for retrieving images. Several current works
[18, 54, 57] learn a mapping from hidden embeddings of an LLM represents for
additional visual outputs into the input space of a frozen pre-trained text-
to-image generation model [41]. In this work, we fed multi-modal LLM with
interleaved image and referential text descriptions as input and aligned the
output with a character-aware fused embedding from our first-stage Char-LDM,
guiding the LLM in implicitly deducing the references.

Story Visualization. StoryGAN [22] pioneers the story generation task, which
proposes a sequential conditional GAN framework with dual frame and story
level discriminators to improve image quality and narrative coherence.
DuCoStoryGAN [27] introduces a dual-learning framework that utilizes video
captioning to enhance semantic alignment between descriptions and generated
images. VLCStoryGAN [26] used video captioning for semantic alignment between
text and frames. Recently, StoryDALL-E [28] retrofits the cross-attention
layers of the pre-trained text-to-image model to promote generalizability to
unseen visual attributes of the generated story. These methods do not consider
ambiguous references in text descriptions. Story-LDM [38] first introduced
reference resolution in story visualization tasks and proposed an
autoregressive diffusion-based framework with a memory-attention module to
resolve ambiguous references. Nevertheless, it struggled with accurately
resolving references and was memory-intensive, as it required retaining all
previous context in pixel space. In our work, we employ a powerful causal
inference LLM for reference resolution, and it efficiently maintains context
by mapping visual features into several token embeddings as LLM inputs rather
than operating in latent pixel space.

Figure 2: (a) In the first stage, a fused embedding is created by integrating
character visuals with text embeddings, serving as the Char-LDM’s conditional
input, and the cross-attention maps of Char-LDM will be guided by
corresponding character segmentation mask for accurate and high-quality
character generation (Section 3.2). (b) In the second stage, the LLM takes the
interleaved image and text context as input and generates $R$ [IMG] tokens.
These tokens are then projected by LDM Mapper into an intermediate output,
which will be encouraged to align with fused embedding as Char-LDM’s input.
The figure intuitively shows how the character-augmented fused embedding and
the casual language modeling aid LLM for reference resolution (Section 3.3).

##  3 Methods

The objective of story visualization is to transform a textual narrative,
composed of a series of $N$ descriptions ${S_{1},...S_{N}}$, into a sequence
of corresponding visual frames ${I_{1},...,I_{N}}$ that illustrate the story.
We’ve developed a two-stage method aimed at generating temporally consistent
visual stories with accurate and high-quality characters. First, we augment
text representation with characters’ visual features and refine a character-
aware LDM [41] (Char-LDM) towards high-quality character generation. This is
achieved by directing the cross-attention maps of specific tokens associated
with the corresponding characters, using character segmentation mask
supervision (Section 3.2). Then, we leverage the reasoning ability of LLM to
resolve ambiguous references by aligning the output of LLM with Char-LDM input
space for temporal consistent story visualization (Section 3.3).

###  3.1 Preliminaries

Cross-attention in text-conditioned Diffusion Models. In diffusion models [15,
46], each diffusion step $t$ involves predicting noise $\epsilon$ from the
noise code $z_{t}\in\mathbb{R}^{(h\times w)\times d_{v}}$ conditioned on text
embedding $\psi(S)\in\mathbb{R}^{L\times d_{c}}$ via U-shaped Network [42],
where $\psi$ is the text encoder, $h$ and $w$ are the latent spatial
dimensions and $L$ is the sequence length. Within U-Net, the cross-attention
layer accepts the spatial latent code $z$ and the text embeddings ${\psi}(S)$
as inputs, then projects them into $Q=W^{q}z$, $K=W^{k}{\psi}(S)$ and
$V=W^{v}{\psi}(S)$, where $W^{q}\in\mathbb{R}^{d_{v}\times d^{\prime}}$,
$W^{k},W^{v}\in\mathbb{R}^{d_{c}\times d^{\prime}}$. The attention scores is
computed as
$A=\mathrm{Softmax}(\frac{QK^{T}}{\sqrt{d^{\prime}}})\in\mathbb{R}^{(h\times
w)\times L}$, where $A[i,j,k]$ represents the attention of $k$-th text token
to the $(i,j)$ latent pixel. In this context, each entry $A[i,j,k]$ within the
cross-attention map $A$ quantifies the magnitude of information propagation
from the $k$-th text token to the latent pixel at position $(i,j)$. This
feature of the interaction between semantic representation and latent pixels
is harnessed in various tasks such as image editing [13, 32], video editing
[24], and fast adaptation [45, 55, 7, 53].

###  3.2 Character-aware LDM with attention control

Integrate visual features with text conditions. To achieve accurate and high-
quality characters in story visualization, we augment text descriptions with
visual features of corresponding characters and guide the attention of text
conditions to focus more on corresponding character synthesis. Given a text
description $S$, suppose there are $K$ characters that should be generated in
image $I$, images of those characters {$I_{c}^{1},...,I_{c}^{K}$}, a list of
token indices indicating each character name located in the description,
denoted as $\\{i_{c}^{1},...i_{c}^{K}\\}$. Inspired by [53, 55, 25], we first
utilize CLIP [36] text encoder ${\psi}$ and image encoder $\phi$ to obtain
text embedding and visual features of the characters appear in the image
respectively. Then, we augment the text embedding if the token represents a
character name. More specifically, we concatenate the token embedding and the
visual features of the corresponding character and feed them into an MLP to
obtain the augmented text embedding. Each augmented token embedding in the
augmented embedding $c$ is formulated as below:

$c^{k}=\textrm{MLP}\left(\mathrm{concat}\left(({\psi}(S[i_{c}^{k}]),\phi(I_{c}^{k})\right)\right)$
(1)

where $i_{c}^{k}$ refers to the index of the text token for character $k$, and
$I_{c}^{k}$ the image corresponding to character $k$. The embeddings for
tokens in $c$ that are unrelated to the character remain identical to the
vanilla CLIP token embeddings. The enhanced embedding $c$ is then employed as
supervision for the second-stage training, which will be further detailed in
Section 3.3, where $c_{1},...c_{N}$ are the corresponding augmented embeddings
for $S_{1},...,S_{N}$.

Controlling attention of text tokens. Previous work [13] has demonstrated that
the visual characteristics of generated images are influenced by the intricate
interplay between latent pixels and text embedding through the diffusion
process of LDM [41]. However, in vanilla LDM [41], a single latent pixel can
unrestrictedly engage with all text tokens. Therefore, we introduce a
constraint to refine this behavior and strengthen the impact of the token
representing the character’s name on certain pixels in the denoising process,
as illustrated in Figure 2 (a). First, we obtain an offline segmentation mask
of corresponding characters denoted as $\\{M_{1},...M_{K}\\}$ as supervision
signals via SAM [17]. We then encourage the cross-attention map $A_{k}$ for
each character $k$ at the token index position $i_{c}^{k}$, to align with the
binary segmentation mask $M_{k}$, whereas diverging from irrelevant regions
$\bar{M}_{k}$, formulated as follows:

$\mathcal{L}_{\textrm{reg}}=\frac{1}{K}\sum_{k=1}^{K}(A_{k}^{-}-A_{k}^{+})$
(2)

where

$A_{k}^{-}=\frac{A_{k}\odot\bar{M}_{k}}{\sum_{i,j}(\bar{M}_{k})_{ij}},\,\,A_{k}^{+}=\frac{A_{k}\odot
M_{k}}{\sum_{i,j}(M_{k})_{ij}}$ (3)

where $K$ is the number of characters to be generated in the image,
$i_{c}^{k}$ is the index of text token representing character $k$ and $\odot$
is the Hadamard product. By reducing the loss, it increases the attention of
character tokens to the relevant pixels of their respective characters, while
reducing their attention to irrelevant areas. Moreover, as the token
embeddings are enriched with the visual features of the corresponding
character, this attention control serves to deepen the connection between the
augmented semantic space and latent pixel denoising, which can consequently
enhance the quality of synthesized characters.

Our first stage Char-LDM focuses solely on the quality of image generation
grounded on a single caption. Yet, there remain challenges that surpass the
abilities of text-to-image generators in visualizing a sequence of stories.
Firstly, story visualization demands character and background consistency, an
aspect not covered by our first-stage enhancements. Moreover, the inherent
nature of lengthy descriptions includes referential terms like he, she, or
they, which presents a significant challenge for LDM in achieving accurate
inference. In contrast, LLMs can adeptly infer the intended character to which
the ambiguous text refers. Therefore, to address this issue, we harness the
formidable reasoning capabilities of LLM to disambiguate such references.

###  3.3 Aligning LLM for reference resolution

To enable an LLM to autoregressively generate images conditioned on prior
context and resolve ambiguous references, the model must be capable of (i)
processing images; (ii) producing images; and (iii) implicitly deducing the
subject of reference. The model could understand the image by learning a
linear mapping from the visual feature to the LLM input space, and generate
images by aligning the hidden states with conditional input required by LDM,
which is the fused embedding encoded by first-stage Char-LDM’s text and visual
encoder. It integrates the visual features of characters into the text
embedding. This character-augmented embedding, along with the causal language
modeling (CLM) [50, 33, 34] will direct the LLM to implicitly deduce and
generate the correct character for the referential input, as depicted in
Figure 2 (b).

More specifically, the LLM input consists of interleaved co-referential text
descriptions and story frames with flexible frame length $n$, in the order of
$(I_{1},S_{1},...,I_{n-1},S_{n-1},S_{n})$, where $2\leq n\leq N$. We first
extract visual embeddings $\phi(I_{i})\in\mathbb{R}^{d_{i}}$ with CLIP [36]
visual backbone, where $i\in[2,n]$, and learn $\texttt{Mapper}_{LLM}$ with
trainable matrix $\mathbf{W}_{v2t}\in\mathbb{R}^{d_{i}\times me}$ which maps
$\phi(I_{i})$ into $m$ $k$-dimensional embeddings reside within LLM input
space [21, 23, 60], where $e$ is the dimension of LLM embedding space.
Additionally, like recent works [18, 54, 57] in enabling LLM to generate
images, we add additional $R$ tokens, denoted as [IMG1], …, [IMGR] to
represent visual outputs and incorporate trainable matrix
$\mathbf{W}_{gen}\in\mathbb{R}^{R\times e}$ into frozen LLM. The training
objective is to minimize the negative log-likelihood of producing [IMG] tokens
conditioned on previously interleaved image/text tokens $\mathcal{T}_{prev}$:

$\mathcal{L}_{gen}=-\sum_{r=1}^{R}\log
p(\texttt{[IMG${}_{r}$]}|\mathcal{T}_{prev},\texttt{[IMG${}_{<r}$]})$ (4)

where

$\mathcal{T}_{prev}=\\{\phi(I_{<i})^{T}\mathbf{W}_{v2t},\psi(S_{1:i})\\}$ (5)

where $i\in[2,n]$ is the number of text descriptions of the current step. To
align [IMG] produced by LLM with LDM input space, we utilize a Transformer-
based $\texttt{Mapper}_{LDM}$ to project [IMG] tokens to the input space of
first-stage finetuned LDM with $L$ learnable query embeddings
$(q_{1},...,q_{L})\in\mathbb{R}^{L\times d}$, where $L$ is the maximum input
sequence length of the LDM, similar to BLIP-2 Q-Former [21]. The training
objective is to minimize the distance between $\mathrm{Mapper}$’s output Gen
Emb and the augmented conditional text representations of LDM, i.e., Fuse Emb
introduced in Section 3.2, formulated as:

$\mathcal{L}_{\textrm{align}}=||\texttt{Mapper}_{LDM}\left(h_{\texttt{[IMG${}_{1:R}$]}},q_{1},...q_{L}\right)-c_{i}||_{2}^{2}$
(6)

where $h_{[\texttt{IMG}_{1:R}]}$ denotes the last hidden states of LLM’s [IMG]
tokens. Suppose we can get access to the original text without reference
$S_{i}^{{}^{\prime}}$. Then, $c_{i}$ is the augmented text embedding of
caption $S_{i}^{{}^{\prime}}$ encoded by the first-stage model’s text and
visual encoder. For instance, if $S_{i}$ is ”They are talking to each other,”
then $S_{i}^{{}^{\prime}}$ would be ”Fred and Wilma are talking to each
other.” This non-referential text, augmented with character visual features,
assists LLM in efficiently disambiguating references using casual language
modeling.

Inference. During the inference process, the model sequentially visualizes
stories grounded on text descriptions. It begins by processing the text
description of the initial frame $S_{1}$. Focusing exclusively on frame
generation, we constrain the LLM to generate only $R$ specific [IMG] tokens
and then feed these token embeddings into the first-stage Char-LDM, resulting
in the generation of the first frame $I_{1}^{gen}$. Subsequently, the LLM
takes a contextual history that includes the text description of the first
frame $S_{1}$, the generated first frame $I_{1}^{gen}$, and the text
description of the second frame $S_{2}$ as input. This process is repeated to
visualize the entire story progressively.

Models Ref text Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$)
BG-F1 ($\uparrow$) FID ($\downarrow$) BLEU4 ($\uparrow$) CIDEr ($\uparrow$)
StoryDALL-E† [28] 69.49 83.35 48.46 55.24 44.24 0.4666 1.4473 LDM [41]
$\times$ 85.66 93.41 54.85 62.04 32.05 0.5230 1.8048 Story-LDM [38] 82.43
91.86 55.3 61.58 36.29 0.4656 1.4335 Char-LDM (Ours) 90.36 95.76 58.36 63.92
21.13 0.5260 1.8361 StoryDALL-E† [28] 61.83 78.36 48.10 54.92 44.66 0.4460
1.3373 LDM [41] ✓ 75.37 87.54 52.57 58.41 32.36 0.4911 1.5103 Story-LDM [38]
77.23 88.26 54.97 60.99 36.34 0.4585 1.4004 StoryGPT-V (Ours) 87.96 94.17
56.01 61.07 21.71 0.5070 1.6607

Table 1: Main experiments on FlintStonesSV [12]. The top portion is evaluated
on the dataset w/o extended referential text. The bottom half displays the
results on the extended dataset with co-reference. †StoryDALL-E [28] takes the
source frame as additional input.

##  4 Experiments

###  4.1 Experimental Setups

Datasets. Our experiments are conducted using two story visualization
datasets: FlintstonesSV [12] and PororoSV [22]. FlintstonesSV [12] contains
20132-training, 2071-validation, and 2309-test stories with 7 main characters
and 323 backgrounds, while PororoSV [22] consists of 10,191 training samples,
2,334 for validation, and 2,208 for testing with 9 main characters. We follow
[38] to extend the datasets with referential text, by replacing the character
names with references, i.e., he, she, or they, wherever applicable. Please
refer to the supplementary for details.

Evaluation metrics. To measure the accuracy of the characters and background
in the generated stories, we consider the following evaluation metrics the
same as previous story visualization literature [26, 28, 38]: Following [26],
we finetune Inception-v3 to measure the classification accuracy and F1-score
of characters (Char-Acc, Char-F1) and background (BG-Acc, BG-F1) respectively.
In addition, we consider the Frechet Inception Distance (FID) score, which
compares the distribution between feature vectors from real and generated
images for quality assessment.

When assessing text-image alignment, the CLIP [36] score falls short in
reliability since it cannot capture fine-grained details. Therefore we choose
the powerful captioning model BLIP2 [21] as the evaluation model and fine-tune
it on the corresponding datasets. We then employ it as a captioner to predict
5 captions for generated images and 5 captions for ground truth images as a
comparison to report the average BLEU4 [31] and CIDEr [51] score to assess
text-image alignment.

Comparison Approaches. We compare our model with state-of-the-art approaches:
VLCStoryGAN [26], StoryDALL-E [28], LDM [41] and Story-LDM [38]. Following
previous research [22, 38], we use 4 consecutive frames for evaluation. For
StoryDALL-E [28], which takes both story descriptions and the initial frame as
input, we use the first frame of a 5-frame story and evaluate using the
generated 4 frames. We finetune vanilla Stable Diffusion (LDM) on
FlintStonesSV [12] and PororoSV [22] as a baseline. Since Story-LDM [38] does
not provide pre-trained checkpoint or cleaned training code, we initiate
training from pre-trained LDM222https://ommer-lab.com/files/latent-
diffusion/nitro/txt2img-f8-large.

Implementation Details. For the first stage training, we freeze CLIP [36] text
encoder and fine-tune the remaining modules for 25k steps with a learning rate
of 1e-5 and batch size of 32 on original non-referential text. To enhance
inference time robustness and flexibility, we adopt a training strategy that
includes $10\%$ unconditional training, i.e., classifier-free guidance [14],
$10\%$ text-only training, and $80\%$ character-augmented fuse training
(Section 3.2).

For the second stage training, we use
OPT-6.7B333https://huggingface.co/facebook/opt-6.7b model as the LLM backbone.
To expedite the second stage alignment training, we first pre-compute non-
referential fused embeddings residing in the input space of the first-stage
Char-LDM. We map visual features into $m=4$ token embeddings as LLM input, set
the max sequence length as 160 and the number of additional [IMG] tokens
represents for LLM’s visual output as $R=8$, batch size as 64 training for 20k
steps. Please refer to the supplementary for more details.

###  4.2 Visual Story Generation

Figure 3: Qualitative comparison on FlintStonesSV [12] (left) and PororoSV
[22] (right) with co-reference descriptions.

Quantitative Results. (i) Generation with original descriptions. The upper
half of Table 1 shows the comparison results on original FlintStonesSV [12]
without referential text descriptions. Our first-stage Char-LDM exhibits
superior performance in generating accurate characters (Char-Acc, Char-F1) and
background scenes (BG-Acc, BG-F1), achieving high fidelity (FID), and
exhibiting better alignment with given text descriptions (BLEU4 [31], CIDEr
[51]). (ii) Generation with co-referenced descriptions. Table 1 (bottom) and
Table 2 show the results on extended FlintStonesSV [12] and PororoSV [22] with
co-referential text descriptions [38] respectively. By harnessing the merit of
reasoning and comprehension abilities of LLM, our model substantially boosts
performance in reference resolution compared to baselines, while maintaining a
strong text-image alignment grounded in the provided text descriptions.

Models Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) FID ($\downarrow$) BLEU4
($\uparrow$) CIDEr ($\uparrow$) StoryDALL-E† [28] 21.03 50.56 40.39 0.2295
0.3666 LDM [41] 27.81 57.02 28.98 0.2560 0.5122 Story-LDM [38] 29.14 57.56
26.64 0.2420 0.4581 StoryGPT-V (Ours) 36.06 62.70 19.56 0.2586 0.5279

Table 2: Performance comparison on PororoSV [22] with co-referenced
descriptions. †StoryDALL-E [28] takes the source frame as additional input.
Figure 4: Human evaluation results on FlintStonesSV [12] w.r.t visual quality,
text-image alignment, character accuracy and temporal consistency.

Qualitative Results. Figure 3 demonstrates qualitative comparison on
FlintStonesSV [12] and PororoSV [22] with co-reference descriptions. LDM [41]
could generate high-quality images but struggles to produce correct characters
in the presence of reference in the captions. Story-LDM [38], despite
incorporating an attention-memory module to handle context, fails to produce
accurate characters in some frames. In comparison, our model excels at
generating frames with pleasing visuals, accurate characters, and maintaining
temporal consistency in the background scenes.

Human Evaluation. In addition, we use Mechanical Turk to assess the quality of
100 stories produced by our methods or Story-LDM [38] on FlintStonesSV [12].
Given a pair of stories generated by Story-LDM [38] and our model, MTurkers
are asked to decide which generated four-frame story is better w.r.t visual
quality, text-image alignment, character accuracy, and temporal consistency.
Each pair is evaluated by 3 unique workers. In Figure 4, our model
demonstrates significantly better story visualization quality with accurate
and temporally coherent synthesis.

###  4.3 Ablation Studies

First stage ablation. We conducted an ablation study for the first stage and
presented results in Table 3. w/o $\mathcal{L}_{reg}$ indicates that we
disabled the $\mathcal{L}_{reg}$ loss (Equation 2), i.e., the model underwent
training without the influence of segmentation masks to direct the cross-
attention maps. w/o augmented text signifies that the model’s conditional
input during its training phase was the standard CLIP [36] text embedding,
rather than the fused embedding incorporating the character’s visual
attributes as discussed in Section 3.2. freeze vis denotes the visual encoder
remained frozen during training. Unless specified, the last two layers of the
visual encoder are made adjustable. The final two rows employ our default
training strategy and the only distinction lies in the inference phase.
Default (w/o img) takes vanilla CLIP [36] text embedding as input condition,
whereas Default (w/ img) employs the fused embedding. As indicated by Table 3,
integrating character visual features during training significantly enhances
the generation performance and the additional cross-attention control propels
the model to achieve its peak on accurate character generation. Note that the
FID score of Default (w/ img) is slightly higher than Default (w/o img). This
is because, during inference, the reference images for corresponding
characters in Default (w/ img) are obtained online, introducing a slight
deviation from the original distribution.

Models Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$) BG-F1
($\uparrow$) FID ($\downarrow$) w/o $\mathcal{L}_{reg}$ 88.86 95.21 55.50
60.77 23.51 w/o augmented text 87.45 94.70 57.67 63.04 21.27 freeze vis 88.67
95.14 56.58 62.46 22.01 Default (w/o img) 89.73 95.56 56.18 62.85 20.96
Default (w/ img) 90.36 95.76 58.36 63.92 21.13

Table 3: Ablation study for the first stage finetuning LDM with cross-
attention control.

Second stage ablation. As shown in Table 4, we conducted an ablation study on
(i) whether to align with the text embedding (Embtext) or the fused embedding
(Embfuse) of our first stage model; (ii) whether the model’s input consists of
a sequence of captions (Caption-) or utilizes interleaved training with both
images and captions (Interleave-) (Equation 5). Experimental results shown in
Table 4 indicate that image-text interleave training can significantly enhance
performance. It is intuitive that taking both images and corresponding
captions as input provides a more profound comprehension of the characters and
their interactions within the image than when provided with sole captions.
This, in turn, amplifies its generative capabilities.

Models Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$) BG-F1
($\uparrow$) FID ($\downarrow$) Caption-Embtext 69.70 83.37 52.67 58.78 21.32
Caption-Embfuse 71.77 84.81 52.57 58.04 24.79 Interleave-Embtext 86.10 93.46
54.92 60.15 21.30 Interleave-Embfuse (default) 87.96 94.17 56.01 61.07 21.71

Table 4: Second stage training strategy ablation. Input only caption or
interleaved text and image. The output of LLM is aligned with our Char-LDM
text embedding (Embtext) or character-augmented fused embedding (Embfuse).

###  4.4 Analysis

We further investigate the impact of first-stage finetuning with cross-
attention control by visualizing averaged cross-attention maps in U-Net latent
pixel space and interpolating them to match the size of the generated images.
As illustrated in Figure 5, vanilla LDM (top) finetune on FlintStonesSV [12]
w/o $\mathcal{L}_{reg}$ (Section 3.2) fails to accurately focus on the
corresponding characters for character tokens. Our model (bottom), which
incorporates cross-attention guidance, is able to precisely direct attention
to generated characters given corresponding character tokens.

Figure 5: Visualization of cross attention maps of corresponding character
tokens.

###  4.5 Properties

Our model could generate longer stories featuring accurate characters, at a
faster speed and with lower computational consumption. Our architecture allows
our model to retain an extensive context requiring minimal computational
resources by efficiently mapping visual features into tokens instead of
operating in pixel space. Figure 6 shows the comparison between our model and
Story-LDM [38] w.r.t GPU memory consumption and inference speed for longer-
frames story generation. Our model is capable of producing sequences exceeding
50 frames with low memory usage, whereas Story-LDM [38] encounters GPU memory
limitations (80G A100) when generating 42 frames. This is because Story-LDM
[38] requires the retention of the entire context, e.g., $n$ frames in latent
pixel space ($n\times h\times w\times d$), whereas our model processes visual
features as four token embedding ($n\times 4\times d$) with the same
dimensions as the text tokens in LLM. Table 5 compares the accuracy of
generated characters and FID score for long story visualizations between our
model and Story-LDM [38]. The performance of Story-LDM [38] significantly
decreases when generating longer stories and reaches the memory limit before
50 frames. In contrast, by utilizing the capacity of LLM to retain extensive
context, our model upholds accurate character consistency in visualizing
lengthy narratives with co-referential text descriptions.

Figure 6: Compare inference speed and GPU memory consumption between our
method and Story-LDM [38]. Story-LDM encounters the 80GB GPU limit when
generating sequences exceeding 40 frames.

Models Metric 4 10 20 40 50 Story-LDM [38] Char-Acc ($\uparrow$) 77.23 74.84
69.01 63.40 N/A FID ($\downarrow$) 36.34 48.92 53.32 60.33 N/A StoryGPT-V
(Ours) Char-Acc ($\uparrow$) 85.44 84.63 82.86 81.04 80.92 FID ($\downarrow$)
27.08 38.91 42.60 48.37 61.23

Table 5: Longer-frames story visualization comparison on FlintStonesSV [12]
with referential text. Story-LDM reaches maximum GPU capacity when generating
50 frames.

Our design could be easily adapted to any LLMs. In our work, we experimented
with OPT-6.7b444https://huggingface.co/facebook/opt-6.7b and
Llama2-7b-chat555https://huggingface.co/meta-llama/Llama-2-7b-chat models. Our
findings, as illustrated in Table 9, indicate an improvement when changing
from OPT [58] to a more powerful model, Llama2 [48].

Models # Params Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$)
BG-F1 ($\uparrow$) FID ($\downarrow$) BLEU4 ($\uparrow$) CIDEr ($\uparrow$)
OPT [58] 6.7b 87.96 94.17 56.01 61.07 21.71 0.5070 1.6607 Llama2 [48] 7b 89.08
95.07 57.29 62.62 21.56 0.5169 1.7516

Table 6: Performance on FlintstonesSV [12] dataset with referential text using
different LLMs.

Our model, StoryGPT-V, is capable for multi-model generation. Owing to
StoryGPT-V design leveraging the advanced capabilities of Large Language
Models (LLMs), it exhibits a unique proficiency in that it can extend visual
stories. StoryGPT-V is not merely limited to visualizing stories based on
provided textual descriptions. Unlike existing models, it also possesses the
innovative capacity to extend these narratives through continuous text
generation. Concurrently, it progressively synthesizes images that align with
the newly generated text segments.

Our model represents a notable advancement in story visualization, being the
first of its kind to consistently produce both high-quality images and
coherent narrative descriptions. This innovation opens avenues for AI-assisted
technologies to accelerate visual storytelling creation experiences by
exploring various visualized plot extensions as the story builds.

##  5 Conclusion

In this paper, we aim at high-quality and consistent character synthesis for
story visualization grounded on co-referential text descriptions. To
accomplish this, we utilize the strengths of the LDM for generating high-
quality images, combined with the reasoning capability of LLM to comprehend
extended contexts, resolve ambiguities, and ensure semantic consistency in the
generation process. We first finetune LDM by guiding the cross-attention map
of LDM with character segmentation masks, which improves the accuracy and
faithfulness of character generation. Next, we facilitate a mapping from the
output of LLM to align with the input space of the first stage LDM, thus
allowing Multi-modal LLM to both process and produce images. This process
leverages the LLM’s logical reasoning to clarify ambiguous references and its
capacity to retain contextual information. Our model reports superior
quantitative results and consistently generates characters with remarkable
quality.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Aone and William [1995] Chinatsu Aone and Scott William.  Evaluating automated and manual acquisition of anaphora resolution strategies.  In _33rd Annual Meeting of the Association for Computational Linguistics_ , pages 122–129, 1995.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2022a] Hong Chen, Rujun Han, Te-Lin Wu, Hideki Nakayama, and Nanyun Peng.  Character-centric story visualization via visual planning and token alignment.  _arXiv preprint arXiv:2210.08465_ , 2022a.
  * Chen et al. [2022b] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.  Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18030–18040, 2022b.
  * Chen et al. [2023] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023.
  * Couairon et al. [2023] Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek.  Zero-shot spatial layout conditioning for text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2174–2183, 2023.
  * Crowson et al. [2022] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  In _European Conference on Computer Vision_ , pages 88–105. Springer, 2022.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34:19822–19835, 2021.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _European Conference on Computer Vision_ , pages 89–106. Springer, 2022.
  * Ge et al. [2023] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.  Planting a seed of vision in large language model.  _arXiv preprint arXiv:2307.08041_ , 2023.
  * Gupta et al. [2018] Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi.  Imagine this! scripts to compositions to videos.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 598–613, 2018.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2023] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Koh et al. [2023a] Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov.  Generating images with multimodal language models.  _arXiv preprint arXiv:2305.17216_ , 2023a.
  * Koh et al. [2023b] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried.  Grounding language models to images for multimodal generation.  _arXiv preprint arXiv:2301.13823_ , 2023b.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023a.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023b.
  * Li et al. [2019] Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao.  Storygan: A sequential conditional gan for story visualization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6329–6338, 2019.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023a.
  * Liu et al. [2023b] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  _arXiv preprint arXiv:2303.04761_ , 2023b.
  * Ma et al. [2023] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu.  Unified multi-modal latent diffusion for joint subject and text conditional image generation.  _arXiv preprint arXiv:2303.09319_ , 2023.
  * Maharana and Bansal [2021] Adyasha Maharana and Mohit Bansal.  Integrating visuospatial, linguistic and commonsense structure into story visualization.  _arXiv preprint arXiv:2110.10834_ , 2021.
  * Maharana et al. [2021] Adyasha Maharana, Darryl Hannan, and Mohit Bansal.  Improving generation and evaluation of visual stories via semantic consistency.  _arXiv preprint arXiv:2105.10026_ , 2021.
  * Maharana et al. [2022] Adyasha Maharana, Darryl Hannan, and Mohit Bansal.  Storydall-e: Adapting pretrained text-to-image transformers for story continuation.  In _European Conference on Computer Vision_ , pages 70–87. Springer, 2022.
  * McCarthy and Lehnert [1995] Joseph F McCarthy and Wendy G Lehnert.  Using decision trees for coreference resolution.  _arXiv preprint cmp-lg/9505043_ , 1995.
  * OpenAI [2023] OpenAI.  Dall-e 3.  https://openai.com/dall-e-3/, 2023.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021a.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Rahman et al. [2023] Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal.  Make-a-story: Visual memory conditioned consistent story generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2493–2502, 2023.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Seo et al. [2017] Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal.  Visual reference resolution using attention memory for visual dialog.  _Advances in neural information processing systems_ , 30, 2017.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _arXiv preprint arXiv:2304.03411_ , 2023.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yun-Zhu Song, Zhi Rui Tam, Hung-Jen Chen, Huiao-Han Lu, and Hong-Han Shuai.  Character-preserving coherent story visualization.  In _European Conference on Computer Vision_ , pages 18–33. Springer, 2020b.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Tsimpoukelli et al. [2021] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ , 34:200–212, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Wang et al. [2023] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  _arXiv preprint arXiv:2305.11175_ , 2023.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.
  * Xiao et al. [2023] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han.  Fastcomposer: Tuning-free multi-subject image generation with localized attention.  _arXiv preprint arXiv:2305.10431_ , 2023.
  * Zeng et al. [2019] Gangyan Zeng, Zhaohui Li, and Yuan Zhang.  Pororogan: An improved story visualization model on pororo-sv dataset.  In _Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence_ , pages 155–159, 2019.
  * Zeqiang et al. [2023] Lai Zeqiang, Zhu Xizhou, Dai Jifeng, Qiao Yu, and Wang Wenhai.  Mini-dalle3: Interactive text to image by prompting large language models.  _arXiv preprint arXiv:2310.07653_ , 2023.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhang et al. [2023] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.  Llavar: Enhanced visual instruction tuning for text-rich image understanding.  _arXiv preprint arXiv:2306.17107_ , 2023.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

## Appendix

Figure 7: Our model StoryGPT-V extending stories in both language and vision:
Gray part represents the text descriptions from datasets. Blue part
corresponds to the frames and the continued written stories based on the
previous captions generated by our model StoryGPT-V. This is the first model
capable of story visualization and multi-modal story generation (continuation)
by leveraging an LLM.

##  Appendix A Multi-modal Story Generation

Owing to StoryGPT-V design leveraging the advanced capabilities of Large
Language Models (LLMs), it exhibits a unique proficiency in that it can extend
visual stories. StoryGPT-V is not merely limited to visualizing stories based
on provided textual descriptions. Unlike existing models, it also possesses
the innovative capacity to extend these narratives through continuous text
generation. Concurrently, it progressively synthesizes images that align with
the newly generated text segments.

Figure 7 presents an example of a multi-modal story generation. Initially, the
first four frames are created according to the text descriptions from the
FlintstonesSV [12] dataset (gray part). Subsequently, the model proceeds to
write the description for the next frame (blue part), taking into account the
captions provided earlier, and then creates a frame based on this new
description (blue part). This method is employed iteratively to generate
successive text descriptions and their corresponding frames.

Our model represents a notable advancement in story visualization, being the
first of its kind to consistently produce both high-quality images and
coherent narrative descriptions. This innovation opens avenues for AI-assisted
technologies to accelerate visual storytelling creation experiences by
exploring various visualized plot extensions as the story builds.

##  Appendix B Ablation Studies

###  B.1 Effect of first-stage design.

In Table 7 lower half, we conducted an ablation study on how the stage-1
design contributes to the final performance. In the first line, the stage-2
LLM is aligned with vanilla LDM fine-tuned on FlintstonesSV [12]. The second
line aligns the LLM output with our Char-LDM’s text embedding (Embtext), while
the last line aligns with character-augmented fused embedding (Embfuse) of our
Char-LDM. The first two lines align to the same text embedding encoded by the
CLIP [36] text encoder, however, our Char-LDM enhanced with cross-attention
control ($\mathcal{L}_{reg}$) produces more precise characters. Different from
Embtext, the last line is aligned with Embfuse, which is augmented with
characters’ visual features. This visual guidance helps LLM to interpret
references more effectively by linking “he, she, they” to the previous
language and image context.

Models Aligning space Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc
($\uparrow$) BG-F1 ($\uparrow$) FID ($\downarrow$) Vanilla LDM [41] $\times$
75.37 87.54 52.57 58.41 32.36 Our Stage-2 Vanilla LDM Embtext 84.06 92.54
53.18 58.29 22.94 Char-LDM Embtext 86.10 93.46 54.92 60.15 21.30 Char-LDM
Embfuse (default) 87.96 94.17 56.01 61.07 21.71

Table 7: The output of our stage-2 model is aligned with conditional input of
vanilla LDM [41] (finetuned on FlintstonesSV [12]), our Char-LDM text
embedding (Embtext) or character-augmented fused embedding (Embfuse).

###  B.2 Number of [IMG] Tokens

We further examined the impact of the number of added [IMG] tokens. As
indicated in Table 8, aligning with the fused embedding and setting $R=8$
yields the best performance.

Models R Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$) BG-F1
($\uparrow$) FID ($\downarrow$) Embtext 4 82.14 90.18 54.28 59.58 21.33
Embtext 8 86.10 93.46 54.92 60.15 21.30 Embtext 16 83.77 91.07 54.08 60.21
21.58 Embfuse 4 86.23 93.43 54.57 59.61 21.97 Embfuse 8 87.96 94.17 56.01
61.07 21.71 Embfuse 16 85.35 91.96 52.93 58.86 23.73

Table 8: StoryGPT-V Ablations: Impact of $R$, the number of added [IMG]
tokens. Embtext: the output of LLM is aligned with text embedding extracted
from the text encoder; Embfuse: aligned with fused embedding Embfuse of first
stage model.

###  B.3 Different LLMs (OPT vs Llama2)

Models # Params Char-Acc ($\uparrow$) Char-F1 ($\uparrow$) BG-Acc ($\uparrow$)
BG-F1 ($\uparrow$) FID ($\downarrow$) BLEU4 ($\uparrow$) CIDEr ($\uparrow$)
OPT [58] 6.7b 87.96 94.17 56.01 61.07 21.71 0.5070 1.6607 Llama2 [48] 7b 89.08
95.07 57.29 62.62 21.56 0.5169 1.7516

Table 9: Performance on FlintstonesSV [12] dataset with referential text using
different LLMs.

Our primary contribution lies in leveraging Large Language Models (LLMs) for
reference resolution for consistent story visualization. In our work, we
experimented with OPT-6.7b666https://huggingface.co/facebook/opt-6.7b and
Llama2-7b-chat777https://huggingface.co/meta-llama/Llama-2-7b-chat models.
It’s important to note that the utilization of Llama2 was specifically to
demonstrate its additional capability for multi-modal generation. The ablation
study of different LLMs was not the main focus of our research.

Our findings, as illustrated in Table 9, indicate only a slight improvement
when changing from OPT [58] to Llama2 [48]. This marginal difference is
attributed to the evaluation metric’s emphasis on image-generation
capabilities, which assesses whether the model’s visual output aligns well
with first-stage Char-LDM’s conditional input space.

##  Appendix C Evaluation

###  C.1 Text-image alignment.

CLIP [36] is trained on large-scale image-caption pairs to align visual and
semantic space. However, a domain gap exists between pre-train data and the
story visualization benchmark. Therefore, we finetune CLIP [36] on the story
visualization datasets. However, we found it still hard to capture fine-
grained semantics, either text-image (T-I) similarity or image-image
similarity (I-I), i.e., the similarity between visual features of generated
images and corresponding ground truth images.

Upon this observation, we choose the powerful captioning model BLIP2 [21] as
the evaluation model. We finetune BLIP2 on FlintstonesSV [12] and PororoSV
[22], respectively, and employ it as an image captioner for generated visual
stories. We avoided direct comparisons to bridge the gap between BLIP2’s
predictions and the actual ground truth captions. Instead, we used the fine-
tuned BLIP2 to generate five captions for each ground truth image and one
caption for each generated image. and report average BLEU4 [31] or CIDEr [51]
score based on these comparisons.

Models CLIP (T-I) ($\uparrow$) CLIP (I-I) ($\uparrow$) BLEU4 ($\uparrow$)
CIDEr ($\uparrow$) StoryDALL-E [28] 0.4417 0.8112 0.4460 1.3373 LDM [41]
0.5007 0.8786 0.4911 1.5103 Story-LDM [38] 0.4979 0.8795 0.4585 1.4004
StoryGPT-V (Ours) 0.5106 0.889 0.5070 1.6607

Table 10:  Text-image alignment score for FlintstonesSV [12] with referential
text descriptions in terms of CLIP [36] similarity, BLEU4 [31] and CIDEr [51].

###  C.2 Human evaluation.

We use Mechanical Turk to assess the quality of 100 stories produced by our
methods or Story-LDM [38] on FlintStonesSV [12]. Given a pair of stories
generated by Story-LDM [38] and our model, people are asked to decide which
generated four-frame story is better w.r.t visual quality, text-image
alignment, character accuracy and temporal consistency. Each pair is evaluated
by 3 unique workers. The human study interface is illustrated in Figure 8.

Figure 8: Human study interface.

##  Appendix D Implementation Details

###  D.1 Data preparation

FlintstonesSV [12] provides the bounding box location of each character in the
image. We fed the bounding boxes into SAM [17] to obtain the segmentation map
of corresponding characters. This offline supervision from SAM [17] is
efficiently obtained without the need for manual labeling efforts.
Furthermore, we enhance the original datasets from resolution of 128x128 to
512x512 via a super-resolution model888https://huggingface.co/CompVis/ldm-
super-resolution-4x-openimages and then we proceed to train and evaluate all
models on this enhanced dataset.

###  D.2 Extending dataset with referential text

We follow Story-LDM [38] to extend the datasets with referential text by
replacing the character names with references, i.e., he, she, or they,
wherever applicable as shown in Algorithm 1. The statistics before and after
the referentail extension are shown in Table 11. Please refer to Story-LDM
[38] implementation999https://github.com/ubc-vision/Make-A-
Story/blob/main/ldm/data for more details on how the referential dataset is
extended.

Dataset # Ref (avg.) # Chars # Backgrounds FlintstonesSV [12] 3.58 7 323
Extended FlintstonesSV 4.61 7 323 PororoSV [22] 1.01 9 None Extended PororoSV
1.16 9 None

Table 11: Dataset statistics of FlintstonesSV [12] and PororoSV [22]

###  D.3 First stage training

We built upon pre-trained Stable Diffusion [41]
v1-5101010https://huggingface.co/runwayml/stable-diffusion-v1-5 and use CLIP
[35] ViT-L to extract characters’ visual features. We freeze the CLIP text
encoder and fine-tune the remaining modules for 25,000 steps with a learning
rate of 1e-5 and batch size of 32. The first stage utilizes solely the
original text description without extended referential text. To enhance
inference time robustness and flexibility, with or without reference images,
we adopt a training strategy that includes $10\%$ unconditional training,
i.e., classifier-free guidance [14], $10\%$ text-only training, and $80\%$
augmented text training, which integrates visual features of characters with
their corresponding token embeddings.

###  D.4 Second stage training

We use OPT-6.7B111111https://huggingface.co/facebook/opt-6.7b model as the LLM
backbone in all experiments in the main paper. To expedite the second stage
alignment training, we first pre-compute non-referential fused embeddings
residing in the input space of the first-stage Char-LDM. We map visual
features into $m=4$ token embeddings as LLM input, set the max sequence length
as 160 and the number of additional [IMG] tokens as $R=8$, batch size as 64
training for 20k steps. Llama2 is only trained for the experiments highlighted
in the supplementary materials, demonstrating its capability for multi-modal
generation and the ablation of different LLMs. The training configuration is
almost the same as OPT, except for batch size 32. All experiments are executed
on a single A100 GPU.

Please refer to all the details at the source code.

Algorithm 1 Character Replacement Algorithm

Definitions:

$i$: index for frames, ranging from 1 to $N$

$S_{i}$: text description of frame $i$

$\mathcal{C}_{i}$: a set contains immediate character(s) in the current frame

for $i\in\\{1,2,\ldots,N\\}$ do

if $i=1$ then

$\mathcal{C}_{i}\leftarrow$ immediate character of $S_{i}$

else

if $\mathcal{C}_{i}\subseteq\mathcal{C}_{i-1}$ then

if $\mathrm{length}(\mathcal{C}_{i})=1$ then

Replace $\mathcal{C}_{i}$ in $S_{i}$ with “he” or “she”

else if $\text{length}(c)>1$ then

Replace $\mathcal{C}_{i}$ in $S_{i}$ with “they”

end if

end if

$\mathcal{C}_{i}\leftarrow\mathcal{C}_{i-1}$

end if

end for

Figure 9: DALL-E 3 [30] zero-shot inference on FlintstonesSV [12] dataset.

##  Appendix E Limitations

Our method demonstrates proficiency in resolving references and ensuring
consistent character and background conditions in the context provided by
guiding the output of a multi-modal Large Language Model (LLM) with character-
augmented semantic embedding. However, several limitations remain. The process
involves feeding the previously generated frame into the LLM to produce a
visual output that aligns with the Latent Diffusion Model (LDM) input
conditional space. This approach guarantees semantic consistency, enabling the
generation of characters and environmental objects that resemble their
originals. Nonetheless, there are minor discrepancies in detail. This is
because the visual output from the Large Language Model (LLM) is aligned with
the semantic embedding space rather than the pixel space, which hinders the
complete reconstruction of all elements in the input image. However, the
current most powerful multi-modal LLM, i.e., DALL-E 3 [30], could not solve
this exact appearance replication in the multi-round image generation task
(Figure 9), indicating an area ripe for further exploration and research.

##  Appendix F Qualitative Results

We provide more generated samples on FlintstonesSV [12] and PororoSV [22] with
referential text as Figure 11-19 show.

Figure 10: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 11: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 12: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 13: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 14: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 15: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 16: Qualitative comparison on FlintstonesSV [12] with co-reference
descriptions.

Figure 17: Qualitative comparison on PororoSV [22] with co-reference
descriptions.

Figure 18: Qualitative comparison on PororoSV [22] with co-reference
descriptions.

Figure 19: Qualitative comparison on PororoSV [22] with co-reference
descriptions.


# GuyLingo: The Republic of Guyana Creole Corpora

Christopher Clarke∗1 Roland Daynauth∗1 Charlene Wilkinson2
Hubert Devonish Jason Mars1
1University of Michigan, Ann Arbor, MI
2University of Guyana, Georgetown, Guyana
{csclarke, daynauth<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

While major languages often enjoy substantial attention and resources, the
linguistic diversity across the globe encompasses a multitude of smaller,
indigenous, and regional languages that lack the same level of computational
support. One such region is the Caribbean. While commonly labeled as "English
speaking", the ex-British Caribbean region consists of a myriad of Creole
languages thriving alongside English. In this paper, we present GuyLingo: a
comprehensive corpus designed for advancing NLP research in the domain of
Creolese (Guyanese English-lexicon Creole), the most widely spoken language in
the culturally rich nation of Guyana. We first outline our framework for
gathering and digitizing this diverse corpus, inclusive of colloquial
expressions, idioms, and regional variations in a low-resource language. We
then demonstrate the challenges of training and evaluating NLP models for
machine translation in Creole. Lastly, we discuss the unique opportunities
presented by recent NLP advancements for accelerating the formal adoption of
Creole languages as official languages in the Caribbean.

GuyLingo: The Republic of Guyana Creole Corpora

Christopher Clarke∗1 Roland Daynauth∗1 Charlene Wilkinson2 Hubert Devonish
Jason Mars1 1University of Michigan, Ann Arbor, MI 2University of Guyana,
Georgetown, Guyana {csclarke, daynauth<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

**footnotetext: Equal contribution. Figure 1: Map of Guyana and its
neighboring territories

##  1 Introduction

Major languages such as English and Chinese frequently receive considerable
attention and resources due to their global prominence and economic influence
Lent et al. (2021, 2022a). The extensive focus on these major languages in
natural language processing (NLP) has resulted in the development of
sophisticated models, extensive datasets, and digital applications consumed by
millions of users today. However, despite this global prominence, the
linguistic landscape of the globe extends far beyond these dominant languages,
encompassing a plethora of smaller, indigenous, and regional languages that
play crucial roles in the cultural heritage and communication of their
respective communities Lent et al. (2022c); Hershcovich et al. (2022). The
countries of the Commonwealth (ex-British) Caribbean Community represent an
example cluster of such countries.

Creole |  English
---|---
“It luk laik nof ting cheenj op,” Seera se. “Somtaim mi doz get fraikn.” |  “So many things feel like they have changed,” said Sara. “I get scared about it sometimes.”
When me lef’ han’ ’cratch me, money a-come |  When my left hand itches, money is coming.
Di leedii prapa nais |  The lady is very pretty

Table 1: Example Guyanese Creole from GuyLingo and its English Translation

Within the diverse linguistic tapestry of the Caribbean Community, a rich
array of languages thrives, reflecting the historical, cultural, and ethnic
diversity of the region Rickford (1987); Holbrook and Holbrook (2001). While
English is commonly used as the official language in many Commonwealth
Caribbean states, the linguistic heritage goes beyond just English,
encompassing a variety of Creole languages, indigenous languages, and
influences from African, Indigenous, European, and Asian languages Devonish
and Thompson (2013).

Creole languages of the Caribbean emerged out of the language contact between
Europeans and Africans arising from colonialism and plantation slavery. These
languages, such as Jamaican, otherwise referred to as Jamaican Creole or
Jamaican Patois/Patwa Armstrong et al. (2022), Trinidadian Creole Michaelis et
al. (2013), and Haitian Creole Hewavitharana et al. (2011), have evolved as
vibrant means of communication, showing language features originating in West
African languages as well as the languages of the colonizing Europeans.
Hagemeijer et al. (2014b).

Despite its prominence as the mother tongue of the majority of the over
700,000 inhabitants of the Republic of Guyana, Creolese (Guyanese English-
lexicon Creole) is a low-status vernacular language that takes second place to
the sole official language, English. This is typical of local vernaculars in
post-colonial situations like Guyana Hershcovich et al. (2022). English has
been traditionally the only language in which Guyanese children are taught to
read and write in school. Written resources in Creolese are limited, making it
a low-resource language within the field of Natural Language Processing (NLP).

In this work, we introduce GuyLingo, a corpus for Creolese curated for
advancing NLP research and development in Creole. Using this resource, we
explore the task of machine translation between English and Creolese. To aid
in this process we design and implement the Guyanese Creole Translation
tool111https://translation.csclarke.com, a web-based GPT-powered machine
translation tool. Lastly, we briefly discuss the insights gained from
developing Guylingo alongside considerations for accelerating the formal
adoption of Creole languages as official languages in the Caribbean.

##  2 GuyLingo Corpus

This section describes the curation of GuyLingo, a corpus of Creolese, the
primary spoken language of Guyana. The creation of this corpus aims to address
the scarcity of resources and attention devoted to indigenous and regional
languages within the NLP community.

###  2.1 Data Collection

The compilation of GuyLingo requires the collecting and digitizing of a series
of linguistic resources. These sources should ideally encompass a spectrum of
Creolese expressions, idiomatic phrases, and regional variations. To ensure
inclusivity and authenticity, we employ a multi-pronged approach.

####  2.1.1 Expert Collaboration

In collaboration with the University of Guyana, Guyanese Languages Unit, a
collection of original Guyanese Creole sources was curated, digitized, and
manually transcribed by a team of researchers. Examples of this include Speirs
(1902) a book of Guyanese proverbs, containing over 1k culturally rich
proverbs from early British Guiana times still used today, and Helen Patuck
(2020) a COVID-19 children’s book transcribed by Creolese experts for primary
education students. In addition, our team of native Creole experts manually
construct a corpus of high-quality common Guyanese Creole sayings and terms.
Table 2 shows a full breakdown of all information sources.

####  2.1.2 Online Resources

Whilst some of the aforementioned sources use the consistent phonemic Cave-GLU
standard writing system Cave (1970) for creole, others do not. This is
particularly true for the many web-based sources such as language forums,
blogs, educational platforms, etc., that contain small excerpts of
colloquialisms, everyday conversations, and idiomatic expressions prevalent in
the Guyanese Creole. These sources were scraped, cleaned, verified, and added
to GuyLingo as shown in 2.

Sources |  Type | # Sentences | Vocab Size
---|---|---|---
Guyanese-Creole-English Vocabulary-Basic words. Polyglot Club (Accessed 2023) |  Corpus | 20 | 71
Guyanese Creole. Wikipedia (Accessed 2023) |  Article | 6 | 28
Gender and Pronominal Variation in an Indo-Guyanese Creole-Speaking Sidnell (1999) |  Journal Article | 21 | 82
Review of Guyanese Creole English Guy (Accessed 2023) |  Presentation | 28 | 96
Guyanese Creole Survey Report. Holbrook and Holbrook (2001) |  Language Survey | 8 | 45
APiCS Online -Structure dataset. Michaelis et al. (2013) |  Report | 344 | 351
Creolese. Devonish and Thompson (2013) |  Journal Article | 69 | 112
Habitual and Imperfective in Guyanese Creole. Sidnell (2002) |  Journal Article | 60 | 103
Tense and aspect in Guyanese Creole: A syntactic, semantic and pragmatic analysis Gibson (1982) |  PhD Thesis | 231 | 374
Two areas of Guyanese Grammar Guyanese Languages Unit (2016) |  Article | 14 | 26
Me Na Able: Creolese 101 Letters from Guyana (2017) |  Blog | 9 | 25
Travel Phrases - Guyanese Creole Travel Phrases  |  Blog | 4 | 9
My Hero is you Helen Patuck (2020) |  Educational | 322 | 831
The Proverbs of British Guiana Speirs (1902) |  Book | 905 | 2054
Common Guyanese Creole Sayings (Manually created by experts) |  Corpus | 332 | 712
Total |  | 2373 | 4177
Table 2: Compilation of Guyanese Creole Language Resources: Sources, Type,
Sentences, and Vocabulary Size

###  2.2 Dataset Characteristics

GuyLingo encapsulates a diverse array of linguistic data, including but not
limited to:

  * •

Conversational dialogues

  * •

Idiomatic expressions and phrases

  * •

Proverbs and folklore

  * •

Regional variations and dialectical nuances

In total, GuyLingo consists of 2373 Guyanese Creole sentences with a
vocabulary size of 4177 unique Creole words.

##  3 GuyLingo for Machine Translation

To investigate the utility of GuyLingo, we conduct experiments on the task of
machine translation assessing the ability of NLP models to facilitate
English$\leftarrow\rightarrow$Guyanese Creole translation. As such to enable
the training and evaluation of these models GuyLingo was further expanded to
include English Creole translation pairs. Of the 2373 sentences, the Common
Guyanese Creole Sayings corpus (expertly curated) was manually transcribed
into English. The remaining Creole sources were extracted alongside their
English translations. In addition, 339 common Creole terms from (Speirs, 1902)
alongside their English pairs were extracted and verified. Using these initial
translation pairs, the Guyanese Creole Translation Tool was built to allow the
initial translation of remaining sentences in GuyLingo. Of the remaining
Creole sources, the largest source of Creolese data from Speirs (1902)
contained no English translations for the proverbs. As such, using the
translation tool these proverbs were machine translated. These machine
translations were reviewed and edited for lexical correspondence but not
semantic meaning given the complexity of translating the contextual meaning of
cultural proverbs.

Figure 2: User Interface of Guyanese Creole Translation Tool. This tool allows
experts to rapidly and iteratively create translation pairs using GPT-4 OpenAI
(2023) as a generator.

###  3.1 Guyanese Creole Translation Tool

The Guyanese Creole Translation tool, as shown in figure 2, is a web-based
application built using Django+React to facilitate easy storing, editing, and
iterative testing of English Creole translations. The UI allows Creolese
experts to easily enter text in English or Creole and get a sample
translation. We utilize GPT-4 OpenAI (2023) to automatically perform these
translations. The advanced prompt includes a subset of example verified
translations from GuyLingo as in-context examples for generation. Once
prompted, the user can modify the generated output before saving it to the
database. Users also have the option to modify the advanced prompts as well as
provide more seed examples for greater control over the translation process.
For instance, users can provide a Guyanese proverb and instruct GPT4 to
consider the nuances of the Guyanese culture while translating the text. As
mentioned previously, a subset of GuyLingo’s English pairs was generated using
this tool.

##### Translation Dataset Statistics

In total, our translation dataset consists of 1969 total translation pairs.
For training and evaluation, we use the 302 manually curated translation pairs
for testing and the remaining GuyLingo translation pairs for model training.
The manually curated translation pairs are all written in the Cave-GLU
standard phonemic system for Creoles Cave (1970).

###  3.2 Experiment Setup

##### Training and Models

We consider the models of T5 Raffel et al. (2023), BART Lewis et al. (2020)
and Pegasus Zhang et al. (2020) for their demonstrated performance on several
machine translation tasks. All models were implemented with PyTorch and
Hugging Face Transformers. We train all models with AdamW (Loshchilov and
Hutter, 2019) and a weight decay of 0.01. We use a learning rate of 2e-5,
batch size of 4, and a linear learning rate warmup over the first 10% steps
with a cosine schedule. We pre-process the data and train all models with
varying random seeds over multiple runs for 10 epochs. Approximately 200 GPU
hours were required to train all hyperparameter variations across all tasks.
Additionally, we evaluate the performance of GPT-4 using in-context learning
on GuyLingo.

##### Evaluation

For evaluation metrics, we adopted common automatic evaluation methods used
for language generation based on n-gram overlap: BLEU Papineni et al. (2002),
ROUGE Lin (2004), METEOR Banerjee and Lavie (2005) and CHRF Popović (2015).

Model | Bleu | Rouge1 | Rouge2 | RougeL | Meteor | CHRF
---|---|---|---|---|---|---
GPT-4 (Zero-shot) | 1.35 | 17.22 | 2.4 | 17.0 | 12.55 | 21.68
GPT-4 (Few-shot) | 1.64 | 20.6 | 3.42 | 20.2 | 14.56 | 22.32
T5-Large | 09.74 | 37.44 | 13.74 | 36.63 | 28.19 | 30.09
Bart-Large | 12.11 | 40.56 | 18.47 | 39.64 | 32.77 | 33.21
Bart-Base | 10.17 | 37.49 | 16.08 | 36.59 | 29.54 | 29.47
Pegasus-Large | 02.67 | 24.15 | 05.30 | 23.16 | 16.38 | 19.69

Table 3: Performance of MT Models on English-Creole Translation

Model | Bleu | Rouge1 | Rouge2 | RougeL | Meteor | CHRF
---|---|---|---|---|---|---
GPT-4 (Zero-shot) | 29.8 | 60.4 | 38.25 | 58.96 | 52.94 | 51.33
GPT-4 (Few-shot) | 30.24 | 60.6 | 39.31 | 58.99 | 54.14 | 51.61
T5-Large | 19.70 | 47.71 | 26.99 | 46.47 | 42.45 | 39.89
Bart-Large | 17.70 | 45.74 | 24.41 | 39.75 | 32.77 | 37.23
Bart-Base | 14.20 | 41.68 | 20.04 | 40.40 | 35.95 | 34.33
Pegasus-Large | 6.10 | 28.96 | 09.88 | 27.91 | 22.53 | 23.72

Table 4: Performance of MT Models on Creole-English Translation

##  4 Results

Table 3 and 4 summarize our evaluation results on automated metrics. For en-
creole translation, our results show that the Bart-Large model achieves the
best performance amongst all models with a BLEU score of 12.11, ROUGE-1 score
of 40.56, ROUGE-2 score of 18.47, ROUGE-L score of 39.64, METEOR score of
32.77 and a CHRF score of 33.21 outperforming other fine-tuned models such as
T5 and large language models such as GPT-4 both in zero and few-shot prompting
settings. The performance of en-creole translation is due to a couple of
factors: 1) Incoherent English to Creole mapping: Many unique words/phrases
found in Creole do not contain English equivalents. 2)  Writing System of
Testing set: The Creolese Cave-GLU writing system Cave (1970) used by the
manually curated testing set is phonemic whereby a particular sound is always
represented by one letter/letter combination. GuyLingo’s training data,
however, contains samples where multiple letter/letter combinations represent
one sound. Thus, the chance of an error is much higher for en-creole
translations.

For creole-eng translation, GPT-4 (Few-shot) using a subset of GuyLingo
training set as in-context learning examples delivers the best performance
with a BLEU score of 30.24. a ROUGE-1 score of 60.6, a ROUGE-2 score of 39.31,
a ROUGE-L score of 58.99, a METEOR score of 54.14, and a CHRF score of 51.61.
This result highlights the power of GPT-4’s large and extensive training on a
diverse and extensive range of text data in addition to its ability to quickly
adapt to new tasks or language pairs with only a few examples.

##  5 Discussion

In this section, we briefly discuss the unique opportunity presented by recent
NLP advancements for accelerating the formal adoption of Creole languages in
the Caribbean.

###  5.1 AI-Driven Applications for Native Languages

One of the major issues affecting the formal adoption is Creolese despite its
prominence as a spoken language is its lack of use in formal communication
outlets such as literature, news, and written texts. AI-driven applications
fueled by rich data sources such as GuyLingo present a major opportunity for
enabling the development of educational content, legal documents, and official
communications in Creolese. Figure 3 showcases a conversational AI Assistant
named IRIS 222https://fb.watch/rbtO5Wocny/ deployed to citizens of Guyana
speaking in Creolese using GuyLingo. Such applications present the ability to
make Creolese more accessible and applicable in various formal contexts
further allowing citizens to feel empowered and foster a sense of national
pride.

Figure 3: Conversational Agent in Whatsapp speaking in Guyanese Creole.

##  6 Related Works

In the context of linguistic diversity, prior works Hershcovich et al. (2022);
Lent et al. (2021, 2022a) have highlighted the challenges faced by lesser-
known languages, emphasizing the importance of recognition and preservation.
Works such as Dabre and Sukhoo (2022), Hagemeijer et al. (2014a), and Liu et
al. (2022) have contributed to advancing NLP research in Creole languages by
building a corpus of text for various Creole languages, fostering machine
translation, and enhancing language modeling techniques specific to these
linguistic varieties. Our work falls into this category. On the other hand,
works such as Lent et al. (2022b) and Lent et al. (2022c) emphasize the
importance of linguistic diversity by documenting the challenges and exploring
the complexities of language modeling for underrepresented languages. The
juxtaposition of these studies with the dominance of major languages in NLP
underscores the need for more inclusive research efforts that consider the
linguistic richness and cultural significance of smaller, indigenous languages
within global technological advancements.

##  7 Conclusion

In this paper, we introduce GuyLingo, a corpus of Guyanese Creolese designed
to facilitate advancements in NLP research. We discuss the process of
gathering and digitizing this diverse corpus while highlighting the unique
opportunities presented by recent NLP advancements for accelerating the formal
adoption of Creole languages in the Caribbean. By providing access to a rich
collection of colloquial language expressions, idioms, and regional
variations, we hope to encourage further research in this field and improve
the representation and understanding of Creole languages in NLP.

##  8 Limitations

While our work aims to contribute to the advancement of NLP for Creole,
several limitations arise:

Limited Representation: Guyana is home to many languages outside of Creolese
such as Wapichan, Makushi, Wai Wai, Akawaio, Arekuna, Patamuna, Kalina
(Carib), Warrau, and Lokono to name a few. Given the cultural significance of
these languages, future research should prioritize their inclusion to ensure a
more inclusive and representative dataset. Additionally, The rich tapestry of
languages in the region extends beyond Guyanese Creole, and efforts should be
made to include additional Creole languages and dialects for a more
comprehensive understanding.

Limited Generalizability: The findings and insights gained from our work,
particularly regarding the formal adoption of Creole languages, may have
limited generalizability to other regions or linguistic contexts.

Language Evolution: Creole languages, by their nature, are dynamic and subject
to continuous evolution. The static nature of a curated corpus and machine
translation models may not fully capture the evolving linguistic landscape,
necessitating regular updates and adaptations to reflect current linguistic
usage.

## Acknowledgements

We thank our anonymous reviewers for their feedback and suggestions. This work
is supported in part by the North American Chapter of the Association for
Computational Linguistics (NAACL) Regional Americas Fund.
333https://www.aclweb.org/portal/content/naacl-regional-americas-fund-cfp-2023

## References

  * Guy (Accessed 2023) Accessed 2023.  Guyanese creole english.  Accessed on December 14, 2023.
  * Armstrong et al. (2022) Ruth-Ann Armstrong, John Hewitt, and Christopher Manning. 2022.  JamPatoisNLI: A jamaican patois natural language inference dataset.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5307–5320, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Cave (1970) George N. Cave. 1970.  Some sociolinguistic factors in the production of standard language in guyana and implications for the language teacher.  _Language Learning_ , 20(2):249–263.
  * Dabre and Sukhoo (2022) Raj Dabre and Aneerav Sukhoo. 2022.  Kreolmorisienmt: A dataset for mauritian creole machine translation.  In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_ , pages 22–29.
  * Devonish and Thompson (2013) Hubert Devonish and Dahlia Thompson. 2013.  Creolese.  In Susanne Maria Michaelis, Philippe Maurer, Martin Haspelmath, and Magnus Huber, editors, _The Survey of Pidgin and Creole Languages, Vol. I: English-based and Dutch-based languages_ , pages 49–60. Oxford University Press, Oxford.
  * Gibson (1982) Kean Amelia Gibson. 1982.  _Tense and aspect in Guyanese Creole: A syntactic, semantic and pragmatic analysis_.  Ph.D. thesis, University of York.
  * Guyanese Languages Unit (2016) Guyanese Languages Unit. 2016.  Two areas of guyanese grammar.  https://guyaneselanguagesunit.com/2016/07/12/two-areas-of-guyanese-grammar/.  Accessed on December 14, 2023.
  * Hagemeijer et al. (2014a) Tjerk Hagemeijer, Michel Généreux, IHE Hendrickx, Amália Mendes, Abigail Tiny, and Armando Zamora. 2014a.  The gulf of guinea creole corpora.
  * Hagemeijer et al. (2014b) Tjerk Hagemeijer, Michel Généreux, Iris Hendrickx, Amália Mendes, Abigail Tiny, and Armando Zamora. 2014b.  The Gulf of Guinea creole corpora.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 523–529, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Helen Patuck (2020) Helen Patuck. 2020.  My Hero is You: How Kids Can Fight COVID-19!  https://www.unicef.org/coronavirus/my-hero-you.  Accessed on: Insert Date Accessed.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022.  Challenges and strategies in cross-cultural nlp.
  * Hewavitharana et al. (2011) Sanjika Hewavitharana, Nguyen Bach, Qin Gao, Vamshi Ambati, and Stephan Vogel. 2011.  CMU Haitian Creole-English translation system for WMT 2011.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 386–392, Edinburgh, Scotland. Association for Computational Linguistics.
  * Holbrook and Holbrook (2001) David J Holbrook and Holly A Holbrook. 2001.  Guyanese creole survey report.  _https://www.sil.org/resources/archives/9001_.
  * Lent et al. (2021) Heather Lent, Emanuele Bugliarello, Miryam de Lhoneux, Chen Qiu, and Anders Søgaard. 2021.  On language models for creoles.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 58–71, Online. Association for Computational Linguistics.
  * Lent et al. (2022a) Heather Lent, Emanuele Bugliarello, and Anders Søgaard. 2022a.  Ancestor-to-creole transfer is not a walk in the park.  In _Proceedings of the Third Workshop on Insights from Negative Results in NLP_ , pages 68–74, Dublin, Ireland. Association for Computational Linguistics.
  * Lent et al. (2022b) Heather Lent, Emanuele Bugliarello, and Anders Søgaard. 2022b.  Ancestor-to-creole transfer is not a walk in the park.  _arXiv preprint arXiv:2206.04371_.
  * Lent et al. (2022c) Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Søgaard. 2022c.  What a creole wants, what a creole needs.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6439–6449, Marseille, France. European Language Resources Association.
  * Letters from Guyana (2017) Letters from Guyana. 2017.  Me na able - creolese 101.  https://lettersfromguyana.wordpress.com/2017/01/29/me-na-able-creolese-101/.  Accessed on December 14, 2023.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2022) Zhengyuan Liu, Shikang Ni, Aiti Aw, and Nancy Chen. 2022.  Singlish message paraphrasing: A joint task of creole translation and text normalization.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3924–3936.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Michaelis et al. (2013) Susanne Maria Michaelis, Philippe Maurer, Martin Haspelmath, and Magnus Huber, editors. 2013.  _APiCS Online_.  Max Planck Institute for Evolutionary Anthropology, Leipzig.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  pages 311–318.
  * Polyglot Club (Accessed 2023) Polyglot Club. Accessed 2023.  Guyanese creole english vocabulary - basic words.  https://polyglotclub.com/wiki/Language/Guyanese-creole-english/Vocabulary/Basic-words.  Accessed on December 14, 2023.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Raffel et al. (2023) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rickford (1987) John R Rickford. 1987.  _Dimensions of a creole continuum: History, texts, and linguistic analysis of Guyanese Creole_.  Stanford University Press.
  * Sidnell (1999) Jack Sidnell. 1999.  Gender and pronominal variation in an indo-guyanese creole-speaking community.  _Language in Society_ , 28(3):367–399.
  * Sidnell (2002) Jack Sidnell. 2002.  Habitual and imperfective in guyanese creole.  _Journal of pidgin and creole languages_ , 17(2):151–189.
  * Speirs (1902) James Speirs. 1902.  _The Proverbs of British Guiana. With an Index of Principal Words, an Index of Subjects, and a Glossary_.  The Argosy Company, Demerara.
  * (34) Travel Phrases.  Guyanese phrases and basics.  http://www.travelphrases.info/languages/guyanese.htm.  Accessed on December 14, 2023.
  * Wikipedia (Accessed 2023) Wikipedia. Accessed 2023.  Guyanese creole.  https://en.wikipedia.org/wiki/Guyanese_Creole.  Accessed on December 14, 2023.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.

##  Appendix A Guyanese Creole Translation Tool

In this section, we further showcase the Guyanese Creole Translation tool
detailing our prompts and user interface.

        Translate the following Guyanese Creole text
        and provide the resulting English translation.
        Please ensure that the translation is clear and
        accurate. Guyanese Creole is spoken in Guyana
        and may include unique vocabulary and grammar.
        Try to capture the original meaning while
        making it comprehensible in English.

        Glossary:
          English: Swallow
          Creole: Swalla

          English: Stagger
          Creole: ’Taggah

          English: Stop-off
          Creole: "Taff-aff
          ...
        Translations
          Translation 1: The beef cooked until it was soft
          Text 1: Di biif kuk kuk kuk til ii saaf

          Translation 2: But my grandfather had a boat
          Text 2: Bo mi granfaada bin ga wan boot

Figure 4: Example GPT-4 Prompt with translation examples from Speirs (1902).