#  _Not (yet) the whole story_ : Evaluating Visual Storytelling Requires More
than Measuring Coherence, Grounding, and Repetition

Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle
Institute for Logic, Language and Computation
University of Amsterdam
<EMAIL_ADDRESS>
###### Abstract

Visual storytelling consists in generating a natural language story given a
temporally ordered sequence of images. This task is not only challenging for
models, but also very difficult to evaluate with automatic metrics since there
is no consensus about what makes a story ‘good’. In this paper, we introduce a
novel method that measures story quality in terms of human likeness regarding
three key aspects highlighted in previous work: visual grounding, coherence,
and repetitiveness. We then use this method to evaluate the stories generated
by several models, showing that the foundation model LLaVA obtains the best
result, but only slightly so compared to TAPM, a 50-times smaller visual
storytelling model. Upgrading the visual and language components of TAPM
results in a model that yields competitive performance with a relatively low
number of parameters. Finally, we carry out a human evaluation study, whose
results suggest that a ‘good’ story may require more than a human-like level
of visual grounding, coherence, and repetition.

_Not (yet) the whole story_ : Evaluating Visual Storytelling Requires More
than Measuring Coherence, Grounding, and Repetition

Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle Institute for Logic,
Language and Computation University of Amsterdam
<EMAIL_ADDRESS>

##  1 Introduction

Visual storytelling is the task of generating a story for a sequence of
several temporally-ordered images or video frames. For both human speakers and
machine learning models, the task requires connecting the visual data
causally, to generate a narrative consistent with the contents of the images.
As for model-generated stories, evaluation is one of the key challenges due to
the inherently creative nature of the task. Since human-written stories are
typically used to train visual storytelling models—under the assumption that
these stories provide a good learning signal—most previous work evaluated
model-generated stories by directly comparing them to human ones using
pattern-matching metrics. However, this approach is simplistic as it ignores
several key aspects of visual stories, such as their degree of visual
grounding, their overall coherence, or how repetitive they are. This problem
has only been addressed recently, with Wang et al. (2022) and Surikuchi et al.
(2023) proposing various metrics to take into account some of these crucial
aspects. These methods assess the appropriateness of a generated story
independently from its overlap with a ground-truth story for the same image
sequence. Given that the same image sequence can possibly give rise to many
different stories, this type of higher-level evaluation that does not rely on
text overlap is clearly desirable.

Nevertheless, we argue that measuring the degree of coherence or visual
grounding of a story may not be sufficiently informative, as there are no
standard conventions that determine the preferable level of such properties.
To address this issue, in this work, we first propose an evaluation method
that assesses the quality of generated stories in terms of their distance from
human-written stories along several relevant dimensions, each measured by an
available reference-free metric. Using this method, we evaluate a range of
models on the visual storytelling task, including models specifically designed
and trained for this task, as well as—for the first time—foundation models
pre-trained to achieve general-purpose language and vision abilities, which we
test in a zero-shot manner. We show that LLaVA (Liu et al., 2024), a powerful
foundation model, performs best on the task, but only slightly so than TAPM
(Yu et al., 2021), a model designed for visual storytelling which is 50 times
smaller than LLaVA. Second, given insights derived from our proposed distance-
based evaluation method, we upgrade the visual and language components on
TAPM, resulting in a model that achieves comparable performance to LLaVA with
a significantly lower number of parameters.

Our results show that the stories generated by LLaVA and the upgraded TAPM
model are very close to human stories regarding their degree of visual
grounding, coherence, and repetition. To further make sense of this finding,
we collect human judgments with regards to comparing human and model stories.
The results of this qualitative study indicate that humans tend to prefer
human-written stories by a significant margin despite the quantitative
closeness we observe.

In sum, we make the following contributions:

  * •

We propose a novel evaluation method for visual storytelling quantifying the
distance between human-written and model-generated stories in terms of visual
grounding, coherence, and non-repetitiveness.

  * •

We use our evaluation to assess the stories generated by various visual
storytelling-specific and foundation models; we find that a foundation model,
LLaVA, achieves the best result (lowest distance).

  * •

We leverage insights from this finding to upgrade a visual storytelling-
specific model, TAPM, by replacing its visual and language components; we show
that doing so results in better performance, on par with or outperforming the
best-performing (and twice larger) LLaVA model.

  * •

Through human evaluation, we validate the scores of our distance-based method;
at the same time, we report that human-written stories are still preferred,
which suggests that the ingredients for a good story may not be limited to a
human-like level of visual grounding, coherence, and non-repetitiveness.

##  2 Related Work

###  2.1 Visual Storytelling

Computational work on visual storytelling was initiated by Huang et al.
(2016), who operationalized the task as generating a textual story given an
ordered sequence of images. The authors proposed the VIST dataset, which
comprises sequences of five natural images collected from Flickr albums, with
corresponding stories provided by human crowd-workers. VIST has been a
catalyst for developing visual storytelling models Kim et al. (2018); Wang et
al. (2018); Yu et al. (2021). Over the last few years, other datasets have
emerged that differ from VIST in some key features. On the one hand, to limit
the complexity of modelling real-world knowledge implicit in natural images,
Ravi et al. (2021) proposed AESOP, a dataset that includes sequences of three
synthetic images (constructed by crowd-workers using clip–art entities from
Abstract Scenes by Zitnick and Parikh (2013)) and corresponding three long-
paragraph stories. More recently, to overcome the possible lack of character
consistency resulting from sampling images from Flickr albums, Hong et al.
(2023) proposed the VWP dataset, which comprises sequences of movie shots
including 5-10 images with corresponding stories provided by crowd-workers.

Regarding modeling, various computational approaches using RNNs and
Transformers have been proposed for the task of generating plausible stories.
Some of these models are trained end-to-end on the VIST dataset Kim et al.
(2018); Wang et al. (2018); Yu et al. (2021), while other approaches utilize
external knowledge sources Hsu et al. (2020, 2021); Chen et al. (2021). We
describe some of these models in detail in Section 4.2. Regardless of the
specific architectures, a challenge common to all computational approaches to
this task is evaluation. In the following subsection, we review existing work
on visual story evaluation in the general context of evaluating visually-
grounded language.

###  2.2 Visually-Grounded Language Evaluation

Since visual storytelling is essentially a vision-to-language task similar to
video/image captioning, evaluation of generated stories typically employed
reference-based pattern-matching metrics such as METEOR Banerjee and Lavie
(2005) and CIDEr Vedantam et al. (2015). However, these $n$-gram based metrics
are shown to correlate poorly with human judgments Novikova et al. (2017);
Wang et al. (2018). Other reference-based evaluation metrics such as BERTScore
Zhang* et al. (2020) and BLEURT Sellam et al. (2020) have also been used.
These leverage pre-trained models to compute similarities between the
generated candidate text and the corresponding references in a high-
dimensional embedding space, which makes them more flexible to paraphrases and
synonyms compared to standard $n$-gram based metrics. Nevertheless, reference-
based metrics are by design only suitable for target-oriented generation tasks
(e.g., machine translation), where it is considered essential for the
generated text to match a curated set of references Pillutla et al. (2021);
Giulianelli et al. (2023). This is not the case in visual storytelling, where
several stories could be plausible for a given image sequence. Recently,
CLIPScore Hessel et al. (2021) has been proposed to quantify the degree of
alignment between an image and a given text without the need for any
reference. It computes a similarity score between the CLIP Radford et al.
(2021) embeddings of the image and the text, and has been shown to correlate
well with human judgments. As such, it is widely adopted for evaluation in the
image captioning task. However, its application in the domain of visual
storytelling is less straightforward, as a visual sequence comprises multiple
images, and stories are made up of multiple related sentences, which makes
evaluating what counts as a ‘good’ story very challenging.

To address some of these challenges, Wang et al. (2022) proposed a
metric—RoViST—specifically for the visual storytelling task, which assesses
three aspects of generated stories: visual grounding, coherence, and
repetition. Subsequently, Surikuchi et al. (2023) proposed GROOViST, a more
advanced method to evaluate grounding in visual storytelling. However, while
this constitutes important progress, in practice it is often not easy to make
sense of these metrics, since they do not offer a reference point for
assessing the quality of a generated story. For example, should stories be
100% grounded in the images? Should they exhibit zero levels of repetition? We
argue that applying the metrics to both model\- and human-generated stories is
essential to boost their capacity to evaluate story-generation models. In the
next section, we propose a novel method that addresses this problem by
building on the existing metrics proposed by Wang et al. (2022) and Surikuchi
et al. (2023).

##  3 Problem Formulation

In this work, we take a human-centric approach and define the quality of
generated stories in terms of their ‘closeness’ to stories produced by humans,
regarding different dimensions that capture abstract properties of interest.
Concretely, we compute dimension-specific scores for model- and for human-
generated stories, measure human-model distance per dimension, and aggregate
these distances to derive an overall distance score. We posit that the lower
the overall distance, the more the generated story complies with high-level
features observed in human stories.

Deciding on which dimensions are most relevant to determine the quality of a
story, let alone how to operationalize such dimensions, is not trivial. In
this study, we consider the three aspects proposed by Wang et al. (2022):
visual grounding, coherence, and repetition. Here we describe how they are
operationalized as reference-free metrics, following which we formally define
our proposed method.

###  3.1 Metrics

#### Visual Grounding

To measure the degree of visual grounding of a story, we use GROOViST
(Surikuchi et al., 2023). For a given <image-sequence, story> pair, GROOViST
first computes the alignment between the noun phrases (NPs) in the story and
the bounding boxes in the images using their corresponding CLIP Radford et al.
(2021) embeddings. For each NP, only the maximum visual alignment score is
retained. To penalize NPs with low visual alignment scores, the mean score of
all the NPs in the dataset is used as a threshold. Specifically, this step is
implemented by calculating the distance of each NP’s score from the threshold.
Resulting scores of the NPs are then weighted using word concreteness ratings
to differentiate abstract words from concrete ones. The overall visual
grounding score of a story is the sum of the concreteness weighted scores of
all NPs normalized by the total number of NPs in the story. The range of the
resulting scores is unbounded with higher values indicating higher degree of
visual grounding.

#### Coherence

We use a slightly modified version of RoViST-C (Wang et al., 2022) to evaluate
the coherence of a given story, which corresponds to the average probability
with which each sentence follows the preceding sentences. These probabilities
are computed using ALBERT Lan et al. (2020) fine-tuned for the sentence order
prediction task using the VIST and ROCStories Mostafazadeh et al. (2016)
datasets. For each sentence ($s_{i}$) in a story, we obtain the probability
that it follows the entire concatenated prefix of all previous sentences
($\\{s_{1},...,s_{i-1}\\}$)—instead of just the previous sentence ($s_{i-1}$)
as done in the original RoViST-C. The overall coherence score of a story is
obtained by taking the average of these probabilities across all its sentences
resulting in a value between 0 and 1 (indicates high coherence).

#### Repetition

We measure the degree of repetition of a story using the RoViST-NR metric,
where NR stands for ‘non-redundancy’ Wang et al. (2022). For two segments of
text, repetition is computed using the Jaccard Similarity (JS) Singh and Singh
(2021) which is defined as the number of co-occurring words between the two
texts normalized by the total number of words in both texts. Inter-sentence
repetition is obtained as the average of JS scores computed between each
sentence $s_{i}$ and all its preceding sentences ($\\{s_{1},...,s_{i-1}\\}$).
For every sentence in the story, the intra-sentence repetition is obtained by
computing the average of JS scores between non-overlapping $4$-gram phrases of
the sentence. The overall repetition score of a story is the average of all
inter- and intra-sentence scores subtracted from $1$. The resulting scores
range between $0$ and $1$ and stories with scores closer to $1$ indicate less
repetition.

###  3.2 Distance between Humans and Models

For a given <image sequence, model story> pair, we compute the coherence
$C_{M}$, visual grounding $G_{M}$, and repetition $R_{M}$ scores using the
three metrics described above. We do the same for the corresponding <image
sequence, human story> pair, and denote the resulting scores as $C_{H}$,
$G_{H}$, and $R_{H}$. We then compute the absolute differences between the
human stories and the model-generated ones to measure metric-level deviations:

$\begin{split}\text{d}^{C}_{HM}&=|C_{H}-C_{M}|,\\\
\text{d}^{G}_{HM}&=|G_{H}-G_{M}|,\\\
\text{d}^{R}_{HM}&=|R_{H}-R_{M}|\end{split}$ (1)

Finally, we compute the overall aggregate distance between the model-generated
story and the corresponding human-annotated story as the average of the
metric-level deviations:

$\text{d}_{HM}=(\text{d}^{C}_{HM}+\text{d}^{G}_{HM}+\text{d}^{R}_{HM})/3$ (2)

##  4 Evaluation of Existing Models

In this section, we evaluate and compare several state-of-the-art models using
our proposed distance measure $\text{d}_{HM}$. We test the models on the
popular VIST dataset, that we describe below.

###  4.1 VIST dataset

VIST (Huang et al., 2016) is the first and most popular dataset for visual
storytelling. The dataset includes images from Flickr albums selected by
filtering titles with “storyable” event types (e.g., _graduation ceremony_).
For each of these selected albums, crowd workers constructed sequences of five
images and provided corresponding five-sentence stories. The stories were then
tokenized by replacing named entities including names of people, with entity
types (`[location]`, `[organization]`) and generic placeholder tokens
(`[male]`, `[female]`). On average, the dataset has 10.2 tokens per story and
an overall vocabulary size of 18200 words. Excluding unavailable images, the
dataset comprises 40071 training, 4988 validation, and 5050 test <image
sequence, story> samples.

###  4.2 Models

We evaluate three end-to-end models that are specifically designed and trained
for the visual storytelling task. In addition, we consider two general-purpose
vision-language foundation models, which are used in a zero-shot manner.

#### GLAC Net

GLocal Attention Cascading Network (Kim et al., 2018) is a model proposed
specifically for the visual storytelling task. Adapting the standard encoder-
decoder architecture, it obtains the global visual context pertaining to every
image sequence position using a bi-LSTM Hochreiter and Schmidhuber (1997)
encoder. The obtained global context embeddings along with the individual
image features (local) are, together, (GLocal) passed on to an LSTM decoder
for story generation. Furthermore, to reduce redundancy in the generated
sentences, GLAC Net samples multiple times from the decoder’s probability
distribution and selects the most frequent word from the pool.

#### AREL

Adversarial REward Learning Wang et al. (2018) is another framework proposed
for the visual storytelling task, which encompasses two modules: a policy
model and a reward model. Similar to GLAC Net, the policy model, which uses
GRUs Cho et al. (2014) instead of LSTMs, takes an image sequence as input and
generates a story. The reward model computes a score for every input <image,
sentence> pair by extracting the sentence representations using
1D-convolutional kernels and concatenating them with the corresponding pre-
trained ResNet-152 (He et al., 2016)) features of the images. Both modules are
trained using an adversarial learning objective: the reward model is trained
to discriminate between the ground truths and the generated stories; the
policy model, to maximize the scores from the reward model.

#### TAPM

Transitional Adaptation of Pretrained Models Yu et al. (2021) is a more recent
approach to visual storytelling that leverages a pre-trained transformer-based
language decoder. First, for every sequence position, the visual encoder pools
together corresponding image features (pre-trained ResNet-101, Faster R-CNN
Ren et al. (2015)) along with features of the images at the previous and next
positions to create enriched visual context representations. Then the visual
contexts are passed as input to the GPT-2small Radford et al. (2019) language
decoder for story generation. To bridge the semantic gap between the pre-
trained image and text representations, TAPM performs an adaptation step prior
to the downstream training for the visual storytelling task. Specifically, for
a pre-determined number of epochs, the visual encoder parameters are fine-
tuned by conditioning them on the outputs of the frozen GPT-2small decoder.

#### BLIP-2

Unlike the previously described models, Bootstrapping Language-Image Pre-
training Li et al. (2023) is a multimodal foundation model designed for
general-purpose vision-language tasks such as visual question answering and
image captioning. It connects a frozen pre-trained vision encoder and a frozen
pre-trained large language model using a connector module called querying
transformer (Q-Former). Q-Former contains an image and a text transformer with
shared self-attention layers and learnable embeddings for querying the frozen
image encoder. It is trained in two stages; in the first stage, it learns
representations that align with the representations of the prompts (e.g.,
questions) of interest. In the second stage, its representations are fine-
tuned based on the loss of the frozen language model generations
(captions/answers). We adapt BLIP-2 for the visual storytelling task in a
zero-shot manner by prompting the model to generate one sentence per each
image in the sequence. We experiment with different settings in which the
prompts contain different degrees of linguistic context.

#### LLaVA

Similar to BLIP-2, Large Language and Vision Assistant Liu et al. (2024) is
another general-purpose multimodal foundation model that connects large pre-
trained vision encoders and large pre-trained language models. However, unlike
BLIP-2, LLaVA focuses on training data and procedure as opposed to the model
architecture. It is the first model that extends instruction-tuning to the
language-image multimodal space. LLaVA achieves this by collecting and
training on vision-language instruction-following data, constructed for
<image, caption> pairs of existing datasets (e.g., COCO), by querying GPT-4
OpenAI (2023) using various in-context-learning prompts. To connect the visual
features with the language embeddings, LLaVA uses a linear layer (single
projection matrix) instead of Q-Former. Similar to BLIP-2, we use LLaVA to
generate stories in a zero-shot manner under different linguistic context
settings.

###  4.3 Experimental Setup

We generate stories for the VIST test set for all models, using greedy
sampling. For GLAC Net and AREL, we leverage the publicly available model
checkpoints. To obtain the model checkpoint for TAPM, we follow the proposed
procedure and train the model from scratch using the VIST dataset.111Using
code: https://github.com/JiwanChung/tapm. The two general-purpose foundation
models are used zero-shot, without training on the VIST dataset or the visual
storytelling task. For BLIP-2, we use the version with ViT-g encoder and
OPT-2.7B decoder and for LLaVA, version 1.6 with CLIP-ViT-L-336px and
Mistral-7B. We prompt these models under different settings that vary with
respect to the amount of linguistic and visual context given in the prompt
(e.g., one image/sentence at a time vs. all images at once). We use three
prompt variations per setting and report the average of the resulting
$\text{d}_{HM}$ values.222All settings and prompts are described in Appendix
A.

###  4.4 Results

Figure 1: Distance between human- and model-generated stories in the VIST test
set according to our proposed measure $\text{d}_{HM}$ (the lower the better).
For BLIP-2 and LLaVA, the best setting is reported; results for all settings
are provided in Fig. 6, Appendix A.

Figure 1 shows the distances between human-written stories and the stories
generated by the models (the lower the better). Examples of model-generated
stories are provided in Figure 2. In Figure 1, we observe that the stories
generated by LLaVA obtain the best overall value ($\text{d}_{HM}=0.099$),
followed by TAPM ($\text{d}_{HM}=0.15$). We notice that GLAC Net-generated
stories exhibit the lowest distance regarding the repetition dimension. We
attribute this to GLAC Net’s inference phase decoding heuristic, which
penalises repetitive expressions (see Section 4.2). BLIP-2 stories are overall
the farthest from stories written by humans.

Regarding the two best performing models, LLaVA and TAPM, two points are worth
highlighting. First, despite a huge difference in model size—LLaVA is a
powerful 7.5B parameter foundation model, while TAPM is 50 times
smaller—TAPM’s $\text{d}_{HM}$ value is only slightly higher than LLaVA’s.
Second, LLaVA outperforms TAPM with respect to coherence and visual grounding.
We hypothesize that this advantage is due to LLaVA’s more powerful language
and vision backbone models. In the next section, we leverage the modular
architecture of TAPM to test this hypothesis.

|  |  |  |  |
---|---|---|---|---|---
Human story |  we invited lots of friends for a barbeque. the fire pit was very large. we roasted hot dogs right over the flame. lots of people were happy. and there was a lot of beer too.
AREL |  the friends were having a great time at the party. we had a lot of fun. we had a lot of beer. we all had a great time and had a great time. we had a great time.
GLAC Net |  the family was having a party. they played some fire. then they had a big bonfire. everyone was happy. it was a great day.
TAPM |  the group of friends got together for a bonfire. we had a lot of fun cooking. the barbecue was delicious. we took a lot of pictures. the night ended with a few drinks.
LLaVA |  in the dark, a group of friends huddled around a fire, their faces lit up with the warmth of the flames. the fire crackled and roared, casting dancing shadows on their faces. one friend, a bit too eager, accidentally dropped a hot dog into the fire, causing a burst of flames and laughter. the friends cheered and clapped, their joy infectious. as the night wore on, they shared stories and laughter, the fire slowly dying down, leaving behind only the memories of their fun-filled evening.
BLIP-2 |  a group of people standing outside drinking beer and talking on cell phones . . a hot dog is being cooked on a stick over a fire . . a group of people standing in a room with a woman making a surprised face

Figure 2: An example from the VIST test set with corresponding model-generated
stories.

##  5 Model Analysis and Improvements

In Section 4, we showed that the $\text{d}_{HM}$ obtained by the visual
storytelling-specific TAPM model is only slightly higher than that of LLaVA, a
50-times larger foundation model. In particular, we notice that LLaVA has an
advantage over TAPM in two dimensions, i.e., visual grounding and coherence.
We hypothesize that this advantage is due to the model’s better language and
vision backbone models—LLaVA builds on a pre-trained, transformer-based
language model and image processor. Thus, we leverage the modular architecture
of TAPM and test whether we can obtain better results (lower distances) by
replacing its original language and vision components with models similar to
those embedded in LLaVA, while keeping the number of parameters significantly
lower. To test whether the results we obtain are consistent across datasets,
we perform this analysis on both VIST and VWP Hong et al. (2023).

###  5.1 Updating the Language Component

By default, TAPM uses GPT-2small as its story decoder. Here, we replace this
language model with Llama 2 Touvron et al. (2023), an auto-regressive
(decoder-only) large language model pre-trained via maximum likelihood
objective for next-token prediction on massive amounts of publicly available
online data. Thanks to the large context window it can take as input and the
Grouped Query Attention mechanism to speed up processing during decoding
Ainslie et al. (2023), this LM is currently the state-of-art on various
downstream tasks in the MMLU benchmark Hendrycks et al. (2021). Here, we use
the 4-bit quantized Dettmers et al. (2023) pre-trained version of the Llama 2
7B model and adapt the parameter dimensions of TAPM’s visual encoder component
to match the updated language decoder. To ensure computational and memory
efficiency, we employ the LoRA (Low-rank Adapter; Hu et al., 2022) fine-tuning
approach (that allows for updating only a subset of the model’s parameters)
and only target the multi-head self-attention blocks—$W^{Q},W^{K},W^{V},W^{O}$
Vaswani et al. (2017)—of the language model during training.333Further details
on training are provided in Appendix B. Henceforth, we refer to this upgraded
TAPM model as (+Llama 2).

###  5.2 Updating the Vision Component

The original TAPM model uses pre-trained ResNet-101 and Faster R-CNN for
extracting the image-level and object-level features, respectively. We
supplement the image-level ResNet features by concatenating them with
representations extracted using pre-trained Vision Transformer model (ViTbase;
Dosovitskiy et al., 2021). ViTbase leverages the transformer architecture for
image processing and is pre-trained on the ImageNet-21K Ridnik et al. (2021)
data. Features extracted using ViTbase have been shown to improve the
performance of models on several computer vision tasks. Henceforth, we refer
to this upgraded TAPM model as (+ViT).

###  5.3 Experimental Setup

We train the (+ViT) and (+Llama 2) models from scratch using the VIST data
training and validation splits for 15 epochs, and obtain the results on the
test split. We conduct the same experiment for the VWP dataset independently,
by following the procedure proposed by the authors of the dataset. As
mentioned in Section 2.1, VWP includes sequences of 5-10 images constructed
using frames obtained from MovieNet Huang et al. (2020). Following the same
procedure used by the authors, we split the dataset into 9606 training, 849
validation, and 586 test <image sequence, story> samples and preprocess the
text to replace recognized named entities with entity types and placeholders.
To compare the performance of the improved TAPM models against LLaVA, we use
LLaVA to generate stories for the VWP test set by prompting it under the
visual context setting.

###  5.4 Results

Figure 3: For the TAPM model and its upgraded versions (+Llama 2) and (+ViT),
we compute the $\text{d}_{HM}$ values on two datasets—VIST and VWP.

Figure 3 shows the $\text{d}_{HM}$ values for the LLaVA model and for all the
different versions of the TAPM model on both the VIST and the VWP datasets.
Firstly, we observe that compared to the original TAPM model, (+Llama 2)
generates stories that are closest to human stories in terms of both coherence
and non-redundancy. Stories by the (+ViT) version achieve the lowest distance
along the dimension of visual grounding. These results are in line with our
intuitions regarding the influence that both modalities have on different
aspects of visual story evaluation. For this analysis, we also considered a
version of the TAPM model in which both the language and vision components are
jointly updated (+Llama 2, +ViT). However, (+Llama 2, +ViT) consistently
under-performed on our distance measure $\text{d}_{HM}$ compared to the
(+Llama 2) and (+ViT) versions, and was only marginally better than the
original TAPM model. Despite the significant difference in the number of
parameters, we notice that (+Llama 2) achieves performance on par with LLaVA
(see Figure 4).

We observe similar results for the TAPM models on the VWP dataset—(+ViT)
obtains the lowest distance in terms of visual grounding and (+Llama 2)
obtains the lowest distance in terms of both coherence and non-redundancy.
Furthermore, on the VWP dataset, the (+Llama 2) model achieves the overall
lowest $\text{d}_{HM}$, performing better than the LLaVA model. These results
quantitatively indicate that the stories generated by the models are close to
the corresponding human-written stories. To better understand if the metrics
are capable of effectively comparing stories generated by models with human-
written ones, we conduct a qualitative evaluation, that we describe in the
next section.

Figure 4: Comparison between the number of model parameters and their
corresponding $\text{d}_{HM}$ values on the VIST test set (the lower the
better). We observe that as the models increase in size, the scores for the
stories they generated get closer to scores of human annotations.

##  6 Qualitative Analysis and Discussion

Our results suggest that the stories generated by the best-performing models
according to $\text{d}_{HM}$ are very close to human levels of visual
grounding, coherence, and degree of repetition. To test whether this aligns
with the perceived overall quality of the stories, we conduct a qualitative
human evaluation. We consider the VIST stories generated by the two
models—TAPM (+Llama 2) and LLaVA—that achieve the best performance in terms of
$\text{d}_{HM}$ (lowest distances). For each model, we select 100 generated
stories, that we randomly sample using the distribution of $\text{d}_{HM}$
values on the VIST test set.444See Figure 12 in Appendix C. We then provide
annotators with <image sequence, model-story> pairs along with corresponding
human-written stories, and ask them to assess whether one is better than the
other, or whether both are similarly fine or bad. Five annotators unrelated to
the project participated in the task.555Instructions and additional details in
Appendix C.

Figure 5 shows the judgments obtained from the annotators for the two models.
We observe that humans consistently prefer human-written stories over stories
generated by the models. This happens slightly more often for TAPM- than for
LLaVA-generated stories (46% vs 57% of cases). Similarly, LLaVA-generated
stories are more frequently preferred over human stories than stories
generated by TAPM, although this happens very seldom for both models (16.6%
vs. 6.2%). These patterns are in line with the results obtained with our
distance metric, according to which LLaVA has a slight advantage over TAPM.
Indeed, we observe that the average $\text{d}_{HM}$ is higher when the human
story is judged as better than when both stories are perceived as having
similar quality; this difference is statistically significant for TAPM (avg
$\text{d}_{HM}$=0.279 vs. 0.243; $t=2.19,p<0.03$) and not so for LLaVA (avg
$\text{d}_{HM}$=0.240 vs. 0.226; $t=1.23,p>0.05$). Overall, this suggests that
our quantitative evaluation effectively captures key aspects of visual
stories.

Figure 5: Aggregated judgments (5 human evaluators per model) comparing human
stories in the VIST test set to stories generated by the two best-performing
models according to our distance measure $\text{d}_{HM}$.

Yet, preference for human-written stories suggests that a good story involves
more than just visual grounding, coherence, and limited repetition. To get
more insight into this issue, we asked the annotators in the study to describe
the properties they considered when comparing stories. Some annotators
reported that they employed their subjective world knowledge in
differentiating between creative and hallucinated stories. Also, stories that
describe events without an overarching narrative were judged as bad despite
being locally coherent, well-grounded, and non-repetitive. In addition,
annotators generally preferred stories that contained phrases or sentences
expressing emotions. We provide examples of these cases in Appendix C.

##  7 Conclusion

We proposed a novel human-centric method ($\text{d}_{HM}$) for evaluating the
quality of model-generated stories in terms of their closeness to human-
written stories along coherence, visual grounding, and non-redundancy. Using
our proposed method, we compared various models and found that the large
foundation model LLaVA obtains slightly better results than the (50 times
smaller) best visual storytelling model TAPM. We showed that upgrading the
components of TAPM boosts its performance (lowers its $\text{d}_{HM}$) across
multiple datasets, confirming the advantage of leveraging last-generation
language and vision pre-trained models. Finally, we conducted a qualitative
human evaluation to explore whether these quantitative findings align with the
overall perceived story quality. We observed that human judgments align with
our quantitative evaluation. Yet, humans still prefer human-written stories
over model-generated ones, which suggests that capturing coherence, visual
grounding, and non-repetitiveness may not (yet) be the whole story.

## Limitations

The human evaluation analysis we performed is arguably small-scale. As such,
we cannot rule out that carrying it out with more annotators and a larger set
of stimuli may lead to different patterns of results. Second, the number of
models we experimented with is quite limited, which is an obvious limitation
of this work. Yet, we defend the selection we made, aimed at a trade-off
between the limited availability of computational resources and the inclusion
of various architecture families. Finally, we experimented with only two
visual storytelling datasets, both in English and both Western-centric. This
is a limitation of our work, which is, however, due to the unavailability of
other datasets with more diverse language and cultural backgrounds. We
strongly support the creation of such resources.

## Acknowledgments

We are immensely grateful to the participants of the qualitative evaluation
study and to our colleagues at the Dialogue Modelling Group for their
invaluable inputs at different stages of this work. AKS was supported by the
TIMELY project under the EU-H2020 grant 101017424. RF was supported by the
European Research Council (ERC Consolidator Grant DREAM 819455).

## References

  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023.  GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4895–4901, Singapore. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Chen et al. (2021) Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. 2021.  Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(2):999–1008.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  QLoRA: Efficient Finetuning of Quantized LLMs.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 10088–10115. Curran Associates, Inc.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Giulianelli et al. (2023) Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023.  What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Singapore. Association for Computational Linguistics.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Massive Multitask Language Understanding.  In _International Conference on Learning Representations_.
  * Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021.  CLIPScore: A Reference-free Evaluation Metric for Image Captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7514–7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780.
  * Hong et al. (2023) Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demberg, and Bernt Schiele. 2023.  Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences.  _Transactions of the Association for Computational Linguistics_ , 11:565–581.
  * Hsu et al. (2020) Chao-Chun Hsu, Zi-Yuan Chen, Chi-Yang Hsu, Chih-Chia Li, Tzu-Yuan Lin, Ting-Hao Huang, and Lun-Wei Ku. 2020.  Knowledge-Enriched Visual Storytelling.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7952–7960.
  * Hsu et al. (2021) Chi-yang Hsu, Yun-Wei Chu, Ting-Hao Huang, and Lun-Wei Ku. 2021.  Plot and Rework: Modeling Storylines for Visual Storytelling.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4443–4453, Online. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Huang et al. (2020) Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020.  MovieNet: A Holistic Dataset for Movie Understanding.  In _Computer Vision – ECCV 2020_ , pages 709–727, Cham. Springer International Publishing.
  * Huang et al. (2016) Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016.  Visual Storytelling.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1233–1239, San Diego, California. Association for Computational Linguistics.
  * Kim et al. (2018) Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung-Wha Park, and Byoung-Tak Zhang. 2018.  GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation.  _CoRR_ , abs/1805.10973.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In _International Conference on Learning Representations_.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 19730–19742. PMLR.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024.  LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849, San Diego, California. Association for Computational Linguistics.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.  Why We Need New Evaluation Metrics for NLG.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 4816–4828. Curran Associates, Inc.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  _OpenAI blog_.
  * Ravi et al. (2021) Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. 2021.  AESOP: Abstract Encoding of Stories, Objects and Pictures.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2052–2063.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Ridnik et al. (2021) Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik. 2021.  ImageNet-21K Pretraining for the Masses.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning Robust Metrics for Text Generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Singh and Singh (2021) Ritika Singh and Satwinder Singh. 2021.  Text Similarity Measures in News Articles by Vector Space Model Using NLP.  _Journal of The Institution of Engineers (India): Series B_ , 102(2):329–338.
  * Surikuchi et al. (2023) Aditya Surikuchi, Sandro Pezzelle, and Raquel Fernández. 2023.  GROOViST: A Metric for Grounding Objects in Visual Storytelling.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3331–3339, Singapore. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _Preprint_ , arXiv:2307.09288.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  CIDEr: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Wang et al. (2022) Eileen Wang, Caren Han, and Josiah Poon. 2022.  RoViST: Learning Robust Metrics for Visual Storytelling.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2691–2702, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2018) Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018.  No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 899–909, Melbourne, Australia. Association for Computational Linguistics.
  * Yu et al. (2021) Youngjae Yu, Jiwan Chung, Heeseung Yun, Jongseok Kim, and Gunhee Kim. 2021.  Transitional Adaptation of Pretrained Models for Visual Storytelling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12658–12668.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT.  In _International Conference on Learning Representations_.
  * Zitnick and Parikh (2013) C. L. Zitnick and Devi Parikh. 2013.  Bringing Semantics into Focus Using Visual Abstraction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.

##  Appendix A Zero-Shot Settings and Prompts

We generated stories using the two foundation models—BLIP-2, LLaVA—zero-shot
by prompting them under different settings. In the _visual_ context setting,
models received the entire image sequence (with its images horizontally
concatenated) as input, along with a prompt designed for this setting. In the
_linguistic_ context settings, models generated one sentence per image in the
sequence. We obtained the results of contextualizing the prompt with different
degrees of linguistic information—sentence generated for the previous image in
the sequence (_prev sentence_), concatenated prefix of sentences generated for
all the preceding images in the sequence (_all sentences_). We used three
variations of the prompts and reported the average of the resulting
$\text{d}_{HM}$ values. Prompts used in all the settings are as provided
below:

⬇

# prompts used for visual context

P1 = ’[INST] <image>\nWrite a story using exactly five sentences for this
image sequence. Do not use more than five sentences. [/INST]’

P2 = ’[INST] <image>\nGenerate a story consisting of five sentences for this
image sequence. Use only five sentences and not more. [/INST]’

P3 = ’[INST] <image>\nOutput a story about this sequence of images using only
five sentences. Make sure the story does not include more than five sentences.
[/INST]’

⬇

# prompts used for linguistic contexts:

## {prev sentence} or {all sentences}

P1 = ’[INST] <image>\nUsing this image, add one sentence to the following
story: {context}<s> [/INST]’

P2 = ’[INST] <image>\nGiven this image, write one sentence as an addition to
the following story: {context}<s> [/INST]’

P3 = ’[INST] <image>\nAdd one sentence based on this image to the following
story: {context}<s> [/INST]’

Figure 6 shows the $\text{d}_{HM}$ values for both the models on the VIST test
set, across all zero-shot settings. Besides the LLaVA model under the _visual_
context setting, models obtained significantly high $\text{d}_{HM}$ values in
other settings. We observed that under the _visual_ context setting, the
BLIP-2 model failed to generate any stories.

Figure 6: $\text{d}_{HM}$ values (the lower the better) for the stories
generated by the BLIP-2 and LLaVA models under different zero-shot settings.

##  Appendix B Improvements to TAPM

To train the (+Llama 2) version, we adapted the model’s dimensionality from
768 to 4096. We used a batch size of 2 and trained the model for 18 epochs—3
for the pre-task adaptation and 15 for the downstream task training. We used
the 4-bit quantized pre-trained version of (+Llama 2) 7B and employed LoRA
fine-tuning. We configured the fine-tuning with a scaling parameter of
$\alpha=8$ and a rank of $r=8$, based on the evidence provided in Hu et al.
(2022). To generate stories using the trained model, we used greedy decoding.
For completing 1 epoch of training and inference on the VIST dataset, the
(+Llama 2) model approximately used 8 compute hours of 1 Nvidia A100 (40GB)
GPU. For all the models we used in Sections 4 and 5, Figure 7 compares the
relationship between the size of the models and the corresponding distances
they obtain along the aspects of coherence, visual grounding, and repetition.

(a) $\text{d}_{HM}$ aggregate (b) Coherence ($\text{d}^{C}_{HM}$) (c) Visual
grounding ($\text{d}^{G}_{HM}$) (d) Non-redundancy ($\text{d}^{R}_{HM}$)

Figure 7:  Comparison between the number of model parameters and their
corresponding $\text{d}_{HM}$ values on the VIST test set (the lower the
better). Plot (a) compares the overall aggregate distance and plots (b)
through (d) show individual metric-level distances. We note that the stories
generated by the overall best performing model—LLaVA—obtains the closest
distance to human stories only in terms of coherence.

##  Appendix C Human Evaluation

Five annotators unrelated to the project participated in the study
voluntarily. We obtained their consent about using the data collected as part
of the study for academic research. Upon expressing their consent, the
annotators received access to an interface with the task description and
instructions, shown in Figure 8. Along with the instructions, we provided one
example for each of the four possible options in the task. After completing
the task, we asked each annotator to describe the properties they considered
for comparing the stories. Specifically, we asked the following question: “
_What are the properties of a story that made you select it as being better?_
”. Based on the responses and judgements obtained from the annotators (see
Section 6), we report qualitative examples in Figures 9, 10, and 11.

Figure 8: Human evaluation task description and instructions.

|  |  |  |  |
---|---|---|---|---|---
S1 |  the table is set for a dinner party. the guests were amazed by the decorations. the food was delicious. they are making a wish. then they had dessert.
S2 |  our first meal together as a married couple was as beautiful as it was delicious. the roses they sent over were of the highest quality like the ingredients in their food. the personal touches like the unique cake designs made it even more special. we were on our way to married life in high class style. and this was literally the icing on the cake.

Figure 9: Example depicting emotions: all annotators evaluated S2 as better
than S1.

|  |  |  |  |
---|---|---|---|---|---
S1 |  it’s halloween and the pumpkins are being carved. i bought a lot of food for it. the house has a lot of decorations. the pumpkin was carved with a scary face. the pumpkins are lit up inside.
S2 |  the pumpkin was angry. someone had stolen all of his seeds. he waited patiently in front of the house for night to fall. once it was night time he made his move. he proceeded into the house to finally get his revenge. there were no survivors.

Figure 10: Example with hallucinations: 4 annotators (out of 5) either
selected S1 as better or evaluated S2 as bad.

|  |  |  |  |
---|---|---|---|---|---
S1 |  our business hosted its winter holiday party at a hotel. the food was served as a buffet. all the young workers sat at one end of the table. the older co-workers sat at the other end of the table. in the end, both young and old co-workers stayed until closing time for the hotel restaurant.
S2 |  in the first image, a man stands in awe of a beautifully decorated Christmas tree. the second image captures a lively dinner party, with guests seated around a table laden with food and wine. the third image shows a bustling restaurant kitchen, where chefs are busy preparing meals. the fourth image reveals a cozy dining room, where a family enjoys a quiet meal together. the final image presents a grand banquet hall, filled with guests and adorned with elegant decorations.

Figure 11: Example with an overarching narrative: all annotators evaluated S1
as better than S2.

(a) TAPM (+Llama 2) (b) LLaVA

Figure 12: $\text{d}_{HM}$ distributions for the VIST test set (left) and for
the 100 randomly sampled instances (right).


††thanks<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
#  Photocurrents of charge and spin in single-layer Fe3GeTe2

M. Merte  Peter Grünberg Institut and Institute for Advanced Simulation,
Forschungszentrum Jülich and JARA, 52425 Jülich, Germany  Department of
Physics, RWTH Aachen University, 52056 Aachen, Germany  Institute of Physics,
Johannes Gutenberg University Mainz, 55099 Mainz, Germany  F. Freimuth
Institute of Physics, Johannes Gutenberg University Mainz, 55099 Mainz,
Germany  Peter Grünberg Institut and Institute for Advanced Simulation,
Forschungszentrum Jülich and JARA, 52425 Jülich, Germany  T. Adamantopoulos
Peter Grünberg Institut and Institute for Advanced Simulation,
Forschungszentrum Jülich and JARA, 52425 Jülich, Germany  D. Go  Institute of
Physics, Johannes Gutenberg University Mainz, 55099 Mainz, Germany  Peter
Grünberg Institut and Institute for Advanced Simulation, Forschungszentrum
Jülich and JARA, 52425 Jülich, Germany  T.G. Saunderson  Institute of Physics,
Johannes Gutenberg University Mainz, 55099 Mainz, Germany  Peter Grünberg
Institut and Institute for Advanced Simulation, Forschungszentrum Jülich and
JARA, 52425 Jülich, Germany  M. Kläui  Institute of Physics, Johannes
Gutenberg University Mainz, 55099 Mainz, Germany  L. Plucinski  Peter Grünberg
Institut and Institute for Advanced Simulation, Forschungszentrum Jülich and
JARA, 52425 Jülich, Germany  O. Gomonay  Institute of Physics, Johannes
Gutenberg University Mainz, 55099 Mainz, Germany  S. Blügel  Peter Grünberg
Institut and Institute for Advanced Simulation, Forschungszentrum Jülich and
JARA, 52425 Jülich, Germany  Y. Mokrousov  Peter Grünberg Institut and
Institute for Advanced Simulation, Forschungszentrum Jülich and JARA, 52425
Jülich, Germany  Institute of Physics, Johannes Gutenberg University Mainz,
55099 Mainz, Germany

###### Abstract

In the realm of two-dimensional materials magnetic and transport properties of
a unique representative $-$ Fe3GeTe2 $-$ attract ever increasing attention.
Here, we use a developed first-principles method for calculating laser-induced
response to study the emergence of photo-induced currents of charge and spin
in single-layer Fe3GeTe2, which are of second order in the electric field. We
provide a symmetry analysis of the emergent photocurrents in the system
finding it to be in excellent agreement with ab-initio calculations. We
analyse the magnitude and behavior of the charge photocurrents with respect to
disorder strength, frequency and band filling. Remarkably, not only do we find
a large charge current response, but also predict that Fe3GeTe2 can serve as a
source of significant laser-induced spin-currents, which makes this material
as a promising platform for various applications in optospintronics.

Introduction. The tremendous progress in realization of robust two-dimensional
(2D) magnetism in van der Waals materials [1, 2, 3, 4] moves various
properties of 2D magnetic materials into the focus of intense attention. Among
the latter, layered Fe3GeTe2 (FGT) is one of the most prolific candidates for
potential 2D magnetic applications, as it is one of the few compounds among 2D
materials that exhibits strong out of plane magnetocrystalline anisotropy, has
one of the largest Curie temperatures among 2D materials [5], and provides a
playground for realization of complex spin textures [6, 7, 8]. Current-induced
switching of magnetization in FGT has been achieved [9], and it was argued
that intrinsic bulk-like spin-orbit torques, arising without a need for an
interface, can be very large in this material [10, 11]. Moreover, it was
predicted that FGT displays very prominent Kerr and Faraday effects, with
magneto-optical properties being qualitatively similar when going from bulk to
the single-layer limit owing to the weak coupling among the layers [12, 13].
These findings motivate an extensive further exploration of intrinsic
properties of FGT, especially in the area of its magnetic response to
electromagnetic fields.

On this front, the properties of laser ignited charge currents are currently
studied very intensively in interfacial systems and 2D materials, since they
mediate THz radiation [14, 15, 16, 17] and carry important information about
intrinsic characteristics of the system [18, 19, 17, 20]. While it is known
that even in nonmagnetic non-centrosymmetric materials such as semiconductors
[21, 22], quantum wells [23], graphene [24] and organic-inorganic halides [25]
light can give rise to spin currents, following the initial suggestion of
enhanced surface spin photocurrents in magnetic systems [26], the physics of
laser-driven spin currents in 2D magnetic materials has started to attract
attention as well [27, 28]. In this work we study the properties of laser-
induced in-plane charge and spin currents in a single-layer FGT from first
principles. For this, we employ an ab-initio implementation of the expressions
for photocurrents of spin and charge that we derived recently [26], which work
equally well for insulating and metallic systems of any given complexity, and
allow for considering the effect of disorder. We compute the charge and spin
current response of FGT for different degree of disorder, and analyze it as a
function of frequency and band filling. Our results provide an important
reference point for exploring future optospintronics applications of this
exciting material.

Method. In order to compute the photocurrents in the system arising as a
response to a continuous laser pulse of frequency $\omega$, we employ an
expression for the second order photocurrent density which was previously
derived by us using Keldysh formalism [26]:

$J_{i}=\frac{a_{0}^{2}e\epsilon_{0}}{2\hbar}\left(\frac{\varepsilon_{\mathrm{H}}}{\hbar\omega}\right)^{2}\operatorname{Im}\sum_{jk}E_{j}E_{k}^{*}\varphi_{ijk},$
(1)

where $a_{0}$ is the Bohr’s radius, $e$ is the elementary charge, $\hbar$ is
the reduced Planck constant, $\varepsilon_{H}=e^{2}/(4\pi\epsilon_{0}a_{0})$
is the Hartree energy, and $E_{i}$ is the $i$’th component of the complex
field amplitude of the pulse. The quantity $\varphi_{ijk}$ is defined as the
following energy integral [26]:

$\displaystyle\varphi_{ijk}=$
$\displaystyle\frac{2}{a_{0}\mathcal{E}_{\mathrm{H}}}\int\frac{\mathrm{d}^{2}k}{(2\pi)^{2}}\int\mathrm{d}\mathcal{E}\operatorname{Tr}[$
(2) $\displaystyle
f(\mathcal{E})v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{j}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}-\hbar\omega)v_{k}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})$
$\displaystyle-$ $\displaystyle
f(\mathcal{E})v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{j}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}-\hbar\omega)v_{k}G_{\boldsymbol{k}}^{\mathrm{A}}(\mathcal{E})$
$\displaystyle+$ $\displaystyle
f(\mathcal{E})v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{k}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}+\hbar\omega)v_{j}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})$
$\displaystyle-$ $\displaystyle
f(\mathcal{E})v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{k}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}+\hbar\omega)v_{j}G_{\boldsymbol{k}}^{\mathrm{A}}(\mathcal{E})$
$\displaystyle+$ $\displaystyle
f(\mathcal{E}-\hbar\omega)v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{j}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}-\hbar\omega)v_{k}G_{\boldsymbol{k}}^{\mathrm{A}}(\mathcal{E})$
$\displaystyle+$
$\displaystyle\left.f(\mathcal{E}+\hbar\omega)v_{i}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})v_{k}G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E}+\hbar\omega)v_{j}G_{\boldsymbol{k}}^{\mathrm{A}}(\mathcal{E})\right]$

where $v_{i}$ is the $i$’th component of the velocity operator,
$f(\mathcal{E})$ is the Fermi-Dirac distribution function and
$G_{\boldsymbol{k}}^{\mathrm{R/A}}$ are equilibrium retarded and advanced
Greens functions respectively. In order to compute the spin photocurrent
$Q_{si}$ propagating in direction $i$ and polarized along axis $s$, we replace
the first of the velocity operators $v_{i}$ appearing in the expression above,
with the operator of the spin velocity $\\{v_{i},\sigma_{s}\\}$, and change
the prefactor $a_{0}^{2}e\epsilon_{0}/2\hbar$ in Eq. 1 to
$-a_{0}^{2}\epsilon_{0}/8$.

In this work, we model the effect of disorder by adapting a model of constant
lifetime broadening of the states $\Gamma$ which results in the following
expressions:
$G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})=\hbar\sum_{n}\frac{|\boldsymbol{k}n\rangle\langle\boldsymbol{k}n|}{\mathcal{E}-\mathcal{E}_{\boldsymbol{k}n}+i\Gamma}$
and
$G_{\boldsymbol{k}}^{\mathrm{A}}(\mathcal{E})=[G_{\boldsymbol{k}}^{\mathrm{R}}(\mathcal{E})]^{\dagger}$
[26], where the energy of the state $\Ket{\boldsymbol{k}n}$ in band $n$ with a
Bloch vector $\boldsymbol{k}$ is $\mathcal{E}_{\boldsymbol{k}n}$. To compute
the photocurrent the integrals in Eq. (2) have to be evaluated. At zero
temperature the Fermi distribution becomes a step function which allows one to
perform the energy integration analytically Ref. [29]. The numerical
evaluation is preformed within the basis of maximally localized Wannier
functions, and the Brillouin zone integration is performed numerically by
employing the efficient technique of Wannier interpolation [30, 31, 32]. A
more detailed description of the methodology will be published elsewhere.
Throughout this work we assume an intensity of the light of 10 GW/cm2, which
corresponds to typical values of the fluence of the order of 0.5 mJ/cm2 for a
50 fs laser pulse [20].

Computational Details. The considered structure of single-layer FGT is shown
in Fig. 1(a-b). Our calculations of single-layer Fe3GeTe2 with point group D3h
predict that in agreement with experiments the FGT layer exhibits a
ferromagnetic ground state with the easy axis pointing out of the plane when
grown at the lattice constant of bulk Fe3GeTe2 [1]. The electronic structure
of the system was calculated including the effect of spin-orbit coupling (SOC)
with the film version of the FLEUR code [33]. The in-plane lattice constant
was set to $a=7.542$ a.u. For self-consistent calculations we used a plane-
wave cutoff of $5.0$ a.u.-1 and the total of 576 $k$-points in the two-
dimensional Brillouin zone. The muffin-tin radii for Fe, Ge, Te were set to
2.08 a.u., 2.19 a.u., and 2.40 a.u., respectively. The nonrelativistic PBE
[34] exchange-correlation functional was used. The computed bandstructure of
the system, shown in Fig. 1, reflects the complex orbital interplay around the
Fermi energy $\mathcal{E}_{F}$ typical of FGT. The impact of low symmetry of
FGT on the electronic structure can be clearly observed along the ${\rm
K}^{\prime}-\Gamma-{\rm K}$ path in Fig. 1, and it ultimately gives rise to
non-vanishing complex non-linear response. After converging the electronic
structure we extracted 48 maximally-localized Wannier functions by using Fe
$d$\- and Ge, Te $p$-orbitals as initial projections, since these orbitals
dominate the bandstructure in a wide energy window around $E_{F}$. Based on
the tight-binding Wannier Hamiltonian constructed from the Wannier functions
we computed the photocurrents on a $2000\times 2000$ interpolation $k$-mesh,
which provides well converged results for the lifetime broadening larger than
$\Gamma=25$ meV.

Figure 1: Geometry and electronic strcuture of single-layer Fe3GeTe2 (FGT).
(a) Side view of FGT layer with red arrows indicating magnetic moments on Fe
atoms directed out of plane. (b) Top view of a 2$\times$2 unit cell of single-
layer FGT. In (a-b) blue spheres, green spheres and red spheres stand for Ge,
Te and Fe atoms, respectively. (c) Bandstructure of ferromagnetic FGT layer in
a $[-2,+2]$ eV energy window around the Fermi energy.

Symmetry analysis. In order to investigate which constraints are imposed on
the photocurrents by the crystal symmetry we expand the photocurrent up to the
first order in the magnetization as follows:

$J_{i}=2\sigma_{ijk}E_{j}E_{k}^{*}=\chi_{ijk}E_{j}E_{k}^{*}+\chi_{ijkl}E_{j}E_{k}^{*}M_{l},$
(3)

where $\sigma_{ijk}$ is the photoconductivity tensor, $\chi_{ijk}$ is a polar
tensor of rank 3 and $\chi_{ijkl}$ is an axial tensor of rank 4. We introduce
the notation for the basis tensors
$\delta^{(nopq)}_{ijkl}=\delta_{in}\delta_{jo}\delta_{kp}\delta_{lq}:=\langle
nopq\rangle$, which allows us to list the tensors that are permitted by
symmetry in a compact form. Only one polar tensor of rank 3 is allowed by
symmetry, namely
$\chi^{(p,1)}_{ijk}=\delta^{(111)}_{ijk}-\delta^{(221)}_{ijk}-\delta^{(212)}_{ijk}-\delta^{(122)}_{ijk}$,
while four axial tensors of rank 4 are consistent with the symmetry:
$\chi^{(a,1)}_{ijkl}=-\delta^{(3211)}_{ijkl}-\delta^{(3121)}_{ijkl}-\delta^{(3112)}_{ijkl}+\delta^{(3222)}_{ijkl}$,
$\chi^{(a,2)}_{ijkl}=-\delta^{(2113)}_{ijkl}-\delta^{(1213)}_{ijkl}-\delta^{(1123)}_{ijkl}+\delta^{(2223)}_{ijkl}$,
$\chi^{(a,3)}_{ijkl}=-\delta^{(2131)}_{ijkl}-\delta^{(1231)}_{ijkl}-\delta^{(1132)}_{ijkl}+\delta^{(2232)}_{ijkl}$,
and
$\chi^{(a,4)}_{ijkl}=\delta^{(2311)}_{ijkl}+\delta^{(1321)}_{ijkl}+\delta^{(1312)}_{ijkl}-\delta^{(2322)}_{ijkl}$.
We do not need to consider $\chi^{(a,1)}$ due to the 2D character of FGT,
where out-of-plane photocurrents are not of interest. Additionally, we may
ignore $\chi^{(a,3)}$ and $\chi^{(a,4)}$ because we choose the magnetization
direction along the $z$ axis. Considering the remaining tensors, expressions
for $\chi^{(p,1)}$ and $\chi^{(a,2)}$ predict vanishing photocurrents for the
circularly polarized pulses, while the sign of $J_{i}$ switches upon the
change in the direction of light from $x$ to $y$ for linearly polarized light.
For the spin current we perform a similar symmetry analysis by expanding it as

$Q_{si}=\chi_{sijk}E_{j}E_{k}^{*}+\chi_{sijkl}E_{j}E_{k}^{*}M_{l},$ (4)

where $\chi_{sijk}$ is an axial tensor of rank 4 while $\chi_{sijkl}$ is a
polar tensor of rank 5. The expressions for axial tensors of rank 4 are listed
above, and due to the 2D character of FGT, we need to consider only
$\chi^{(a,1)}$ in the expansion of $Q_{si}$ (note that in the expansion of
$J_{i}$ we excluded $\chi^{(a,1)}$). There are fifteen polar tensors of rank 5
for the point group of FGT. As discussed above we need to consider only
tensors that predict an effect for magnetization along the $z$ axis.
Additionally, we consider only tensors where the indices $i,j,k$ are all
different from $z$ because of the 2D character of FGT. It turns out that out
of the 15 polar tensors only a single one satisfies these requirements. It is
given by
$\chi^{(p,1)}_{sijkl}=\delta^{(31113)}_{sijkl}-\delta^{(32213)}_{sijkl}-\delta^{(32123)}_{sijkl}-\delta^{(31223)}_{sijkl}$.
As a result, by symmetry only $z$-polarized spin currents are allowed, with
in-plane properties of spin currents identical to those of charge currents.

Figure 2: Dependence of photocurrents on the band broadening $\Gamma$. Shown
are the photocurrent components $J_{x}$ (red solid line) and $J_{y}$ (blue
dashed line) in response to the light pulse of frequency $\hbar\omega=1.55$ eV
and intensity of 10 GW/cm2 polarized along the $x$-axis. The inset depicts the
corresponding photocurrent components multiplied with a factor $\Gamma$.

Results. Overall, within the accuracy of the calculations, the symmetry
properties of computed photocurrents are fully consistent with the symmetry
analysis presented above, and thus in the following we focus only on $J_{x}$
and $J_{y}$ component of the charge current, and $Q_{zx}$, $Q_{zy}$ components
of the spin current, arising in response to a laser pulse polarized along
$x$-direction. First, we calculate and present in Fig. 2 the dependence of the
charge photocurrents on the quasi-particle lifetime as quantified by parameter
$\Gamma$ for the same light frequency of 1.55 eV as used in Ref. [26]. For
small values of $\Gamma$ between 25 $-$ 50 meV the predicted magnitude of the
photocurrent lies in the vicinity of 50 A/m, which is roughly one order of
magnitude larger than the magnitude of photocurrents emerging in the magnetic
Rashba model with the Rashba strength of 100 meV and similar parameters of the
pulse and degree of disorder, as predicted in Ref. [26]. This highlights FGT
as a source of strong intrinsic photocurrents which originate in the symmetry
properties of this material, and which will emerge in addition to interfacial
currents when deposited on a substrate.

Figure 3: Spectral properties of the photocurrents. (a) Frequency dependence
of the conductivity tensor components $\sigma_{xyy}$ and $\sigma_{yxx}$
evaluated for the true position of the Fermi energy in FGT layer for different
values of the broadening. (b) Dependence of the photocurrents $J_{x}$ and
$J_{y}$ on the position of the Fermi level and disorder strength (fading color
of the lines). (c) Same as in (a) for spin photocurrents $Q_{xz}$ and
$Q_{zy}$. In (b-c) the light pulse of frequency $\hbar\omega=1.55$ eV and
intensity of 10 GW/cm2 polarized along the $x$-axis was used. In all plots
$\Gamma$ is indicated with the color scale.

The magnitude of the photocurrents rapidly decreases as the amount of disorder
increases in the system, and the signal decreases ten-fold by going to the
values of $\Gamma$ of about 300 meV, with $J_{y}$ changing sign at the
disorder strength of about 150 meV. Generally, the overall functional
dependence of the photocurrents on lifetime currently presents a subject of
debates, as the non-linear nature of the effect makes it extremely difficult
to disentangle various physically-distinct disorder-driven contributions to
the photocurrents [35, 36, 37, 38, 39, 40, 41]. According to our calculations,
which do not assume any approximations on the nature of photo-induced
electronic processes, in a large range of $\Gamma$ beyond 300 meV the
photocurrents in FGT exhibit a clear $1/\Gamma$ behavior, see inset of Fig. 2.
On the other hand the behavior found for smaller disorder with respect to
$\Gamma$ appears to be very non-linear with higher-order contributions clearly
at play.

In order to examine whether the magnitude of the photocurrents can be
controlled by the frequency of the light, we compute relevant components of
the conductivity tensor as given by Eq. 3, presenting the results in Fig.
3(a). The calculated signal exhibits a very strong variation with frequency,
although the qualitative behavior of the two components is similar. The
observed strong variation can be attributed to the complex orbital composition
of the electronic structure of FGT. In particular, pronounced variations just
below the frequency of 1 eV and 2 eV can be attributed to the transitions
between the groups of bands just above and below the Fermi energy at the
K-point, and the groups positioned at approximately $-1$ eV and $+$0.5 eV at
the $\Gamma$-point $-$ which also mediate the behavior of the magneto-optical
conductivity of FGT [12, 13]. Concerning the overall magnitude of the computed
signal, reaching as much as 200 $\mu$A/V2 for frequencies below 1 eV, it is
comparable to previously reported results in van der Waals monolayers such as
non-magnetic GeS and WS2 [36, 42] and more recently in 2D magnets like CrI3
[43].

Figure 4: Reciprocal-space distribution of photocurrents. Distribution of the
$J_{x}$ (a) and $J_{y}$ (b) charge, and $-Q_{zx}$ (c) and $-Q_{zy}$ (d) spin
photocurrents in the Brillouin zone of single-layer FGT computed for the true
position of the Fermi energy. The light pulse of frequency $\hbar\omega=1.55$
eV and intensity of 10 GW/cm2 polarized along the $x$-axis was used in
combination with disorder strength $\Gamma$ of 25 meV.

The signatures of the orbital structure of FGT can be also clearly seen in the
dependence of the photocurrents on band filling, shown in Fig. 3(b). For both
components of the currents, the suppression of the signal can be clearly
visible for a position of the Fermi energy at about $-0.5$ eV and $-1.5$ eV,
which corresponds to the suppression of corresponding transitions, discussed
above, as the states become unpopulated with decreasing band filling. Notably,
in contrast to $J_{y}$, at an energy of $\approx-1.5$ eV the sign of $J_{x}$
changes, which should result in a drastic change in the direction of the in-
plane photocurrent. In Fig. 3(c) we plot the band filling dependence of
$Q_{zx}$ and $Q_{zy}$ components of the photocurrents of spin. The predicted
magnitude of the spin currents which can be generate by light in FGT is
sizeable, which marks FGT as one of the promising materials for spin
photogalvanic applications.

Interestingly, in a wide energy window just below the true $E_{F}$ the
qualitative behavior of the spin currents is very similar to that of charge
currents in terms of the evolution of the magnitude with band filling and
dependence on disorder strength. However, one difference is that while
$Q_{zx}$ changes sign around $-1.5$ eV in correlation with $J_{x}$, the sign
change occurs also for $Q_{zy}$ but not for $J_{y}$. This stands in sharp
contrast to a simple picture of a spin current being proportional to the spin
polarization of the participating states along $z$, scaled by the
corresponding component of the charge photocurrent: indeed, since just below
the energy of $-1.5$ eV the spin-polarization of Fe states changes sign, this
consideration explains the change in sign of $Q_{zx}$, however, it fails to
mimic the change of sign in $Q_{zy}$.

To understand this behavior better, we plot the reciprocal space distribution
of charge and spin currents at the true Fermi energy in Fig. 4. While from
these plots it is clear that an overall correlation between two types of
currents is present, large parts of the Brillouin zone where charge and spin
currents have a reversed sign correlation, are also visible, as can be most
prominently seen for $y$-components of the currents around the corners of the
Brillouin zone. With decreasing band filling the contribution of the
corresponding parts becomes promoted, which can explain the change of the sign
of $Q_{zy}$ with respect to $J_{y}$. Such non-trivial correlation between the
magnitude and direction of charge and spin currents can be used to engineer a
desired charge and spin transport setup by band filling. We also observe that
the decay of the spin photocurrents with disorder strength can depend strongly
on the band filling: e.g. while both components of the spin currents decay
rapidly with $\Gamma$ around the true Fermi energy, the $Q_{zx}$ component
displays a stronger disorder robustness in the energy region between $-3$ eV
and $-2$ eV. This indicates that the direction of spin photocurrents in FGT
can be tuned not only by band filling but also by the degree of disorder.

To summarize, in this work we used an ab-initio implementation of Keldysh
formalism for second order response to address the properties of charge and
spin photocurrents in a single-layer Fe3GeTe2. Our predictions concerning the
symmetry of the currents are in full agreement with direct first principles
calculations, which predict the magnitude of the photocurrents emerging in FGT
to be comparable to those of a magnetic Rashba model with strong spin-orbit
interaction. The predicted non-trivial response of the currents to such
effects as disorder strength, band filling and frequency marks FGT as a
promising platform for crafting the desired properties of the photocurrents,
which might prove to be important for future optospintronics applications of
2D magnets.

Acknowledgements. This work was supported by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) $-$ TRR 173 $-$
268565370 (projects A11 and A01), TRR 288 – 422213477 (project A06), and the
Sino-German research project DISTOMAT (MO 1731/10-1). This project has
received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Skłodowska-Curie grant agreement No
861300, and under synergy grant ”3D MAGiC” agreement (Grant No. 856538). We
also gratefully acknowledge the Jülich Supercomputing Centre and RWTH Aachen
University for providing computational resources under project jiff40.

## References

  * Huang  _et al._ [2017] B. Huang, G. Clark, E. Navarro-Moratalla, D. R. Klein, R. Cheng, K. L. Seyler, D. Zhong, E. Schmidgall, M. A. McGuire, D. H. Cobden, W. Yao, D. Xiao, P. Jarillo-Herrero, and X. Xu, Nature 546, 270 (2017).
  * Gong and Zhang [2019] C. Gong and X. Zhang, Science 363 (2019).
  * Gong  _et al._ [2017] C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y. Wang, Z. Q. Qiu, R. J. Cava, S. G. Louie, J. Xia, and X. Zhang, Nature 546, 265 (2017).
  * Gibertini  _et al._ [2019] M. Gibertini, M. Koperski, A. F. Morpurgo, and K. S. Novoselov, Nature Nanotechnology 14, 408 (2019).
  * Verchenko  _et al._ [2015] V. Y. Verchenko, A. A. Tsirlin, A. V. Sobolev, I. A. Presniakov, and A. V. Shevelkov, Inorganic Chemistry 54, 8598 (2015).
  * Ding  _et al._ [2020] B. Ding, Z. Li, G. Xu, H. Li, Z. Hou, E. Liu, X. Xi, F. Xu, Y. Yao, and W. Wang, Nano Letters 20, 868 (2020).
  * Wu  _et al._ [2020] Y. Wu, S. Zhang, J. Zhang, W. Wang, Y. L. Zhu, J. Hu, G. Yin, K. Wong, C. Fang, C. Wan, X. Han, Q. Shao, T. Taniguchi, K. Watanabe, J. Zang, Z. Mao, X. Zhang, and K. L. Wang, Nature Communications 11, 3860 (2020).
  * Park  _et al._ [2021] T.-E. Park, L. Peng, J. Liang, A. Hallal, F. S. Yasin, X. Zhang, K. M. Song, S. J. Kim, K. Kim, M. Weigand, G. Schütz, S. Finizio, J. Raabe, K. Garcia, J. Xia, Y. Zhou, M. Ezawa, X. Liu, J. Chang, H. C. Koo, Y. D. Kim, M. Chshiev, A. Fert, H. Yang, X. Yu, and S. Woo, Phys. Rev. B 103, 104410 (2021).
  * Alghamdi  _et al._ [2019] M. Alghamdi, M. Lohmann, J. Li, P. R. Jothi, Q. Shao, M. Aldosary, T. Su, B. P. T. Fokwa, and J. Shi, Nano Letters 19, 4400 (2019).
  * Johansen  _et al._ [2019] O. Johansen, V. Risinggård, A. Sudbø, J. Linder, and A. Brataas, Phys. Rev. Lett. 122, 217203 (2019).
  * Martin  _et al._ [2021] F. Martin, K. Lee, M. Schmitt, A. Liedtke, A. Shahee, H. T. Simensen, T. Scholz, T. G. Saunderson, D. Go, M. Gradhand, Y. Mokrousov, T. Denneulin, A. Kovács, B. Lotsch, A. Brataas, and M. Kläui, Strong bulk spin-orbit torques quantified in the van der Waals ferromagnet Fe3GeTe2 (2021), arXiv:2107.09420 [cond-mat.mtrl-sci] .
  * Yang  _et al._ [2021] X. Yang, X. Zhou, W. Feng, and Y. Yao, Strong magneto-optical and anomalous transport manifestations in two-dimensional van der Waals magnets FenGeTe2 ($n$ = 3, 4, 5) (2021), arXiv:2108.02926 [cond-mat.mtrl-sci] .
  * Jiang and Guo [2020] M.-C. Jiang and G.-Y. Guo, Large magneto-optical effect and magnetic anisotropy energy in two-dimensional ferromagnetic metal Fe3GeTe2 (2020), arXiv:2012.04285 [cond-mat.mtrl-sci] .
  * Kampfrath  _et al._ [2013] T. Kampfrath, M. Battiato, P. Maldonado, G. Eilers, J. Nötzold, S. Mährlein, V. Zbarsky, F. Freimuth, Y. Mokrousov, S. Blügel, M. Wolf, I. Radu, P. M. Oppeneer, and M. Münzenberg, Nature Nanotechnology 8, 256 (2013).
  * Dhillon  _et al._ [2017] S. S. Dhillon, M. S. Vitiello, E. H. Linfield, A. G. Davies, M. C. Hoffmann, J. Booske, C. Paoloni, M. Gensch, P. Weightman, G. P. Williams, E. Castro-Camus, D. R. S. Cumming, F. Simoens, I. Escorcia-Carranza, J. Grant, S. Lucyszyn, M. Kuwata-Gonokami, K. Konishi, M. Koch, C. A. Schmuttenmaer, T. L. Cocker, R. Huber, A. G. Markelz, Z. D. Taylor, V. P. Wallace, J. A. Zeitler, J. Sibik, T. M. Korter, B. Ellison, S. Rea, P. Goldsmith, K. B. Cooper, R. Appleby, D. Pardo, P. G. Huggard, V. Krozer, H. Shams, M. Fice, C. Renaud, A. Seeds, A. Stöhr, M. Naftaly, N. Ridler, R. Clarke, J. E. Cunningham, and M. B. Johnston, Journal of Physics D: Applied Physics 50, 043001 (2017).
  * Vitiello [2019] M. S. Vitiello, Journal of Physics: Materials 3, 014008 (2019).
  * Papaioannou and Beigang [2021] E. T. Papaioannou and R. Beigang, Nanophotonics 10, 1243 (2021).
  * Wahada  _et al._ [2021] M. A. Wahada, E. Sasioglu, W. Hoppe, X. Zhou, H. Deniz, R. Rouzegar, T. Kampfrath, I. Mertig, S. S. P. Parkin, and G. Woltersdorf, Atomic scale control of spin current transmission at interfaces (2021), arXiv:2108.01770 [cond-mat.mes-hall] .
  * Burch  _et al._ [2018] K. S. Burch, D. Mandrus, and J.-G. Park, Nature 563, 47 (2018).
  * Huisman  _et al._ [2016] T. J. Huisman, R. V. Mikhaylovskiy, J. D. Costa, F. Freimuth, E. Paz, J. Ventura, P. P. Freitas, S. Blügel, Y. Mokrousov, T. Rasing, and A. V. Kimel, Nature Nanotechnology 11, 455 (2016).
  * Sipe and Shkrebtii [2000] J. E. Sipe and A. I. Shkrebtii, Phys. Rev. B 61, 5337 (2000).
  * Mu  _et al._ [2021] X. Mu, Y. Pan, and J. Zhou, npj Computational Materials 7, 61 (2021).
  * Sherman  _et al._ [2005] E. Y. Sherman, A. Najmaie, and J. E. Sipe, Applied Physics Letters 86, 122103 (2005).
  * Yin and Peng [2019] J. Yin and H. Peng, Nature Nanotechnology 14, 105 (2019).
  * Tyznik  _et al._ [2021] C. Tyznik, J. Lee, J. Sorli, X. Liu, E. K. Holland, C. S. Day, J. E. Anthony, Y.-L. Loo, Z. V. Vardeny, and O. D. Jurchescu, ACS Applied Materials & Interfaces 13, 10231 (2021).
  * Freimuth  _et al._ [2021] F. Freimuth, S. Blügel, and Y. Mokrousov, Phys. Rev. B 103, 075428 (2021).
  * Xu  _et al._ [2021] H. Xu, H. Wang, J. Zhou, and J. Li, Nature Communications 12, 4330 (2021).
  * Xiao  _et al._ [2021] R.-C. Xiao, D.-F. Shao, Y.-H. Li, and H. Jiang, npj Quantum Materials 6, 35 (2021).
  * Freimuth  _et al._ [2016] F. Freimuth, S. Blügel, and Y. Mokrousov, Phys. Rev. B 94, 144432 (2016).
  * Pizzi  _et al._ [2020] G. Pizzi, V. Vitale, R. Arita, S. Blügel, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, J. Ibañez-Azpiroz, H. Lee, J.-M. Lihm, D. Marchand, A. Marrazzo, Y. Mokrousov, J. I. Mustafa, Y. Nohara, Y. Nomura, L. Paulatto, S. Poncé, T. Ponweiser, J. Qiao, F. Thöle, S. S. Tsirkin, M. Wierzbowska, N. Marzari, D. Vanderbilt, I. Souza, A. A. Mostofi, and J. R. Yates, Journal of Physics: Condensed Matter 32, 165902 (2020).
  * Freimuth  _et al._ [2008] F. Freimuth, Y. Mokrousov, D. Wortmann, S. Heinze, and S. Blügel, Phys. Rev. B 78, 035120 (2008).
  * Wang  _et al._ [2006] X. Wang, J. R. Yates, I. Souza, and D. Vanderbilt, Phys. Rev. B 74, 195118 (2006).
  * [33] www.flapw.de.
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Kraut and von Baltz [1979] W. Kraut and R. von Baltz, Phys. Rev. B 19, 1548 (1979).
  * Ibañez Azpiroz  _et al._ [2018] J. Ibañez Azpiroz, S. S. Tsirkin, and I. Souza, Phys. Rev. B 97, 245143 (2018).
  * Matsyshyn and Sodemann [2019] O. Matsyshyn and I. Sodemann, Phys. Rev. Lett. 123, 246602 (2019).
  * Morimoto and Nagaosa [2016] T. Morimoto and N. Nagaosa, Science Advances 2, 10.1126/sciadv.1501524 (2016).
  * Ahn  _et al._ [2020] J. Ahn, G.-Y. Guo, and N. Nagaosa, Phys. Rev. X 10, 041041 (2020).
  * Ahn  _et al._ [2021] J. Ahn, G.-Y. Guo, N. Nagaosa, and A. Vishwanath, Riemannian geometry of resonant optical responses (2021), arXiv:2103.01241 [cond-mat.mes-hall] .
  * Asgari and Culcer [2021] R. Asgari and D. Culcer, Unidirectional valley-contrasting photo-current in the non-linear optical response of transition metal dichalcogenide monolayers (2021), arXiv:2108.10438 [cond-mat.mes-hall] .
  * Wang  _et al._ [2017] C. Wang, X. Liu, L. Kang, B.-L. Gu, Y. Xu, and W. Duan, Phys. Rev. B 96, 115147 (2017).
  * Zhang  _et al._ [2019] Y. Zhang, T. Holder, H. Ishizuka, F. de Juan, N. Nagaosa, C. Felser, and B. Yan, Nature Communications 10, 3783 (2019).