# MoVQA: A Benchmark of Versatile Question-Answering for
Long-Form Movie Understanding

Hongjie Zhang1 Yi Liu1,3∗ Lu Dong1,4∗ Yifei Huang1
Zhenhua Ling4 Yali Wang3,1✉ Limin Wang2,1✉ Yu Qiao1✉
1OpenGVLab, Shanghai AI Laboratory 2Nanjing University
3Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
4University of Science and Technology of China
https://movqa.github.io/
Equal contribution ✉ Corresponding authors

###### Abstract

While several long-form VideoQA datasets have been introduced, the length of
both videos used to curate questions and sub-clips of clues leveraged to
answer those questions have not yet reached the criteria for genuine long-form
video understanding. Moreover, their QAs are unduly narrow and modality-
biased, lacking a wider view of understanding long-term video content with
rich dynamics and complex narratives. To remedy this, we introduce MoVQA, a
long-form movie question-answering dataset, and benchmark to assess the
diverse cognitive capabilities of multimodal systems rely on multi-level
temporal lengths, with considering both video length and clue length.
Additionally, to take a step towards human-level understanding in long-form
video, versatile and multimodal question-answering is designed from the
moviegoer-perspective to assess the model capabilities on various perceptual
and cognitive axes. Analysis involving various baselines reveals a consistent
trend: the performance of all methods significantly deteriorate with
increasing video and clue length. Meanwhile, our established baseline method
has shown some improvements, but there is still ample scope for enhancement on
our challenging MoVQA dataset. We expect our MoVQA to provide a new
perspective and encourage inspiring works on long-form video understanding
research.

##  1 Introduction

Figure 1: A causal reasoning QA example from MoVQA about the movie
_Godfather_. Answering this question not only requires capturing clues from a
long temporal span, but also requires high-level processing of these relevant
clues. Figure 2: An overview of our MoVQA. We designed a multi-level question-
answering to systematically evaluate the long-form video understanding
capabilities of models. These QA pairs are sourced from single-scene, multi-
scene and full-scene, and cover various aspects of video understanding,
including information synopsis, temporal perception, spatial perception,
causal reasoning, hypothetical reasoning and external knowledge.

Movies are a mosaic of life, and to truly understand their content, one must
piece together the various clues scattered throughout [1, 44, 39, 41, 29, 12,
42, 60]. For example, in Fig.1, answering the question in the movie
_Godfather_ requires amalgamating multiple clues from the long storyline, and
these clues may be tens of minutes away from the clip where the question
arises. Yet being able to capture these clues, reason about their causal and
temporal relations, which lies at the core of human intelligence [49, 2],
remains a great challenge for multimodal systems and is also much less
explored by existing long-form video understanding tasks [30, 44, 10, 40, 47,
46, 39, 62].

It is worth noting that, in what aspects does long-form VideoQA differ from
general VideoQA in terms of “long”? We argue that there are two indicators:
clue length and video length. The clue length, called as temporal certificate
length in [30], represents the total length of the minimum set of sub-clips of
the video that are both necessary and sufficient to convince a human verifier
that the annotated information for that data is indeed correct, without having
to watch the rest of sub-clips outside of the clue set. Undeniably, this
indicator plays a crucial role in measuring the inherent temporal difficulty
of long-form VideoQA task. However, assuming two VideoQA tasks with same clue
length, and without considering other factors, intuitively, answering the
question within a longer video length is more challenging. This is because a
longer video length implies a longer contextual temporal relationship,
resulting in more information redundancy and increasing the difficulty of
capturing relevant clues.

In this work, considering both indicators, we introduce MoVQA, a new dataset
and benchmark, contains 21,953 manually annotated QA pairs sourced from 100
movies with diverse genres, to assess the model capabilities on long-form
video understanding with multi-level temporal lengths. As shown in Fig.2, QAs
in our MoVQA come from three-level lengths: single-scene, multi-scene and
full-scene. Annotators are asked to manually segment the movie into multiple
consecutive, non-overlapping single-scene intervals of varying length based on
the storyline and visual presentation, with clear start and end timestamps. A
median clue length and video length of these single-scenes are about 200
seconds and 7.5 minutes. For longer video understanding, adjacent and closely
related single-scenes are merged to form multi-scenes with longer durations,
with clue length and video length of about 320 seconds and 20 minutes,
respectively. Our MoVQA also includes specifically designed QAs for the full-
scene, which have a clue length of about 540 seconds and a video length of
about 120 minutes. Compared to [30], which has a median clue length of about
100 seconds and a video length of 3 minutes, our MoVQA exceeds it on both two
indicators. Additionally, the types of QAs in [30] are relatively narrow,
primarily focusing on summarizing the overarching behavior of the video.

To take a step towards human-level long-video understanding, QAs in our MoVQA
are designed from the perspective of the interconnected abilities required by
moviegoers to understand movie content. They are grouped into six types:
information synopsis, temporal perception, spatial perception, causal
reasoning, hypothetical reasoning and external knowledge. Compared to VideoQA
dataset that also target movies [41], where QAs are solely collected based on
text summaries, making them very plot-focused and less relevant for visual
information. In contrast, ours are relatively more versatile and require
integration of multimodal information for answering. The primary objective of
our MoVQA is to incentivize the development of multimodal systems that can
tackle versatile QAs presented by moviegoers, spanning long periods of time
and relying on a variety of modal information.

The main contributions of this paper are as follows: (1) We focus on long-form
with multi-level QAs by considering both clue length and video length, to
support the research on video understanding across different temporal lengths.
(2) Versatile and multimodal question-answering is designed from the
moviegoer-perspective to assess the model capabilities on various perceptual
and cognitive axes. (3) We benchmark various state-of-the-art multimodal
systems on MoVQA and find that even the most advanced current systems achieve
unsatisfactory results. Moreover, our established baseline with keyframe
selection and contextual modelling achieves the state-of-the-art performance,
which we hope can guide future directions in establishing long-form
understanding methods.

##  2 Related Work

Video Question-Answering Datasets. VideoQA is a powerful tool for evaluating a
model’s comprehensive understanding of a given video[32, 16, 29, 51, 5, 7, 41,
20, 21, 48, 32]. Early VideoQA dataset, such as TGIF-QA [16], MovieFiB [29],
MSRVTT-QA [51] primarily focused on visual descriptions, repetition, and
transitions of single objects or actions. Their videos typically consisted of
relatively short videos capturing a single event. Recent advancements in
VideoQA aim for more causal and temporal reasoning on multiple events.
Datasets like MovieQA [41] and TVQA [20] design causal questions on movies and
TV shows respectively, requiring plot understanding or actor dialogue
understanding. LifeQA [4] generates question answering pairs about real-life
scenarios. ActivityNet-QA [58] generates much longer motion, spatial, and
temporal QA based on ActivityNet dataset [11]. NExT-QA [48] emphasizes causal
and temporal actions in daily life videos, while STAR [43] automatically
generates situation abstraction and logic-grounded question answering, i.e.,
person-object-action relations in daily life videos. Nevertheless, these
datasets ask questions on relatively short videos (ranging from 10 to 203
seconds), while our benchmark ask questions on much longer videos (956 seconds
on average) with much richer video content.

Long-form Video Understanding Datasets. Recently, long video understanding has
begun to rise [31, 14, 44, 39, 30, 37, 12, 42, 50, 60, 9]. [44] proposes a
long-form video understanding benchmark based on [1], where the tasks are much
specific, such as speaker style and movie release year. [39] proposes MAD, a
language grounding dataset designed to ground queries on an average clip
duration of 110 minutes. However, their language queries are much shorter
(averaging 4 seconds) and lack long-form contextual relationships or high-
level semantics. [30] proposes EgoSchema with a length of 180 seconds, which
has a certificate length of about 100 seconds, i.e., the sub-clips of the
videos that are both necessary and sufficient for answering the question.
Nevertheless, their questions consist mainly of summary questions and lack
variety in question types. In contrast, our MoVQA generates six diverse types
of questions on longer videos, resulting in a much longer certificate length.

Video Question-Answering Method. VideoQA is a typical type of vision-language
task that has been studied for many years. Early, most studies analyze the
video or text backbones and the fusion of video and text [59, 19, 17, 33].
[59] proposes a deep modular co-attention network for efficient interaction
between questions and videos. [33] utilizes motion and appearance features to
build a question to video graph. Recently, advanced VideoQA models are built
upon pretrained vision-language model [52, 53, 23]. [53] leverages a frozen
language model and trainable adapters to train a video-language backbone. [23]
adopts a frozen image encoder, a frozen large language model, and a trainable
Q-Former to train an image-language backbone. These pretrained vision-language
models demonstrate strong performance on VideoQA tasks.

Long-form Video Understanding The main challenge in long-form video
understanding is how to handle long video inputs [44]. There are two main
directions. The first approach is to model long video input using an end-to-
end network [13, 45, 26, 8, 38]. Specifically, [45] uses non-local network to
attend the long video features with short video feature query. [26] models
long video using sparsely sampled high-cost video frames and dense sampled
low-cost audio. [8] utilizes cascaded segment and region selection modules to
select frames and image regions that are closely relevant to the question.
Another approach is to select keyframes from the long videos [54, 34, 18, 15,
27, 35]. For example, [57] uses grounding datasets for keyframe localizer
pretraining and leverages QA feedback to refine the keyframe localizer. We
combine the merit of existing works to design a dual perceiver modeling
framework for efficient long video understanding.

##  3 MoVQA Benchmark

We carefully selected 100 movies from the highly rated movies on
IMDb***http://www.imdb.com/, covering a wide range of genres, years and
countries. The average length of the movies is 125.7 minutes. To ensure the
purity of the visual information in our experiments, our movie data does not
contain embedded subtitles. Additionally, we collect the external subtitles
for each movie, and each subtitle contains the start and end timestamps
corresponding to the dialogue in the video content. Only during the annotation
process, the movie data includes embedded subtitles to facilitate the
understanding of the movie content by annotators. The collected movies are
publicly available from the internet, and are allowed to be crawled according
to the service contract of the website. Considering copyright issue, we will
only release the pretrained features of movies. Furthermore, our data source
does not contain any information that names or uniquely identifiable
individuals or offensive content.

###  3.1 Question-Answering Design

Dataset | Annotation | #QA Pairs | Avg.Len.(s) | Multi-level | Multimodal | QA Type
---|---|---|---|---|---|---
Causal | Hypothetical | Synopsis | Knowledge | Spatial | Temporal
TGIF-QA [16] | Auto | 165,165 | 3 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
MSRVTT-QA [51] | Auto | 243,690 | 15 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
How2QA [25] | Human | 44,007 | 60 | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓
NExT-QA [25] | Human | 52,044 | 44 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
EgoSchema [30] | Auto | 5,000 | 180 | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗
MovieQA [41] | Human | 6,462 | 203 | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗
TVQA [20] | Human | 152,545 | 76 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓
MoVQA | Human | 21,953 | 992 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

Table 1: Comparison of our MoVQA to various VideoQA datasets. MoVQA is unique
since its QAs are sourced from multi-level and longer temporal lengths, and
the QA types are more diverse.

Scene Partition. To evaluate model capabilities on long-form video
understanding across different temporal lengths, each movie was first
segmented into clips of varying durations. Our annotators were asked to first
select a movie from the list that they were familiar with, and then manually
segment the movie into multiple consecutive, non-overlapping single-scene
intervals of varying length based on the storyline and visual presentation,
with clear start and end timestamps. The average duration of single-scene is
about 7.8 minutes. For longer video understanding, adjacent and closely
related single-scenes are merged to form multi-scenes with longer durations,
typically spanning an average length of 22 minutes. Our MoVQA also
specifically designed QAs with a super long-form for the full-scene (average
duration of 125.7 minutes).

QA Annotation. For each scene, the annotator had the freedom of forming 9 to
10 questions based on the 6 types, and these questions require both vision and
language understanding to answer. The correct answer to each question is given
by the annotator. Another annotator was presented with a clip and a question
at a time, but not provided with the correct answer, and asked to answer the
question correctly. Finally, the two annotators jointly discussed and
reorganized or corrected the QA pairs based on their answers. The reason for
inviting another annotator to answer the question is that assuming that the QA
pairs are completed by the same annotator, biases in understanding from the
annotator inevitably arise, leading to incorrect answers.

QA Types. To assess model capabilities on various perceptual and cognitive
axes in movie understanding, QAs in our MoVQA are designed from the
perspective of the interconnected abilities required by moviegoers to
understand the movie content. They are grouped into six types:

$\bullet$ Information Synopsis. Information synopsis questions require the
model to extract key information related to the question from the movie and
summarize them concisely in language. This involve aspects such as the movie
theme, plot synopsis, character relationships, and motivations.

$\bullet$ Temporal Perception. Temporal perception questions require the model
to understand the timeline of events and the plot in the movie, and accurately
locate the temporal location of the answer on the timeline. Compared with the
temporal action localization task [61, 6], which requires the model to
recognize and locate specific actions in a video, our temporal perception task
places more emphasis on understanding the complex narrative sequence in the
movie.

$\bullet$ Spatial Perception. Spatial perception questions require the model
to recognize and understand the visual elements in movies, which often have
multiple characters, objects, and scenes. It may require combining the context
of the story to infer the visual information related to the question.

$\bullet$ Causal Reasoning. Causal reasoning questions require the model to
understand and infer cause-and-effect relationships between events in the
storyline, and causality is the key driving force in the development of the
story.

$\bullet$ Hypothetical Reasoning. Hypothetical reasoning questions require the
model to understand the preconditions and post-effects of storylines. Based on
causal tracing of storylines, we designed hypothetical reasoning questions
with the assumption that certain events in the story are not happened, and how
the storyline will develop under this condition.

$\bullet$ External Knowledge. External knowledge questions require the model
to understand additional knowledge that could not be found in movies but has
relevance to the movie content. These external knowledge may involve cultural
background, actor information, history, and social response.

We showcase examples of the various types mentioned above in our supplementary
materials.

###  3.2 Distractor Options Generation

We evaluate long-form video understanding as a multiple-choice question-
answering task, where each question has five candidate answers, only one of
which is correct. The role of distractor options in multiple-choice question-
answering is crucial because they determine the difficulty of QA. Well-
designed distractor options should closely resemble the correct answer in
terms of plausibility, making it challenging for participants to select the
correct option.

We initially attempted to collect distractor options using ChatGPT, so as to
save labor costs. However, we found that the generated candidates were
unsatisfactory and had significant shortcuts. More details are shown in Sec
5.4. Therefore, we opted to engage human annotators in the process of writing
distractor options within our dataset. We established three guiding principles
to avoid the bias from the distractor options generated by ChatGPT. Firstly,
we aimed to ensure that the length of the distractor options closely matched
that of the correct options, to avoid overly long or short distractor bias.
Secondly, we ensured that the semantic content of the distractor options were
different from that of the correct answers. To achieve this, we employed
SentenceTransformer [36] to calculate the semantic differences and instructed
annotators to verify and correct options with a semantic similarity exceeding
0.9. Finally, for longer answers, we mandated that annotators modify at least
two distinct parts of the response to create a more robust set of distractor
options. We show in Table 10 that human-corrected distractor options following
these three principles are more challenging than those directly generated by
the ChatGPT.

Figure 3:  (a) The distribution of QA types based on multi-level scenes. (b)
The distribution of clue length and video length in different VideoQA
datasets.

One addition point is that we did not ask human workers to generate
distractors for our temporal perception QA, because the answer format of
temporal perception QA is a time interval that can be generated according to
rules. To generate difficult distractors for temporal perception QA, we
randomly chose four equal-length, non-overlap intervals that are before or
after the time interval of the correct answer to generate the distractors. The
equal-length design can avoid bias in the length distribution of time
intervals.

Quality Control. To construct a high-quality dataset which offers accurate
annotations at all scenes and types, we adopt a series of mechanisms,
including pretesting the annotators rigorously before formal annotation,
preparing informative reference slides as well as demonstrations, and cross-
validating across annotators.

Q Type | #QA Pairs | Q-Len | CO-Len | DO-Len
---|---|---|---|---
Synopsis | 6,786 | 11.0 | 28.9 | 28.7
Temporal | 6,786 | 12.1 | 4.0 | 4.0
Spatial | 2,262 | 10.8 | 13.9 | 13.4
Causal | 2,262 | 10.9 | 15.0 | 15.3
Hypothetical | 2,262 | 14.9 | 14.7 | 14.8
Knowledge | 1,595 | 8.7 | 26.9 | 26.6
Total | 21,953 | 11.5 | 16.6 | 16.5

Table 2: Statistics for different QA types. Q = question, CO = correct option,
DO = distractor option. Length is defined as the number of words.

###  3.3 Data Statistics

A comparison of our MoVQA to various popular VideoQA datasets is presented in
Table 1. QAs in our MoVQA not only span multi-level scenes and encompass
diverse types, but also have the longest video length. We collect a total of
21,953 QAs, including 15,950 single-scene, 5,103 multi-scene, and 900 full-
scene. The distribution of different types of QAs based on multi-level scenes
is shown in Fig.3 (a). The number of QAs in three scenes accounted for 73% in
single-scene, 23% in multiple-scene, and 4% in full-scene. The distribution of
clue length and video length across different VideoQA datasets is shown in
Fig.3 (b). We follow [30] and benchmark the clue length for 50 hours of clips
(200 QAs) chosen randomly. Our MoVQA has a median clue length of about 200
seconds for single-scene, 320 seconds for multi-scene, and 540 seconds for
full-scene, which are $2\times$ (single-scene), $3.2\times$ (multi-scene), and
$5.4\times$ (full-scene) longer than the dataset with the second-longest clue
length. In general, the median clue length of all QAs is approximately 230
seconds. Additionally, our MoVQA has a median video length of about 8 minutes
for all QAs, which is $2.6\times$ longer than the second-longest video length.
Specifically, 7.5 minutes for single-scene, 20 minutes for multi-scene, and
120 minutes for full-scene. Table 2 presents statistics of the QAs based on
types. On average, both questions and answers in our MoVQA are longer compared
to those in previous datasets, especially the answers, which are approximately
$3\times$ longer than those in TVQA [20] and MovieQA [41]. This indicates that
our dataset provides richer and more detailed information, requiring more
clues to describe and answer the questions.

##  4 Proposed Method

Figure 4: An overview of our DPM framework. A similar process is used for
subtitle modality and thus omitted for simplicity.

From the moviegoer-perspective, when answering a question about a movie, it is
essential to consider both the key clues about the question and the contextual
elements of the movie. Based on this perspective, we propose a multimodal Dual
Perceiver Modeling (DPM) framework, which consists of two branches: one is the
keyframe branch for selecting the most important clues, and the other is the
context branch for modeling contextual information of long videos. An overview
of our proposed method is shown in Fig.4. The inputs are a long video clip
$V$, a subtitle $S$, a question $Q$ and five candidate answers
$\\{a_{i}\\}_{i=1}^{5}$.

###  4.1 Keyframe Branch

To capture clues relevant to the question from the long video, we refer to the
keyframe selection module in [57], which serves as a localizer to select the
top-k keyframes related to the question as clues from long video frames.
Specifically, this localizer generates the probability of each frame by asking
BLIP-2 [23] with a localization question and selects the top-k frames as
keyframes. For video, the keyframe-aware perceiver module consists of the
image encoder and the Q-Former used in BLIP-2 [23], while for text modality,
it is the word embedding used in the LLM. Considering that our movie data
contains both videos and subtitles of dialogues, relying solely on visual
information to locate keyframes can be suboptimal, we choose to combine visual
and subtitle information in our keyframe generator module to locate keyframes
relevant to the question. Then, we adopt the selected top-k keyframes along
with the corresponding top-k subtitles for the next operation.

###  4.2 Context Branch

To model the contextual information on long videos, we first extract features
from the video and subtitles using our feature extractor module, which has a
similar structure to the keyframe perceiver module in the keyframe branch.
Then, we proceed to model the context of this long sequence features using a
context-aware perceiver module, which is inspired by the design of Q-Former
used in [23]. Specifically, the context-aware perceiver module is a
transformer decoder, where the query is a set of learnable embeddings, and
both the key and value are the long video token.

Setting | Method | Single-Scene | Multi-Scene | Full-Scene |
---|---|---|---|---|---
Zero-shot | FrozenBiLM [53] | 28.1 | 25.0 | 25.3 |
BLIP-2 [23] | 41.3 | 39.3 | 38.3 |
SEVILA [57] | 42.0 | 40.0 | 39.5 |
Finetuning | FrozenBiLM [53] | 42.2 | 39.0 | 38.9 |
BLIP-2 [23] | 48.7 | 44.1 | 40.0 |
SEVILA [57] | 49.2 | 45.6 | 40.7 |
DPM | 50.3 | 49.0 | 42.7 |

Table 3: Performance of VideoQA methods on our MoVQA.

Method | Synopsis | Temporal | Spatial | Causal | Hypothetical | Knowledge
---|---|---|---|---|---|---
FrozenBiLM [53] | 50.4 | 22.8 | 48.3 | 47.0 | 47.9 | 40.8
BLIP-2 [23] | 56.7 | 22.3 | 62.4 | 56.7 | 52.8 | 53.0
SEVILA [57] | 57.2 | 23.3 | 61.6 | 54.7 | 55.2 | 54.8
DPM | 60.2 | 23.6 | 63.1 | 57.6 | 56.3 | 55.0

Table 4: Performance of VideoQA methods on different question types of our
MoVQA.

Finally, we concatenate the output of the two branches and the QA prompt as
the input to the LLM used in [23] to generate an answer. The QA prompt follows
[57].

##  5 Experiments

Our setting is as follows. We used a 6:2:2 train/val/test data to split the
100 movies of MoVQA dataset, such that movies and their corresponding QA pairs
appear in only one split. This results in 13,780 QA pairs for training, 4,133
QA pairs for validation, and 4,040 QA pairs for testing. For multiple-choice
question-answering task, we use the question-answering accuracy as the
evaluation metrics.

Setting | Method | Single-Scene | Multi-Scene | Full-Scene
---|---|---|---|---
Zero-shot | Random | 20.0 | 20.0 | 20.0
Mplug-Owl [55] | 25.2 | 23.5 | 22.1
Otter [22] | 23.1 | 22.1 | 21.3
VideoChatGPT [28] | 23.4 | 22.7 | 22.3
VideoChat [24] | 24.9 | 24.1 | 21.6
Instruction Tuning | VideoChatGPT [28] | 28.2 | 26.5 | 25.1
VideoChat [24] | 29.7 | 29.2 | 25.6

Table 5: Performance of multimodal LLMs on our MoVQA.

Method | Synopsis | Temporal | Spatial | Causal | Hypothetical | Knowledge
---|---|---|---|---|---|---
Random | 20.0 | 20.0 | 20.0 | 20.0 | 20.0 | 20.0
Mplug-Owl [55] | 25.1 | 19.9 | 25.3 | 21.9 | 23.5 | 27.5
Otter [22] | 22.6 | 20.7 | 19.6 | 26.1 | 24.2 | 21.8
VideoChatGPT [28] | 23.8 | 20.2 | 22.1 | 22.1 | 21.4 | 24.1
VideoChat [24] | 25.2 | 20.1 | 26.7 | 25.8 | 22.9 | 26.0

Table 6: Performance of multimodal LLMs methods on different question types of
our MoVQA.

###  5.1 Implementation Details

Following [57], we build our DPM upon BLIP-2 [23], using its flan-t5 xl LLM
variant. In the keyframe branch, we choose 32 keyframes along with their
respective subtitles, from 512 uniformly sampled frames and their
corresponding subtitles. In the context branch, we uniformly sample 512 frames
and truncated concatenated subtitles to a length of 512. Then, we use 32
learnable query embeddings to model the context of long videos and long
subtitles, respectively. All experiments are conducted on 8 A100 GPUs. Due to
the large computation of the BLIP-2, we can only use a batch size of 1.
Regarding other hyperparameters, we refer to the settings in [57]. As for
other VideoQA methods and all multimodal LLMs methods, their experimental
settings follow the original paper.

###  5.2 VideoQA Methods

We report the performance of the state-of-the-art VideoQA methods, including
FrozenBiLM [53], BLIP-2 [23], SEVILA [57] and our DPM. Table 3 shows the
results on different scenes. Firstly, we can see that the SEVILA achieves the
best zero-shot performance with other baselines and our DPM achieves the best
finetuning results across all scenes, with zero-shot accuracies ranging from
39.5% to 42.0% and finetuning accuracies ranging from 42.7% to 50.3%.
Comparing with the second-ranked SEVILA in finetuning, our DPM exhibits
improvements of 1.1% in single-scene, 3.4% in multi-scene and 2.0% in full-
scene, respectively. Secondly, all methods significantly deteriorate with an
increasing number of scenes, indicating that the difficulty of question-
answering increases with longer clue lengths and video lengths. Table 4 shows
the results on different QA types. We can see that the biggest challenge of QA
type is temporal perception. This could be attributed to the fact that in the
temporal perception type, the content of answer options has a relatively
simple form compared to other types, e.g., “From $00\colon 36\colon 49$ to
$00\colon 37\colon 30$.” in Fig.2. There is limited linguistic distinction
between the correct and distractor options, making it difficult for models to
eliminate certain distractor options based solely on linguistic cues.

###  5.3 Multimodal LLMs Methods

Unlike VideoQA methods, multimodal LLMs methods do not calculate the
probability values for each option. Instead, it directly answers the question
and generates the corresponding option. We find that multimodal LLMs methods
face significant challenges on our MoVQA dataset. We report the performance of
the state-of-the-art multimodal LLMs methods for which the codes are publicly
available, including VideoChat [24], Video-ChatGPT [28], mPLUG-Owl [56] and
Otter[22], and systematically explore the performance of these models on our
MoVQA.

Setting | Video token num | Single-Scene | Multi-Scene | Full-Scene
---|---|---|---|---
Zero-shot | 32 $\rightarrow$ 4 | 41.5 | 38.6 | 37.0
32 $\rightarrow$ 10 | 42.0 | 40.0 | 39.5
64 $\rightarrow$ 10 | 42.1 | 39.6 | 38.9
64 $\rightarrow$ 30 | 39.7 | 37.4 | 34.0
| 512 $\rightarrow$ 30 | 42.2 | 39.8 | 38.3
Finetuning | 32 $\rightarrow$ 4 | 45.5 | 42.3 | 39.7
32 $\rightarrow$ 10 | 48.1 | 45.0 | 40.4
64 $\rightarrow$ 10 | 48.6 | 45.2 | 40.6
64 $\rightarrow$ 30 | 49.2 | 45.6 | 40.7
| 512 $\rightarrow$ 30 | 49.3 | 45.3 | 40.7

Table 7: Ablation study on video tokens. The notation $X\rightarrow Y$
represents the number of frames uniformly sampled initially, followed by the
number of the selected keyframes determined by the localizer in SEVILA.

Choice Extraction. During inference, although the problems are presented as
clear single-choice questions with well-formatted options, many LLMs still
provide answers in free-form text. To address this issue, we first extract
choices from the predictions of LLMs by employing exact matching techniques.
For instance, when trying to match the choice “a0”, we check if the output of
LLMs contains any occurrences of “a0”, “a0.”, “a0)”, “a0:”, and so on. Once a
match is found, we consider it a successful extraction of the model’s choice.
If the match strategy fails, we then try to extract the choice by semantic
similarity. We enhance semantic similarity by calculating the semantic
similarity between the output answer and multiple options. We choose the
option with the highest similarity as the final selection.

Zero-shot Results. We evaluated the performance of several common multimodal
models on our MoVQA dataset under zero-shot setting. The results are
summarized in Table 5. It can be observed that under the zero-shot setting,
the results are not very satisfactory, and the performance also deteriorates
as the length of the video segments increases. Moreover, we utilize
instruction-tuning of the LLMs on the prediction tokens, leveraging its
original autoregressive training objective. The pretrained model is fine-tuned
using carefully selected, high-quality video-text pairs. Moreover, as shown in
Table 6, we report the performance of different types on our MoVQA. Current
multimodal LLMs models have their own strengths and weaknesses in different
types of QA, which are closely related to the data they were pretrained on.
And in some types, such as temporal perception, their performance is poor, to
the point where it is not much different from random guessing.

Instruction Tuning. We also employ predetermined prompts based on the
following template like [28]. The questions come from the annotation of our
MoVQA. Throughout the training process, the weights of both the video encoder
and LLMs remain fixed, and the model maximizes the likelihood of predicting
tokens that represent the answer by adapting the linear layer. We can see that
in Table 5 after instruction tuning, the model’s performance has improved to
some extent. However, there is still a significant gap between model’s
performance and human performance.

###  5.4 Qualitative Analysis

Setting | Modality input | Single-Scene | Multi-Scene | Full-Scene
---|---|---|---|---
Zero-shot | Q | 22.4 | 20.9 | 29.0
V + Q | 23.0 | 21.2 | 23.5
S + Q | 28.5 | 25.8 | 23.5
V + S + Q | 28.1 | 25.0 | 25.3
Finetuning | Q | 36.0 | 33.7 | 32.2
V + Q | 36.8 | 34.4 | 33.3
S + Q | 41.3 | 37.7 | 36.9
V + S + Q | 42.2 | 39.0 | 38.9

Table 8: Ablation study on modality input. The modality input Q, V, and S
indicate question, video, and subtitle respectively.

Video Sample Strategy. We analyze the effect of video frame sampling on long
video using the SEVILA model with different number of sampled frame and
keyframes. As shown in Table 7, it suggests that more video tokens may not
work well in the zero-shot setting, while more video tokens works better in
the finetuning setting. For the zero-shot setting, more video tokens may not
work well because the base model BLIP-2 in SEVILA is pretrained with image
data (single-frame), leading to an enlarged zero-shot performance gap when
more video frames are input. In the finetuning setting, we observe a monotonic
relationship between performance and the number of frames. However, the
improvement reaches a saturation point at 64 frames.

Modality Dependency. To investigate whether MoVQA requires multimodal
dependency, we adopt FrozenBiLM [53] model with different modality inputs for
ablation. As shown in Table 8, we can find that the model performs best with
only subtitle input in the zero-shot setting and performs best with both
subtitle and video input in the finetuning setting. The results from the
finetuning setting suggest that our dataset requires multimodal processing
abilities. For the zero-shot setting, our movie video inputs do not provide
assistance. This may be due to the domain gap between the pretraining dataset
WebVid [3] of FrozenBiLM and our movie dataset.

Model variance | Single-Scene | Multi-Scene | Full-Scene
---|---|---|---
SEVILA [57] | 49.2 | 45.6 | 40.7
DPM-C | 49.8 | 46.7 | 40.1
DPM-K | 50.1 | 48.6 | 42.6
DPM | 50.3 | 49.0 | 42.7

Table 9: Ablation study of our DPM method. The variants DPM-K and DPM-C
indicate that only keyframe branch or context branch is used for DPM.

Setting | Distractor option Type | Single-Scene | Multi-Scene | Full-Scene
---|---|---|---|---
Zero-shot | QA-based | 28.7 | 27.8 | 36.7
Sentence-based | 28.6 | 26.9 | 27.0
Human | 28.1 | 25.0 | 25.3
Finetuning | QA-based | 79.2 | 80.9 | 79.4
Sentence-based | 55.3 | 51.2 | 52.3
Human | 41.5 | 36.9 | 38.3

Table 10: Results on MoVQA dataset with different distractor option type.

Variants of DPM. To analyze the effectiveness of different components in our
DPM, we conducted ablation experiments on the keyframe branch and the context
branch for comparison. According to the results in Table 9, the DPM-K with
only relying on the keyframe branch demonstrates the second-best performance.
Compared to SEVILA [57], which relies solely on visual information to locate
keyframes, DPM-K, combining visual and subtitle information, can better
identify keyframes relevant to the question. Additionally, DPM-C can
effectively improve performance by modeling contextual information. Overall,
the DPM that combines both branches demonstrates the best performance.

Human-Written vs ChatGPT. We initially attempted to generate distractor
options using ChatGPT. We conducted preliminary experiments using FrozenBiLM
model [53] with its original hyperparameters. As shown in Fig.5, we tried two
types of prompts for generating distractor options using ChatGPT, one that
generates distractor options that cannot answer the original question and are
different from the correct answer (QA-based), and another that generates
distractor options that are different from the correct answer (Sentence-
based). From Table 10, we can see that both ChatGPT-prompt-generated QA
datasets have very high QA accuracy, and our manual distractor options lead to
a much lower accuracy. This indicates there may exist significant QA shortcuts
for ChatGPT-generated distractor options. We found that many distractor
options generated by ChatGPT are consistently either too short or too long
compared to the right answer, and sometimes use different words but convey a
similar semantic meaning as the correct answers. Generating reliable multiple
choice QA based on ChatGPT may be a good avenue for future work.

Figure 5: Two types of prompts for generating distractor options.

##  6 Conclusions

We present our MoVQA, a long-form movie question-answering dataset, and
benchmark it to assess the versatile cognitive capabilities of multimodal
systems. The assessment relies on multi-level temporal lengths (single-scene,
multi-scene and full-scene), considering both clue length and video length.
Compared to the second longest dataset, which has about a median clue length
of 100 seconds and a video length of 3 minutes, our MoVQA surpasses it in both
clue length (230 seconds) and video length (8 minutes). Additionally, to take
a step towards human-level understanding, QAs in our MoVQA are designed from
the perspective of the interconnected abilities required by moviegoers to
understand movie content. We also benchmark various methods on MoVQA and find
that even the most advanced methods achieve unsatisfactory results. Our
established method has shown some improvements, but there is still ample scope
for enhancement on our challenging MoVQA dataset. We believe that our MoVQA
will have a significant impact on the advancement and assessment of
forthcoming long-form video understanding models.

Limitations. There are a thousand Hamlets in a thousand people’s eyes. Despite
conducting multiple rounds of cross-validation, the understanding of the
essence of a movie may vary among individuals. As a result, for some
challenging abstract questions, the answers may not be entirely accurate.
Further, there are inevitable distribution biases of distractor options in our
MoVQA. Finally, it is important to acknowledge that human curation is an
imperfect process. Despite implementing multiple rounds of checks to minimize
false positives, it is inevitable that the collected MoVQA may include some
mislabeled or improperly formatted QA pairs. In our future work, we will
further expand the scale of MoVQA by including more movies in multiple
languages with detailed QA annotations. We will also explore long-form video
modeling techniques for enhanced movie understanding.

## References

  * mov [2021] https://www.youtube.com/c/movieclips.  2021\.
  * Argaw et al. [2023] Dawit Mureja Argaw, Joon-Young Lee, Markus Woodson, In So Kweon, and Fabian Caba Heilbron.  Long-range multimodal pretraining for movie understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13392–13403, 2023.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1728–1738, 2021.
  * Castro et al. [2020] Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, and Rada Mihalcea.  Lifeqa: A real-life dataset for video question answering.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4352–4358, 2020.
  * Castro et al. [2022] Santiago Castro, Naihao Deng, Pingxuan Huang, Mihai Burzo, and Rada Mihalcea.  Wildqa: In-the-wild video question answering.  _arXiv preprint arXiv:2209.06650_ , 2022.
  * Cheng and Bertasius [2022] Feng Cheng and Gedas Bertasius.  Tallformer: Temporal action localization with a long-memory transformer.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV_ , pages 503–521. Springer, 2022.
  * Choi et al. [2021] Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Minsu Lee, and Byoung-Tak Zhang.  Dramaqa: Character-centered video story understanding with hierarchical qa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1166–1174, 2021.
  * Gao et al. [2023] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou.  Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14773–14783, 2023.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18995–19012, 2022.
  * Han et al. [2022] Tengda Han, Weidi Xie, and Andrew Zisserman.  Temporal alignment networks for long-term video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2906–2916, 2022.
  * Heilbron et al. [2015] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _2015 IEEE conference on computer vision and pattern recognition (CVPR)_ , pages 961–970. IEEE, 2015.
  * Huang et al. [2020a] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin.  Movienet: A holistic dataset for movie understanding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 709–727. Springer, 2020a.
  * Huang et al. [2018] Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato.  Predicting gaze in egocentric video by learning task-dependent attention transition.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 754–769, 2018.
  * Huang et al. [2020b] Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Improving action segmentation via graph-based temporal reasoning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14024–14034, 2020b.
  * Huang et al. [2023] Yifei Huang, Lijin Yang, and Yoichi Sato.  Weakly supervised temporal sentence grounding with uncertainty-guided self-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18908–18918, 2023.
  * Jang et al. [2017] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2758–2766, 2017.
  * Kim et al. [2020] Hyounghun Kim, Zineng Tang, and Mohit Bansal.  Dense-caption matching and frame-selection gating for temporal localization in videoqa.  _arXiv preprint arXiv:2005.06409_ , 2020.
  * Korbar et al. [2019] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Scsampler: Sampling salient clips from video for efficient action recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6232–6242, 2019.
  * Le et al. [2020] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.  Hierarchical conditional relation networks for video question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9972–9981, 2020.
  * Lei et al. [2018] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg.  Tvqa: Localized, compositional video question answering.  _arXiv preprint arXiv:1809.01696_ , 2018.
  * Lei et al. [2019] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  Tvqa+: Spatio-temporal grounding for video question answering.  _arXiv preprint arXiv:1904.11574_ , 2019.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023a.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023b.
  * Li et al. [2023c] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023c.
  * Li et al. [2020] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  _arXiv preprint arXiv:2005.00200_ , 2020.
  * Lin et al. [2022] Yan-Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius.  Eclipse: Efficient long-range video retrieval using sight and sound.  In _European Conference on Computer Vision_ , pages 413–430. Springer, 2022.
  * Liu et al. [2022] Xiaolong Liu, Song Bai, and Xiang Bai.  An empirical study of end-to-end temporal action detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20010–20019, 2022.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * Maharaj et al. [2017] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal.  A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6884–6893, 2017.
  * Mangalam et al. [2023] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.  Egoschema: A diagnostic benchmark for very long-form video language understanding.  _arXiv preprint arXiv:2308.09126_ , 2023.
  * Mullapudi et al. [2019] Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian.  Online model distillation for efficient video inference.  In _Proceedings of the IEEE/CVF International conference on computer vision_ , pages 3573–3582, 2019.
  * Mun et al. [2017] Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han.  Marioqa: Answering questions by watching gameplay videos.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2867–2875, 2017.
  * Park et al. [2021] Jungin Park, Jiyoung Lee, and Kwanghoon Sohn.  Bridge to answer: Structure-aware graph interaction network for video question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 15526–15535, 2021.
  * Pei et al. [2017] Wenjie Pei, Tadas Baltrusaitis, David MJ Tax, and Louis-Philippe Morency.  Temporal attention-gated model for robust sequence classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6730–6739, 2017.
  * Potapov et al. [2014] Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid.  Category-specific video summarization.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 540–555. Springer, 2014.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Shou et al. [2021] Mike Zheng Shou, Stan Weixian Lei, Weiyao Wang, Deepti Ghadiyaram, and Matt Feiszli.  Generic event boundary detection: A benchmark for event segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8075–8084, 2021.
  * Shou et al. [2016] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1049–1058, 2016.
  * Soldan et al. [2022] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  Mad: A scalable dataset for language grounding in videos from movie audio descriptions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5026–5035, 2022.
  * Sun et al. [2022] Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu.  Long-form video-language pre-training with multimodal temporal contrastive learning.  _Advances in neural information processing systems_ , 35:38032–38045, 2022.
  * Tapaswi et al. [2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  Movieqa: Understanding stories in movies through question-answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4631–4640, 2016.
  * Vicol et al. [2018] Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler.  Moviegraphs: Towards understanding human-centric situations from videos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8581–8590, 2018.
  * Wu et al. [2021] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.  Star: A benchmark for situated reasoning in real-world videos.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Wu and Krahenbuhl [2021] Chao-Yuan Wu and Philipp Krahenbuhl.  Towards long-form video understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1884–1894, 2021.
  * Wu et al. [2019] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.  Long-term feature banks for detailed video understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 284–293, 2019.
  * Wu et al. [2022] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13587–13597, 2022.
  * Xiao et al. [2022] Fanyi Xiao, Kaustav Kundu, Joseph Tighe, and Davide Modolo.  Hierarchical self-supervised representation learning for movie understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9727–9736, 2022.
  * Xiao et al. [2021a] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9777–9786, 2021a.
  * Xiao et al. [2021b] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9777–9786, 2021b.
  * Xiong et al. [2019] Yu Xiong, Qingqiu Huang, Lingfeng Guo, Hang Zhou, Bolei Zhou, and Dahua Lin.  A graph-based framework to bridge movies and synopses.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4592–4601, 2019.
  * Xu et al. [2017] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 1645–1653, 2017.
  * Yang et al. [2021a] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1686–1697, 2021a.
  * Yang et al. [2022] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  _arXiv preprint arXiv:2206.08155_ , 2022.
  * Yang et al. [2021b] Lijin Yang, Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Stacked temporal attention: Improving first-person action recognition by emphasizing discriminative clips.  _arXiv preprint arXiv:2112.01038_ , 2021b.
  * Ye et al. [2023a] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality, 2023a.
  * Ye et al. [2023b] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_ , 2023b.
  * Yu et al. [2023] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal.  Self-chained image-language model for video localization and question answering.  _arXiv preprint arXiv:2305.06988_ , 2023.
  * Yu et al. [2019a] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 9127–9134, 2019a.
  * Yu et al. [2019b] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian.  Deep modular co-attention networks for visual question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6281–6290, 2019b.
  * Yue et al. [2023] Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin.  Movie101: A new movie understanding benchmark.  _arXiv preprint arXiv:2305.12140_ , 2023.
  * Zhang et al. [2022] Chen-Lin Zhang, Jianxin Wu, and Yin Li.  Actionformer: Localizing moments of actions with transformers.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV_ , pages 492–510. Springer, 2022.
  * Zhang et al. [2016] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_ , pages 766–782. Springer, 2016.

## Appendix

In the supplementary material, we describe the following parts:

(1) We give more data statistics about our MoVQA, and show the drawbacks of
the distractor options generated by the ChatGPT.

(2) Examples of the 6 types of QA designed for the MoVQA dataset are
showcased, along with the results predicted by our model.

(3) A list of the 100 movies utilized in the MoVQA dataset is provided,
including their respective names and release years.

##  7 More data statistics of our MoVQA.

Fig.6 shows the word cloud of the nouns/verbs in the questions and answers of
our MoVQA. We can find that both the questions and answers cover various
aspects and exhibit a high level of diversity in terms of word cloud. This is
also attributed to the fact that our question-answering system itself
originates from different types of questions.

Fig.7 shows distractor options generated from the ChatGPT. Compared to the
correct option, on the one hand, these distractor options tend to have longer
sentence lengths, which is due to the redundancy of the ChatGPT compared to
humans. On the other hand, these distractor options exhibit very similar
wording in terms of sentence structure. These differences from the correct
option impact the overall difficulty of multiple-choice questions. Therefore,
we ultimately chose human annotators to design and generate the distractor
options.

Figure 6: Word cloud of the nouns/verbs in the questions and answers of MoVQA.
(a): Question, (b): Answer. Figure 7: Distractor options generated from
ChatGPT. Noted that the distractor options are more likely to be longer and
express similar sentence structures. Best viewed in color.

##  8 More examples

Fig. 8-13 show the 6 types of QA designed in the MoVQA, which are Information
Synopsis, Temporal Perception, Spatial Perception, Causal Reasoning,
Hypothetical Reasoning and External Knowledge. We also show the ground truth
answers and the results predicted by our model.

Figure 8: A Information Synopsis QA example from MoVQA dataset, as well as the
prediction from our DPM. The ground truth answer is highlighted in green, and
the model prediction is indicated by ✓. Best viewed in color. Figure 9: A
Temporal Perception QA example from MoVQA dataset, as well as the prediction
from our DPM. The ground truth answer is highlighted in green, and the model
prediction is indicated by ✓. Best viewed in color. Figure 10: A Spatial
Perception QA example from MoVQA dataset, as well as the prediction from our
DPM. The ground truth answer is highlighted in green, and the model prediction
is indicated by ✓. Best viewed in color. Figure 11: A Causal Reasoning QA
example from MoVQA dataset, as well as the prediction from our DPM. The ground
truth answer is highlighted in green, and the model prediction is indicated by
✓. Best viewed in color. Figure 12: A Hypothetical Reasoning QA example from
MoVQA dataset, as well as the prediction from our DPM. The ground truth answer
is highlighted in green, and the model prediction is indicated by ✓. Best
viewed in color. Figure 13: A External Knowledge QA example from MoVQA
dataset, as well as the prediction from our DPM. The ground truth answer is
highlighted in green, and the model prediction is indicated by ✓. Best viewed
in color.

##  9 The 100 movies used in our MoVQA.

Table 11 shows the names and release years of the 100 movies used in the MoVQA
dataset. We can see that movies in our MoVQA cover a wide range of genres,
years and countries. We order them by the IMDb score.

ID | Movie Name | Year
---|---|---
1 | The Shawshank Redemption | 1994
2 | Forrest Gump | 1994
3 | Farewell My Concubine | 1993
4 | Life Is Beautiful | 1997
5 | The Legend of 1900 | 1998
6 | Schindler’s List | 1993
7 | Spirited Away | 2001
8 | Wall-E | 2008
9 | Titanic | 1997
10 | Inception | 2010
11 | Three Idiots | 2009
12 | The Choir Boys | 2004
13 | Hachi: A Dog’s Tale | 2010
14 | My Neighbor Totoro | 1988
15 | The Godfather | 1972
16 | A Chinese Odyssey Part 2: Cinderella | 1994
17 | Gone with the Wind | 1939
18 | Cinema Paradiso | 1988
19 | Fight Club | 1999
20 | The Truman Show | 1998
21 | The Lord of the Rings (The Return of the King) | 2003
22 | Roman Holiday | 1953
23 | The Cove | 2009
24 | Lock, Stock and Two Smoking Barrels | 1998
25 | Intouchables | 2011
26 | 12 Angry Men | 1957
27 | A Chinese Odyssey Part 1- Pandora’s Box | 1994
28 | Flipped | 2010
29 | The Lives of Others | 2006
30 | Amelie | 2001
31 | V For Vendetta | 2005
32 | Infernal Affairs | 2002
33 | Scent of a Woman | 1992
34 | The Dark Knight | 2008
35 | The Lord of the Rings (The Two Towers) | 2002
36 | The Lord of the Rings (The Fellowship of the Ring) | 2001
37 | Silenced | 2011
38 | A Beautiful Mind | 2001
39 | The Godfather Part II | 1974
40 | Edward Scissorhands | 1990
41 | 7 Deadly Sins | 1995
42 | Life of Pi | 2012
43 | Braveheart | 1995
44 | The Pianist | 2002
45 | The Prestige | 2006
46 | Memories of Matsuko | 2006
47 | Pulp Fiction | 1994
48 | The Butterfly Effect | 2004
49 | The Curious Case of Benjamin Button | 2008
50 | The Matrix | 1999

ID | Movie Name | Year
---|---|---
51 | The Sixth Sense | 1999
52 | City of God | 2002
53 | Eat Drink Man Woman | 1994
54 | Good Will Hunting | 1997
55 | Grave of the Fireflies | 1988
56 | The Monkey King | 2012
57 | Almost A Love Story | 1996
58 | Chungking Express | 1994
59 | Legends of the Fall | 1995
60 | Pirates of the Caribbean | 2003
61 | Identity | 2003
62 | Kikujiro | 1999
63 | Shutter Island | 2010
64 | The Terminal | 2004
65 | Slumdog Millionaire | 2008
66 | Catch Me If You Can | 2002
67 | Hotel Rwanda | 2004
68 | The Godfather: Part III | 1990
69 | The Attorney | 2013
70 | Pride and Prejudice | 2005
71 | Once A Thief | 1991
72 | A Better Tomorrow | 1986
73 | A Perfect World | 1993
74 | Harry Potter and the Sorcerer’s Stone | 2001
75 | Memories of Murder | 2003
76 | The Last Emperor | 1987
77 | Kekexili: Mountain Patrol | 2004
78 | Lord of War | 2005
79 | Hope | 2013
80 | The Rock | 1996
81 | Taare Zameen Par | 2007
82 | E. T. The Extra-Terrestrial | 1982
83 | The Bourne Identity | 2002
84 | Face Off | 1997
85 | The Bourne Supremacy | 2004
86 | Days Of Being Wild | 1990
87 | The Grand Budapest Hotel | 2014
88 | The Man From Earth | 2007
89 | In the Mood for Love | 2000
90 | King of Comedy | 1999
91 | The Matrix Revolutions | 2003
92 | Triangle | 2009
93 | The King’s Speech | 2010
94 | Blood Diamond | 2006
95 | The Terror Live | 2013
96 | Black Hawk Down | 2001
97 | The Mission | 1999
98 | Echoes Of The Rainbow | 2009
99 | Fist of legend | 1994
100 | The Warlords | 2007

Table 11: 100 movies used in MoVQA dataset.