#  Video Generation Beyond a Single Clip

Hsin-Ping Huang1,2, Yu-Chuan Su1, Ming-Hsuan Yang1,2,3
1Google Research 2UC Merced 3Yonsei University

###### Abstract

We tackle the long video generation problem, i.e. generating videos beyond the
output length of video generation models. Due to the computation resource
constraints, video generation models can only generate video clips that are
relatively short compared with the length of real videos. Existing works apply
a sliding window approach to generate long videos at inference time, which is
often limited to generating recurrent events or homogeneous content. To
generate long videos covering diverse content and multiple events, we propose
to use additional guidance to control the video generation process. We further
present a two-stage approach to the problem, which allows us to utilize
existing video generation models to generate high-quality videos within a
small time window while modeling the video holistically based on the input
guidance. The proposed approach is complementary to existing efforts on video
generation, which focus on generating realistic video within a fixed time
window. Extensive experiments on challenging real-world videos validate the
benefit of the proposed method, which improves over state-of-the-art by up to
9.5% in objective metrics and is preferred by users more than 80% of time.

##  1 Introduction

Figure 1:  Long video generation. Video generation models can only generate a
relatively short video clip. Existing works [15] generate long videos using a
sliding window approach that extends the previously generated video clip,
which often leads to videos with repetitive patterns. Instead, we study the
problem of generating long videos using additional guidance, which helps to
generate videos covering multiple, non-recurrent events. We also propose a
two-stage approach for the long video generation problem, which is
complementary to existing efforts on video generation.

Video generation has recently attracted increasing attention. As a natural
extension of image generation, most existing works treat video generation as a
3D volume prediction problem and focus on generating realistic video clips.
While this paradigm has demonstrated impressive progress in a wide range of
video generation tasks such as frame prediction [39, 13, 42, 31, 29], class-
conditional generation, and unconditional video generation [46, 32, 41, 33,
6], the length of the video clip that can be generated is inevitably limited
by the computation resource. Due to the significant computation and memory
overhead incurred by state-of-the-art video generation models, they often
generate video clips that are much shorter than real videos.

In order to generate long videos that match the length of real videos,
existing works adopt a sliding window approach. Specifically, the model
generates one video clip at a time in temporal order while taking the
previously generated frames as input. The previously generated frames serve as
the condition for the model to ensure that the generated video is consistent
across video clips. This approach has been successfully applied to generate
long videos [15]. However, the results are far from satisfactory, particularly
in real data domains. The synthesized videos are often limited to homogeneous
videos of natural scenes and videos with recurrent human actions. In contrast,
many real videos contain dynamic scenes with multiple, non-recurrent events.

The existing sliding window approach is sub-optimal from two aspects. First,
it implicitly assumes that video generation is a Markov process, where a new
clip only depends on the previous clip or even the last frame of the previous
clip. This is not true in many videos, e.g. an object may leave and re-enter
the video which introduces a long-range dependency to the video. Second, it
assumes that the initial frames are a sufficient condition for generating the
video clip. However, it is well known that there may be multiple valid futures
given the initial condition for a video [55], and it is unlikely to infer the
correct future purely from the initial frames. To overcome these problems, it
is essential to use additional guidance to control the generation process of
long videos.

In this work, we propose to tackle the _long video generation_ problem. Given
a video clip generation model, a series of guidance, and a reference frame
representing the initial condition of the video, the goal is to generate a
long video that is beyond the capability of the clip generation model. In this
work, we choose to use object labels as the input guidance, where the guidance
describes the set of objects that will appear in the video clip. While other
types of guidance are also possible, we choose object labels because they can
be provided by users easily and naturally supports video manipulation such as
content insertion or removal. The long video generation problem aims to extend
the ability of existing video generation models to generate realistic video
covering diverse content and multiple events and is complimentary to existing
efforts that focus on generating high-quality videos within a fixed temporal
window.

To solve the long video generation problem, we propose a two-stage approach by
decomposing the problem into a keyframes generation problem followed by a
frame interpolation problem. We first predict all the keyframes jointly based
on the input guidance and reference frame. These keyframes represent the
starting frames of each video clip. We then generate the entire video by
predicting the intermediate frames between keyframes, using the video clip
generation model. See Fig. 1. The two-stage approach allows us to utilize
existing video generation models that are highly optimized and can generate
high-quality, realistic videos within a short temporal window. It also allows
us to model the full video jointly and capture long-range dependencies through
keyframe generation. Our evaluations show that the holistic keyframe modeling
help to maintain consistency throughout the video.

We conduct extensive quantitative and qualitative studies on both real and
synthetic data. In particular, we evaluate our method on the EPIC Kitchen
dataset, which is challenging for video generation due to the rapid motion and
complex scenes. Empirical results verify the advantage of the proposed long
video generation method and show that it outperforms state-of-the-art video
generation models by 9.5% on LPIPS.

Our main contributions are as follows. First, we study the long video
generation problem which aims to extend the capability of existing video
generation models to generate long videos with dynamic senses and non-
recurrent events. Second, we propose a two-stage approach for the long video
generation problem. Finally, we conduct extensive evaluations to validate the
efficacy of our proposed framework.

##  2 Related Work

Video synthesis. Video prediction, class-conditional video generation and
unconditional video generation have been widely studied as the sub-tasks of
video generation. GAN-based methods [46, 32, 41, 6, 33, 36, 40, 3] have
demonstrated early success to generate short video clips, while the generation
quality decreases significantly when applying to long videos. Diffusion models
[45, 52, 17] are recently introduced for video generation. However, the slow
sampling speed of diffusion models limits their ability to generate long
videos. Auto-regressive models are first developed to synthesize raw pixels in
videos [22, 47, 1]. Thanks to the development of vector quantization [12, 43]
and transformer [9] models, auto-regressive methods are adapted to predict
discrete tokens in the latent space [51, 30, 15] with impressive visual
quality. In this work, we build upon recent advances of VQVAE and non-
autoregressive transformer models [4, 53].

Figure 2: Approach overview. The proposed method consists of three stages: 1)
generating the series of layouts from the series of object label sets and a
reference first frame image, 2) synthesizing the keyframes of the video from
the predicted layout sequence and the reference image, and 3) interpolating
the synthesized keyframe sequence to obtain the complete video.

Despite the recent success in video generation, these models mostly focus on
generating short clips (_e.g_. 16-frame videos) and are limited to
synthesizing videos in specific domains [44, 15], such as human actions [38,
34], sky timelapse [50], robotics videos [10]. Most recently, a few text-to-
video models [48, 49, 19, 35, 16] are developed to generate videos given
natural language inputs. However, these models are limited to videos with
single scenes without a meaningful storyline. In contrast, our work aims to
generate videos with diverse content and novel events.

There are few works exploring complex video generation conditioned on input
guidance at multiple timesteps while limited to synthetic environments [2, 54,
15]. Our work focuses on the real-world dataset.

Story visualization and image manipulation. Story visualization [26, 37, 28,
27] focuses on synthesizing a sequence of images that visualize a story of
multiple sentences. Each sentence in the story corresponds to one synthesized
image. GeNeVA [11, 14, 56, 7] is a conditional text-to-image generation task
developed on CoDraw [24] dataset. It studies the problem of constructing a
scene iteratively based on a sequence of descriptions. However, these two
lines of work are limited to experiments on synthetic and cartoon data. These
approaches focus on generating a few frames of a visual story instead of
videos. In addition, the inputs to these methods are natural language
descriptions which might exist in ambiguity and do not clearly describe the
objects in the image, unlike object labels. In contrast, we focus on
experiments on real-world data and generating videos given a series of object
labels as inputs.

##  3 Approach

In this section, we introduce the proposed method for long video generation.
We first give an overview of the approach and then describe the three main
components.

###  3.1 Overview

We define our problem as follows. The model takes 1) a series of $N$ sets of
object labels $\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N}\\}$, and
2) a reference frame, $I_{0}$ as input. The reference frame is the starting
frame of the video and is the same initial condition as existing video
prediction task. The object label sets serve as the guidance for video
generation. Each object label set $\mathbf{x}_{n}$ contains $k^{n}$ object
labels which indicates the objects that will appear in the video. Note that
$k^{n}$ may vary across different timestep $n$. Our goal is to synthesize a
video $\mathbf{V}$ that covers the content provided in the guidance while
maintaining the same style as the initial frame $\mathbf{I}_{0}$.

We present the overview of the long video generation framework in Fig. 2. To
tackle the long video generation problem, our core idea is to model the entire
video jointly through keyframe generation and utilize existing models to
generate high-quality videos within a short temporal window. We achieve these
by 1) generating a series of $N$ keyframes in the video given the object label
sets, each keyframe presents the starting frames of each video clip, and 2)
predicting the intermediate frames between adjacent keyframes to obtain the
complete video. To reduce the difficulty of keyframe generation, we use
layouts as the intermediate representation to bridge the gap of generating 2D
images from symbolic object labels [18, 21]. We introduce an additional stage
of layout generation that creates an explicit 2D structure of the scene as an
intermediate input to constrain the keyframe generation. It brings an
additional benefit that the users can manipulate the generated videos by
editing the layouts such as content insertion or removal. The proposed
approach consists of three steps: layout generation, keyframe generation, and
frame interpolation.

###  3.2 Layout Generation

We first generate a series of layouts from the object label sets to explicitly
constrain the keyframe generation. Given a series of $N$ object label sets
$\\{\mathbf{x}_{n}\\}$, and a reference first frame $\mathbf{I}_{0}$, we
synthesize a series of layouts
$\\{\mathbf{L}_{1},\mathbf{L}_{2},\cdots,\mathbf{L}_{N}\\}$, which represents
the layouts of the $N$ keyframes in the video. Since the reference first frame
$\mathbf{I}_{0}$ is given, we assume $\mathbf{L}_{0}$ is known as well and can
be used as a reference layout to synthesize
$\\{\mathbf{L}_{1},\mathbf{L}_{2},\cdots,\mathbf{L}_{N}\\}$. We define the
layout $\mathbf{L}$ as a set of bounding boxes with variable length $k^{n}$,
_i.e_. $\\{\mathbf{b}_{1},\mathbf{b}_{2},\cdots,\mathbf{b}_{k^{n}}\\}$. The
bounding box contains the attribute of its object label, x-coordinate of the
center, y-coordinate of the center, width and height, _i.e_.
$\mathbf{b}=\\{\mathbf{c},\mathbf{x},\mathbf{y},\mathbf{w},\mathbf{h}\\}$.

Our layout generator is motivated by BLT [25]. We first apply tokenization to
encode the object label sets $\\{\mathbf{x}_{n}\\}$ into discrete tokens. We
then learn a transformer model that predicts the layout tokens given the
object label sets as input. The ground truth tokens are obtained from
tokenizing the layouts $\\{\mathbf{L}_{n}\\}$. Though the object label sets
and layouts have variable length $k^{n}$, we pad the token sequences to fixed
length in practice. Specifically, the input series of $N$ object label sets
with $k^{n}$ labels in each set can be flattened into a sequence of tokens.
Similarly, the layout sequences can be flattened into token sequences as well,
where the values of the bounding box attributes are simply discretized by
uniform quantization [25].

The tokens of the reference layout $\mathbf{L}_{0}$ and the object labels
$\mathbf{c}$ are known and the other attributes of the bounding boxes
$\\{\mathbf{x},\mathbf{y},\mathbf{w},\mathbf{h}\\}$ are replaced with a [MASK]
token. The transformer model is trained to predict the missing tokens for the
layout series $\hat{\mathbf{L}}_{1}-\hat{\mathbf{L}}_{N}$ [25]. Different from
BLT [25] which takes a single object label set as input and predicts a single
layout, our model takes the reference layout and a series of object label sets
as input, and predicts multiple layouts at the same time. Thus, our model
preserves the temporal consistency of the layout sequences while BLT fails to.

###  3.3 Keyframe Generation

Next, we generate a sequence of keyframes from the predicted layout sequences.
Given a reference first frame $\mathbf{I}_{0}$ and a series of layouts
$\\{\hat{\mathbf{L}}_{1},\hat{\mathbf{L}}_{2},\cdots,\hat{\mathbf{L}}_{N}\\}$
synthesized in the previous stage, we aim to generate a sequence of keyframes
$\mathbf{I}_{1}$-$\mathbf{I}_{N}$. Each pair of adjacent keyframes will be
used as the start and end frame to generate one video clip out of the $N$
clips in the entire video. Following [4], we convert the keyframe generation
into tokenization and sequence prediction. We use a VQVAE [12] model to encode
the raw image pixels $\mathbf{I}$ into discrete visual tokens $\mathbf{e}$,
and a bidirectional transformer model is used to predict the masked image
tokens _i.e_. target keyframes. Finally, the decoder of VQVAE is used to map
the visual token $\hat{\mathbf{e}}$ into raw images $\hat{\mathbf{I}}$.

Specifically, the input series of layouts
$\\{\mathbf{L}_{0},\hat{\mathbf{L}}_{1},\cdots,\hat{\mathbf{L}}_{N}\\}$ are
flattened into a sequence of discrete tokens. The reference keyframe
$\mathbf{I}_{0}$ and the target keyframe sequence
$\mathbf{I}_{1}$-$\mathbf{I}_{N}$ are transformed into discrete visual tokens
and flattened into sequences as well, _i.e_.
$\\{\mathbf{e}_{0},\mathbf{e}_{1},\cdots,\mathbf{e}_{N}\\}$. At training time,
the tokens of input layouts and the visual tokens of the target keyframes are
concatenated. The tokens are randomly masked out and the transformer model is
trained to predict the missing token. Specifically, given a sequence
$\mathbf{s}=\\{\mathbf{e}_{0},\mathbf{L}_{0},\hat{\mathbf{L}}_{1},\cdots,\hat{\mathbf{L}}_{N},\mathbf{e}_{1},\cdots,\mathbf{e}_{N}\\}$
in dataset $\mathcal{D}$, we replace the tokens in the sequence with the
[MASK] token and obtains the masked sequence $\mathbf{s}_{M}$. We minimize the
negative log-likelihood of predicting the masked tokens $\mathbf{s}_{t},t\in
M$.

$\mathcal{L}={-}\mathop{\mathbb{E}}\limits_{s\in\mathcal{D}}\sum_{t\in M}\log
p(s_{t}|s_{M})$ (1)

At test time, the tokens of layout sequence and the first frame
$\\{\mathbf{e}_{0},\mathbf{L}_{0},\hat{\mathbf{L}}_{1},\cdots,\hat{\mathbf{L}}_{N}\\}$
are given, and the model predicts the tokens of the following keyframes
$\hat{\mathbf{e}}_{1}-\hat{\mathbf{e}}_{N}$. Finally, the decoder of VQGAN is
used to reconstruct the target keyframes
$\hat{\mathbf{I}}_{1}-\hat{\mathbf{I}}_{N}$. Fig. 3 shows our keyframe
generation approach.

Compare with [4], our keyframe generation method generates all the frames
jointly. This provides a more holistic model for the entire video. As we show
in the experiment, this helps to improve the consistency across keyframes and
improves the coherency of the video.

Figure 3: Keyframe generation. The reference frame and predicted layouts are
first converted into discrete tokens. A transformer takes the tokens as input
and predicts the tokens of the keyframes, which are then decoded into the
final keyframes. Note that the transformer generates all keyframes jointly,
which provides a more holistic model for the video.

###  3.4 Frame Interpolation

Finally, given the reference first frame $\mathbf{I}_{0}$ and a sequence of
generated keyframes $\mathbf{\hat{I}}_{1}\ldots\mathbf{\hat{I}}_{N}$, we apply
an existing video generation model to generate the complete video. In
particular, we use MAGVIT [53] to generate intermediate frames following the
video interpolation task of the original model. Specifically, the model takes
the initial and final frame of the video as input. It first converts the input
frames into discrete video tokens using 3D-VQVAE. A transformer model is then
used to predict the tokens of intermediate frames. Finally, the interpolated
video tokens are mapped back to the raw videos by the 3D decoder.

We re-train the video generation model on our data. During inference time,
given two consecutive keyframes $\mathbf{\hat{I}}_{n-1}$ and
$\mathbf{\hat{I}}_{n}$, the model predicts the video token sequences that
connect between the two keyframes, _i.e_. $\mathbf{\hat{z}}_{n}$. We predict
$N$ clips of video tokens
$\mathbf{\hat{z}}_{1},\mathbf{\hat{z}}_{2},\cdots,\mathbf{\hat{z}}_{N}$.
Finally, the video token sequences are concatenated and then mapped to the raw
video pixels $\mathbf{V}$ by the 3D decoder.

##  4 Experiments

We validate our approach on both real and synthetic data. We first evaluate
the video generation results on challenging real-world videos. Next, we study
the keyframe generation results on both real and synthetic data.

Dataset. We validate our method on two datasets.

EPIC Kitchen [8] is a real video dataset consisting of egocentric videos of
kitchen activities. It contains 700 videos with a total length of 100 hours.
Compared with other commonly used datasets for video generation research, e.g.
UCF [38], Kinetics [23], and BAIR [10], the content of EPIC Kitchen videos are
more dynamic. The objects move in and out of the camera field-of-view
frequently, and the scene and camera viewpoint may change rapidly. To
synthesize such video, the video generation model needs to generate multiple,
non-recurrent events within a short time window. This aligns with our goal for
long video generation, and EPIC Kitchen serves as an ideal test bed for the
problem and approach. Therefore, we choose EPIC Kitchen over other datasets
for our experiments.

We preprocess the EPIC Kitchen videos as follows. We follow the original train
and test split and cut the videos into 64-frame sequences. We first re-sample
the sequences with double the frame rate, so that each 64-frame sequence
covers a 5-second video with the five keyframes sampled equidistantly. This
leads to 276k sequences for training and 3k sequences of non-overlapping
frames for test. We use MaskFormer [5] to extract the semantic map for each
frame and convert them into the object labels and bounding boxes, which serve
as the input guidance and ground truth layout. Please refer to the
supplementary material for details.

CoDraw [24] is a synthetic dataset consisting of virtual scenes made by clip
objects. It contains 10k scenes consisting of 58 different objects. Each scene
comes with a sequence of images showing the step-by-step construction process
of the scene. While it is not a video dataset, it contains diverse object
classes and provides complete annotation for the object labels and layouts,
which is ideal for controlled experiments. We use the sequence of images as
video keyframes and evaluate keyframe and layout generation on CoDraw. To
analyze the temporal consistency between the generated keyframes, we extend
the original CoDraw data by creating 6 different appearances for each clip
object class and re-render the data using the original layouts. This resulting
dataset consists of 67k training sequences and 4k test sequences.

Table 1: Quantitative results for video generation. We report the metrics on the EPIC Kitchen dataset [8]. Image metrics are averaged across all video frames. Our method consistently outperforms state-of-the-art video generation methods.  Methods |  FVD $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---|---
MAGVIT (frame pred.) [53] | 421.7 | 11.316 | 0.175 | 0.482
MAGVIT (class cond.) [53] | 400.9 | 11.398 | 0.178 | 0.476
Ours | 380.8 | 12.037 | 0.206 | 0.431
Ours (GT Layouts) | 363.5 | 13.566 | 0.287 | 0.350
Ours (GT Keyframes) | 314.7 | 15.591 | 0.386 | 0.253
Figure 4: Per-frame results for video generation. X-axis shows the frame
number, ranging from 0 to 64. Y-axis shows the LPIPS of a specific frame
averaging over all test videos. The performance of MAGVIT drops rapidly over
time, while our approach can slow down the quality degradation. Figures are
best viewed in color.

Evaluation metrics. We use the following metrics for performance evaluation:

  * $\bullet$

Frèchet Video Distance (FVD)—assesses the quality of generated videos.
Specifically, it measures whether the distribution of generated videos is
close to that of real videos in the feature space. Following the original
paper, we use I3D model trained on Kinetics-400 for video features.

  * $\bullet$

Frèchet Inception Distance (FID)—assesses the quality of generated images,
similar to FVD. We use inception V3 for image features. FID is used to
evaluate the keyframe quality.

  * $\bullet$

Learned Perceptual Image Patch Similarity (LPIPS)—assesses the perceptual
similarity between the generated video frames and the ground truth video
frames. We compute these metrics for every frame and report the average.

  * $\bullet$

PSNR, SSIM—assess the similarity between the generated frames and ground truth
frame, similar to LPIPS.

Implementation details. In our experiments, each video sequence contains 64
frames, and the videos are generated at 128$\times$128\. We sample one
keyframe every 16 frames in the 64-frame clips so that the keyframe sequence
contains $N=4$ synthesized keyframes at 256$\times$256\. More implementation
details are in the supplementary materials.

Figure 5: Visual results for video generation. MAGVIT (frame pred./class
cond.) generates video with homogeneous contents, and the quality degrades for
long sequences. Our method generates video with scene change (floor to table),
object deletion (utensils on left hand) and object insertion (knife on right
hand), showing the ability of our model to generate videos with multiple
events. We further show that Our (GT Layouts) generates videos close to the
upper bound results of Our (GT Keyframes). On the other hand, Ours generates
videos that match the content but with different layouts. The results validate
the ability of our model to generate videos that satisfy different levels of
the input guidance _i.e_. object label sets or (more constrained) layouts.
Figure 6:  Video manipulation through layouts. Users can generate different
videos by sampling different layouts, as shown in the first two rows. Users
can even manipulate the videos by editing the layouts. The yellow cross in the
third row shows object deletion, and the yellow arrow in the last row shows
object movement.

###  4.1 Video Generation

First, we evaluate the video generation results on the EPIC Kitchen dataset.
The goal is to verify the effectiveness of additional guidance and holistic
video modeling in long video generation.

Baselines. We compare with the following state-of-the-art video generation
methods

  * $\bullet$

MAGVIT (frame prediction) [53]: given the reference frame as input, we apply
MAGVIT to generate a 16-frame clip. We then take the last predicted frame as
input to iteratively generate the entire video. This is the standard sliding
window approach for long video generation.

  * $\bullet$

MAGVIT (class conditional) [53]: we condition the MAGVIT model on both the
reference frame and the object label. This extends the sliding window approach
to take additional guidance similar to our method.

To understand how the quality of layout and keyframe generation affects the
video generation results, we also compare with two variants of our method that
take the ground truth layouts and keyframes as inputs respectively.

Quantitative results. The results are in Tab. 1. Our method consistently
outperforms the baselines in terms of video quality, and the generated videos
are closer to the ground truth videos. The results verify the effectiveness of
the proposed approach. Note that MAGVIT (class conditional) performs better
than MAGVIT (frame prediction), which shows the benefit of additional
guidance. Our approach further improves MAGVIT (class conditional), which
suggests that both the additional guidance and the holistic video modeling
provided by our approach are helpful for long video generation.

On the other hand, Ours (GT Layouts) and Ours (GT Keyframes) significantly
improve the output video quality. The results suggest that long video
generation has significant room for improvement given the very same video
generation model, and improving the video generation model alone may not be
sufficient for solving the video generation problem. It verifies the
importance of the long video generation problem. Note that our multi-stage
approach allows users manually improve the intermediate representations, e.g.
provide more detailed layouts, which allows further improvement for the
generated videos in an interactive generation setting.

Fig. 4 shows the per-frame LPIPS score. The generated image quality degraded
rapidly in MAGVIT, especially when we try to generate videos beyond the length
of the training clip (i.e. 16 frames). In contrast, our approach experiences a
slower quality degradation, which further verifies its benefit in generating
video beyond the training clip.

Qualitative results. Fig. 5 show the qualitative results. MAGVIT tends to
predict a relatively static video, which is consistent with the observations
in prior works. On the other hand, our method generates video with scene
change (wall to table), object deletion (plate at the bottom) and object
insertion (left/right hand), showing the ability of our model to generate
videos with multiple events. Compared with the upper bound results of Ours (GT
Keyframes), our method generates videos that match the content but with
different layouts or locations.

Fig. 6 shows that our method allows users to generate different videos by
sampling different layouts from the model. It also allows video manipulation
through layout, where the user may remove an object, change the size and
position of the objects, _etc_.

User study. We also conduct a user study to augment the quantitative
evaluation. In the study, we present two videos generated by different methods
together with the ground truth video and ask the raters 1) which video has the
better visual quality, and 2) which video better reproduces the content of the
ground truth video. We conduct the study with 40 videos and 11 participants.
The results are in Tab. 2, which is consistent with the quantitative results
and further validates the superior performance of our method

Table 2: User study. We report the percentage of raters that consider our method generates better video quality and better reproduces the ground truth videos respectively.  Methods | Quality | Reproduction
---|---|---
Ours _vs_. MAGVIT (frame pred.) | 76.3% | 82.1%
Ours _vs_. MAGVIT (class cond.) | 68.4% | 66.7%
Table 3: Quantitative results for keyframe generation. The metrics are averaged across keyframes. Please refer to the supplementary material for additional metrics.  | CoDraw | EPIC Kitchen
---|---|---
Methods |  FID $\downarrow$ |  LPIPS $\downarrow$ |  FID $\downarrow$ |  LPIPS $\downarrow$
MaskGIT [4] | 10.8 | 0.304 | 46.9 | 0.633
HCSS [20] | 9.8 | 0.425 | 50.2 | 0.653
Ours | 7.4 | 0.325 | 34.8 | 0.575
Ours-GT | 3.9 | 0.106 | 24.2 | 0.416
Ours-GT (Single) | 4.6 | 0.156 | 27.5 | 0.480
Figure 7: Qualitative results for keyframe generation. Compared with our
method, MaskGIT tends to predict repetitive content which shows the importance
of guidance. On the other hand, HCSS predicts inconsistent keyframes and fails
to maintain temporal consistency. When we compare Ours-GT with Ours-GT
(Single), we can see the iterative approach fails to generate consistent
keyframes. The results show the importance of modeling the entire video
jointly.

###  4.2 Keyframes Generation

Next, we evaluate the performance of our keyframe generation model. The goal
is to verify 1) the importance of generating all the keyframes jointly, and 2)
the importance of additional guidance for generating content across a large
temporal window.

Baselines. We compare the following baselines and variants of our method:

  * $\bullet$

MaskGIT [4]: the model takes the reference as input and iteratively predicts
the next keyframe. This model represents keyframe generation without input
guidance.

  * $\bullet$

HCSS [20]: the model takes a single layout as input and generates a single
keyframe. We apply HCSS to generate each keyframe independently from the
predicted layouts. This model represents keyframe generation without full
video modeling.

  * $\bullet$

Ours: our keyframe generation given the predicted layouts as inputs.

  * $\bullet$

Ours-GT: our keyframe generation using the ground truth layouts as inputs
(upper bound performance).

  * $\bullet$

Ours-GT (Single): our keyframe generator that predicts keyframes iteratively
conditioning on the previous keyframe and ground truth layouts. This model
represents keyframe generation without full video modeling.

Quantitative results. The results are in Tab. 3. Our method consistently
outperforms HCSS on both real and synthetic data, which verifies the
importance of joint prediction for all keyframes. Our method also performs
better than MaskGIT except for the perceptual similarity with ground truth
frames in the synthetic dataset. After taking a closer look at the generated
frames, we observed that MaskGIT tends to predict repetitive keyframes with
little changes across frames. This implies that the video will remain static,
which is not suitable for video generation. These results show that our method
can generate better keyframes for video generation than MaskGIT, which
verifies the importance of guidance. We also compare different variants of our
method. In particular, the superior performance of Ours-GT over Ours-GT
(Single) further verifies that a model that considers the entire video jointly
leads to better keyframe generation.

Qualitative results. Fig. 7 shows the qualitative results. As mentioned
before, MaskGIT tends to generate keyframes with similar content, which shows
the importance of providing input guidance at multiple timesteps. Comparing
HCSS and our method, we can see that HCSS fails to generate consistent results
across the keyframes, e.g. the color of the fire changes. Similarly, when we
compare Ours-GT with Ours-GT (Single), we can see the iterative approach fails
to generate consistent keyframes. The examples clearly demonstrate the
importance of joint modeling for the entire video. Please refer to the
supplementary material for additional evaluation, including the evaluation for
layout generation.

##  5 Conclusions

We tackle the problem of long video generation which aims to generate videos
beyond the output length of video generation models. We show that the existing
sliding window approach is sub-optimal, and there is significant room for
improvement using the same video generation model. To improve long video
generation, we propose to use additional guidance to control the generation
process. We further propose a two-stage approach which can utilize existing
video generation models while capturing long-range dependency within the
video. Empirical results validate our model design and show favorable results
over state-of-the-art video generation methods.

## References

  * [1] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan.  Fitvid: Overfitting in pixel-level video prediction.  arXiv preprint arXiv:2106.13195, 2020.
  * [2] Amir Bar, Roei Herzig, Xiaolong Wang, Gal Chechik, Trevor Darrell, and Amir Globerson.  Compositional video synthesis with action graphs.  In ICML, 2021.
  * [3] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras.  Generating long videos of dynamic scenes.  In NeurIPS, 2022.
  * [4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.  Maskgit: Masked generative image transformer.  In CVPR, 2022.
  * [5] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In NeurIPS, 2021.
  * [6] Aidan Clark, Jeff Donahue, and Karen Simonyan.  Adversarial video generation on complex datasets.  arXiv preprint arXiv:1907.06571, 2019.
  * [7] Gaoxiang Cong, Liang Li, Zhenhuan Liu, Yunbin Tu, Weijun Qin, Shenyuan Zhang, Chengang Yan, Wenyu Wang, and Bin Jiang.  Ls-gan: Iterative language-based image manipulation via long and short term consistency reasoning.  In ACM MM, 2022.
  * [8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  IJCV, 130:33–55, 2022.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [10] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine.  Self-supervised visual planning with temporal skip connections.  In CoRL, 2017.
  * [11] Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W. Taylor.  Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction.  In ICCV, 2019.
  * [12] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [13] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In NeurIPS, 2016.
  * [14] Tsu-Jui Fu, Xin Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang.  SSCR: Iterative language-based image editing via self-supervised counterfactual reasoning.  In EMNLP, 2020.
  * [15] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh.  Long video generation with time-agnostic vqgan and time-sensitive transformer.  In ECCV, 2022.
  * [16] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  In ICLR Workshop, 2022.
  * [18] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee.  Inferring semantic layout for hierarchical text-to-image synthesis.  In CVPR, 2018.
  * [19] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  In ICLR, 2023.
  * [20] Manuel Jahn, Robin Rombach, and Björn Ommer.  High-resolution complex scene synthesis with transformers.  In CVPRW, 2021.
  * [21] Justin Johnson, Agrim Gupta, and Li Fei-Fei.  Image generation from scene graphs.  In CVPR, 2018.
  * [22] Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu.  Video pixel networks.  In ICML, 2017.
  * [23] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [24] Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In ACL, 2019.
  * [25] Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa.  Blt: Bidirectional layout transformer for controllable layout generation.  In ECCV, 2022.
  * [26] Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao.  Storygan: A sequential conditional gan for story visualization.  In CVPR, 2019.
  * [27] Adyasha Maharana and Mohit Bansal.  Integrating visuospatial, linguistic and commonsense structure into story visualization.  In EMNLP, 2021.
  * [28] Adyasha Maharana, Darryl Hannan, and Mohit Bansal.  Improving generation and evaluation of visual stories via semantic consistency.  In NAACL, 2021.
  * [29] Michael Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  In ICLR, 2015.
  * [30] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid.  CCVS: Context-aware controllable video synthesis.  In NeurIPS, 2021.
  * [31] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh.  Action-conditional video prediction using deep networks in atari games.  In NeurIPS, 2015.
  * [32] Masaki Saito, Eiichi Matsumoto, and Shunta Saito.  Temporal generative adversarial nets with singular value clipping.  In ICCV, 2017.
  * [33] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi.  Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan.  IJCV, 2020.
  * [34] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  In NeurIPS, 2019.
  * [35] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  In ICLR, 2023.
  * [36] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny.  Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2.  In CVPR, 2022.
  * [37] Yun-Zhu Song, Zhi-Rui Tam, Hung-Jen Chen, Huiao-Han Lu, and Hong-Han Shuai.  Character-preserving coherent story visualization.  In ECCV, 2020.
  * [38] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [39] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.  Unsupervised learning of video representations using LSTMs.  In ICML, 2015.
  * [40] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov.  A good image generator is what you need for high-resolution video synthesis.  In ICLR, 2021.
  * [41] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz.  MoCoGAN: Decomposing motion and content for video generation.  In CVPR, 2018.
  * [42] Joost van Amersfoort, Anitha Kannan, Marc’Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala.  Transformation-based models of video sequences.  arXiv preprint arXiv:1701.08435, 2017.
  * [43] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In NeurIPS, 2017.
  * [44] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and D. Erhan.  Phenaki: Variable length video generation from open domain textual description.  In ICLR, 2023.
  * [45] Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal.  Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation.  In NeurIPS, 2022.
  * [46] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Generating videos with scene dynamics.  In NeurIPS, 2016.
  * [47] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit.  Scaling autoregressive video models.  In ICLR, 2020.
  * [48] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.  Godiva: Generating open-domain videos from natural descriptions.  arXiv preprint arXiv:2104.14806, 2021.
  * [49] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan.  Nüwa: Visual synthesis pre-training for neural visual world creation.  In ECCV, 2022.
  * [50] Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo.  Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks.  In CVPR, 2018.
  * [51] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [52] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  arXiv preprint arXiv:2203.09481, 2022.
  * [53] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang.  Magvit: Masked generative video transformer.  arXiv preprint arXiv:2212.05199, 2022.
  * [54] Wei Yu, Wenxin Chen, Songhenh Yin, Steve Easterbrook, and Animesh Garg.  Modular action concept grounding in semantic video prediction.  In CVPR, 2022.
  * [55] Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth.  Markov decision process for video generation.  In ICCVW, 2019.
  * [56] Liang Li Shaofei Cai Qianqian Xu Shuhui Wang Qingming Huang. Zhenhuan Liu, Jincan Deng.  Ir-gan: Image manipulation with linguistic instruction by increment reasoning.  In ACM MM, 2020.