Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning. In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations. With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets. In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme. Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.
 Generative models have gained much interest in the research community over the last few years as they provide a promise for unsupervised representation learning. Generative Adversarial Networks (GANs) BID0 have been one of the most successful generative models till date. Following its introduction in 2014, significant progress has been made towards improving the stability, quality and the diversity of the generated images BID11 BID2 . While GANs have been successful in the image domain, recent efforts have extended it to other modalities such as texts BID14 , graphs (Wang et al., 2018b) , etc.In this work, we focus on the less studied domain of videos. Generating videos are much harder than images because the additional temporal dimension makes generated data extremely high dimensional, and the generated sequences must be both photo-realistically diverse and temporally consistent. We tackle the problem of text-conditioned video synthesis where the input is a text description and the goal is to synthesize a video corresponding to the input text. This problem has many potential applications, some of which include producing multimedia special effects, generating synthetic data for model-based Reinforcement Learning systems and domain adaptation, etc.Two recent works that address the problem of text-conditioned video generation include BID6 and BID9 . Both these methods are variants of conditional GAN model applied to the video data. In spite of some successes, they have the following limitations: (1) They employ 3D transposed convolution layers in the generator network, which constrains them to only produce fixed-length videos. (2) Their models are trained on low-resolution videos -results are shown only at a 64×64 resolution. (3) Text conditioning is performed using a simple concatenation of video and text features in the discriminator: Such a conditioning scheme may perform well on certain datasets, but has difficulty in capturing rich video-text variations.In this work, we aim to address all the concerns above. First, to model videos of varying length, we use a recurrent neural network in the latent space and employ a shared frame generator network similar to BID12 . Second, we present a model for generating high-resolution videos by using a Resnet-style architecture in the generator and the discriminator network. Third, we propose a new multi-scale text-conditioning scheme based on convolutional filter generation to strengthen the associations between the conditioned text and the generated video. We call our model Text-Filter conditioning GAN (TFGAN). Finally, we construct a benchmark synthetic moving shapes dataset to extensively evaluate the effectiveness of the new conditioning scheme we proposed. Text representations are extracted from the input text and passed to a GRU network to get a trajectory in the latent space. These latent vectors are fed to a shared frame generator to produce the video sequence. The generated videos are then passed to conditional discriminator networks. The box highlighted in red is where the conditioning is performed and is expanded in FIG1 In summary, our contributions in this work are as follows: (i) A new conditional GAN with an effective multi-scale text-conditioning scheme based on convolutional filter generation is proposed; (ii) A benchmark synthetic dataset for studying text conditioning in video generation is presented; (iii) Photo-realistic video synthesis is achieved using a deeper generator-discriminator architecture. In this work, we address the problem of generating videos conditioned on text. We propose a novel text-conditioning framework whereby conditioning is performed using convolution operations on image feature maps with filters generated from text. To better understand the text conditioning, we construct a synthetic dataset and show that our conditioning scheme achieves superior performance compared to other techniques. Finally, by using deeper architectures in the discriminator and generator networks, we generate photo-realistic videos on the challenging Kinetics dataset. Sample N b real samples with incorrect video-text correspondence DISPLAYFORM0 Update G: DISPLAYFORM1 Update T : DISPLAYFORM2
An effective text-conditioning GAN framework for generating videos from text This paper presents a GAN-based method for video generation conditioned on text description, with a new conditioning method that generates convolution filters from the encoded text, and uses them for a convolution in the discriminator. This paper proposes conditional GAN models for text-to-video synthesis: developing text-feature-conditioned CNN filters and constructing moving-shape dataset with improved performance on video/image generation.