#  VideoCoCa: Video-Text Modeling with Zero-Shot Transfer
from Contrastive Captioners

Shen Yan† Tao Zhu†∗ Zirui Wang† Yuan Cao† Mi Zhang‡
Soham Ghosh† Yonghui Wu† Jiahui Yu†
†Google Research ‡The Ohio State University
{shenyan, tzhu, ziruiw, yuancao<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

We explore an efficient approach to establish a foundational video-text model.
We present VideoCoCa that maximally reuses a pretrained image-text contrastive
captioner (CoCa) model and adapt it to video-text tasks with minimal extra
training. While previous works adapt image-text models with various cross-
frame fusion modules, we find that the generative attentional pooling and
contrastive attentional pooling layers in CoCa are instantly adaptable to
flattened frame embeddings, yielding state-of-the-art results on zero-shot
video classification and zero-shot text-to-video retrieval. Furthermore, we
explore lightweight finetuning on top of VideoCoCa, and achieve strong results
on video question-answering and video captioning.

##  1 Introduction

Recently large-scale pretrained foundation models [5] have achieved successes
across a wide range of tasks on natural language processing and multimodal
problems, with versatility to transfer on a diverse set of downstream tasks.
For image-text models, pioneering works have achieved strong performance by
utilizing noisy web-crawled image-text pairs, such as CLIP[39] and ALIGN[19]
for crossmodal retrieval and SimVLM[56] for multimodal understanding. More
recent works [70, 68, 50, 52, 49] further improved the pretraining approaches
to enable better and more downstream vision-language tasks. In particular,
Contrastive Captioners (CoCa)[68] is directly applicable to classification,
crossmodal retrieval and multimodal understanding through a unified
contrastive and captioning learning schemes, achieving superior performance on
vision and vision-language tasks including image classification, image-text
retrieval, and visual question answering.

Figure 1: Left: Overview of VideoCoCa. All weights of the pretrained CoCa
model are reused, without the need of learning new modules. We compute frame
token embeddings offline from the frozen CoCa image encoder. These tokens are
then processed by a generative pooler and a contrastive pooler on all
flattened frame tokens, yielding a strong zero-shot transfer video-text
baseline. When continued pretraining on video-text data, the image encoder is
frozen, while the attentional poolers and text decoders are jointly optimized
with the contrastive loss and captioning loss, thereby saving heavy
computation on frame embedding. Right: An illustration of the attentional
poolers and flattened frame token embeddings. We flatten $N\times T$ token
embeddings as a long sequence of frozen video representations.

To extend image-text models to video, recent work has focused on either the
visual domain by aggregating temporal information with new modules [51, 1,
35], or on the textual domain by proposing different sampling methods to
curate higher-relevance text-video pairs [67, 60] for efficient contrastive
training. Given a well-pretrained image-text foundation model, it is natural
to question whether any heavy video-specific adaptor or many video-specific
data is needed when transferring to video-text modelling

In this paper, we explore an efficient approach to establish a foundational
video-text model for tasks including open-vocabulary video classification,
text-to-video retrieval, video captioning and video question-answering. We
present VideoCoCa, a minimalist approach that extends the image-text
contrastive captioners (CoCa) [68] to video-text tasks. The design principle
of VideoCoCa is to maximally reuse a pretrained image-text CoCa model and
minimize additional training overhead. Figure 1 (left) shows an overview of
our proposed model. As illustrated, VideoCoCa reuses the architecture and
pretrained model weights of CoCa and applies the image encoder to a sequence
of sampled frames of a video independently. The outputs are flattened and
concatenated to form a long sequence of frozen token embeddings, with the same
attentional poolers applied on top of them.

Without any additional video-specific parameter or early temporal fusion, and
without accessing any video or video-text data, VideoCoCa consistently
outperforms existing video-text models on video recognition tasks. It achieves
new state-of-the-art results on zero-shot video classification on Kinetics
400/600/700 [21], UCF101 [45], HMDB51 [23], and Charades [43], as well as
zero-shot text-to-video retrieval on MSR-VTT [61], ActivityNet Captions [22]
and VATEX [54]. We also explore different lightweight finetuning approaches on
video-text data to further boost results and achieve the best quality-
efficiency trade-offs. VideoCoCa is able to obtain further improvement on top
of its zero-shot transferred model architecture and achieves strong results on
the tasks of video question answering (iVQA [65], MSRVTT-QA [58], MSVD-QA
[58], ActivityNet-QA [69]) and video captioning (MSR-VTT [61], ActivityNet
[22], VATEX [54], Youcook2 [75]).

##  2 Related Work

#### Efficient video learners from image-text models.

CLIP [39] and ALIGN [19] were the first two works to gain popularity on zero-
shot image classification and crossmodal alignment tasks, by training dual-
encoder models with contrastive loss on noisy image-text pairs. Since then,
many works [32, 51, 20, 29, 62] have reported that retraining a new model for
video-text models is extremely challenging due to efficiency considerations,
and they have investigated ways to transfer the knowledge from the image-text
CLIP model to end-to-end action recognition and video-text retrieval. More
recently, there have been works that show efficient ways of adapting strong
image-text models to video foundation models with minimal extra computation.
For example, X-CLIP [35] uses a cross-frame attention and video-specific
prompting for video classification. CLIP-Hitchhiker [3] aggregates video
features by using the weighted mean of image embeddings with text as
similarity queries, achieving strong results on text-to-video retrieval. In
line with these research, we investigate efficient adaptation methods for
zero-shot transfer in the video-text domain by adapting attentional poolers.

#### Adapting image transformers to videos.

The success of these efficient video learners is largely due to the fact that
they initialize their video models using pretrained image models. Early works
[7, 53, 57] on 3D CNNs have demonstrated that image classification models can
be inflated into spatio-temporal feature extractors and their pretrained
weights provide a valuable initialization. For Transformers, several prior
works [4, 2, 64, 30, 27] illustrated the effectiveness of this approach for
video recognition. A common idea among these works is the adaptation of the
ViT [14] architecture for videos by extending self-attention from 2D image
space to 3D spatio-temporal volume. For example, TimeSformer [4] uses divided
space-time attention, where temporal and spatial attention layers in each
block are initialized with the same weights from the corresponding attention
layer in ViT. ViViT [2] initializes the weights of the tubelet embedding using
the central frame initialization from the image model, where all temporal
positions are initialized with zeros except at the center position. MTV [64]
further extends this idea by initializing each view of the multiview encoder.
Building on these research, we explore the use of all component weights of
CoCa for the model initializations.

#### Zero-shot transfer for video understanding.

LiT [72] shows that a locked pre-trained image model with an unlocked text
model is the most efficient to turn existing vision backbones into zero-shot
learners. Flamingo [1] freezes the weights of the pretrained vision and
language models, and only trains the cross-attention layers and perceiver
resampler [18] for efficient few-shot learning on video understanding.
FrozenBiLM [66] improves zero-shot videoQA by learning additional lightweight
adapter modules of the frozen bidirectional language models. Socratic models
[71] enable zero-shot multimodal-informed prompting using multiple visual-
audio-language pretrained models even without requiring finetuning. Similar to
the perceiver resampler [18] used in Flamingo [1] and the query learner used
in PaLI[11], we adopt the generative pooler in CoCa as the downsampler to
learn a fixed set of spacetime tokens. It does not introduce new parameters
but still allows for lightweight adaptation.

##  3 VideoCoCa

###  3.1 Preliminaries: CoCa

Contrastive Captioners (CoCa) [68] is an encoder-decoder architecture that
combines contrastive pretraining approaches like CLIP [39] with generative
pretraining apporaches like SimVLM [56]. It is designed to facilitate image,
text and image-text representation learning. CoCa adopts a cascaded decoder
design, where the bottom half unimodal decoder encodes the text context with
causally masked self-attention, and the top half multimodal decoder uses
cross-attention to align image and text. The model is trained with joint
contrastive loss and captioning loss. For text representations, the [CLS]
token from the unimodal decoder is used as a global text representation for
the contrastive loss, and the captioning loss is applied per text token to
learn fine-grained visual-textual information.

CoCa uses two attentional pooling layers [24] (poolers in short) to extract
image representations, in which a $n_{query}=256$ generative attentional
pooling layer is used to generate embeddings for captioning loss, and a
$n_{query}=1$ contrastive attentional pooling layer together with its outputs
as contrastive image embedding. The benefit of such an architecture design is
two-fold. First, the pooler yields a fixed number of tokens (for example 256
tokens as generative embedding and 1 token as contrastive embedding)
regardless of input image resolutions, making the image encoder and the text
decoder more modular and adaptable to other modalities. Second, the pooler
serves as a lightweight adaptor so that the pretrained ViT as backbone remains
frozen for many downstream tasks. For example, it is shown that by only
finetuning the poolers, a pretrained CoCa can already achieve 90.6% top-1
ImageNet accuracy, in which case the frozen ViT did not see any ImageNet data.
In this work, we adopt this design and explore to adapt to video-text domain.

###  3.2 Transferring CoCa to Video-Text Tasks

We illustrate how image CoCa can be quickly turned into VideoCoCa by tuning a
small portion of parameters. We denote our input minibatch of videos as
$\mathbf{V}\in\mathbb{R}^{B\times T\times H\times W\times C}$, where $T$ is
the number of frames uniformly sampled from a video. We follow ViT [14] and
extract tokens from images by partitioning an image into non-overlapping
patches and linearly projecting them. All tokens are then concatenated
together to form a sequence, resulting in a minibatch of the sequence tokens
$\mathbf{\tilde{z}}$ of shape $\mathbb{R}^{(B\times T)\times N\times d}$,
where $N=\lfloor\frac{H}{h}\rfloor\times\lfloor\frac{W}{w}\rfloor$. A
positional embedding $\mathbf{p}\in\mathbb{R}^{N\times d}$ is next added to
this sequence to obtain $\mathbf{z}$. We follow [68] to extract the text
representations (see Section 3.1). The frame-level representations
$\mathbf{z}^{L}\in\mathbb{R}^{(B\times T)\times N\times d}$ are obtained by
forwarding $\mathbf{z}$ into the CoCa image encoder, where $L$ is the number
of encoder layers. We consider the following alternatives to adapt CoCa to
videos.

#### Attentional poolers.

To obtain video representations, we concatenate all spatial tokens together
along the temporal dimension into
$\mathbf{z}^{L}\in\mathbb{R}^{B\times(T\times N)\times d}$, which is then fed
into the generative and the contrastive poolers. See Figure 1 (right) for a
detailed illustration. This model corresponds to a late fusion of temporal
information, similar to the factorized encoder. Compared to [68], where
additional new poolers are added on top of the frame-level representations to
learn video representations, this model does not add any novel learnable
layers, allowing reuse of all parameters from the pretrained CoCa model with
minimal extra computation, thereby enabling zero-shot transfer to video-text
tasks from image-text models.

#### Factorized encoder.

This model additionally adds a Transformer encoder on top of the contrastive
pooler. The frame-level representations $\mathbf{z}^{L}$, is first fed into
the generative pooler and the contrastive pooler to get the spatial embeddings
$\mathbf{z}^{L_{s}}\in\mathbb{R}^{(B\times T)\times d}$. The spatial
embeddings $\mathbf{z}^{L_{s}}$ are then reshaped to into
$\mathbf{z}^{L_{s}}\in\mathbb{R}^{B\times T\times d}$ and fed into a
transformer encoder consisting of $L_{t}$ layers to model interactions between
tokens from different frames. The output tokens are used for the captioning
loss and their global average pooled embeddings over the temporal dimension
are used for the contrastive loss. Following [2, 64], we choose $L_{t}=4$.
Note that different from [2, 64], where the spatial embeddings are summarized
by the prepended learnable class token $\mathbf{z}_{cls}\in\mathbb{R}^{d}$
[13], we use the output of the contrastive pooler as representation.

#### Joint space-time encoder.

This model is commonly known as spatio-temporal attention model [2] or joint
space-time model [4]. Specifically, we reshape the sequence tokens into
$\mathbf{\tilde{z}}\in\mathbb{R}^{B\times(T\times N)\times d}$ and then add
the positional embedding $\mathbf{p}\in\mathbb{R}^{(T\times N)\times d}$ to
obtain $\mathbf{z}$. $\mathbf{p}$ is initialized by temporally repeating the
positional embedding from the pretrained image model. This allows CoCa image
encoder to encode pairwise interactions between all spatial-temporal tokens
from the first layer. The spatial-temporal representations $\mathbf{z}^{L}$,
is then fed into the generative pooler and the contrastive pooler to get the
final task-specific representations. This model corresponds to an early fusion
of temporal information and does not add any new learnable layers. However, it
makes the self-attention computation in the encoder much heavier due to the
linearly increased number of tokens.

#### Mean pooling.

The frame-level representations $\mathbf{z}^{L}\in\mathbb{R}^{(B\times
T)\times N\times d}$, are simply separately average pooled over the temporal
dimension after the attentional poolers. It ignores the temporal information
and is used as our adaptor baseline. Note that [68] also uses this method to
compute zero-shot text-to-video retrieval metrics.

###  3.3 Finetuning VideoCoCa on Video-Text Data

In addition to direct zero-shot transfer from image-text CoCa to video-text
tasks, it is natural to consider further pushing the limit of VideoCoCa by
continued pretraining on web-scale video-text paired data. We use VideoCC3M
[34] as our main source of video-text data which is detailed in Section 4.1.
We explore four different learning choices and conclude with the best setup
with analysis.

#### Finetuning (FT).

Under this setup, we unfreeze all parameters of the pretrained CoCa model
during continued video-text pretraining, including the parameters of the
encoder and the decoder, as well as the parameters of the generative pooler
and the contrastive pooler. These parameters are finetuned together with the
newly added learnable layers. Note that this could lead to less stable
training [72]. We use it as our baseline tuning method.

#### Frozen encoder-decoder tuning (Frozen).

We freeze the parameters of the encoder and the decoder, and only tune the
parameters of the generative and the contrastive poolers. This allows us to
re-use most parameters of the pretrained CoCa model.

#### Frozen tuning then finetuning (Frozen + FT).

Frozen encoder-decoder tuning may converge very fast given the small amount of
parameters of the pooler (See Figure 2(a)). We explore to first conduct frozen
feature tuning and then finetuning. In this way, the parameters of the pooler
can be quickly trained, thus making the finetuning more stable. Note that it
is a two-step tuning method.

#### Frozen encoder tuning (LiT).

Similar to the best practice in [72], we only freeze the parameters of the
pretrained CoCa image encoder, and tune the parameters of the poolers as well
as the decoder. As the computation of image representations is much heavier
than text representations, this not only allows us to precompute frame-level
embeddings once to save TPU memory and computations for development, but also
provides sufficient amount of learnable parameters for task adaptation (See
Figure 2(b)).

(a) Total parameters of each component in CoCa.

(b) Learable parameters of different tuning methods.

Figure 2: An illustration of the parameters of each component (left), and
learnable parameters of different tuning methods (right). LiT is adopted for
VideoCoCa tuning.

##  4 Experiments

###  4.1 Experimental setup

#### Dataset for pretraining.

In our experiments, we continue training the model on video-text datasets for
one epoch to mitigate catastrophic forgetting. We report the performance of
VideoCoCa models that are pretrained on the the following datasets:

_HowTo100M_ [33] contains around 136 million video clips sourced from over
1.2M narrated videos associated with automatically transcribed speech.

_VideoCC3M_ [34] is mined from the image captioning dataset Conceptual
Captions 3M (CC3M) [41]. It includes around 8.7M video clip-text pairs and
900K unique captions. Unlike HowTo100M [33], which is restricted to
instructional videos, VideoCC3M is created from a more diverse corpus of
online videos and is thus more balanced.

#### Zero-shot video classification.

We use Kinetics[21], UCF101 [45], HMDB51 [23], and Charades [43] to evaluate
zero-shot transfer performance on video classification. We report results on
Kinetics with 400, 600, and 700 classes, respectively. Following prior work
[39, 19], we use aligned video/text embeddings to perform zero-shot video
classification by matching videos with label names without finetuning. We
follow the same setup in [39] and apply the same set of prompts used for label
class names. For Kinetics-400/600/700, we report Top-1 and Top-5 accuracy on
the validation set. For UCF101 and HMDB51, we follow [39, 35] to evaluate on
the three splits of the test data, and report the average Top-1 and Top-5
accuracy. For Charades, as it is a long-form video dataset with varied action
classes over the duration (30 seconds in average), we follow [51, 3] to report
the multi-label classification results in mean average precision (mAP) on the
test data.

#### Zero-shot text-video retrieval.

We report the zero-shot score of Recall at K (R@K) on MSR-VTT [61],
ActivityNet-Captions [22] and Youcook2 [75]. MSR-VTT contains 10K YouTube
videos clips with 20 English descriptions each. We follow [37, 71, 68] to
report results on the available subset of the full test set. ActivityNet-
Captions contains 20K YouTube videos with 100K captions. We evaluate
paragraph-to-video retrieval [32, 25] on the val1 split where the captions in
the same video are concatenated to form a paragraph. Youcook2 contains 2K
cooking videos (14K video clips) and each video clip is annotated with one
caption. We follow [33, 1] to evaluate on the validation set. VATEX-v1.1 [54]
includes 26K/3K/6K videos for training/validation/test. There are 10 sentences
in English and Chinese to describe each video. We follow [10, 15] to only use
the English annotations for evaluation but on the entire validation set.

#### Video captioning.

We report both zero-shot and finetuning performance of BLEU-4 [36], CIDEr [47]
and ROUGE [28] on MSR-VTT [61], ActivityNet-Captions [22] and Youcook2 [75].
For MSR-VTT, we follow the standard splits in [31, 40] for a fair comparison.
For ActivityNet-Captions, we concatenate captions for each segments to form a
paragraph caption [26, 63] and use train/val1/val2 as training/validation/test
splits. For Youcook2, we follow splits used in [31, 59, 1] for finetuning and
evaluation. For VATEX, we use train/validation/public test as the splits used
in [44, 74].

#### Finetuning for video question answering.

We report the finetuning performance on the open-ended VideoQA datasets IVQA
[65], MSRVTT-QA [58], MSVD-QA [58] and ActivityNet-QA [69]. We use original
splits for training, validation and test. We report top-1 test accuracy, which
requires the model to learn a new task-specific attentional pooler on top of
the decoder. Therefore, we only have finetuned results for the VideoQA task.

#### Model variants.

For main results, we initialize the parameters of VideoCoCa from CoCa setup
[68] (”VideoCoCa” in short), where the model consists of 1B parameters in the
image encoder and 1.1B parameters in the text decoder. We also explore Base
and Large setup used in [68], which are detailed in Table 5. For ablation
studies, we use the smaller model variant CoCa-Small in [68] where the decoder
layers of CoCa-Base is reduced to 12. We fix 256 tokens as the generative
embedding and 1 token as the contrastive embedding for all our experiments.

#### Training and inference.

We uniformly sample $8$ frames for each clip in all our experiments. Ablations
with varying number of input frames are detailed in Appendix. We use the
smaller model variant in [68] (_i.e._ CoCa-Small) with the image resolution
$224\times 224$ and patch size $16\times 16$ for our ablation study in Section
4.2. For the main results in Section 4.3, we use CoCa with the image
resolution $576\times 576$ and patch size $18\times 18$. For pretraining, we
use a batch size of $128$ video-text pairs, and only apply center cropping to
the training data. All models are trained on the combined contrastive and
captioning objectives with the same optimization configuration as used in
[68], except with a lower learning rate of $1\times 10^{-7}$ and weight decay
of $1\times 10^{-6}$. We continue pretraining VideoCoCa for one epoch. The
training with precomputed frame token embeddings as input takes about $6$
hours on $128$ CloudTPUv4 chips. For finetuning, we use standard
configurations and report the details in Appendix. During the evaluation, we
always use a single view inference for simplicity. Specifically, we use the
center cropping for each video frame, and no temporal croppings of a video is
used.

Table 1: Comparisons of 3 tasks zero-shot transfer performance with different
adaptors on Kinetics-400 (video classification) and MSR-VTT (text-to-video
retrieval, video captioning). All models are initialized from pretrained CoCa-
Small and then finetuned on VideoCC3M. Without the need of learning new
modules, attentional pooler consistently performs the best.

Adaptors | VideoCls. | VideoRet. | VideoCap.
---|---|---|---
Top-1 | Top-5 | R@1 | R@5 | BLEU-4 | CIDEr
Mean Pooling | 40.3 | 69.3 | 24.5 | 45.3 | 15.5 | 13.4
Factorized Enc. | 43.3 | 72.7 | 24.9 | 45.6 | 15.5 | 17.2
Joint Space-Time | 38.1 | 65.5 | 22.6 | 43.8 | 14.7 | 12.1
Attentional Pooler | 45.6 | 73.4 | 26.4 | 46.8 | 16.8 | 19.9

###  4.2 Ablation study

We conduct ablation studies on the Kinetics-400, MSR-VTT, ActivityNet-Captions
and Youcook2. We report zero-shot video classification accuracies, text-to-
video retrieval R@K and video captioning results.

#### Adaptor choices.

Table 1 shows the comparison of different adaptation methods. In all cases,
the model is initialized from pretrained CoCa-Small and then continued
pretraining on VideoCC3M using ”Finetuning” method in Section 3.3. Compared to
the ”Mean Pooling” baseline, which ignores temporal information, ”Factorized
Encoder” clearly outperforms on video classification ($+3\%$ and $+3.4\%$ on
top-1 and top-5 accuracy, respectively) and video captioning ($+3.8$ CIDEr
score), indicating that a late fusion of temporal information is useful to
learn high-quality video representation better than the image-text approach.
We observe that ”Attentional Pooler”, which also corresponds to the late
fusion but does not introduce any new video-specific parameters, outperforms
all other methods on three video-text transfer tasks ($+5.3\%$ top-1 on video
classification, $+1.9$ R@1 on text-to-video retrieval, $+6.5$ CIDEr on video
captioning). ”Joint Space-Time Encoder”, which corrsponds to an early fusion
of temporal information, is the worst performing method of the four, and we
think it is because early reshaping of the spatial-temporal features causes
the input pattern to change, making the pretrained image-text model harder to
adapt. Given the strong zero-shot transfer results from ”Attentional Pooler”
and its minimalist design, we use it in subsequent experiments.

Table 2: Comparisons of 3 tasks zero-shot transfer performance with different
tuning choices on Kinetics-400 (video classification) and MSR-VTT (text-to-
video retrieval, video captioning). All models are initialized from a
pretrained CoCa-Small. Frozen image encoder tuning (LiT) is adopted as it
allows faster and lightweight task adaptation.

Tuning choices | VideoCls. | VideoRet. | VideoCap.
---|---|---|---
Top-1 | Top-5 | R@1 | R@5 | BLEU-4 | CIDEr
FT | 45.6 | 73.4 | 26.4 | 46.8 | 16.8 | 19.9
Frozen | 45.4 | 73.2 | 27.3 | 47.0 | 16.2 | 19.7
Frozen + FT | 45.8 | 73.5 | 28.4 | 49.3 | 16.8 | 20.5
LiT | 45.7 | 73.9 | 28.8 | 49.3 | 17.0 | 20.6

#### Tuning choices.

Table 2 shows the comparison on zero-shot video-text tasks transfer with
different tuning choices. VideoCoCa is initialized from CoCa-Small and
continue pretrained on VideoCC3M. Compared to ”Finetuning” (FT) baseline, we
surprisingly find that ”Frozen Encoder-Decoder Tuning” (Frozen) alone already
achieves comparable results to the baseline on all tasks. This allows us to
freeze most of the parameters of CoCa and only tune a few parameters in the
attentional pooler. ”Frozen Tuning then Finetuning” (Frozen + FT), is able to
further improve the performance on top of frozen tuning ($+2.0$ on R@1),
however, is inconvenient in practice as it needs to first conduct frozen
tuning then finetuning. Frozen Encoder Tuning (LiT), which freezes the
parameters of image encoder and only tunes the parameters of text decoder and
attentional poolers, achieves the best performance on text-to-video retrieval
($+2.4$ on R@1) and video captioning ($+0.7$ on CIDEr) among four tuning
methods, indicating that a strong frozen image encoder can be enough for
video-text tasks adaptation. Another advantage is that it allows us to
precompute frame-level embeddings, and thus saves most of the TPU memory in
the actual training. Therefore, we adopt LiT for subsequent experiments.

#### Pretraining with mixed VideoCC3M and HowTo100M.

We experimented with mixing VideoCC3M and HowTo100M in each minibatch during
pretraining. The model is initialized from the pretrained CoCa-Small and then
continued pretraining on the mixed datasets. Table LABEL:tab:mix_ablations
shows results on zero-shot text-to-video retrieval in R@10. We find that
increasing the mixing ratio of HowTo100M in each minibatch gives monotonically
increased results on nearby domain dataset YouCook2 but decreased results on
generic Youtube videos MSR-VTT and ActivityNet Captions. For example,
comparing to using 100% VideoCC3M for pretraining, using 100% HowTo100M gives
$+1.7\%$ improvement on Youcook2, but $-2.5\%$ on MSR-VTT and $-6.0\%$ on
ActivityNet Captions. We observed that there is no optimal mixing ratio that
provides the best results on all three datasets. Therefore, we always use
VideoCC3M as our pretraining dataset.

Table 3:  Results on zero-shot text-to-video retrieval with different mixing
ratios between VideoCC3M and HowTo100M in each minibatch of pretraining. All
results are reported in R@10 on MSR-VTT Full, ActivityNet-Captions and
Youcook2. All VideoCoCa models are initialized from a pretrained CoCa-Small
model then trained with LiT.

Mixing Ratio [%] | MSR-VTT Full | ActivityNet-Captions | YouCook2
---|---|---|---
VideoCC3M | HT100M |  |  |
100 | 0 | 59.5 | 66.8 | 29.3
70 | 30 | 59.0 | 66.1 | 29.6
50 | 50 | 58.1 | 65.6 | 30.0
30 | 70 | 57.6 | 63.4 | 30.5
0 | 100 | 57.0 | 60.8 | 31.0

Table 4: Comparisons to state-of-the-art results on zero-shot video
classification. Results are reported in top accuracy [%] or mean average
precision (mAP). _Avg._ denotes averaged Top-1/5 accuracy. ”CoCa” refers to
CoCa image-text model that is directly applied to zero-shot video
classification using the frame-level average pooling. We use the notation
“w.o. videos” to refer to the VideoCoCa model that is initialized from the
pretrained CoCa model without continued pretraining. In line with [35], we use
single view evaluation for simplicity.

(a) Kinetics 400

Method | Top 1 | Top 5
---|---|---
REST [6] | 63.9 | 81.0
CoCa | 66.4 | 87.1
VideoCoCa (w.o. videos) | 72.0 | 90.3
VideoCoCa | 72.0 | 90.5

(b) HMDB 51

Method | Top 1 | Top 5
---|---|---
ER-ZSAR [9] | 35.3 | –
ActionCLIP [51] | 40.8 | –
X-CLIP [35] | 44.6 | –
X-Florence [35] | 48.4 | –
REST [6] | 49.7 | –
MOV-L [38] | 57.8 | –
CoCa | 57.2 | 82.6
VideoCoCa (w.o. videos) | 57.4 | 82.7
VideoCoCa | 58.7 | 84.5

(c) Kinetics 600

Method | Top 1 | Top 5
---|---|---
ER-ZSAR [9] | 42.1 | 73.1
X-CLIP [35] | 65.2 | 86.1
X-Florence [35] | 68.8 | 88.4
CoCa | 65.1 | 87.1
VideoCoCa (w.o. videos) | 70.0 | 89.0
VideoCoCa | 70.1 | 88.9

(d) UCF 101

Method | Top 1 | Top 5
---|---|---
ER-ZSAR [9] | 51.8 | –
ActionCLIP [51] | 58.3 | –
REST [6] | 69.1 | –
X-CLIP [35] | 72.0 | –
X-Florence [35] | 73.2 | –
MOV-L [38] | 80.9 | –
CoCa | 85.1 | 97.5
VideoCoCa (w.o. videos) | 86.6 | 98.5
VideoCoCa | 86.6 | 98.4

(e) Kinetics 700

Method | Top 1 | Top 5 | Avg.
---|---|---|---
NFNet-F6 [1] | – | – | 62.9
CLIP-L [1] | – | – | 69.6
CoCa | 60.2 | 82.2 | 71.2
VideoCoCa (w.o. videos) | 62.2 | 83.7 | 73.0
VideoCoCa | 62.5 | 83.8 | 73.2

(f) Charades

Method | mAP
---|---
CLIP-Hitchhiker [3] | 21.1
CoCa | 23.1
VideoCoCa (w.o. videos) | 25.2
VideoCoCa | 25.8

Table 5: Model scaling results. We report results on zero-shot video
classification (top-1 accuracy), zero-shot text-to-video retrieval (R@1),
zero-shot video captioning (CIDEr) and finetuning for video question answering
(top-1 accuracy) across three model scales. VideoCoCa consistently outperforms
CoCa baseline (mean pooling) with the same number of parameters.

Model | Params | TFLOPs | VideoCls. | VideoRet. | VideoCap. | VideoQA
---|---|---|---|---|---|---
K400 | K600 | K700 | HMDB51 | UCF101 | MSRVTT | ActivityNet | Youcook2 | MSRVTT | ActivityNet | Youcook2 | IVQA | MSRVTT-QA | MSVD-QA
CoCa-B | 383M | 2.01 | 57.5 | 56.5 | 47.6 | 51.1 | 78.3 | 27.5 | 21.7 | 11.2 | 12.8 | 8.5 | 7.0 | 28.7 | 41.2 | 52.2
VideoCoCa-B | 383M | 2.22 | 61.2 | 59.6 | 51.0 | 53.7 | 80.4 | 31.2 | 29.6 | 16.5 | 20.6 | 14.1 | 14.4 | 29.4 | 42.6 | 53.6
CoCa-L | 787M | 6.89 | 62.9 | 60.7 | 50.5 | 52.2 | 84.6 | 29.0 | 24.2 | 13.8 | 16.5 | 9.1 | 15.7 | 31.2 | 43.3 | 55.0
VideoCoCa-L | 787M | 7.39 | 68.4 | 67.4 | 58.9 | 56.9 | 85.3 | 33.9 | 31.5 | 18.9 | 24.3 | 17.4 | 20.7 | 32.6 | 44.3 | 56.2
CoCa | 2.1B | 28.74 | 66.4 | 65.1 | 60.2 | 57.2 | 85.1 | 30.0 | 28.5 | 16.8 | 15.9 | 11.2 | 18.2 | 33.6 | 45.9 | 55.4
VideoCoCa | 2.1B | 29.82 | 72.0 | 70.1 | 62.5 | 58.7 | 86.6 | 34.3 | 34.5 | 20.3 | 27.1 | 19.3 | 34.3 | 39.0 | 46.3 | 56.9

Table 6: Comparison to state-of-the-art results on zero-shot text-video
retrieval. All results are reported on R@1 / R@5 / R@10. “w.o. videos” refers
to the VideoCoCa model that is initialized from the pretrained CoCa model
without continued pretraining. For Youcook2 results, see Table
LABEL:tab:mix_ablations for the analysis.

---

(a) MSR-VTT Full Method | Text-to-Video | Video-to-Text
---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
CLIP [37] | 23.3 | 44.2 | 53.6 | 43.3 | 73.3 | 81.8
Socratic Models [71] | – | – | – | 46.9 | 73.5 | 81.3
CoCa [68] | 30.0 | 52.4 | 61.6 | 49.9 | 73.4 | 81.4
VideoCoCa (w.o. videos) | 32.9 | 56.0 | 65.0 | 59.7 | 81.8 | 89.3
VideoCoCa | 34.3 | 57.8 | 67.0 | 64.7 | 85.2 | 91.4

(b) ActivityNet Captions Method | Text-to-Video | Video-to-Text
---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
SINGULARITY [25] | 30.6 | 55.6 | 66.9 | – | – | –
InternVideo [55] | 30.7 | – | – | 31.4 | – | –
CoCa | 28.5 | 58.4 | 70.5 | 21.9 | 49.5 | 64.1
VideoCoCa (w.o. videos) | 32.2 | 61.0 | 74.6 | 26.8 | 54.7 | 70.0
VideoCoCa | 34.5 | 63.2 | 76.6 | 33.0 | 61.6 | 75.3

---

(c) VATEX Method | Text-to-Video | Video-to-Text
---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
InternVideo [55] | 49.5 | – | – | 69.5 | – | –
CoCa | 47.2 | 79.0 | 86.9 | 60.9 | 88.1 | 82.4
VideoCoCa (w.o. videos) | 50.7 | 81.3 | 88.5 | 64.1 | 90.2 | 95.6
VideoCoCa | 53.2 | 83.3 | 90.1 | 73.6 | 93.2 | 97.2

(d) Youcook2 Method | Text-to-Video
---|---
R@1 | R@5 | R@10
FitCLIP [8] | 5.8 | 15.5 | 22.1
TACo [67] | 19.9 | 43.2 | 55.7
VideoCLIP [60] | 22.7 | 50.4 | 63.1
CoCa | 16.8 | 35.8 | 46.4
VideoCoCa (w.o. videos) | 21.7 | 43.9 | 55.2
VideoCoCa | 20.3 | 43.0 | 53.3

Table 7: Comparison to state-of-the-art zero-shot and finetuned results on
video captioning. All results are reported on BLEU-4 (B-4), CIDEr (C) and
ROUGE (R).

---

(a) MSR-VTT Method | B-4 | C | R
---|---|---|---
zero-shot results
AVR [34] | 13.2 | 8.2 | –
VideoCoCa | 19.7 | 27.1 | 35.4
finetuning results
UniVL [31] | 41.8 | 50.0 | 60.8
MV-GPT [40] | 48.9 | 60.0 | 64.0
GIT2 [50] | 54.8 | 75.9 | 68.2
VideoCoCa | 53.8 | 73.2 | 68.0

(b) ActivityNet Captions Method | B-4 | C | R
---|---|---|---
zero-shot results
CoCa | 6.9 | 11.2 | 19.8
VideoCoCa | 8.5 | 19.3 | 22.5
finetuning results
MART [26] | 9.8 | 22.2 | 30.9
VLCap [63] | 13.4 | 30.3 | 36.0
VideoCoCa | 14.7 | 39.3 | 35.0

(c) VATEX Method | B-4 | C | R
---|---|---|---
zero-shot results
CoCa | 6.2 | 12.4 | 21.0
VideoCoCa | 9.5 | 22.8 | 29.5
finetuning results
NITS-VC [44] | 20.0 | 24.0 | 42.0
ORG-TRL [74] | 32.1 | 49.7 | 48.9
VideoCoCa | 39.7 | 77.8 | 54.5

(d) Youcook2 Method | B-4 | C | R
---|---|---|---
zero-shot results
Flamingo-80B [1] | – | 60.1 | –
VideoCoCa | 7.7 | 34.3 | 16.5
finetuning results
UniVL [31] | 11.2 | 127.0 | 40.1
VLM [59] | 12.3 | 138.7 | 41.5
OmniVL [49] | 8.7 | 116.0 | 36.1
GIT2 [50] | 9.4 | 131.2 | 37.5
VideoCoCa | 14.2 | 128.0 | 37.7

Table 8: Comparison to state-of-the-art results on finetuning for video
question answering. All results are in top-1 accuracy [%].

Method | IVQA | MSRVTT-QA | MSVD-QA | ActivityNet-QA
---|---|---|---|---
JustAsk [65] | 35.4 | 41.8 | 47.5 | 38.9
SINGULARITY [25] | – | 43.5 | – | 44.1
VIOLET [16] | – | 43.9 | 47.9 | –
All-in-one [48] | – | 46.8 | 48.3 | –
FrozenBiLM [66] | 39.6 | 47.0 | 54.8 | 43.2
GiT2 [50] | – | 45.6 | 58.2 | –
VideoCoCa | 39.0 | 46.3 | 56.9 | 56.1

#### Scaling up the model.

To make the notation consistent with [68], we use VideoCoCa, VideoCoCa-L, and
VideoCoCa-B to refer to model parameters initialized from CoCa, CoCa-Large,
and CoCa-Base, respectively. Table 5 shows our model scaling results on four
video-text tasks. All models are trained with a spatial resolution of
$576\times 576$ and the total FLOPs (TFLOPs) are reported. With the same
number of parameters, VideoCoCa consistently achieves higher performance than
the CoCa model at every model scale. Furthermore, note how our Large model
(transformer depth of 24 layers) outperforms CoCa (40 layers), while using
3.9$\times$ less TFLOPs. Similarly, our Base model outperforms CoCa-L on most
datasets. This result indicates that we can achieve greater video-text tasks
improvements by adapting attentional poolers for videos with continued
training. When using our largest 2.1B model, our VideoCoCa substantially
advances the best results obtained from CoCa with less than 4% extra TFLOPs
compute.

###  4.3 Main Results

We compare VideoCoCa to the state-of-the-arts on fourteen different datasets
and four tasks, including zero-shot video classification, zero-shot text-video
retrieval, zero-shot and finetuning for video captioning, and finetuning for
video question answering. We evaluate models with single view per video clip,
following [35] (detailed in Section 4.1).

#### Zero-shot video classification.

We use ”CoCa” to denote CoCa image-text model that is directly applied to
zero-shot video classification using frame-level average pooling as the
feature aggregation. We use notation ”w.o. videos” to denote VideoCoCa model
that is initialized from pretrained CoCa model without continued pretraining
on VideoCC3M. On Kinetics 700, ”Avg. Top 1/5” refers to averaged top-1 and
top-5 accuracy, following [1].

Table 4(f) summarizes our zero-shot video classification results on
Kinetics-400/600/700, UCF101, HMDB51, and Charades. On Kinetics, we observe
that CoCa image-text model alone is already a strong baseline, leading to a
$+2.5\%$ top-1 accuracy on Kinetics 400 and a $+1.6\%$ averaged top-1 and
top-5 accuracy on Kinetics 700 compared to the previous state-of-the-art
models. Compared to CoCa, We surprisingly find that VideoCoCa (w.o. videos)
with the plugin attentional poolers as the video adaptor is instantly
adaptable to the flattened frame embeddings, yielding significant improvement
on top of CoCa. Furthermore, when continually pretrained on VideoCC3M,
VideoCoCa can be further improved, leading to $+5.6\%$, $+5.0\%$, $+2.3\%$
top-1 accuracy on Kinetics 400, Kinetics 600 and Kinetics 700 over the CoCa
baseline. On UCF101 and HMDB51, CoCa can already achieve competitive results,
mostly due to less human action variations in the video clip [21]. VideoCoCa
is again able to give further improvements on top of CoCa. On the long-form
video dataset Charades, VideoCoCa achieves state-of-the-art results ($+4.7$
mAP) with only $8$ frames.

#### Zero-shot text-video retrieval.

We summarize our results on text-to-video retrieval in Table 6(d). We also
report video-to-text retrieval results following [71]. On MSR-VTT, ActivityNet
Captions and VATEX, VideoCoCa achieves the highest zero-shot retrieval metrics
for both text-to-video and video-to-text retrieval. We find that continued
training on VideoCC3M is able to further improve on top of VideoCoCa (w.o.
videos), for example, by $+1.4$ R@1 on MSR-VTT text-to-video retrieval, $+2.3$
R@1 on ActivityNet Captions and $+2.5$ R@1 on VATEX. We observe even bigger
gains on video-to-text retrieval. On Youcook2, VideoCoCa does not achieve
better results. As we we observed in Table LABEL:tab:mix_ablations, increasing
the ratio of HowTo100M can improve the results on Youcook2 but decrease the
performance on MSR-VTT and ActivityNet Captions. This explains works such as
[67, 60], which are solely pretrained on HowTo100M, have strong results on
Youcook2 but underperformed results on MSR-VTT. Similar findings are also
reported in [8]. We conclude that the improvement on Youcook2 can be
attributed to the strong visual-language domain match between YouCook2 and
HowTo100M.

#### Video captioning.

We report both zero-shot and finetuned video captioning results in Table 7(d).
VideoCoCa significantly outperforms other models on zero-shot video captioning
on MSR-VTT, ActivityNet Captions and VATEX. On Youcook2, we report results
without using transcripts in the encoder for a fair comparison. For
finetuning, VideoCoCa achieves competitive results among previous works, in
some cases outperform them by large margins (_e.g._ ActivityNet Captions,
VATEX).

#### Finetuning for video question answering.

Table 8 lists our results on open-ended VideoQA. VideoCoCa, again, achieves
strong performance among state-of-the-art models. We also note that there is
no single video-text model that performs significantly better than the others
on all four video-text tasks.

##  5 Conclusion

In this work, we presented VideoCoCa, a simplified approach to transfer
pretrained image-text CoCa models to video-text tasks in a zero-shot manner.
Without any video-only or video-text data, VideoCoCa zero-shot transfer
baseline already achieves state-of-the-art results on zero-shot video
classification and zero-shot text-to-video retrieval. Furthermore, we
investigated efficient finetuning on video-text data with VideoCoCa and
achieved strong results on video captioning and video question answering. We
conducted extensive ablations on other architecture options and tuning methods
on a wide range of video-text tasks. We hope our proposed approach and
empirical results can establish a strong video-text baseline on leveraging
pretrained image-text foundation models for video-text tasks.

#### Acknowledgements.

We thank Anja Hauth, Arsha Nagrani, David Ross, Bryan Seybold, Xuehan Xiong,
Lucas Smaira, AJ Piergiovanni, Anelia Angelova, Radu Soricut, Yin Cui, Rui
Qian, Antoine Yang, Mojtaba Seyedhosseini, Paul Natsev, Cordelia Schmid and
Tom Duerig for their valuable feedback on dataset evaluation. We also
appreciate Vijay Vasudevan for the helpful discussions on embedding
extractions. We thank Xingyi Zhou, Wei Han and Claire Cui for their
proofreading efforts, Erica Moreira, Victor Gomes for their assistance with
resource coordination. Our implementation was made possible thanks to the
support of the infrastructures team, and we want to specially thank Laurent El
Shafey for his guidance on the infrastructures and model scalings.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  In NeurIPS, 2022.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In ICCV, 2021.
  * [3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  A clip-hitchhiker’s guide to long video retrieval.  In arXiv:2205.08508, 2022.
  * [4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  In arXiv:2108.07258, 2021.
  * [6] Adrian Bulat, Enrique Sanchez, Brais Martinez, and Georgios Tzimiropoulos.  Rest: Retrieve & self-train for generative action recognition.  In arXiv:2209.15000, 2022.
  * [7] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [8] Santiago Castro and Fabian Caba Heilbron.  Fitclip: Refining large-scale pretrained image-text models for zero-shot video understanding tasks.  In BMVC, 2022.
  * [9] Shizhe Chen and Dong Huang.  Elaborative rehearsal for zero-shot action recognition.  In ICCV, 2021.
  * [10] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In CVPR, 2020.
  * [11] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  In ICLR, 2023.
  * [12] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In NeurIPS, 2020.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [15] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen.  Clip2video: Mastering video-text retrieval via image clip.  In arXiv:2106.11097, 2021.
  * [16] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu.  Violet: End-to-end video-language transformers with masked visual-token modeling.  In arXiv:2111.1268, 2021.
  * [17] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.  Deep networks with stochastic depth.  In ECCV, 2016.
  * [18] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.  Perceiver: General perception with iterative attention.  In ICML, 2021.
  * [19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [20] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In ECCV, 2022.
  * [21] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  In arXiv:1705.06950, 2017.
  * [22] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In ICCV, 2017.
  * [23] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre.  Hmdb: A large video database for human motion recognition.  In ICCV, 2011.
  * [24] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In ICML, 2019.
  * [25] Jie Lei, Tamara L. Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning.  In arXiv:2206.03428, 2022.
  * [26] Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L. Berg, and Mohit Bansal.  Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning.  In ACL, 2020.
  * [27] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao.  Uniformer: Unifying convolution and self-attention for visual recognition.  In ICLR, 2022.
  * [28] Chin-Yew Lin.  Rouge: a package for automatic evaluation of summaries.  In ACL workshop, 2004.
  * [29] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Frozen clip models are efficient video learners.  In arXiv:2208.03550, 2022.
  * [30] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In CVPR, 2022.
  * [31] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  In arXiv:2002.06353, 2020.
  * [32] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An empirical study of clip for end to end video clip retrieval.  In arXiv:2104.08860, 2021.
  * [33] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, 2019.
  * [34] Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid.  Learning audio-video modalities from image captions.  In ECCV, 2022.
  * [35] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  In ECCV, 2022.
  * [36] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In ACL, 2002.
  * [37] Jesús Andrés Portillo-Quintero, José Carlos Ortiz-Bayliss, and Hugo Terashima-Marín.  A straightforward framework for video retrieval using clip.  In arXiv:2102.12443, 2021.
  * [38] Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui.  Multimodal open-vocabulary video classification via pre-trained vision and language models.  In arXiv:2207.07646, 2022.
  * [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [40] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.  End-to-end generative pretraining for multimodal video captioning.  In CVPR, 2022.
  * [41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [42] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In arXiv:1804.04235, 2018.
  * [43] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In ECCV, 2016.
  * [44] Alok Singh, Thoudam Doren Singh, and Sivaji Bandyopadhyay.  Nits-vc system for vatex video captioning challenge 2020.  In arXiv:2006.04058, 2020.
  * [45] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  In arXiv:1212.0402, 2012.
  * [46] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [47] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, 2015.
  * [48] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Ge Yuying, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  In arXiv:2203.07303, 2022.
  * [49] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan.  Omnivl: One foundation model for image-language and video-language tasks.  In arXiv:2209.07526, 2022.
  * [50] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  In TMLR, 2022.
  * [51] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  Actionclip: A new paradigm for video action recognition.  In arXiv:2109.08472, 2021.
  * [52] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  In arXiv:2208.10442, 2022.
  * [53] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [54] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In ICCV, 2019.
  * [55] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao.  Internvideo: General video foundation models via generative and discriminative learning.  In arXiv:2212.03191, 2022.
  * [56] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  In ICLR, 2022.
  * [57] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In ECCV, 2018.
  * [58] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In ACM Multimedia, 2017.
  * [59] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer.  Vlm: Task-agnostic video-language model pre-training for video understanding.  In ACL Findings, 2021.
  * [60] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In EMNLP, 2021.
  * [61] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In CVPR, 2016.
  * [62] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo.  Clip-vip: Adapting pre-trained image-text model to video-language representation alignment.  In arXiv:2209.06430, 2022.
  * [63] Kashu Yamazaki, Sang Truong, Khoa Vo, Michael Kidd, Chase Rainwater, Khoa Luu, and Ngan Le.  Vlcap: Vision-language with contrastive learning for coherent video paragraph captioning.  In ICIP, 2022.
  * [64] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.  Multiview transformers for video recognition.  In CVPR, 2022.
  * [65] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In ICCV, 2021.
  * [66] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  In NeurIPS, 2022.
  * [67] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao.  Taco: Token-aware cascade contrastive learning for video-text alignment.  In ICCV, 2021.
  * [68] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  In TMLR, 2022.
  * [69] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In AAAI, 2019.
  * [70] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  In arXiv:2111.11432, 2021.
  * [71] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Socratic models: Composing zero-shot multimodal reasoning with language.  In arXiv:2204.00598, 2022.
  * [72] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In CVPR, 2022.
  * [73] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * [74] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha.  Object relational graph with teacher-recommended learning for video captioning.  In CVPR, 2020.
  * [75] Luowei Zhou, Chenliang Xu, and Jason J. Corso.  Towards automatic learning of procedures from web instructional videos.  In AAAI, 2018.

##  Appendix A Varying the number of input frames.

The ablations on the zero-shot transfer performance with the increasing number
of frames are shown in Table 9. We evaluate on the Kinetics-400 and MSR-VTT
datasets and report zero-shot video classification accuracies and text-to-
video retrieval R@K. Results are obtained by VideoCoCa-Small and VideoCoCa-
Base without continued pretraining on VideoCC3M. The image resolution
$224\times 224$ and patch size $16\times 16$ is used. We observe that the
performance gain on both video classification and text-to-video retrieval
becomes smaller starting from 8 frames, and this is consistent across two
model scales. Therefore, we keep 8 frames for each video clip to make the
larger model trained faster and save more resources.

##  Appendix B Finetuning hyperparameters.

Table 10 details the hyperparamters used in all of our VideoCoCa finetuning
experiments for video captioning and video question answering. We use
Adafactor [42] optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.99$ and decoupled
weight decay ratio of 1e-4, a cosine learning rate schedule with linear
warmup, and a batch size of $64$ or $128$ for our experiments. We precompute
the frame embeddings with only center cropping applied on frames. We do not
use other data augmentation methods (_e.g._ random flip, color jittering,
random augment [12]) and other regularizations (_e.g._ mixup [73], label
smoothing [46], stochastic layer drop [17]). For video question answering, we
additionally learn a fully connected layer (with identity activation)
optimized with Softmax loss, and the learning rate of the classifier is set to
be the same as the base learning rate for the decoder and the poolers.

Table 9: Comparisons of zero-shot transfer performance with varying the number
of frames on Kinetics-400 and MSR-VTT.

Model |  Num. Frames ($T$) | VideoCls. | VideoRet.
---|---|---|---
Top-1 | Top-5 | R@1 | R@5 | R@10
VideoCoCa-S (w.o. videos) | 1 | 33.1 | 59.2 | 13.5 | 29.3 | 37.6
2 | 38.5 | 66.0 | 16.1 | 34.6 | 44.0
4 | 42.1 | 70.4 | 21.7 | 42.8 | 52.8
8 | 43.5 | 71.8 | 23.6 | 45.2 | 55.2
16 | 44.2 | 72.3 | 24.3 | 45.5 | 55.7
32 | 44.2 | 72.3 | 24.0 | 45.7 | 55.8
64 | 44.3 | 72.4 | 24.1 | 45.6 | 55.6
| 128 | 44.2 | 72.4 | 24.0 | 45.5 | 55.6
VideoCoCa-B (w.o videos) | 1 | 49.4 | 72.8 | 18.5 | 36.5 | 44.9
2 | 55.7 | 79.1 | 21.5 | 42.2 | 51.6
4 | 59.3 | 82.5 | 24.7 | 47.3 | 57.4
8 | 60.7 | 83.4 | 30.3 | 53.3 | 62.5
16 | 61.1 | 83.8 | 30.7 | 53.6 | 62.7
32 | 61.0 | 83.7 | 30.5 | 53.6 | 62.7
64 | 61.1 | 83.8 | 30.5 | 53.6 | 62.6

Table 10: Finetuning hyperparamters of VideoCoCa for video captioning and video question answering. We only use the center cropping for each video frame. We do not use other data augmentation methods (_e.g._ random flip, color jittering, random augment) and other regularizations (_e.g._ mixup, label smoothing, stochastic layer drop). Values which are constant across all columns are listed once. Datasets are denoted as follows: MSRVTT, ActivityNet-Captions, VATEX and YouCook2 for video captioning, IVQA, MSRVTT-QA, MSVD-QA, and ActivityNet-QA for video question answering. | MSRVTT | ActivityNet-Captions | VATEX | YouCook2 | IVQA | MSRVTT-QA | MSVD-QA | ActivityNet-QA
---|---|---|---|---|---|---|---|---
Optimizer | Adafactor with decoupled weight decay
Gradient clip | 1.0
EMA decay rate | 0.99
Weight decay rate | 1e-4
Learning rate schedule | Cosine with linear warmup
Linear warmup epochs | 1000
Batch size | 64 | 64 | 64 | 64 | 64 | 128 | 128 | 128
Base learning rate | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 5e-6 | 1e-5 | 1e-5 | 1e-5
Classifier learning rate | – | – | – | – | 5e-6 | 1e-5 | 1e-5 | 1e-5
Training steps | 5000 | 5000 | 10000 | 10000 | 10000 | 10000 | 10000 | 10000