# Multiview Transformers for Video Recognition

Shen Yan‡ Xuehan Xiong† Anurag Arnab† Zhichao Lu† Mi Zhang‡
Chen Sun†§ Cordelia Schmid†
†Google Research ‡Michigan State University §Brown University
{xxman, aarnab, lzc, chensun<EMAIL_ADDRESS>{yanshen6,
<EMAIL_ADDRESS>
This work was done while the first author was an intern at Google.

###### Abstract

Video understanding requires reasoning at multiple spatiotemporal resolutions
– from short fine-grained motions to events taking place over longer
durations. Although transformer architectures have recently advanced the
state-of-the-art, they have not explicitly modelled different spatiotemporal
resolutions. To this end, we present Multiview Transformers for Video
Recognition (MTV). Our model consists of separate encoders to represent
different views of the input video with lateral connections to fuse
information across views. We present thorough ablation studies of our model
and show that MTV consistently performs better than single-view counterparts
in terms of accuracy and computational cost across a range of model sizes.
Furthermore, we achieve state-of-the-art results on six standard datasets, and
improve even further with large-scale pretraining. Code and checkpoints are
available at: https://github.com/google-research/scenic.

##  1 Introduction

Vision architectures based on convolutional neural networks (CNNs), and now
more recently transformers, have made great advances in numerous computer
vision tasks. A central idea, that has remained constant across classical
methods based on handcrafted features [9, 14, 38] to CNNs [43, 42, 84] and now
transformers [73, 11, 44], has been to analyze input signals at multiple
resolutions.

In the image domain, multiscale processing is typically performed with
pyramids as the statistics of natural images are isotropic (all orientations
are equally likely) and shift invariant [30, 66]. To model multiscale temporal
information in videos, previous approaches such as SlowFast [23] have
processed videos with two streams, using a “Fast” stream operating at high
frame rates and a “Slow” stream at low frame rates, or employed graph neural
networks to model long-range interactions [76, 4].

Figure 1:  Overview of our Multiview Transformer. We create multiple input
representations, or “views”, of the input, by tokenizing the video using
tubelets of different sizes (for clarity, we show two views here). These
tokens are then processed by separate encoder streams, which include lateral
connections and a final global encoder to fuse information from different
views. Note that the tokens from each view may have different hidden sizes,
and the encoders used to process them can vary in architecture too.

When creating a pyramidal structure, spatio-temporal information is partially
lost due to its pooling or subsampling operations. For example, when
constructing the “Slow” stream, SlowFast [23] subsamples frames, losing
temporal information. In this work, we propose a simple transformer-based
model without relying on pyramidal structures or subsampling the inputs to
capture multi-resolution temporal context. We do so by leveraging multiple
input representations, or “views” of the input video. As shown in Fig. 1, we
extract tokens from the input video over multiple temporal durations.
Intuitively, tokens extracted from long time intervals capture the gist of the
scene (such as the background where the activity is taking place), whilst
tokens extracted from short segments can capture fine-grained details (such as
the gestures performed by a person).

We propose a multiview transformer (Fig. 1) to process these tokens, and it
consists of separate transformer encoders specialized for each “view”, with
lateral connections between them to fuse information from different views to
each other. We can use transformer encoders of varying sizes to process each
view, and find that it is better (in terms of accuracy/computation trade-offs)
to use a smaller encoder (_e.g_. smaller hidden sizes and fewer layers) to
represent the broader view of the video (Fig. 1 left) while an encoder with
larger capacity is used to capture the details (Fig. 1 right). This design
therefore poses a clear contrast to pyramid-based approaches where model
complexity increases as the spatio-temporal resolution decreases. Our design
is verified by our experiments which show clear advantages over the former
approach.

Our proposed method, of processing different “views” of the input video is
simple, and in contrast to previous work [23] generalizes readily to a
variable number of views. This is significant, as our experiments show that
accuracy increases as the number of views grows. Although our proposed
architecture increases the number of tokens processed by the network according
to the number of input views, we show that we can consistently achieve
superior accuracy/computation trade-offs compared to the current state of the
art [3], across a spectrum of model sizes, ranging from “Small” to “Huge”. We
show empirically that this is because processing more views in parallel
enables us to achieve larger accuracy improvements than increasing the depth
of the transformer network. We perform thorough ablation studies of our design
choices, and achieve state-of-the-art results on six standard video
classification datasets. Moreover, we show that these results can be further
improved with large-scale pretraining.

##  2 Related Work

##### Evolution of video understanding models

Early works [37, 34, 72] relied on hand-crafted features to encode motion and
appearance information. With the emergence of large labelled datasets like
ImageNet [16], Convolutional Neural Networks (CNNs) [39] showed their
superiority over the classic methods. Since AlexNet [36] won the ImageNet
challenge by a large margin, CNNs have been quickly adopted to various vision
tasks, their architectures have been refined over many generations [58, 63,
27, 12] and later improved by Neural Architecture Search (NAS) [54, 85, 65].
At the same time, CNNs and RNNs have quickly become the de-facto backbones for
video understanding tasks [32, 57, 50]. Since the release of the Kinetics
dataset [33], 3D CNNs [68, 24, 10] have gained popularity, and many variants
[62, 70, 78, 69, 22] have been developed to improve the speed and accuracy.
Convolution operations can only process one local neighborhood at a time, and
consequently, transformer blocks [71] have been inserted into CNNs as
additional layers to improve modeling of long range interactions among spatio-
temporal features [74, 75]. Although achieving great success in natural
language [17, 8, 53], pure transformer architectures had not gained the same
popularity in computer vision until Vision Transformers (ViT) [18]. Inspired
by ViT, ViViT [3] and Timesformer [6] were the first two works that
successfully adopted a pure transformer architecture for video classification,
advancing the state of the art previously set by 3D CNNs.

##### Multiscale processing in computer vision

“Pyramid” structures [1] are one of the most popular multiscale
representations for images and have been key in the early computer vision
works, where their use has been widespread in multiple domains including
feature descriptors [45], feature tracking [46, 7], image compression [9],
etc. This idea has also been successfully adopted for modern CNNs [58, 63, 27]
where the spatial dimension of the network is gradually reduced while the
network “depth” is gradually increased to encode more semantically rich
features. Also, this technique has been used to produce higher resolution
output features for downstream tasks[42, 84, 43]. Multiscale processing is
necessary for CNNs because a convolution operation only operates on a sub-
region of the input and a hierarchical structure is required to capture the
whole view of the image or video. In theory, such a hierarchy is not required
for transformers as each token “attends” to all other positions. In practice,
due to the limited amount of training data, applying similar multiscale
processing in transformers [21, 44, 11, 73] to reduce complexity of the model
has proven to be effective.

Our model does not follow the pyramid structure but directly takes different
views of the video and feeds them into cross-view encoders. As our experiments
validate, this alternative multiview architecture has consistently
outperformed its single-view counterpart in terms of accuracy/FLOP trade-offs.
This is because processing more views in parallel gives us larger accuracy
improvements than increasing the depth of the transformer network.
Significantly, such improvement persists as we scale the model capacity to
over a billion parameters (_e.g_., our “Huge” model), which has not been shown
by the previous pyramid-structured transformers [21, 44, 73]. Conceptually,
our method is most comparable to SlowFast [23] where a two-stream CNN is used
to process two views of the same video clip (densely sampled and sparsely
sampled frames). Instead of sampling the input video at different frame rates,
we obtain different view by linearly projecting spatio-temporal “tubelets” [3]
of varying sizes for each view. Furthermore, we empirically show that our
proposed method outperforms [23] when using transformer backbones.

##  3 Multiview Transformers for Video

We begin with an overview of vision transformer, ViT [18], and its extension
to video, ViViT [3], which our model is based on, in Sec. 3.1. As shown in
Fig. 1, our model constructs different “views” of the input video by
extracting tokens from spatio-temporal tubelets of varying dimensions (Sec.
3.2). These tokens are then processed by a multiview transformer, which
incorporates lateral connections to efficiently fuse together information from
multiple scales (Sec. 3.3).

###  3.1 Preliminaries: ViT and ViViT

We denote our input video as $\mathbf{V}\in\mathbb{R}^{T\times H\times W\times
C}$. Transformer architectures [71] process inputs by converting inputs into
discrete tokens which are subsequently processed by multiple transformer
layers sequentially.

ViT [18] extracts tokens from images by partitioning an image into non-
overlapping patches and linearly projecting them. ViViT [4] extends this to
video by extracting $N$ non-overlapping, spatio-temporal “tubes”[3] from the
input video, $x_{1},x_{2},\ldots x_{N}\in\mathbb{R}^{t\times h\times w\times
c}$ where
$N=\lfloor\frac{T}{t}\rfloor\times\lfloor\frac{H}{h}\rfloor\times\lfloor\frac{W}{w}\rfloor$.

Each tube, $x_{i}$, is then projected into a token,
$\mathbf{z}_{i}\in\mathbb{R}^{d}$ by a linear operator $\mathbf{E}$, as
$\mathbf{z}_{i}=\mathbf{E}x_{i}$. All tokens are then concatenated together to
form a sequence, which is prepended with a learnable class token
$\mathbf{z}_{cls}\in\mathbb{R}^{d}$ [17]. As transformers are permutation
invariant, a positional embedding $\mathbf{p}\in\mathbb{R}^{(N+1)\times d}$,
is also added to this sequence. Therefore, this tokenization process can be
denoted as

$\mathbf{z}^{0}=[\mathbf{z}_{cls},\mathbf{E}x_{1},\mathbf{E}x_{2},\ldots,\mathbf{E}x_{N}]+\mathbf{p}.$
(1)

Note that the linear projection $\mathbf{E}$ can also be seen as a 3D
convolution with a kernel of size $t\times h\times w$ and stride of $(t,h,w)$
in the time, height and width dimensions respectively.

The sequence of tokens $\mathbf{z}$ is then processed by a transformer encoder
consisting of $L$ layers. Each layer, $\ell$, is applied sequentially, and
consists of the following operations,

$\displaystyle\mathbf{y}^{\ell}$
$\displaystyle=\text{MSA}\left(\text{LN}\left(\mathbf{z}^{\ell-1}\right)\right)+\mathbf{z}^{\ell-1},$
(2) $\displaystyle\mathbf{z}^{\ell}$
$\displaystyle=\text{MLP}\left(\text{LN}\left(\mathbf{y}^{\ell}\right)\right)+\mathbf{y}^{\ell}$
(3)

where MSA denotes multi-head self-attention [71], LN is layer normalization
[5] and MLP consists of two linear projections separated by GeLU [28] non-
linearity.

Finally, a linear classifier, $\mathbf{W}^{\text{out}}\in\mathbb{R}^{d\times
C}$ maps the encoded classification token, $\textbf{z}^{\ell}_{cls}$ to one of
$C$ classes.

(a) An example of CVA for fusion.

(b) An example of bottleneck tokens for fusion.

(c) An example of MLP fusion.

Figure 2: An illustration of our proposed cross-view fusion methods. In all
three subfigures, view $i$ (left) refers to a video representation using
larger tubelets, and thus less input tokens and view $i+1$ (right) corresponds
to the representation with smaller tubelets and more input tokens. “$+$”
denotes summation. Tokens extracted from tubelets are colored red and
bottleneck tokens are colored blue. MSA is short for Multihead Self-Attention
and CVA stands for Cross View Attention.

###  3.2 Multiview tokenization

In our model, we extract multiple sets of tokens,
$\mathbf{z}^{0,(1)},\mathbf{z}^{0,(2)},\ldots,\mathbf{z}^{0,(V)}$ from the
input video. Here, $V$ is the number of views, and thus
$\mathbf{z}^{\ell,(i)}$ denotes tokens after $\ell$ layers of transformer
processing for the $i^{th}$ view. We define a view as a video representation
expressed by a set of fixed-sized tubelets. A larger view corresponds to a set
of larger tubelets (and thus fewer tokens) and a smaller view corresponds to
smaller tubelets (and thus more tokens). The $0^{th}$ layer corresponds to the
tokens that are input to the subsequent transformer. As shown in Fig. 1, we
tokenize each view using a 3D convolution, as it was the best tokenization
method reported by [3]. We can use different convolutional kernels, and
different hidden sizes, $d^{(i)}$, for each view. Note that smaller
convolutional kernels correspond to smaller spatio-temporal “tubelets”, thus
resulting in more tokens to be processed for the $i^{th}$ view. Intuitively,
fine-grained motions can be captured by smaller tubelets whilst larger
tubelets capture slowly-varying semantics of the scene. As each view captures
different levels of information, we use transformer encoders of varying
capacities for each stream with lateral connections between them to fuse
information, as described in the next section.

###  3.3 Multiview transformer

After extracting tokens from multiple views, we have
$\mathbf{Z}^{0}=[\mathbf{z}^{0,(1)},\mathbf{z}^{0,(2)},\ldots,\mathbf{z}^{0,(V)}]$
from the input, which are processed with a multiview transformer as shown in
Fig. 1. As self-attention has quadratic complexity [71], processing tokens
from all views jointly is not computationally feasible for video. As a result,
we first use a multiview encoder, comprising of separate transformer encoders
(consisting of $L^{(i)}$ transformer layers) for the tokens between views,
with lateral connections between these encoders to fuse information from each
view (Fig. 2). Finally, we extract a token representation from each view, and
process these jointly with a final global encoder to produce the final
classification token, which we linearly read-off to obtain the final
classification.

####  3.3.1 Multiview encoder

Our multiview encoder consists of separate transformer encoders for each view
which are connected by lateral connections to fuse cross-view information.
Each transformer layer within the encoders follows the same design as the
original transformer of Vaswani  _et al_. [71], except for the fact that we
optionally fuse information from other streams within the layer as described
in Sec. 3.3.2. Note that our model is agnostic to the exact type of
transformer layer used. Furthermore, within each transformer layer, we compute
self-attention only among tokens extracted from the same temporal index,
following the Factorised Encoder of [3]. This significantly reduces the
computational cost of the model. Furthermore, self-attention along all spatio-
temporal tokens is unnecessary, as we fuse information from other views within
the multiview encoder, and also because of the subsequent global encoder which
aggregates tokens from all streams.

####  3.3.2 Cross-view fusion

We consider the following three cross-view fusion methods. Note that the
hidden dimensions of the tokens, $d^{(i)}$, can vary between views.

##### Cross-view attention (CVA)

A straight-forward method of combining information between different views is
to perform self-attention jointly on all $\sum_{i}N^{(i)}$ tokens where
$N^{(i)}$ is the number of tokens in the $i^{th}$ view. However, due to the
quadratic complexity of self-attention, this is prohibitive computationally
for video models, and hence we perform a more efficient alternative.

We sequentially fuse information between all pairs of two adjacent views, $i$
and $i+1$, where the views are ordered in terms of increasing numbers of
tokens (_i.e_. $N^{(i)}\leq N^{(i+1)}$). Concretely, to update the tokens from
the larger view, $\mathbf{z}^{(i)}$, we compute attention where the queries
are $\mathbf{z}^{(i)}$, and the keys and values are $\mathbf{z}^{(i+1)}$ (the
tokens from the smaller view). As the hidden dimensions of the tokens between
the two views can be different, we first project the keys and values to the
same dimension, as denoted by

$\displaystyle\mathbf{z}^{(i)}$
$\displaystyle=\text{CVA}(\mathbf{z}^{(i)},\mathbf{W}^{\text{proj}}\mathbf{z}^{(i+1)}),$
(4) $\displaystyle\text{CVA}(\mathbf{x},\mathbf{y})$
$\displaystyle=\text{Softmax}\left(\frac{\mathbf{W}^{Q}\mathbf{x}\mathbf{W}^{K}\mathbf{y}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{W}^{V}\mathbf{y}.$
(5)

Note that $\mathbf{W}^{Q}$, $\mathbf{W}^{K}$ and $\mathbf{W}^{V}$ are the
query-, key- and value-projection matrices used in the attention operation
[71]. As shown in Fig. 2(a), we also include a residual connection around the
cross-view attention operation, and zero-initialize the parameters of this
operation, as this helps when using image-pretrained models as is common
practice [3, 6]. Similar studies on cross stream attention have been done by
[11] for images.

##### Bottleneck tokens

An efficient method of transferring information between tokens from two views,
$\mathbf{z}^{(i)}$ and $\mathbf{z}^{(i+1)}$, is by an intermediate set of $B$
bottleneck tokens. Once again, we sequentially fuse information between all
pairs of two adjacent views, $i+1$ and $i$, where the views are ordered in
terms of increasing numbers of tokens.

In more detail, we initialize a sequence of bottleneck tokens,
$\mathbf{z}_{B}^{(i+1)}\in\mathbb{R}^{B^{(i+1)}\times d^{(i+1)}}$ where
$B^{(i+1)}$ is the number of bottleneck tokens in the $(i+1)^{th}$ view and
$B^{(i+1)}\ll N^{(i+1)}$. As shown in Fig. 2(b) (where $B=1$), the bottleneck
tokens from view $i+1$, $\mathbf{z}_{B}^{(i+1)}$, are concatenated to the
input tokens of the same view, $\mathbf{z}^{(i+1)}$, and processed with self-
attention. This effectively transfers information between all tokens from view
$i+1$. Thereafter, these tokens, $\mathbf{z}_{B}^{(i+1)}$ are linearly
projected to the depth of view $i$, and concatenated to $\mathbf{z}^{(i)}$
before performing self-attention again. This process is repeated between each
pair of adjacent views as shown in Fig. 2(b), and allows us to efficiently
transfer information from one view to the next.

As with cross-view attention, we sequentially perform fusion between all pairs
of adjacent views, beginning from the view with the largest number of tokens,
and proceeding in order of decreasing token numbers. Intuitively, this allows
the view with the fewest tokens to aggregate fine-grained information from all
subsequent views.

Note that the only parameters introduced into the model from this fusion
method are the linear projections of bottleneck tokens from one view to the
next, and the bottleneck tokens themselves which are learned from random
initialization. We also note that “bottleneck” tokens have also been used by
[31, 49].

##### MLP fusion

Recall that each transformer encoder layer consists of a multi-head self
attention operation (Eq. 2), followed by an MLP block (Eq. 3). A simple method
is to fuse before the MLP block within each encoder layer.

Concretely, as shown in Fig. 2(c), tokens from view $i+1$,
$\mathbf{z}^{(i+1)}$ with hidden dimension $d^{(i+1)}$ are concatenated with
tokens from view $i$ along the hidden dimension. These tokens are then fed
into the MLP block of layer $i$ and linearly projected to the depth $d^{(i)}$.
This process is repeated between adjacent views of the network, where once
again, views are ordered by increasing number of tokens per view.

##### Fusion locations

We note that it is not necessary to perform cross-view fusion at each layer of
the cross-view encoder to transfer information among the different views,
since each fusion operation has a global “receptive field” that considers all
the tokens from the previous views. Furthermore, it is also possible for the
encoders for each individual view to have different depths, meaning that
fusion can occur between layer $l$ of view $i$ and layer $l^{\prime}$ of view
$j$ where $l\neq l^{\prime}$. Therefore, we consider the fusion locations as a
design choice which we perform ablation studies on.

####  3.3.3 Global encoder

Finally, we aggregate the tokens from each of the views with the final global
encoder, as shown in Fig. 1, effectively fusing information from all views
after the cross-view transformer. We extract the classification token from
each view, $\\{\mathbf{z}_{cls}^{(i)}\\}_{i=1}^{V}$, and process them further
with another transformer encoder, following Vaswani  _et al_. [71], that
aggregates information from all views. The resulting classification token is
then mapped to one of $C$ classification outputs, where $C$ is the number of
classes.

##  4 Experiments

###  4.1 Experimental setup

##### Model variants

For the backbone of each view, we consider five ViT variants, “Tiny”, “Small”,
“Base”, “Large”, and “Huge”. Their settings strictly follow the ones defined
in BERT [17] and ViT [18, 59], _i.e_. number of transformer layers, number of
attention heads, hidden dimensions. See Appendix A.4 for the detailed
settings. For convenience, each model variant is denoted with the following
abbreviations indicating the backbone size and tubelet length. For example,
B/2+S/4+Ti/8 denotes a three-view model, where a “Base”, “Small”, and “Tiny”
encoders are used to processes tokens from the views with tubelets of sizes
$16\times 16\times 2$, $16\times 16\times 4$, and $16\times 16\times 8$,
respectively. Note that we omit 16 in our model abbreviations because all our
models use $16\times 16$ as the spatial tubelet size except for the “Huge”
model, which uses $14\times 14$, following ViT [18]. All model variants use
the same global encoder which follows the “Base” architecture, except that the
number of heads is set to 8 instead of 12. The reason is that the hidden
dimension of the tokens should be divisible by the number of heads for multi-
head attention, and the number of hidden dimensions across all standard
transformer architectures (from “Tiny” to “Huge” [59, 18]) is divisible by 8.

(a) Effects of different model-view assignments.

Model variants | GFLOPs | MParams | Top-1
---|---|---|---
B/8+Ti/2 | 81 | 161 | 77.3
B/2+Ti/8 | 337 | 221 | 81.3
B/8+S/4+Ti/2 | 202 | 250 | 78.5
B/2+S/4+Ti/8 | 384 | 310 | 81.8
B/4+S/8+Ti/16 | 195 | 314 | 81.1

(b) Effects of the same model applied to different views.

Model variants | GFLOPs | MParams | Top-1
---|---|---|---
B/4+S/8+Ti/16 | 195 | 314 | 81.1
B/4+B/8+B/16 | 324 | 759 | 81.1
B/2+Ti/8 | 337 | 221 | 81.3
B/2+B/8 | 448 | 465 | 81.5
B/2+S/4+Ti/8 | 384 | 310 | 81.8
B/2+B/4+B/8 | 637 | 751 | 81.7

(c) Comparison of different cross-view fusion methods.

Model variants | Method | GFLOPs | MParams | Top-1
---|---|---|---|---
B/4 | N/A | 145 | 173 | 78.3
S/8 | 20 | 60 | 74.1
Ti/16 | 3 | 13 | 67.6
B/4+S/8+Ti/16 | Ensemble | 168 | 246 | 77.7
Late fusion | 187 | 306 | 80.6
MLP | 202 | 323 | 80.6
Bottleneck | 188 | 306 | 81.0
CVA | 195 | 314 | 81.1

(d) Comparison to SlowFast multi-resolution method.

Model variants | GFLOPs | MParams | Top-1
---|---|---|---
SlowFast (transformer backbone) |
Slow-only (B) | 79 | 87 | 78.0
Fast-only (Ti) | 63 | 6 | 74.6
Slowfast (B+Ti) | 202 | 105 | 79.7
B/4+Ti/16 (ours) | 168 | 224 | 80.8

(e) Effects of increasing number of views.

Model variants | GFLOPs | Top-1
---|---|---
B/4 | 145 | 78.3
B/4+Ti/16 | 168 |  80.8 (+2.5)
B/4+S/8+Ti/16 | 195 |  81.1 (+2.8)
B/4 (14) | 168 |  78.1 (-0.2)
B/4 (17) | 203 |  78.4 (+0.1)

(f) Effects of applying CVA at different layers.

Fusion layers | GFLOPs | MParams | Top-1
---|---|---|---
0 | 195 | 314 | 80.96
5 | 81.08
11 | 81.00
0, 1 | 203 | 323 | 80.91
5, 6 | 80.96
10, 11 | 80.81
5, 11 | 81.14
0, 5, 11 | 210 | 331 | 80.95

Table 1: Ablation studies of our method. (a) Assigning larger models to
smaller tubelet sizes achieves the highest accuracy. (b) We apply the same
“Base” encoder to all views, and show that there is minimal accuracy
difference to the alternatives from (a), but a large increase in computation.
(c) A comparison of different cross-view fusion methods, shows that Cross-View
Attention (CVA) is the best. The “Ensemble” and “late fusion” baselines are
detailed in the text. (d) We compare our approach to the alternate temporal
multi-resolution method of [23], implemented in the context of transformers,
and show signficant improvements. (e) We achieve substantial accuracy by
adding more views, and this improvement is larger than that obtained by adding
more layers to a single encoder. (f) The optimal fusion layers are at the
middle and late stages of the network.

##### Training and inference

We follow the training settings of ViViT reported in the paper and public code
[3], unless otherwise stated. Namely, all models are trained on 32 frames with
a temporal stride of 2. We train our model using synchronous SGD with momentum
of 0.9 following a cosine learning rate schedule with a linear warm up. The
input frame resolution is set to be $224\times 224$ in both training and
inference. We follow [3] and apply the same data augmentation and
regularization schemes [29, 82, 13, 64], which were used by [67] to train
vision transformers more effectively. During inference, we adopt the standard
evaluation protocol by averaging over multiple spatial and temporal crops. The
number of crops is given in the results tables. For reproducibility, we
include exhaustive details in Appendix A.3.

##### Initialization

Following previous works [3, 6, 51], we initialize our model from a
corresponding ViT model pretrained on large-scale image datasets [16, 61]
obtained from the public code of [18]. The initial tubelet embedding operator,
$\mathbf{E}$, and positional embeddings, $\mathbf{p}$, have different shapes
in the pretrained model and we use the same technique as [3] to adapt them to
initialize each view of our multiview encoder (Sec. 3.3.1). The final global
encoder (Sec. 3.3.3) is randomly initialized.

##### Datasets

We report the performance of our proposed models on a diverse set of video
classification datasets:

_Kinetics_ [33] is a collection of large-scale, high-quality datasets of 10s
video clips focusing on human actions. We report results on Kinetics 400, 600,
and 700, with 400, 600, and 700 classes, respectively.

_Moments in Time_ [48] is a collection of 800,000 labeled 3 second videos,
involving people, animals, objects or natural phenomena, that capture the gist
of a dynamic scene.

_Epic-Kitchens-100_ [15] consists of 90,000 egocentric videos, totaling 100
hours, recorded in kitchens. Each video is labeled with a “noun” and a “verb”
and therefore we predict both categories using a single network with two
“heads”. Three accuracy scores (“noun”, “verb”, and “action”) are commonly
reported for this dataset with action accuracy being the primary metric. The
“action” label is formed by selecting the top-scoring noun and verb pair.

_Something-Something V2_ [26] consists of more than 220,000 short video clips
that show humans interacting with everyday objects. Similar objects and
backgrounds appear in videos across different classes. Therefore, in contrast
to other datasets, this one challenges a model’s capability to distinguish
classes from motion cues.

###  4.2 Ablation study

We conduct ablation studies on the Kinetics 400 dataset. In all cases, the
largest backbone in the multiview encoder is “Base” for faster
experimentation. We report accuracies when averaging predictions across
multiple spatio-temporal crops, as standard practice [3, 6, 23, 10]. In
particular, we use $4\times 3$ crops, that is 4 temporal crops, with 3 spatial
crops for each temporal crop. We used a learning rate of 0.1 for all
experiments for 30 epochs, and used no additional regularization as done by
[3].

(a) Accuracy[%] - GFLOPs comparison between MTV and ViViT-FE.

(b) Accuracy[%] - Throughput comparison between MTV and ViViT-FE.

Figure 3:  Accuracy/computation trade-off between ViViT-FE [3] (blue) and our
MTV (red). Figure 3(a) shows that MTV is consistently better and requires less
FLOPs than ViViT-FE to achieve higher accuracy across different model scales
(shown by the dotted green arrows pointing upper-left). With additional FLOPs,
MTV shows larger accuracy gains (shown by the dotted green arrows pointing
upper-right). Similarly, Fig. 3(b) shows that MTV can have higher throughput
than ViVIT-FE, whilst still improving its accuracy, across all model scales.
All speed comparisons are measured with the same hardware (Cloud TPU-v4),
whilst the accuracy is computed from $4\times 3$ view testing.

##### Model-view assignments

Recall that a view is a video representation in terms of tubelets, and that a
larger view equates to larger tubelets (and hence fewer transformer tokens)
and smaller views correspond to smaller tubelets (and thus more tokens).

We considered two model-view assignment strategies: larger models for larger
views (_e.g_., B/8+Ti/2, the larger “Base” model is used to encode $16\times
16\times 8$ tubelets and the smaller “Tiny” model encodes $16\times 16\times
2$ tubelets) and smaller models for larger views (_e.g_., B/2+Ti/8). Table
1(b) shows that assigning a larger model to smaller views is superior. For
example, B/2+S/4+Ti/8 scores 81.8% while B/8+S/4+Ti/2 only scores 78.5%. One
may argue that this is due to the increase of the FLOPs but B/4+S/8+Ti/16
still outperforms B/8+S/4+Ti/2 by a large margin under similar FLOPs. Our
explanation is that larger views capture the gist of the scene, which requires
less complexity to learn while the details of the scene are encapsulated by
smaller views so a larger-capacity model is needed.

Another strategy is to assign the same model to all views. Table 1(b) shows
that in all three examples there is little difference between assigning a
“Base” model and assigning a “Small” or “Tiny” model to larger views. This
result is surprising yet beneficial since we can reduce the complexity of the
model at almost no cost of accuracy.

##### What is the best cross-view fusion method?

Table 1(d) shows the comparison of different fusion methods on a three-view
model. We use one late fusion and an ensemble approach as the baselines.
“Ensemble” simply sums the probabilities produced from each view, where the
models from each view are trained separately. We also tried summing up the
logits and majority voting but both obtained worse results. This method
actually decreases the performance compared to the B/4 model since “Small” and
“Tiny” models perform not comparably well. “Late fusion” concatenates the
final embeddings produced by the transformer encoder from each view without
any cross-view operations before feeding it into the global encoder. It
improves the B/4 model from 78.3% to 80.6%. All of our fusion methods except
MLP outperform the baselines while CVA is the best overall. Based on this
observation, we choose CVA as the fusion method for all subsequent
experiments. MLP fusion is the worst performing method of the three and we
think it is because concatenation in the MLP blocks introduces additional
channels that have to be randomly initialized, making model optimization more
difficult.

##### Effect of the number of views

Table 1(f) shows performance on Kinetics-400 as we increase the number of
views. With two views we achieve a +2.5% in Top-1 accuracy over the baseline
B/4 model. As we increase to three views, the improvement widens to 2.8%.
Furthermore, we show that such improvement is non-trivial. For example, we
also train a 14-layer and a 17-layer variants of the “Base” model. They share
similar FLOPs with our two-view and three-view counterparts but their
performance remains similar to that of the baseline.

##### Which layers to apply cross-view fusion?

Motivated by Tab. 1(d), we fix the fusion method to CVA, and vary the
locations and number of layers where we apply CVA, when using a three-view
B+S+Ti model (each encoder thus has 12 layers) in Tab. 1(f). The choices are
in the early-, mid-, and late-stages of the transformer encoders and the
number of fusion layers is set to be one and two. When using one fusion layer,
the best location for fusion is mid followed by late, then early. Adding more
fusion layers in the same stage does not improve the performance but combining
mid and late fusion improves the performance. For example, fusion at 5th and
11th layers achieve the best result. Based on this observation, we set the
fusion layers to be {11, 23} for L+B+S+Ti and {11, 23, 31} for H+B+S+Ti model
variants, respectively, in subsequent experiments.

##### Comparison to SlowFast

SlowFast [23] proposes a two-stream CNN architecture that takes frames sampled
at two different frame rates. The “Slow” pathway, built with a larger encoder,
processes the low frame rate stream to capture the semantics of the scene
while the “Fast” pathway that takes in high frame rate inputs is used to
capture motion information. To make a fair comparison, we implement [23] in
the context of transformers where we use “Base” and “Tiny” models as the
encoders for the Slow and Fast paths respectively and use CVA for lateral
connections. The Slow path takes four frames as inputs sampled with a temporal
stride of 16 and the Fast path takes 16 frames sampled with a stride of 4. As
SlowFast captures multiscale temporal information by varying the frame rate to
the two streams, the temporal duration for the tubelets is set to 1 in this
case. Table 1(d) shows that our method is significantly more accurate than the
SlowFast method whilst also using fewer FLOPs.

###  4.3 Comparison to the state of the art

Table 2: Comparisons to state-of-the-art. For “views”, $x\times y$ denotes $x$
temporal views and $y$ spatial views. We report the total TFLOPs to process
all spatio-temporal views. We use shorter notation, MTV-B, L, H to denote
variants, B/2+S/4+Ti/8, L/2+B/4+S/8+Ti/16, and H/2+B/4+S/8+Ti/16,
respectively. Models use a spatial resolution of $224\times 224$, unless
explicitly stated by MTV ($x$p), which refers to a spatial resolution of
$x\times x$. Models are pretrained on ImageNet-21K unless explicitly stated in
parenthesis.

(a) Kinetics 400 Method | Top 1 | Top 5 | Views | TFLOPs
---|---|---|---|---
TEA [40] | 76.1 | 92.5 | $10\times 3$ | 2.10
TSM-ResNeXt-101 [41] | 76.3 | – | – | –
I3D NL [74] | 77.7 | 93.3 | $10\times 3$ | 10.77
VidTR-L [83] | 79.1 | 93.9 | $10\times 3$ | 10.53
LGD-3D R101 [52] | 79.4 | 94.4 | – | –
SlowFast R101-NL [23] | 79.8 | 93.9 | $10\times 3$ | 7.02
X3D-XXL [22] | 80.4 | 94.6 | $10\times 3$ | 5.82
OmniSource [20] | 80.5 | 94.4 | – | –
TimeSformer-L [6] | 80.7 | 94.7 | $1\times 3$ | 7.14
MFormer-HR [51] | 81.1 | 95.2 | $10\times 3$ | 28.76
MViT-B [21] | 81.2 | 95.1 | $3\times 3$ | 4.10
MoViNet-A6 [35] | 81.5 | 95.3 | $1\times 1$ | 0.39
ViViT-L FE [3] | 81.7 | 93.8 | $1\times 3$ | 11.94
MTV-B | 81.8 | 95.0 | $4\times 3$ | 4.79
MTV-B (320p) | 82.4 | 95.2 | $4\times 3$ | 11.16
Methods with web-scale pretraining |
VATT-L [2] (HowTo100M) | 82.1 | 95.5 | $4\times 3$ | 29.80
ip-CSN-152 [69] (IG) | 82.5 | 95.3 | $10\times 3$ | 3.27
R3D-RS (WTS) [19] | 83.5 | – | $10\times 3$ | 9.21
OmniSource [20] (IG) | 83.6 | 96.0 | – | –
ViViT-H [3] (JFT) | 84.9 | 95.8 | $4\times 3$ | 47.77
TokenLearner-L/10 [55] (JFT) | 85.4 | 96.3 | $4\times 3$ | 48.91
Florence [79] (FLD-900M) | 86.5 | 97.3 | $4\times 3$ | –
CoVeR (JFT-3B) [81] | 87.2 | – | $1\times 3$ | –
MTV-L (JFT) | 84.3 | 96.3 | $4\times 3$ | 18.05
MTV-H (JFT) | 85.8 | 96.6 | $4\times 3$ | 44.47
MTV-H (WTS) | 89.1 | 98.2 | $4\times 3$ | 44.47
MTV-H (WTS 280p) | 89.9 | 98.3 | $4\times 3$ | 73.57

(b) Kinetics 600

Method | Top 1 | Top 5
---|---|---
SlowFast R101-NL [23] | 81.8 | 95.1
X3D-XL [22] | 81.9 | 95.5
TimeSformer-L [6] | 82.2 | 95.6
MFormer-HR [51] | 82.7 | 96.1
ViViT-L FE [3] | 82.9 | 94.6
MViT-B [21] | 83.8 | 96.3
MoViNet-A6 [35] | 84.8 | 96.5
MTV-B | 83.6 | 96.1
MTV-B (320p) | 84.0 | 96.2
R3D-RS (WTS) [19] | 84.3 | –
ViViT-H [3] (JFT) | 85.8 | 96.5
TokenLearner-L/10 [55] (JFT) | 86.3 | 97.0
Florence [79] (FLD-900M) | 87.8 | 97.8
CoVeR (JFT-3B) [81] | 87.9 | –
MTV-L (JFT) | 85.4 | 96.7
MTV-H (JFT) | 86.5 | 97.3
MTV-H (WTS) | 89.6 | 98.3
MTV-H (WTS 280p) | 90.3 | 98.5

(c) Something-Something v2

Method | Top 1 | Top 5
---|---|---
SlowFast R50 [23, 77] | 61.7 | –
TimeSformer-HR [6] | 62.5 | –
VidTR [83] | 63.0 | –
ViViT-L FE [3] | 65.9 | 89.9
MViT [21] | 67.7 | 90.9
MFormer-L [51] | 68.1 | 91.2
MTV-B | 67.6 | 90.1
MTV-B (320p) | 68.5 | 90.4

(d) Kinetics 700

| Top 1 | Top 5
---|---|---
VidTR-L [83] | 70.2 | –
SlowFast R101 [23] | 71.0 | 89.6
MoViNet-A6 [35] | 72.3 | –
MTV-L | 75.2 | 91.7
CoVeR (JFT-3B) [81] | 79.8 | –
MTV-H (JFT) | 78.0 | 93.3
MTV-H (WTS) | 82.2 | 95.7
MTV-H (WTS 280p) | 83.4 | 96.2

(e) Epic-Kitchens-100 Top 1 accuracy

Method | Action | Verb | Noun
---|---|---|---
ViViT-L FE [3] | 44.0 | 66.4 | 56.8
MFormer-HR [51] | 44.5 | 67.0 | 58.5
MoViNet-A6 [35] | 47.7 | 72.2 | 57.3
MTV-B | 46.7 | 67.8 | 60.5
MTV-B (320p) | 48.6 | 68.0 | 63.1
MTV-B (WTS 280p) | 50.5 | 69.9 | 63.9

(f) Moments in Time

| Top 1 | Top 5
---|---|---
AssembleNet-101 [56] | 34.3 | 62.7
ViViT-L FE [3] | 38.5 | 64.1
MoViNet-A6 [35] | 40.2 | –
MTV-L | 41.7 | 69.7
VATT-L (HT100M) [2] | 41.1 | 67.7
MTV-H (JFT) | 44.0 | 70.2
MTV-H (WTS) | 45.6 | 74.7
MTV-H (WTS 280p) | 47.2 | 75.7

We compare to the state-of-the-art across six different datasets. We evaluate
models with four temporal- and three spatial-views per video clip, following
[3]. To make the notation more concise, we now use MTV-B to refer to
B/2+S/4+Ti/8, MTV-L to refer to L/2+B/4+S/8+Ti/16 and MTV-H to refer to
H/2+B/4+S/8+Ti/16. Except for Kinetics, all our models start from a Kinetics
400 checkpoint and then are fine-tuned on the target datasets following [3,
51, 21].

##### Accuracy/computation trade-offs

Figure 3 compares our proposed MTV to its single-view counterpart, ViViT
Factorized Encoder (FE) [3] at every model scale on Kinetics 400. We compare
to ViViT-FE using tubelets with a temporal dimension of $t=2$, as the authors
obtained the best performance with this.

We can control the complexity of MTV by increasing or decreasing $t$ used in
each view. For example, increasing $t$ from 2 to 4 for the smallest view (and
proportionally increasing $t$ for all other views) will roughly reduce the
input tokens by half for each view, and thus halve the total FLOPs for
processing each input. Our method with $t=4$ for the smallest view
consistently achieves higher accuracy than ViViT-FE at every complexity level
while using fewer FLOPs, indicated by the green arrows pointing to the upper-
left in Fig. 3(a). This further validates that processing more views in
parallel enables us to achieve larger accuracy improvements than increasing
the number of input tokens. If we set $t=2$ as in ViViT-FE, we use additional
FLOPs, but increase significantly in accuracy too, as indicated by the green
arrow pointing to the upper-right in Fig. 3(a).

Furthermore, note how our B/2 model (transformer depth of 12 layers)
outperforms ViViT-L/2 (24 layers), whilst using less FLOPs. Similarly, our L/2
model outperforms ViViT-H/2. This shows that we can achieve greater accuracy
improvements by processing multiple views in parallel than by increasing the
depth for processing a single view.

Finally, note that Fig. 3(b) shows that our conclusions are also consistent
when using the inference time to measure our model’s efficiency. Appendix A.1
also shows that these trends also hold when using an unfactorized [3] backbone
architecture of ViViT and MTV.

##### Kinetics

We compare to methods that are pretrained on ImageNet-1K, ImageNet-21K [16]
and those that do not utilize pretraining at all in the first part of Tab.
2(f). In the second part of the tables, we compare to methods that are
pretrained on web-scale datasets such as Instagram 65M [25], JFT-300M [61],
JFT-3B [80], WTS [60], Florence [79] or HowTo100M [47]. Observe that we
achieve state-of-the-art results both with and without web-scale pretraining.

On Kinetics 400, our ImageNet-21K pretrained “Base” model improves the “Large”
ViViT-FE model [3], which corresponds to a deeper, single-view equivalent of
our model by 0.1% and 1.2% in Top-1 and Top-5 accuracy, whilst using 40% of
the total FLOPs. Our higher resolution version improves further by 0.7% and
1.4% while still using slightly fewer FLOPs. On Kinetics 600, our “Base” model
scores second to [35] whose model structure is derived using architecture
search on Kinetics 600 itself. We show significant improvements over [35] on
both Kinetics 400 and 700 for which the architecture of [35] was not directly
optimized for.

When using additional JFT-300M pretraining, our “Huge” model outperforms other
recent transformer models using the same pretraining dataset [3, 55]. And when
we utilize the Weak Textual Supervision (WTS) dataset of [60] for pre-
training, we substantially advance the best reported results on Kinetics: On
Kinetics 400, we achieve a Top-1 accuracy of 89.9%, which improves upon the
previous highest result (CoVeR [81]) by 2.7%. Similarly, on Kinetics 600, we
achieve a Top-1 of 90.3%, which is an absolute improvement of 2.4% on [81]. On
Kinetics 700, we achieve 83.4%, which improves even further by 3.6% over [81].
We also improve upon R3D-RS [19], which also used WTS pretraining, by 6.4% and
6.0% on Kinetics-400 and -600.

##### Epic-Kitchens-100

Following the standard protocol [15], we report Top-1 action-, verb- and noun-
accuracies with action accuracy being the primary metric. Our results are
averaged over $4\times 1$ crops as additional spatial crops did not help. Both
our MTV-B and MTV-B(320p) significantly improve the previous state-of-the-art
on noun classes, and MTV-B(320p) achieves a new state-of-the-art of 48.6% on
actions. With WTS pretraining and increasing resolution, we improved the
results to 50.5%. We found that additional data augmentation (detailed in
Appendix A.3) has to be used to achieve good performance (as also observed by
[3, 51]) as this is the smallest dataset of all six with 67,000 training
examples.

##### Something-Something V2

This dataset consists of class labels such as “move to left” and “pointing to
right” [26]. As the model needs to explicitly reason about direction, we do
not perform random horizontal or vertical flipping as data augmentation on
this dataset as also done by [21]. We improve substantially over ViViT-L-FE
[3], which corresponds to a deeper single-view equivalent of our model by
2.6%, and also improve upon MFormer [51] by 0.4%.

##### Moments in Time

Our MTV-L model significantly improves over the previous state-of-the-art [35]
by 1.5% in Top-1 accuracy. Moreover, our model with ImageNet-21K pretraining
even outperforms VATT [2], which was pretrained on HowTo100M [47], a dataset
consisting of around 100M video clips. When using WTS pre-training, we improve
our accuracy even further, achieving 47.2%.

##  5 Conclusion

We have presented a simple method for capturing multi-resolution temporal
context in transformer architectures, based on processing multiple “views” of
the input video in parallel. We have demonstrated that our approach performs
better, in terms of accuracy/computation trade-offs than increasing the depth
of current single-view architectures. Furthermore, we have achieved state-of-
the-art results on six popular video classification datasets. These results
were then further improved with large-scale pretraining [61, 60].

##### Limitations and future work

Although we have improved upon the state-of-the-art, there is still a large
room for improvement on datasets other than Kinetics. Furthermore, we have
relied on models pretrained on large image- or video-datasets for
initialization. Reducing this dependence on supervised pretraining is a clear
avenue of future research. We have conducted thorough ablations on standard
transformer architectures [18, 3], and will investigate if our approach is
complementary to recent, spatial-pyramid based multiscale transformer encoders
such as MViT [21] and Swin [44].

##### Societal impact

Video classification models can be used in a wide range of applications. We
are unaware of all potential applications, but are mindful that each
application has its own merits, and that also depends on the intentions of the
individuals building and using these systems. We also note that training
datasets may contain biases that models trained on them are unsuitable for
certain applications.

## References

  * [1] Edward H Adelson, Charles H Anderson, James R Bergen, Peter J Burt, and Joan M Ogden.  Pyramid methods in image processing.  RCA engineer, 29(6):33–41, 1984.
  * [2] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  In NeurIPS, 2021.
  * [3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In ICCV, 2021.
  * [4] Anurag Arnab, Chen Sun, and Cordelia Schmid.  Unified graph structured models for video understanding.  In ICCV, 2021.
  * [5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  In arXiv preprint arXiv:1607.06450, 2016.
  * [6] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [7] Jean-Yves Bouguet et al.  Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm.  Intel corporation, 5(1-10):4, 2001.
  * [8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [9] Peter J Burt and Edward H Adelson.  The laplacian pyramid as a compact image code.  In Readings in computer vision, pages 671–679. Elsevier, 1987.
  * [10] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [11] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  In ICCV, 2021.
  * [12] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In CVPR, 2017.
  * [13] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In NeurIPS, 2020.
  * [14] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In CVPR, 2005.
  * [15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  In IJCV, 2021.
  * [16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [19] Xianzhi Du, Yeqing Li, Yin Cui, Rui Qian, Jing Li, and Irwan Bello.  Revisiting 3d resnets for video recognition.  In arXiv preprint arXiv:2109.01696, 2021.
  * [20] Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin.  Omni-sourced webly-supervised learning for video recognition.  In ECCV, 2020.
  * [21] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In ICCV, 2021.
  * [22] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In CVPR, 2020.
  * [23] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [24] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.  Spatiotemporal residual networks for video action recognition.  In NeurIPS, 2016.
  * [25] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan.  Large-scale weakly-supervised pre-training for video action recognition.  In CVPR, 2019.
  * [26] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The” something something” video database for learning and evaluating visual common sense.  In ICCV, 2017.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [28] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  In arXiv preprint arXiv:1606.08415, 2016.
  * [29] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.  Deep networks with stochastic depth.  In ECCV, 2016.
  * [30] Jinggang Huang and David Mumford.  Statistics of natural images and models.  In CVPR, 1999.
  * [31] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira.  Perceiver io: A general architecture for structured inputs & outputs.  In arXiv preprint arXiv: 2107.14795, 2021.
  * [32] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In CVPR, 2014.
  * [33] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  In arXiv preprint arXiv:1705.06950, 2017.
  * [34] Alexander Klaser, Marcin Marszałek, and Cordelia Schmid.  A spatio-temporal descriptor based on 3d-gradients.  In BMVC, 2008.
  * [35] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong.  Movinets: Mobile video networks for efficient video recognition.  In CVPR, 2021.
  * [36] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, 2012.
  * [37] Ivan Laptev.  On space-time interest points.  In IJCV, 2005.
  * [38] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce.  Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories.  In CVPR, 2006.
  * [39] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural computation, 1(4):541–551, 1989.
  * [40] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang.  Tea: Temporal excitation and aggregation for action recognition.  In CVPR, 2020.
  * [41] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In ICCV, 2019.
  * [42] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [43] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, 2016.
  * [44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [45] David G Lowe.  Distinctive image features from scale-invariant keypoints.  In IJCV, 2004.
  * [46] Bruce D Lucas, Takeo Kanade, et al.  An iterative image registration technique with an application to stereo vision.  In IJCAI, 1981.
  * [47] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, 2019.
  * [48] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al.  Moments in time dataset: one million videos for event understanding.  In PAMI, 2019.
  * [49] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  In NeurIPS, 2021.
  * [50] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In CVPR, 2015.
  * [51] Mandela Patrick, Dylan Campbell, Yuki M Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, Jo Henriques, et al.  Keeping your eye on the ball: Trajectory attention in video transformers.  In NeurIPS, 2021.
  * [52] Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, and Tao Mei.  Learning spatio-temporal representation with local and global diffusion.  In CVPR, 2019.
  * [53] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  In JMLR, 2020.
  * [54] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In ICML, 2017.
  * [55] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova.  Tokenlearner: What can 8 learned tokens do for images and videos?  In NeurIPS, 2021.
  * [56] Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova.  Assemblenet: Searching for multi-stream neural connectivity in video architectures.  In ICLR, 2019.
  * [57] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NeurIPS, 2014.
  * [58] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [59] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  In arXiv preprint arXiv:2106.10270, 2021.
  * [60] Jonathan C. Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David A. Ross.  Learning video representations from textual web supervision.  In arXiv 2007.14937, 2020.
  * [61] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era.  In ICCV, 2017.
  * [62] Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi.  Human action recognition using factorized spatio-temporal convolutional networks.  In ICCV, 2015.
  * [63] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [64] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [65] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, 2019.
  * [66] Antonio Torralba and Aude Oliva.  Statistics of natural image categories.  Network: computation in neural systems, 2003.
  * [67] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [68] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [69] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video classification with channel-separated convolutional networks.  In ICCV, 2019.
  * [70] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In CVPR, 2018.
  * [71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [72] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu.  Dense trajectories and motion boundary descriptors for action recognition.  In IJCV, 2013.
  * [73] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In ICCV, 2021.
  * [74] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [75] Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua.  Attentionnas: Spatiotemporal attention cell search for video classification.  In ECCV, 2020.
  * [76] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.  Long-term feature banks for detailed video understanding.  In CVPR, 2019.
  * [77] Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl.  A multigrid method for efficiently training video models.  In CVPR, 2020.
  * [78] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In ECCV, 2018.
  * [79] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  In arXiv preprint arXiv:2111.11432, 2021.
  * [80] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In arXiv preprint arXiv:2106.04560, 2021.
  * [81] Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha.  Co-training transformer with videos and images improves action recognition.  In arXiv preprint arXiv:2112.07175, 2021.
  * [82] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  Mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * [83] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe.  Vidtr: Video transformer without convolutions.  In ICCV, 2021.
  * [84] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [85] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  In ICLR, 2017.

##  Appendix A Additional experiments

In this Appendix, we provide additional experimental details. Section A.1
provides accuracy-FLOPs and accuracy-throughput comparison between two model
variants of ViViT and MTV. Section A.2 provides the effect of spatial
resolution of tubelets. Section A.3 and Section A.4 provides details of our
training hyperparameters and model configurations used in our experiments.

###  A.1 Changing transformer encoder architecture

We present additional results by changing the transformer architecture used
within our multiview encoder. Specifically, we use the unfactorized ViViT
transformer encoder (Model 1 of [3]). In this variant, each transformer
encoder layer computes self-attention over all spatio-temporal tokens. This
makes our multiview transformer encoder cover a wide range of spatial and
temporal dimensions across different views. A one-layer MLP with hidden
dimension of $3072$ is used as the global encoder for our unfactorized MTV
model.

As shown in Fig. 4, MTV (unfactorized) consistently outperforms its single-
view counterpart (_i.e._ ViViT unfactorized) for every scale (see Fig. 4(a))
and corresponds to a better accuracy-throughput curve as shown in Fig. 4(b).
Note how MTV can more than double the throughput of ViViT unfactorized, whilst
still improving its accuracy, for each model scale. Specifically, MTV
(unfactorized) H/4+B/8+S/16+Ti/32 model leads to a significant speed-up by
$172\%$ while still keeping a higher accuracy of $0.4\%$ improvement compared
to ViViT-H.

Moreover, we report the accuracy-throughput comparison between MTV and ViViT
factorized model (ViViT-FE) in Fig. 4(d). Note that the accuracy-FLOPs
comparison is already reported in paper Section 4.3. The improvements in
accuracy-throughput, and accuracy-FLOPs remain significant in this setting.

Note that the unfactorized ViViT transformer encoder, which attends to all
spatio-temporal tokens, is less efficient than the Factorized Encoder
architecture that we used in the main paper. However, we achieve larger
relative improvements in accuracy/computation trade-offs compared to the
corrsponding single-view ViViT baseline when using this encoder architecture.

(a) Accuracy[%] - GFLOPs comparison between MTV (unfactorized) and ViViT.

(b) Accuracy[%] - Throughput comparison between MTV (unfactorized) and ViViT.

(c) Accuracy[%] - GFLOPs comparison between MTV and ViViT-FE.

(d) Accuracy[%] - Throughput comparison between MTV and ViViT-FE.

Figure 4: Accuracy/complexity trade-off between ViViT / ViViT-FE [3] (blue)
and our MTV (unfactorized) / MTV (red). MTV (unfactorized) is consistently
better and requires less FLOPs (see Fig. 4(a)) than ViViT to achieve higher
accuracy across different model scales (indicated by the dotted green arrows
pointing upper-left). With additional FLOPs, MTV shows larger accuracy gains
(shown by the dotted green arrows pointing upper-right). The lower number of
FLOPs is translated to higher throughput (clips per second), as indicated by
the green arrows in Fig. 4(b). Note how MTV can more than double the
throughput of ViViT unfactorized, whilst still improving its accuracy, across
all model scales. Similar findings are also observed by the comparison between
ViViT-FE and MTV model in Fig. 4(c) and Fig. 4(d). Note that Fig. 4(c)
appeared as Figure 3 in the main paper, and is included here for clarity and
consistency. All speed comparisons are measured with the same hardware (Cloud
TPU-v4). The complexity is for a single $32\times 224\times 224\times 3$ input
video (denoted as $T\times H\times W\times C$), and the accuracy is obtained
by $4\times 3$ view testing.

###  A.2 Spatial resolution of tubelets

We study the effect of the spatial resolution of tubelets in Tab.
LABEL:tab:spatial_ablations. We use our B/4 + Ti/16 model variant, and vary
the spatial resolution of the tubelets. Our results indicate that the accuracy
is primarily impacted by the spatial resolution of the large encoder. We also
note that processing more tokens, and thus using more computation, typically
results in higher accuracies.

Table 3:  Effect of spatial resolution of tubelets. All experiments are conducted on Kinetics 400 using the model variant B/4+Ti/16. Accuracies are for $4\times 3$ crops. Tubelet spatial size | GFLOPs | Top-1
---|---|---
B | Ti |  |
$24\times 24$ | $16\times 16$ | 68 | 78.1
$16\times 16$ | $24\times 24$ | 165 | 80.5
$16\times 16$ | $16\times 16$ | 168 | 80.5
$16\times 16$ | $12\times 12$ | 169 | 80.6
$12\times 12$ | $16\times 16$ | 295 | 81.0

###  A.3 Hyperparameters for each datasets

Table 4: Training hyperparamters for experiments in the main paper. “–” indicates that the regularisation method was not used at all. Values which are constant across all columns are listed once. Datasets are denoted as follows: K400: Kinetics 400. K600: Kinetics 600. K700: Kinetics 700. MiT: Moments in Time. EK: Epic Kitchens. SSv2: Something-Something v2. |  K400 |  K600 |  K700 |  MiT |  EK |  SSv2
---|---|---|---|---|---|---
Optimization
Optimizer | Synchronous SGD
Momentum | 0.9
Batch size |  64 |  64 |  64 |  256 |  128 |  512
Learning rate schedule | cosine with linear warmup
Linear warmup epochs | 2.5
Base learning rate |  0.1 |  0.1 |  0.1 |  0.1 |  0.2 |  0.5
Epochs |  30 |  30 |  30 |  30 |  80 |  100
Data augmentation
Random crop probability | 1.0
Random flip probability |  0.5 |  0.5 |  0.5 |  0.5 |  0.5 |  –
Scale jitter probability | 1.0
Maximum scale | 1.33
Minimum scale | 0.9
Colour jitter probability |  0.8 |  0.8 |  0.8 |  0.8 |  – |  –
Rand augment number of layers [13] |  – |  – |  – |  – |  3 |  1
Rand augment magnitude [13] |  – |  – |  – |  – |  10 |  15
Other regularisation
Stochastic droplayer rate [29] |  0.1 |  0.1 |  0.1 |  0.1 |  0.1 |  0.3
Label smoothing [64] |  – |  – |  – |  – |  0.2 |  0.2
Mixup [82] |  – |  – |  – |  – |  0.1 |  0.3

Table 4 details the hyperparamters used in all of our experiments. We use
synchronous SGD with momentum, a cosine learning rate schedule with linear
warmup, and a batch size of 64 for all experiments on the Kinetic datasets. We
found that larger batch size and additional regularization are helpful when
training on the smaller Epic Kitchens and Something-Something v2 datasets, as
also noted by [3].

###  A.4 Model configurations

Table 5 summarizes our model configurations of each view for our multiview
transformer encoder. For the backbone of each view, we consider five ViT
variants, “Tiny”, “Small”, “Base”, “Large”, and “Huge”. Their settings
strictly follow the ones defined in BERT [17] and ViT [18, 59]. For the global
encoder, all model variants of MTV use the same global encoder which follows
the “Base” architecture, except that the number of heads is set to 8 instead
of 12. The reason is that the hidden dimension of the tokens should be
divisible by the number of heads for multi-head attention, and the number of
hidden dimensions across all backbone sizes is divisible by 8 (as shown in
Tab. 5). All model variants of MTV (unfactorized) use a one-layer MLP with the
same hidden dimension as the “Base” architecture.

Table 5: Model configurations for each view of MTV. Model name |  Hidden size |  MLP dimension |  Number of attention heads |  Number of encoder layers |  Tubelet spatial size
---|---|---|---|---|---
Tiny |  192 |  768 |  3 |  12 |  16
Small |  384 |  1536 |  6 |  12 |  16
Base |  768 |  3072 |  12 |  12 |  16
Large |  1024 |  4096 |  16 |  24 |  16
Huge |  1280 |  5120 |  16 |  32 |  14