Edward<EMAIL_ADDRESS>Jon<EMAIL_ADDRESS>Andrew<EMAIL_ADDRESS>The University of Surrey

# Two-Stream Transformer Architecture for Long Form Video Understanding

###### Abstract

Pure vision transformer architectures are highly effective for short video
classification and action recognition tasks. However, due to the quadratic
complexity of self attention and lack of inductive bias, transformers are
resource intensive and suffer from data inefficiencies. Long form video
understanding tasks amplify data and memory efficiency problems in
transformers making current approaches unfeasible to implement on data or
memory restricted domains. This paper introduces an efficient Spatio-Temporal
Attention Network (STAN) which uses a two-stream transformer architecture to
model dependencies between static image features and temporal contextual
features. Our proposed approach can classify videos up to two minutes in
length on a single GPU, is data efficient, and achieves SOTA performance on
several long video understanding tasks.

##  1 Introduction

Long form video understanding (LVU) is a sub-domain of video recognition
concerned with understanding contextual information across contiguous shots
which can contain multiple locations, scenes, interactions, and actions. While
a blurb can give a snapshot of a book, stories are enriched by the development
of characters and their interactions with objects, people, and locations. The
same applies in video recognition where understanding the context of
individual moments in relation to a whole video can provide valuable
information for tasks such as classification, speaker recognition, character
understanding, and video retrieval. However, current video recognition methods
and datasets have tended to focus on short videos [Marszalek et
al.(2009)Marszalek, Laptev, and Schmid, Fouhey et al.(2018)Fouhey, Kuo, Efros,
and Malik, Ji et al.(2012)Ji, Xu, Yang, and Yu, Cascante-Bonilla et
al.(2019)Cascante-Bonilla, Sitaraman, Luo, and Ordonez, Huang et
al.(2020)Huang, Xiong, Rao, Wang, and Lin, Zhou et al.(2010)Zhou, Hermans,
Karandikar, and Rehg, Hori et al.(2017)Hori, Hori, Lee, Zhang, Harsham,
Hershey, Marks, and Sumi, Feichtenhofer(2020), Qiu et al.(2017)Qiu, Yao, and
Mei], effectively aggregating convolutional image features via late fusion and
inflation [Feichtenhofer(2020), Karpathy et al.(2014)Karpathy, Toderici,
Shetty, Leung, Sukthankar, and Fei-Fei, Ji et al.(2012)Ji, Xu, Yang, and Yu,
Qiu et al.(2017)Qiu, Yao, and Mei, Tran et al.(2019)Tran, Wang, Torresani, and
Feiszli], or extending attention based image classification methods to the
tasks of action recognition, object tracking, and segmentation [Sun et
al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid, Lu et al.(2019)Lu, Batra,
Parikh, and Lee, Ho et al.(2019)Ho, Kalchbrenner, Weissenborn, and Salimans,
Girdhar and Grauman(2021), Bertasius et al.(2021)Bertasius, Wang, and
Torresani, Wang et al.(2020)Wang, Xiong, Neumann, Piergiovanni, Ryoo,
Angelova, Kitani, and Hua].

Figure 1: An overview of our approach, STAN. We encode video scenes into two
feature representations using a two-stream spatio-temporal convolutional
network. A transformer encoder is then used to model temporal dependencies
between the tokens via an additional classification token randomly
initialised. The proposed method allows us to model long term dependencies
between individual frames of long videos which feature multiple actions and
environments.

Intuitively, an image can provide a good summary of a moment in time, and so
it is logical that these methods perform well at classifying short videos from
only a few frames [Yue-Hei Ng et al.(2015)Yue-Hei Ng, Hausknecht,
Vijayanarasimhan, Vinyals, Monga, and Toderici]; however, many video
classification tasks require, or can be improved with, long-term temporal
reasoning [Wu and Krahenbuhl(2021)]. The temporal fusion of features via
recurrent networks [Rumelhart et al.(1985)Rumelhart, Hinton, and Williams,
Donahue et al.(2015)Donahue, Anne Hendricks, Guadarrama, Rohrbach,
Venugopalan, Saenko, and Darrell] provides one solution by aggregating frames
over longer sequences of frames, but these architectures suffer from
computational inefficiencies due to their recursive design while 3D
convolutional approaches also do not scale effectively, primarily due to their
high computational footprint and the linear growth of the receptive field in
the temporal dimension.

More recently, transformers [Devlin et al.(2018)Devlin, Chang, Lee, and
Toutanova, Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.]
have been adapted to the video domain [Arnab et al.(2021)Arnab, Dehghani,
Heigold, Sun, Lučić, and Schmid, Sun et al.(2019)Sun, Myers, Vondrick, Murphy,
and Schmid, Girdhar and Grauman(2021)], achieving SOTA results on multiple
short video tasks. But due to the lack of inductive bias in transformer
architectures, it has been observed that video transformers are not as data
efficient as their CNN counterparts [Touvron et al.(2021)Touvron, Cord, Douze,
Massa, Sablayrolles, and Jégou], requiring more data and thus more time to
train. Furthermore, it has been shown that transformer image networks trained
on natural videos work just as effectively when inputs are shuffled [Fan et
al.(2021)Fan, Xiong, Mangalam, Li, Yan, Malik, and Feichtenhofer],
demonstrating weak temporal bias. This issue is compounded when addressing LVU
tasks, as we encounter larger datasets with fewer opportunities for parameter
updates during training caused by a small target to data ratio.

The following question then arises.How can we leverage inductive bias from
image CNN’s to make video transformer networks more data and memory efficient
for long form video understanding tasks?

Our solution is a two-stream Spatio-Temporal Attention Network (STAN) with
which we gain data and computational efficiency by introducing inductive bias
via convolution. Existing methods for image classification with transformers
such as [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] split images
into $16\times 16$ pixel regions encoded with a positional embedding to
introduce permutation and translation invariance. This method has been
extended to video by expanding these regions temporally to create 3D tokens
with 2D positional embeddings [Arnab et al.(2021)Arnab, Dehghani, Heigold,
Sun, Lučić, and Schmid]. Inspired by work in two-stream slow-fast
convolutional video networks for short video classification [Feichtenhofer et
al.(2019)Feichtenhofer, Fan, Malik, and He], our approach replaces this
tokenisation method with both image spatial and context temporal scene
features extracted from pre-trained convolutional neural networks. We then use
a two-stream transformer architecture to model temporal dependencies between
the scene features for classifying long videos of up to two minutes in length.
Our method is data and memory efficient and achieves SOTA results on a range
of long form video understanding tasks.

##  2 Related Work

Early video classification works focused on non-temporal aggregation, which
included clustering [Jégou et al.(2010)Jégou, Douze, Schmid, and Pérez,
Perronnin and Dance(2007), Miech et al.(2017b)Miech, Laptev, and Sivic] or
fusing [Fish et al.(2021)Fish, Weinbren, and Gilbert] spatial features
obtained from convolutional neural networks. Since a short video will share a
similar distribution of pixels over concurrent frames, these networks perform
well for video classification and object detection tasks. Naturally, fusing
these output features using RNN’s [Rumelhart et al.(1985)Rumelhart, Hinton,
and Williams, Baccouche et al.(2011)Baccouche, Mamalet, Wolf, Garcia, and
Baskurt, Chao et al.(2018)Chao, Vijayanarasimhan, Seybold, Ross, Deng, and
Sukthankar, Wu et al.(2015)Wu, Wang, Jiang, Ye, and Xue] such as an LSTM
[Donahue et al.(2015)Donahue, Anne Hendricks, Guadarrama, Rohrbach,
Venugopalan, Saenko, and Darrell, Yue-Hei Ng et al.(2015)Yue-Hei Ng,
Hausknecht, Vijayanarasimhan, Vinyals, Monga, and Toderici] improves
performance by introducing temporal information. However, as discussed in
[Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai,
Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.], these model
architectures align with steps in computational time, and are inefficient at
longer sequence lengths as memory constraints limit batching across sequences.

Later works explored extending convolution to video, inflating image CNN’s via
a temporal channel [Feichtenhofer(2020), Karpathy et al.(2014)Karpathy,
Toderici, Shetty, Leung, Sukthankar, and Fei-Fei, Ji et al.(2012)Ji, Xu, Yang,
and Yu, Qiu et al.(2017)Qiu, Yao, and Mei, Carreira and Zisserman(2017)] and
using two stream convolutional networks to aggregate spatial and temporal
information introduced via optical flow [Simonyan and Zisserman(2014)], or
from various sampling and aggregation intervals [Tran et al.(2018)Tran, Wang,
Torresani, Ray, LeCun, and Paluri, Pan et al.(2021)Pan, Chen, Shou, Liu, Shao,
and Li, Xie et al.(2018)Xie, Sun, Huang, Tu, and Murphy]. 3D Convolutional
Neural Networks (CNN’s) are highly effective at video classification, object
detection, and action recognition tasks but are computationally intensive to
train. For example, in [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and
Paluri] the authors process just 16 frames at the cost of 40 GFLOPS per single
pass, making the approach infeasible for sequences longer than a few seconds.
To address computational overhead in training convolutional video networks, in
[Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] the authors
present an efficient and high performance CNN for video classification, which
utilises a fast and slow temporal sampling stream. The slow stream can process
higher resolution images and extract key spatial information over a few key
frames, while the fast stream maps low resolution frames to infer temporal
information. This method is highly effective at video and action
classification on short 10 second clips at a low computational cost.

Transformers [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.,
Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova, Vaswani et
al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and
Polosukhin, Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal,
Neelakantan, Shyam, Sastry, Askell, et al.] use self-attention to model
dependencies between inputs and have recently been shown to work effectively
for video classification tasks when implemented with temporal positional
information [Sun et al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid, Lu et
al.(2019)Lu, Batra, Parikh, and Lee, Ho et al.(2019)Ho, Kalchbrenner,
Weissenborn, and Salimans, Girdhar and Grauman(2021), Bertasius et
al.(2021)Bertasius, Wang, and Torresani, Wang et al.(2020)Wang, Xiong,
Neumann, Piergiovanni, Ryoo, Angelova, Kitani, and Hua]. Unlike convolutional
methods, they lack inductive bias and, as such, take longer to model
dependencies between neighbouring pixel regions which extends training time
and harms data efficiency [Touvron et al.(2021)Touvron, Cord, Douze, Massa,
Sablayrolles, and Jégou] thus making them innapropriate for LVU tasks.
Introducing inductive bias via convolution, shifting windows, and gaussian
bias to transformers has shown to be effective in the image domain [Yuan et
al.(2021)Yuan, Chen, Wang, Yu, Shi, Jiang, Tay, Feng, and Yan, Guo et
al.(2021)Guo, Han, Wu, Xu, Tang, Xu, and Wang, Wang et al.(2017)Wang, Jiang,
Qian, Yang, Li, Zhang, Wang, and Tang, Yang et al.(2018)Yang, Tu, Wong, Meng,
Chao, and Zhang, Lin et al.(2019)Lin, Gan, and Han], and hybrid networks for
video have also been proposed in [Neimark et al.(2021)Neimark, Bar, Zohar, and
Asselmann, d’Ascoli et al.(2021)d’Ascoli, Touvron, Leavitt, Morcos, Biroli,
and Sagun, Girdhar et al.()Girdhar, Carreira, Doersch, and Zisserman], but
only on short segments of video. Finally, LVU networks have been explored in
[Wu and Krahenbuhl(2021), Zhao et al.(2019)Zhao, Zhang, Xiao, Xiao, Yan, Yu,
Cai, and Wu] where the authors leverage short-term detection and tracking to
form instance representations or use LSTM’s for the task of long form video-
question answering respectively.

##  3 Method

Overview We aim to classify long videos by splitting them into discrete scenes
and extracting spatial and temporal representations for temporal fusion via a
two-stream transformer encoder. To do so, we first analyse the video for
changes in average frame intensity/brightness using a running average over RGB
video channels. We use these time stamps as scene segmentation points and
uniformly sample 12 frames from each scene segment using a 3D CNN encoder to
generate a low-resolution temporal feature tokens for each scene. We also
extract the central frame of each scene at a higher resolution and use a 2D
CNN to obtain a spatial feature token. The spatial and temporal scene tokens
are encoded with a positional embedding and temporally aggregated using a two-
stream transformer encoder. For classification, we randomly initialise an
additional token prepended to the spatial and temporal sequence of embedding
tokens, which learns to model the temporal inter-dependency of the individual
scenes. An overview of the sampling methodology is shown in Fig 2.

Figure 2: For each scene $s$ in the video $v$ we obtain a temporal ($\hat{t}$)
and spatial ($\hat{x}$) feature embedding using 2D and 3D convolutional neural
networks denoted here as $h(.)$ and $g(.)$. A shared positional embedding,
$p$, is added to every scene embedding to generate the spatial tokens,
$z_{s}$, and the temporal tokens, $z_{t}$. A two stream spatio-temporal
transformer with two layers ($l$), learns the dependency between the sequence
of spatial tokens and the temporal tokens. Following normalisation and a
linear projection, the features are fused for classification. In practice, we
only fuse and classify the prepended CLS token as discussed in Section 3. When
training the STAN-Small model, we do not back-propagate through the 3D CNN
represented here by the stop-gradient line. For STAN-Large we continue to
fine-tune both convolutional encoders.

Spatial and Temporal Tokens Given a uniformly sampled set of 12 video frames
from a scene defined here as $t\in\mathbb{R}^{12\times H\times W\times C}$ we
implement a R(2+1)d Video ResNet encoder [Tran et al.(2018)Tran, Wang,
Torresani, Ray, LeCun, and Paluri] pretrained on the Kinetics400 Dataset
[Carreira and Zisserman(2017)], defined as $g(t)$, to encode each set of
frames into a single feature embedding token $\hat{t}$ which represents the
projected temporal features of the scene $s_{i}$ from a set of scenes $s$. We
obtain an embedding for each scene in the video using the above method to
generate a set ${z_{t}}$ which represents the set of all temporal embedding
vectors for the scene $s_{i}\in s$.

$z_{t}=[z_{cls},\hat{t_{0}},\hat{t_{1}},...,\hat{t_{n}}]+p$ (1)

Where ${z}_{cls}$ is a randomly initialized vector in the same dimension as
$\hat{t_{i}}$ and $p_{i}\in\mathcal{PE}$ is a positional embedding added to
the sequence as described in [Vaswani et al.(2017)Vaswani, Shazeer, Parmar,
Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] as,

$\mathcal{PE}_{pos,2_{i}}=sin(\frac{pos}{10000^{2i/d_{model}}})$

$\mathcal{PE}_{pos,2_{i+1}}=cos(\frac{pos}{10000^{2i/d_{model}}})$ (2)

Where $d_{model}$ is a common dimension for both the spatial and temporal
tokens. In [Girdhar et al.()Girdhar, Carreira, Doersch, and Zisserman, Zhang
et al.(2021)Zhang, Li, Liu, Shuai, Zhu, Brattoli, Chen, Marsic, and Tighe,
Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai,
Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.], positional
information is used to infer the relationship between cropped regions in an
individual frame. As we have introduced inductive bias via convolution, we do
not need to model the position of pixel regions and instead extend the
positional embedding method to infer the position of each short temporal
sequence. We argue that this is logical for longer videos that feature a
narrative composed of multiple dynamic actions and environments such as those
found in movies. In section 4 we show that this positional information
benefits the model’s performance on such a task.

Spatial Encoding Token To obtain a spatial encoding token, we perform a linear
projection of the high resolution frame $x$, which is sampled from the center
of $s_{i}$, to a spatial embedding token $\hat{x}$ using a ResNet18[He et
al.(2016)He, Zhang, Ren, and Sun] encoder model pretrained on the ImageNet
Dataset [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] so that
$\hat{x}=h(x)$. As in the case of the temporal token, we form a sequence of
spatial embedding tokens $z_{x}$ from spatial feature embeddings obtained via
$h(.)$ so that $z_{x}={\\{\hat{x_{0}},\hat{x_{1}},...,\hat{x_{n}}\\}}$ where
$n$ is the total number of scenes in the video $z$. In Eq 3 $z_{x}^{cls}$ is
the randomly initialized spatial cls token used for classification.

${z_{s}}=[z_{x}^{cls},\hat{x_{0}},\hat{x_{1}},...,\hat{x_{n}}]+p$ (3)

The positional embedding vector $p$ in Eq 3 is the same as defined in Eq 1 and
as such the positional information is shared between the spatial and temporal
tokens. This ensures that positional information is consistent between the
spatial and temporal streams during fusion. We now define a spatial and
temporal transformer for temporal aggregation of the input tokens and then
discuss techniques to fuse the output classification from both streams.

Spatial and Temporal Transformer Encoders For the spatial and temporal
transformer encoder architecture, we implement the transformer model
introduced in [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin] originally designed for natural language
processing. As described in [Vaswani et al.(2017)Vaswani, Shazeer, Parmar,
Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] we generate Query ($Q$), Key
($K$), and Value ($V$) matrices from both the spatial and temporal embedding
representations where each row in the matrix represents a corresponding scene,
with the first row representing our classification token. A matrix of outputs
is computed as,

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}V})$ (4)

The output matrix $A(Q,K,V)$ is summed with the input embedding matrix via a
residual connection and normalised using Layer Normalisation. Finally, the
output is summed with a second residual connection before a classification MLP
head. In practice, we only apply the final classification MLP to the temporal
and spatial transformer classification tokens which learn a representation of
the input via self-attention. As such, we can discard the other rows of the
output matrix and only backpropagate via the classification MLP. We use Multi-
Headed-Self-Attention to model further representations in spatial and temporal
domains by replicating the attention mechanism in Eq 4 and concatenating the
heads. This process is described in detail in [Vaswani et al.(2017)Vaswani,
Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]. In [Girdhar
et al.()Girdhar, Carreira, Doersch, and Zisserman, Zhang et al.(2021)Zhang,
Li, Liu, Shuai, Zhu, Brattoli, Chen, Marsic, and Tighe, d’Ascoli et
al.(2021)d’Ascoli, Touvron, Leavitt, Morcos, Biroli, and Sagun] the authors
use a linear layer for classification based on ${z}_{cls}\in\mathbb{R}^{d}$.
In our work, we linearly project ${z}_{cls}$ to $\mathbb{R}^{d}_{model}$ for
both the spatial and temporal features so we can experiment with several
fusion methods, which we will describe next.

Fusion and Classification For fusion, we normalise the output embeddings and
project to a common dimension with a linear layer before classification via a
three-layer MLP separated by a gated non-linearity. The loss function can be
defined as a binary cross entropy loss between the targets $y$ and the scaled
sum of the feature embeddings,

$\mathcal{L}^{Fusion}={L}_{BCE}(h(Norm([q(z_{s}^{cls})+\lambda
q(z_{t}^{cls})]),y)$ (5)

Where $\lambda<1.0$ acts as a hyper-parameter to scale the influence of the
temporal network in generating the output logits and set to 0.6. $q(.)$ Is an
MLP with one hidden layer and weights shared for both the temporal and spatial
streams, while the function $h(.)$ represents the final MLP used for
classification after fusion. To improve training time we use transfer learning
and pre-train the 2D CNN on ImageNet [Deng et al.(2009)Deng, Dong, Socher, Li,
Li, and Fei-Fei] and the 3D CNN on Kinetics400 [Carreira and Zisserman(2017)].
We present both a large, and small version of the model. For the STAN-Small,
we do not update the parameters in the temporal stream layers but back-
propagate through the two-stream transformer to the 2D CNN. This reduces the
number of trainable parameters by 48 Million, making the whole network
trainable on a single Nvidia RTX5000 GPU with 16GB of memory. The larger
model, STAN-Large, which back-propagates via both streams, improves
performance by 11% but requires 92.5 million trainable parameters compared
with 45 million for STAN-Small. In Table 1 we experiment with a number of
additional methods for fusion including collaborative gating [Liu et
al.(2019)Liu, Albanie, Nagrani, and Zisserman] and distillation [Hinton et
al.(2015)Hinton, Vinyals, Dean, et al.]. A full experimental analysis follows
in the next section.

##  4 Results

We evaluate the proposed method on several long video classification tasks. As
discussed in [Wu and Krahenbuhl(2021)] video datasets have typically focused
on short video tasks, therefore following [Wu and Krahenbuhl(2021)] we define
long videos as videos which feature more than three shots or scenes, are
longer than a minute in length, and in which classification relies on, or is
improved by, contextual understanding of the relationship between the content
of shots and their order. To evaluate our method, we use both the MMX-
Trailer-20 Dataset (MMX) [Fish et al.(2021)Fish, Weinbren, and Gilbert], and
the Long Video Understanding Benchmark (LVU) [Wu and Krahenbuhl(2021)].

MMX Dataset The MMX-Trailer-20 Dataset [Fish et al.(2021)Fish, Weinbren, and
Gilbert] is a multi-modal movie trailer dataset with 7555 movie trailers
spanning many scenes, locations, actions, and narratives. The MMX-Trailer
dataset features movie trailers from 20 genres with six genre labels per video
from 1900 $-$ 2021 with an average length of two minutes. The training set is
6047 videos, with the validation and test sets both at 754 samples. The
dataset consists of 37 million frames in total.

LVU Benchmark The Long Video Understanding Benchmark [Wu and Krahenbuhl(2021)]
contains 30K videos with an average length of 120 seconds. The benchmark
comprises of content understanding, user engagement prediction, and movie
metadata prediction tasks, demonstrating various requirements for long
temporal modelling. We test our approach on the content understanding tasks
including character relationship identification, speaking style, and scene
recognition.

Compared Methods To demonstrate the effectiveness of the two-stream network,
we implement several existing methods for video classification using the MMX-
Trailer-20 dataset. They include extracting convolutional image features from
a ResNet18 [He et al.(2016)He, Zhang, Ren, and Sun] for each scene and simply
using average pooling for classification. We also implement this same method
with features obtained from a two-stream Inflated 3D Convolutional Neural
Network [Ji et al.(2012)Ji, Xu, Yang, and Yu] and a SENet [Hu et al.(2018)Hu,
Shen, and Sun]. We also show results for comparative temporal networks,
including a Temporal Pyramid Network, which extracts multiple level features
from a CNN to model dynamic temporal movements in videos, and a vanilla LSTM
[Donahue et al.(2015)Donahue, Anne Hendricks, Guadarrama, Rohrbach,
Venugopalan, Saenko, and Darrell] to model the temporal relationship between
the feature representations from an 18 layer ResNet. More complex temporal
aggregation strategies are also explored, such as the Fine-Grained Semantic
[Fish et al.(2021)Fish, Weinbren, and Gilbert] architecture which uses
concatenation to aggregate multi-modal features as well as the effectiveness
of earlier audio-visual works such as audio-visual classification using
support vector machines [Huang and Wang(2012)] and audio VGG network as
described in [Abu-El-Haija et al.(2016)Abu-El-Haija, Kothari, Lee, Natsev,
Toderici, Varadarajan, and Vijayanarasimhan]. We also explore distillation as
described in [Touvron et al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles,
and Jégou], using the temporal transformer encoder as a teacher network to the
spatial transformer. For the LVU Benchmark task we compare our method with
those presented in the paper including videoBERT [Sun et al.(2019)Sun, Myers,
Vondrick, Murphy, and Schmid], R101-SlowFast [Feichtenhofer et
al.(2019)Feichtenhofer, Fan, Malik, and He] and Object Transformer [Wu and
Krahenbuhl(2021)].

Metrics As outlined in [Fish et al.(2021)Fish, Weinbren, and Gilbert] the MMX
dataset is imbalanced and as such we follow other works [Dong et
al.(2016)Dong, Li, and Snoek, Miech et al.(2017a)Miech, Laptev, and Sivic,
Mithun et al.(2018)Mithun, Li, Metze, and Roy-Chowdhury] and use Mean Average
Precision $mAP$ to evaluate the effectiveness of our classifier. To calculate
the $mAP$, we average the area under the precision-recall curve per genre,
weighting instances according to the class frequencies. We also show weighted
Precision ($P_{w}$), weighted Recall ($R_{w}$), and weighted F1-Score
($F1_{w}$) in Table 3. With all metrics, a higher value demonstrates improved
accuracy. For LVU we compute the standard error averaged over five runs as
proposed in the original paper [Wu and Krahenbuhl(2021)] where higher values
represent improved performance.

Evaluation First, in Table 1 we evaluate our approach against existing methods
for video classification using the MMX-Trailer-20 Dataset. We demonstrate that
our method outperforms pooling of convolutional spatial features by 10%.
Secondly, we show our method improves performance on scene level features
extracted via an inflated 3D CNN [Feichtenhofer(2020)] by 11%. Third, we
compare our approach with other methods for combining convolutional features
temporally, including an LSTM [Donahue et al.(2015)Donahue, Anne Hendricks,
Guadarrama, Rohrbach, Venugopalan, Saenko, and Darrell], S(TPN) [Zheng et
al.(2019)Zheng, An, Wu, and Ruan], concatenation [Fish et al.(2021)Fish,
Weinbren, and Gilbert], and collaborative gating [Liu et al.(2019)Liu,
Albanie, Nagrani, and Zisserman]. Finally, we show results for an alternative
method for two-stream aggregation using a distillation network as described in
[Touvron et al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and Jégou]. We
outperform all existing methods for the genre classification task.

width=1  Method CNN Feature Extractor Frames per Scene Aggregation Method
$\overline{mAP}$ ResNet [He et al.(2016)He, Zhang, Ren, and Sun] ResNet18 1
Avg Pool 0.434 SqueezeExcite [Hu et al.(2018)Hu, Shen, and Sun] SE-ResNet 1
Avg Pool 0.544 I3D [Carreira and Zisserman(2017)] I3D 12 Avg Pool 0.487
Collaborative Gating [Liu et al.(2019)Liu, Albanie, Nagrani, and Zisserman]
ResNet18 12 Gated Unit 0.4723 S(TPN) [Zheng et al.(2019)Zheng, An, Wu, and
Ruan] ResNet18 12 Temporal Pyramid 0.492 Fine-Grained Semantic [Fish et
al.(2021)Fish, Weinbren, and Gilbert] Multi-Modal 16 Concatenation 0.583 LSTM
[Donahue et al.(2015)Donahue, Anne Hendricks, Guadarrama, Rohrbach,
Venugopalan, Saenko, and Darrell] ResNet18 1 LSTM 0.596 Distillation [Hinton
et al.(2015)Hinton, Vinyals, Dean, et al., Touvron et al.(2021)Touvron, Cord,
Douze, Massa, Sablayrolles, and Jégou] ResNet18 + R2+1D 12 Transformer 0.601
STAN-Small ResNet18 + R2+1D 12 Transformer 0.640 STAN-Large ResNet18 + R2+1D
12 Transformer 0.750

Table 1: Comparison of our proposed approach with existing methods for video classification using CNN feature extractors and evaluated on the MMX-Trailer-20 Dataset. We implement several feature extraction and aggregation methods to evaluate their effectiveness for the long video classification task. Method | Relation | Speaking | Scene
---|---|---|---
R101-SlowFast+NL [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 52.4 | 35.8 | 54.7
VideoBERT [Sun et al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid] | 52.8 | 37.9 | 54.9
Object Transformer [Wu and Krahenbuhl(2021)] | 53.1 | 39.4 | 56.9
STAN-Large (ours) | 56.25 | 41.41 | 58.33

Table 2: Accuracy of our approach on long video understanding tasks using the
Long Video Understanding Dataset. The reader can find references and further
details in [Wu and Krahenbuhl(2021)]. We outperform current approaches for
classifying conversation (speaking), character relationships (relation), and
locations of scenes (scene).

In Table 2 we show that we achieve SOTA results for three long-form video
understanding tasks on the LVU Benchmark outperforming other methods which
also utilise transformer architectures such as VideoBERT [Sun et al.(2019)Sun,
Myers, Vondrick, Murphy, and Schmid] and [Wu and Krahenbuhl(2021)] which
utilises pre-trained CNN backbones plus self-supervised masked pre-training.
Our architecture performs particularly well on the relationship identification
task $(\bf{+3.1})$ despite having no prior knowledge of the domain in pre-
training.

Ablation Experiments In Table 4 we perform several ablation studies to assess
the impact of the spatial and temporal features and the data efficiency of the
model. To measure the data efficiency, we randomly sampled 2000 trailers from
the MMX-Trailer-20 training partition, with the entire test partition of 754.
Table 4 shows that the model still achieves SOTA performance on the MMX-
Trailer-Dataset despite being trained on only a quarter of the samples
demonstrating high data efficiency. We infer that data efficiency is improved
by using convolutional encoding as the transformer network only needs to map
self attention between the scene feature tokens rather than pixel localities.
Furthermore, the proposed network architecture can infer translation in-
variance within the convolutional encoding.

In Table 4 we also provide further results for models that only use either the
spatial or temporal convolutional encoder to assess the impact of propagating
gradients through the convolutional encoders. We observe that back-propagating
through the spatial convolutional encoder provides the most significant
performance gain with the most negligible effect on the number of trainable
parameters. Training both the CNN encoders end to end (STAN-Large) is the most
effective method for achieving high accuracy. Still, it comes at a cost,
increasing the number of trainable parameters by 48 million. We find that
using just the temporal transformer encoder and introducing a stop gradient
before the convolutional feature extractor performs well but is improved with
spatial features for 28 million additional trainable parameters. We also show
genre specific results in Table 3.

width=1  Model Actn Advnt Animtn Bio Cmdy Crme Doc Drma Famly Fntsy Hstry
Hrror Mystry Music SciFi Sprt Shrt Thrll War $F1_{w}$ $mAP$ $P_{w}$ $R_{w}$
Support 130 197 46 13 224 102 87 267 117 115 44 104 41 86 107 30 45 12 21 - -
- - Random 0.29 0.41 0.11 0.03 0.46 0.24 0.21 0.52 0.27 0.26 0.11 0.24 0.1 0.2
0.25 0.08 0.11 0.03 0.05 0.318 0.134 0.19 1 ResNet [He et al.(2016)He, Zhang,
Ren, and Sun] 0.43 0.55 0.74 0 0.49 0.38 0.63 0.55 0.51 0.28 0.24 0.42 0.3
0.28 0.41 0.22 0.19 0.11 0.33 0.434 0.489 0.437 0.48 VGG-Audio [Abu-El-Haija
et al.(2016)Abu-El-Haija, Kothari, Lee, Natsev, Toderici, Varadarajan, and
Vijayanarasimhan] 0.47 0.51 0.40 0.10 0.61 0.38 0.58 0.55 0.51 0.37 0.11 0.34
0.39 0.30 0.35 0.16 0.15 0.13 0.12 0.454 0.449 0.400 0.537 I3D [Carreira and
Zisserman(2017)] 0.5 0.59 0.74 0 0.62 0.33 0.63 0.56 0.55 0.36 0.2 0.38 0.45
0.24 0.37 0.23 0.14 0.10 0.13 0.463 0.487 0.448 0.494 SqueezeExcite [Hu et
al.(2018)Hu, Shen, and Sun] 0.48 0.63 0.79 0.12 0.65 0.41 0.60 0.59 0.55 0.42
0.25 0.47 0.42 0.29 0.50 0.34 0.19 0.12 0.31 0.516 0.554 0.493 0.572 Naive
Concat [He et al.(2016)He, Zhang, Ren, and Sun] 0.56 0.61 0.64 0.09 0.64 0.35
0.69 0.60 0.58 0.39 0.19 0.49 0.45 0.21 0.48 0.39 0.28 0.27 0.41 0.525 0.497
0.522 0.551 Fine-Grained Semantic [Fish et al.(2021)Fish, Weinbren, and
Gilbert] 0.62 0.69 0.71 0.11 0.71 0.53 0.73 0.62 0.51 0.34 0.56 0.60 0.45 0.50
0.64 0.30 0.11 0.13 0.55 0.597 0.583 0.554 0.697 STAN-Small 0.71 0.68 0.92
0.21 0.61 0.65 0.62 0.69 0.86 0.49 0.46 0.58 0.43 0.39 0.53 0.13 0.20 0.85
0.50 0.65 0.64 0.62 0.73

Table 3:  Genre classification performance for each genre on the MMX-Trailer-20 dataset. We observe high performance gains on genres in which temporal information can be considered an important classifier such as Action $+9$ and Animation $+11$. We also observe that other network architectures perform very poorly on classification of the genre thriller while we improve accuracy by $+58$. We conclude that long-term temporal modelling performs well on this task as the content is difficult to classify when features are presented in isolation. Method | Samples | Parameters (Millions) | mAP
---|---|---|---
Spatial with backprop | 2000 | 28 | 0.5903
Temporal with backprop | 2000 | 48 | 0.6221
Spatial no backprop | 2000 | 16.5 | 0.4024
Temporal no backprop | 2000 | 16.5 | 0.59
Distillation Network | 2000 | 44.5 | 0.6005
Gated Fusion | 2000 | 45 | 0.4728
STAN-Small | 2000 | 45 | 0.6151
STAN-Small | 6047 | 45 | 0.6401
STAN-Large | 6047 | 92.5 | 0.7506

Table 4: Ablation experiments assess the network quality under a constrained
data training protocol. Each model is trained using only 2000 samples, while
the test length remained consistent at 754 samples. The network continues to
outperform existing methods with fewer data. We also show results for
individual spatial and temporal streams with back-propagation.  Figure 3:
Class activation maps of randomly selected samples from the MMX-Trailer-20
test partition. Each sub-figure represents a series of input tokens from the
spatial encoder $h$. The class activation maps show pixel regions in red to
blue, with red regions representing high activation and, therefore, greater
contribution to the predicted class labels.

Qualitative Results In Fig 3 we show class activation maps of the input 2D CNN
encoder, shown in Fig 2 as $h()$ and the predicted genres for a given input
video. We observe that the model predicts the labels correctly and learns
cohesive features for the input tokens. For example in Fig 3 we see that the
Family genre is predicted when we have a strong activation on animals faces.
For the Horror classification, spatial features include people screaming and
dark scenes. We also notice that text is an important feature for the
classification task, acting as a strong temporal marker in the sequence of
tokens. Class activation maps are obtained via Grad-CAM[Selvaraju et
al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra] using the code
provided by [Gildenblat and contributors(2021)]. We refer the reader to the
appendix for further qualitative results and ablation experiments.

##  5 Conclusions

We present a data and memory efficient spatio-temporal attention network for
long video classification which combines the advantages of convolutional
inductive bias with the computational advantages of transformer networks for
the task of long video classification. We show that by using static image and
temporal context convolutional tokens we are able to create a data efficient
architecture capable of classifying videos of up to two minutes in length and
trained on a single GPU while achieving SOTA results on a range of long-form
video understanding tasks.

## References

  * [Abu-El-Haija et al.(2016)Abu-El-Haija, Kothari, Lee, Natsev, Toderici, Varadarajan, and Vijayanarasimhan] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  _arXiv preprint arXiv:1609.08675_ , 2016.
  * [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun, Lučić, and Schmid] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Baccouche et al.(2011)Baccouche, Mamalet, Wolf, Garcia, and Baskurt] Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, and Atilla Baskurt.  Sequential deep learning for human action recognition.  In _International workshop on human behavior understanding_. Springer, 2011.
  * [Bertasius et al.(2021)Bertasius, Wang, and Torresani] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  _arXiv preprint arXiv:2102.05095_ , 2021.
  * [Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, et al.] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * [Carreira and Zisserman(2017)] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [Cascante-Bonilla et al.(2019)Cascante-Bonilla, Sitaraman, Luo, and Ordonez] Paola Cascante-Bonilla, Kalpathy Sitaraman, Mengjia Luo, and Vicente Ordonez.  Moviescope: Large-scale analysis of movies using multiple modalities.  _arXiv preprint arXiv:1908.03180_ , 2019.
  * [Chao et al.(2018)Chao, Vijayanarasimhan, Seybold, Ross, Deng, and Sukthankar] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [d’Ascoli et al.(2021)d’Ascoli, Touvron, Leavitt, Morcos, Biroli, and Sagun] Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  _arXiv preprint arXiv:2103.10697_ , 2021.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR-09_ , 2009.
  * [Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * [Donahue et al.(2015)Donahue, Anne Hendricks, Guadarrama, Rohrbach, Venugopalan, Saenko, and Darrell] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015.
  * [Dong et al.(2016)Dong, Li, and Snoek] Jianfeng Dong, Xirong Li, and Cees GM Snoek.  Word2visualvec: Image and video to sentence matching by visual feature prediction.  _arXiv preprint arXiv:1604.06838_ , 2016.
  * [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * [Fan et al.(2021)Fan, Xiong, Mangalam, Li, Yan, Malik, and Feichtenhofer] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Feichtenhofer(2020)] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019.
  * [Fish et al.(2021)Fish, Weinbren, and Gilbert] Edward Fish, Jon Weinbren, and Andrew Gilbert.  Rethinking genre classification with fine grained semantic clustering.  In _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021.
  * [Fouhey et al.(2018)Fouhey, Kuo, Efros, and Malik] David F Fouhey, Wei-cheng Kuo, Alexei A Efros, and Jitendra Malik.  From lifestyle vlogs to everyday interactions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [Gildenblat and contributors(2021)] Jacob Gildenblat and contributors.  Pytorch library for cam methods.  https://github.com/jacobgil/pytorch-grad-cam, 2021.
  * [Girdhar and Grauman(2021)] Rohit Girdhar and Kristen Grauman.  Anticipative video transformer.  _arXiv preprint arXiv:2106.02036_ , 2021.
  * [Girdhar et al.()Girdhar, Carreira, Doersch, and Zisserman] Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  _CoRR_.  URL http://arxiv.org/abs/1812.02707.
  * [Guo et al.(2021)Guo, Han, Wu, Xu, Tang, Xu, and Wang] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang.  Cmt: Convolutional neural networks meet vision transformers.  _arXiv preprint arXiv:2107.06263_ , 2021.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * [Hinton et al.(2015)Hinton, Vinyals, Dean, et al.] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * [Ho et al.(2019)Ho, Kalchbrenner, Weissenborn, and Salimans] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans.  Axial attention in multidimensional transformers.  _arXiv/1912.12180_ , 2019.
  * [Hori et al.(2017)Hori, Hori, Lee, Zhang, Harsham, Hershey, Marks, and Sumi] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi.  Attention-based multimodal fusion for video description.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4193–4202, 2017.
  * [Hu et al.(2018)Hu, Shen, and Sun] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * [Huang et al.(2020)Huang, Xiong, Rao, Wang, and Lin] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin.  Movienet: A holistic dataset for movie understanding.  In _European Conference on Computer Vision_. Springer, 2020.
  * [Huang and Wang(2012)] Yin-Fu Huang and Shih-Hao Wang.  Movie genre classification using svm with audio and video features.  In _International Conference on Active Media Technology_. Springer, 2012.
  * [Jégou et al.(2010)Jégou, Douze, Schmid, and Pérez] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez.  Aggregating local descriptors into a compact image representation.  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , 2010.
  * [Ji et al.(2012)Ji, Xu, Yang, and Yu] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 2012.
  * [Karpathy et al.(2014)Karpathy, Toderici, Shetty, Leung, Sukthankar, and Fei-Fei] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014.
  * [Lin et al.(2019)Lin, Gan, and Han] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7083–7093, 2019.
  * [Liu et al.(2019)Liu, Albanie, Nagrani, and Zisserman] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  _BMVC-19_ , 2019.
  * [Lu et al.(2019)Lu, Batra, Parikh, and Lee] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_ , 2019.
  * [Marszalek et al.(2009)Marszalek, Laptev, and Schmid] Marcin Marszalek, Ivan Laptev, and Cordelia Schmid.  Actions in context.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2009.
  * [Miech et al.(2017a)Miech, Laptev, and Sivic] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with Context Gating for video classification, jun 2017a.
  * [Miech et al.(2017b)Miech, Laptev, and Sivic] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  _arXiv preprint arXiv:1706.06905_ , 2017b.
  * [Mithun et al.(2018)Mithun, Li, Metze, and Roy-Chowdhury] Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval.  In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_ , 2018.
  * [Neimark et al.(2021)Neimark, Bar, Zohar, and Asselmann] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann.  Video transformer network.  _arXiv preprint arXiv:2102.00719_ , 2021.
  * [Pan et al.(2021)Pan, Chen, Shou, Liu, Shao, and Li] Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li.  Actor-context-actor relation network for spatio-temporal action localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [Perronnin and Dance(2007)] Florent Perronnin and Christopher Dance.  Fisher kernels on visual vocabularies for image categorization.  In _2007 IEEE conference on computer vision and pattern recognition_ , 2007.
  * [Qiu et al.(2017)Qiu, Yao, and Mei] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In _proceedings of the IEEE International Conference on Computer Vision_ , 2017.
  * [Rumelhart et al.(1985)Rumelhart, Hinton, and Williams] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation.  Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  * [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * [Simonyan and Zisserman(2014)] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  _Advances in neural information processing systems_ , 27, 2014.
  * [Sun et al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7464–7473, 2019.
  * [Touvron et al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and Jégou] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_. PMLR, 2021.
  * [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , 2015.
  * [Tran et al.(2018)Tran, Wang, Torresani, Ray, LeCun, and Paluri] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018.
  * [Tran et al.(2019)Tran, Wang, Torresani, and Feiszli] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video classification with channel-separated convolutional networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * [Wang et al.(2017)Wang, Jiang, Qian, Yang, Li, Zhang, Wang, and Tang] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * [Wang et al.(2020)Wang, Xiong, Neumann, Piergiovanni, Ryoo, Angelova, Kitani, and Hua] Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua.  Attentionnas: Spatiotemporal attention cell search for video classification.  In _European Conference on Computer Vision_ , pages 449–465. Springer, 2020.
  * [Wu and Krahenbuhl(2021)] Chao-Yuan Wu and Philipp Krahenbuhl.  Towards long-form video understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1884–1894, 2021.
  * [Wu et al.(2015)Wu, Wang, Jiang, Ye, and Xue] Zuxuan Wu, Xi Wang, Yu-Gang Jiang, Hao Ye, and Xiangyang Xue.  Modeling spatial-temporal clues in a hybrid deep learning framework for video classification.  In _Proceedings of the 23rd ACM international conference on Multimedia_ , pages 461–470, 2015.
  * [Xie et al.(2018)Xie, Sun, Huang, Tu, and Murphy] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 305–321, 2018.
  * [Yang et al.(2018)Yang, Tu, Wong, Meng, Chao, and Zhang] Baosong Yang, Zhaopeng Tu, Derek F Wong, Fandong Meng, Lidia S Chao, and Tong Zhang.  Modeling localness for self-attention networks.  _arXiv preprint arXiv:1810.10182_ , 2018.
  * [Yuan et al.(2021)Yuan, Chen, Wang, Yu, Shi, Jiang, Tay, Feng, and Yan] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  _arXiv preprint arXiv:2101.11986_ , 2021.
  * [Yue-Hei Ng et al.(2015)Yue-Hei Ng, Hausknecht, Vijayanarasimhan, Vinyals, Monga, and Toderici] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015.
  * [Zhang et al.(2021)Zhang, Li, Liu, Shuai, Zhu, Brattoli, Chen, Marsic, and Tighe] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe.  Vidtr: Video transformer without convolutions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13577–13587, 2021.
  * [Zhao et al.(2019)Zhao, Zhang, Xiao, Xiao, Yan, Yu, Cai, and Wu] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, and Fei Wu.  Long-form video question answering via dynamic hierarchical reinforced networks.  _IEEE Transactions on Image Processing_ , 28(12):5939–5952, 2019.
  * [Zheng et al.(2019)Zheng, An, Wu, and Ruan] Zhenxing Zheng, Gaoyun An, Dapeng Wu, and Qiuqi Ruan.  Spatial-temporal pyramid based convolutional neural network for action recognition.  _Neurocomputing_ , 358:446–455, 2019.
  * [Zhou et al.(2010)Zhou, Hermans, Karandikar, and Rehg] Howard Zhou, Tucker Hermans, Asmita V Karandikar, and James M Rehg.  Movie genre classification via scene categorization.  In _Proceedings of the 18th ACM international conference on Multimedia_ , 2010.


# Routing and QoS Policy Optimization in SD-WAN

1Pham Tran Anh Quang, 1Xu Gong, 1Chuang Liu, 1Kun Li, 1Jérémie Leguay
2Xinhui Zhang, 2Yong Zhang, 2Jiayan Li, 2Kai Ye

1Huawei Technologies Ltd., 2Industrial and Commercial Bank of China

###### Abstract

In modern SD-WAN networks, a global controller continuously optimizes
application and user intents by selecting the proper routing policies for each
application. Nevertheless, the competition between flows can still occur at
each overlay link and it may degrade Quality of Service (QoS). To mitigate
performance degradations in case of congestion, QoS policies can also be
dynamically optimized to limit the rate of low-priority flows and share the
bandwidth among all flows in a fair manner. This demonstration presents a
comprehensive control plane architecture to coordinate path selection and rate
allocation, in order to meet application requirements while optimizing a
global objective (e.g., low congestion, best quality, and minimum cost).

##  I Introduction

Financial companies, as other businesses, are transforming their WAN (Wide
Area Network) to reduce costs while optimizing user experience. To do so, they
are migrating from fully dedicated private lines, using MSTP (Multi-service
SDH-based Transport Platform) for instance, to a cost-efficient mix of
multiple access networks using SD-WAN. In particular, they are smoothly
upgrading their WAN by 1) downsizing some of the existing MSTP lines to reduce
cost and 2) combining them with cheaper and shared private leased lines based
on MPLS-VPN (MV). While MSTP lines can guarantee strict Service Level
Agreements (SLA) requirements, the reduction of their capacity along with the
use of shared private lines, offering weaker QoS guarantees, is calling for
advanced traffic engineering mechanisms based on dynamic load balancing and
queuing. Indeed, the main challenge is to intelligently use all the available
capacity to meet applications requirements and optimize user experience.

This demonstration will consider a use case from the Industrial and Commercial
Bank of China (ICBC), illustrated in Fig. 1, where $2$ hubs (regional
branches) are connected to 3 spokes (other sites except regional branches). In
the original setup, MSTP lines at $18$ Mpbs connect hubs to spokes (capacities
have been scaled down to keep simulations tractable but proportions are kept).
The Metropolitan Area Network (MAN) between hubs is not used as all flows can
only be routed over a single path (flow-level multi-path traffic steering is
not available). After migration to SD-WAN, existing MSTP lines are downsized
by $66\%$ to save cost and they are combined with other access networks: 1)
existing hub-to-hub connectivity can now be used as one routing option between
hubs and spokes, and 2) shared MV lines at $12$ Mbps are subscribed. The total
cost after network transformation is reduced by $26$% approximately.

We previously demonstrated a policy optimization algorithm for Smart Policy
Routing (SPR) [1] to optimize global intents (e.g., minimum cost, best delay),
while meeting QoS requirements of applications [2, 3]. This demonstration is a
further improvement with a joint optimization of SPR and QoS policies. Indeed,
while load balancing greatly improves performance, dynamic QoS policies can
further improve performance in case of congestion or tight SLA requirements.
In this demonstration, the network controller periodically decides the set of
overlay links that each application, also called flow group, is allowed to use
(i.e., SPR policies), and the QoS policy that is applied to each flow group on
each overlay link.

(a) Access network with MSTP only

(b) SD-WAN network combining MSTP, MV, and MAN

Figure 1: Network transformation scenario from a) to b).

A number of Active Queue Management (AQM) techniques have been proposed to
help sustain delay and throughput requirements [4]. Most of them uses implicit
(packet dropping) or explicit signals (ECN marking) to control sources sharing
bottleneck link. They do not aim at tuning queuing parameters and optimizing
performance globally. Other works on Adaptive Weighted Fair Queuing (AWFQ)
have proposed the dynamic adaptation of scheduling parameters. However, they
consider the control of only one device (local approach, no coordination with
among devices) [5], a distributed approach using bandwidth pricing to
coordinate multiple devices [6] or a distributed protocol to trigger queue
adjustments [7]. In the later solution, agents at the destination informs
about delay violations so that upstream agents adjust their weights.

Our solution goes beyond state-of-the-art, as it defines a global framework to
optimize routing and queuing parameters at the centralized controller to
coordinate all edge devices in the context of overlay networks. Existing
solutions do not consider a global optimization and do not perform a joint
optimization of routing and QoS.

Figure 2: System architecture with the controller and routers.

##  II Joint policy optimization

As depicted in Fig. 2, a centralized controller can modify periodically
routing policies and queuing parameters with corresponding agents. In this
demonstration, we consider the case where these agents operate independently
and in a time-triggered manner. Routing policies are updated at a slow pace to
leave some time for the load balancing to converge, while queuing policies are
updated more frequently as they immediately impact performance.

As presented in Fig. 3, we consider a hierarchical QoS scheduler based on
Class-Based Queuing (CBQ) that handles high priority flows with Priority
Queuing (PQ) and low priority flows with Weighted Fair Queuing (WFQ). A CBQ
scheduler is deployed at each port of each edge router (at hubs and spokes)
while the other devices in the middle (in the underlay) operate using a single
First In First Out (FIFO) queue at each port. The bandwidth allocation for
each of the low priority flow groups is controlled thanks to shapers (i.e.,
maximum rate) and scheduling weights (minimum rate).

Intent-based routing and smart queuing. Ingress routers are configured with
SPR and QoS policies for each flow group. The SPR policy [1] contains the set
of allowed overlay links that a flow group can use. The QoS policy defines
queuing parameters, e.g. queue weights and shaper rates. At each ingress
device, the sub-set of active paths over which each flow group is load
balanced is determined using link-level measurements (i.e., delay, jitter,
loss). In case of congestion, the performance of high-priority flows can
degrade and bandwidth starvation can occur for low-priority flows. Therefore,
the dynamic tuning of QoS policies aims at protecting high-priority flow
groups and while ensuring fairness in performance degradation for low-priority
flow groups. In this demonstration, we present an automatic configuration
scheme for both SPR and QoS policies to enhance SLA satisfaction for all flow
groups.

Problem formulations. We consider a set of overlay links $E$ of capacity
$C_{e}$ and a set of flow groups $K$. For each flow group $k\in K$, the
measured traffic demand is denoted $b^{k}$ and the set of all possible overlay
links is given by $E_{k}\subseteq E$ (e.g., outgoing links of ingress
routers). The delay requirement of flow group $k$ is $D_{k}$.

1) SPR policy optimization problem. The following model optimizes a global
intent (e.g. minimize congestion or maximize quality) while meeting SLAs (see
[3] for model details).

$\displaystyle\min\quad$ $\displaystyle\sum_{k\in K}\sum_{e\in
E_{k}}\bigl{[}\sum_{i\in I_{e}}MLU\text{ (congestion) or }f_{e}^{k}(x)\text{
(quality)}\bigr{]}$ s.t. $\displaystyle MLU\geq\sum_{i\in
I_{e}}LU_{e}^{i}=\sum_{k\in K_{e}}b_{k}x_{e}^{k}\leq C_{e},\quad\forall e\in
E,$ (1) $\displaystyle f_{e}^{k}(x)\leq D_{k}\quad\forall k\in K,\forall e\in
E_{k},$ (2) $\displaystyle\sum_{e\in E_{k}}x_{e}^{k}=1,\quad\forall k\in K$
(3)

Once SPR policies have been optimized through a slow control loop, the traffic
for flow group $k$ over overlay link $e$ is expected to be
$d^{k}_{e}=b^{k}x_{e}^{k}$. To further optimize bandwidth sharing at each
port, QoS policies are optimized over a faster control loop thanks to the
following model.

Figure 3: Class-Based Queuing (CBQ) scheduler

2) QoS policy optimization problem.

$\displaystyle\min\quad$
$\displaystyle\sum_{e}\left(-d^{k}_{e}\log{z^{k}_{e}}+\sum_{k:d^{k}_{e}>0}\left(\alpha\frac{y^{k}_{e}}{D_{k}}+M^{k}h^{k}_{e}\right)\right)$
s.t. $\displaystyle\sum_{k}z^{k}_{e}\leq C_{e},\quad\forall e\in E,$ (4)
$\displaystyle f^{k}_{e}(z)\leq D_{k}+y_{e}^{k}\quad\forall k\in K,\forall
e\in E_{k},$ (5) $\displaystyle z^{k}_{e}+h^{k}_{e}\geq d^{k}_{e},\quad\forall
k\in K,\forall e\in E$ (6)

The rate allocation violation $h_{e}^{k}$ identifies the gap between the
demand and the allocated rate. The SLA violation $y_{e}^{k}$ is the gap
between SLA requirements (e.g., end-to-end delay) and prediction. The SLA
prediction $f_{e}^{k}(z)$ is a function of the rate allocation that could be
derived from queuing theory or a data-driven model. The objective of the
optimization problem is to (i) satisfy the traffic demand ($M^{k}$ is large to
prioritize this objective), (ii) ensure fairness, and (iii) minimize SLA
violations. We introduce $\alpha$ to trade-off fairness with SLA violations.
In this demonstration, we select $\alpha=0.01$ to give more priority to the
bandwidth fairness. The output is the rate allocation for each flow group,
$z^{k}_{e}$, which is used to tune queuing weights and shaping.

Figure 4: Visualization of the total traffic and the end-to-end delay for each
application, and the MLU.

SPR and QoS optimization problems can be solved using a solver (e.g. SCIP) and
heuristic algorithms like the local search algorithm are able to optimize SPR
policies efficiently.

App Policy |  All-TNs(a) |  All-TNs(b) |  MLU(b) |  MLU-QoS(b)
---|---|---|---|---
| APP SLA satisfaction rate (%) / Avg E2E delay (ms) / $95^{th}$ E2E delay
(ms)
Critical |  100 / 10.6 / 10.8 |  78 / 27.9 / 127 |  100 / 11.0 / 11.3 |  1.00 / 11.7 / 13.3
VoIP |  100 / 10.6 / 10.8 |  84 / 27.3 / 105 |  95 / 15.0 / 17.9 |  100 / 14.4 / 17.4
Video |  100 / 11.0 / 11.2 |  79 / 56.2 / 249 |  93 / 28.8 / 157.3 |  100 / 16.6 / 18.3
Interactive |  100 / 11.1 / 11.5 |  77 / 65.4 / 262 |  95 / 24.2 / 43.5 |  100 / 17.2 / 19.8
Bulk |  100 / 11.2 / 12.0 |  92 / 45.8 / 192 |  100 / 17.7 / 18.9 |  98 / 23.6 / 69.4
Office |  100 / 12.0 / 14.0 |  85 / 64.1 / 291 |  91 / 50.8 / 243.0 |  98 / 33.2 / 121.0
Table I: Performance for all policy optimization scenarios

##  III Demonstration

We use NS3 [8] and applications are classified into $6$ flow groups with
increasing end-to-end delay requirements and priority: Critical ($15$ms), VoIP
($20$ms), Video ($40$ms), Interactive ($30$ms), Bulk ($100$ms) and Office
($150$ms). Each spoke receives $6$ flow groups from servers located either at
hub 1 or 2. The traffic demand is the same in scenarios of Fig. 1 (a&b). The
transport layer is TCP. The microflow inter-arrival time in each flow group
varies to generate diurnal traffic patterns. The propagation delay of MSTP and
MV links are $10$ms and $15$ms, respectively. A Hierarchical QoS scheduler is
deployed to prioritize traffic. The $4$ highest priority applications can
preempt other applications (strict priorities). Bandwidth sharing among the
$2$ lowest priority flow groups is ensured using WFQ and their individual rate
can be shaped. Packet priorities are marked using DSCP. Link-level
measurements are collected every $1$s. SPR and QoS policies are updated by the
controller every $50$s and $10$s, respectively (periods may depend on traffic
variations).

Table I provides End-to-End (E2E) delay (average and $95^{th}$ percentile) and
SLA satisfaction rate (% of epochs SLA requirement is met) for each flow
group. Three configurations of SPR and QoS policies are considered: (i) All-
TNs where all links can be used by ingress devices (SPR policies are not
optimized by the controller) (ii) MLU where SPR policies are optimized to
minimize congestion (i.e., Maximum Link Utilization) but fixed queuing
parameters are used, and (iii) MLU-QoS where SPR and QoS policies are jointly
optimized. For All-TNs, the two scenarios of Figs. 1a&b are compared, denoted
as All-TNs(a) and All-TNs(b). The first scenario has higher MSTP capacities
but higher deployment cost. As a consequence, it can satisfy all SLA
requirements. MLU-QoS has the best performance for the SD-WAN network
configuration. It can guarantee SLAs in more than $95\%$ of times. Without QoS
policy optimization, low-priority flows experience bandwidth starvation, which
causes SLA violations. In addition, high-priority flows group are not
protected against low-priority flows.

Fig. 4 presents the graphical interface of our demo: a dashboard to visualize
for all policy optimization strategies the total traffic and the end-to-end
delay, and the congestion (i.e., MLU). The video is here:
https://tinyurl.com/3atckfpf

## References

  * [1] Huawei Technologies, “Smart Policy Routing,” Tech. Rep., 2021. [Online]. Available: https://support.huawei.com/enterprise/en/doc/EDOC1000174111/2ed71c78/smart-policy-routing
  * [2] P. Quang, S. Martin, J. Leguay, X. Gong, and F. Zeng, “Intent-based policy optimization in sd-wan,” in _Proc. of SIGCOMM’21 Poster and Demo Sessions_ , 2021, pp. 74–75.
  * [3] P. Quang, S. Martin, J. Leguay, X. Gong, and H. Xu, “Intent-based routing policy optimization in sd-wan,” in _Proc. of IEEE ICC_ , 2022.
  * [4] R. Adams, “Active queue management: A survey,” _IEEE communications surveys & tutorials_, vol. 15, no. 3, pp. 1425–1476, 2012.
  * [5] T. Frantti and M. Jutila, “Embedded fuzzy expert system for adaptive weighted fair queueing,” _Expert Systems with Applications_ , vol. 36, no. 8, pp. 11 390–11 397, 2009.
  * [6] A. Sayenko, T. Hämäläinen, J. Joutsensalo, and L. Kannisto, “Comparison and analysis of the revenue-based adaptive queuing models,” _Computer Networks_ , vol. 50, no. 8, pp. 1040–1058, 2006.
  * [7] S. Hussain and A. Marshall, “An agent-based control mechanism for wfq in ip networks,” _Control engineering practice_ , vol. 11, no. 10, pp. 1143–1151, 2003.
  * [8] G. F. Riley and T. R. Henderson, _The ns-3 Network Simulator_ , K. Wehrle, M. Güneş, and J. Gross, Eds., 2010.