(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  OpenGVLab, Shanghai AI Laboratory 22institutetext: Shenzhen
Institute of Advanced Technology, Chinese Academy of Sciences 33institutetext:
University of Chinese Academy of Sciences 44institutetext: State Key
Laboratory for Novel Software Technology, Nanjing University
https://github.com/OpenGVLab/VideoMamba

#  VideoMamba: State Space Model for Efficient Video Understanding

Kunchang Li 22331♠1♠ Xinhao Li 441♠1♠ Yi Wang 1♡1♡ Yinan He
Yali Wang 1 https://github.com/OpenGVLab/VideoMamba 1221♡1♡ Limin Wang 441♡1♡
Yu Qiao 1♡1♡

###### Abstract

††$\spadesuit$ Interns at Shanghai AI Laboratory. $\heartsuit$ Corresponding
authors.

Addressing the dual challenges of local redundancy and global dependencies in
video understanding, this work innovatively adapts the Mamba to the video
domain. The proposed VideoMamba overcomes the limitations of existing 3D
convolution neural networks and video transformers. Its linear-complexity
operator enables efficient long-term modeling, which is crucial for high-
resolution long video understanding. Extensive evaluations reveal VideoMamba’s
four core abilities: (1) Scalability in the visual domain without extensive
dataset pretraining, thanks to a novel self-distillation technique; (2)
Sensitivity for recognizing short-term actions even with fine-grained motion
differences; (3) Superiority in long-term video understanding, showcasing
significant advancements over traditional feature-based models; and (4)
Compatibility with other modalities, demonstrating robustness in multi-modal
contexts. Through these distinct advantages, VideoMamba sets a new benchmark
for video understanding, offering a scalable and efficient solution for
comprehensive video understanding. All the code and models are available.

##  1 Introduction

The core objective for video understanding lies in mastering spatiotemporal
representations, which inherently presents two formidable challenges: the
large spatiotemporal redundancy within short video clips, and the complex
spatiotemporal dependencies among long contexts. Although the once-dominant 3D
convolutional neural networks (CNNs)[76, 9, 19] and video transformers[4, 2],
effectively tackle one of the challenges mentioned by leveraging either local
convolution or long-range attention, they fall short in addressing both
simultaneously. UniFormer [44] attempts to integrate the advantages of both
methods, but it struggles with modeling long videos, which has been the major
trend in recent research on video understanding [72, 48] and generation [5,
92].

Figure 1: Comparisons of throughput and memory. The TimeSformer-Ti [4] is
built based on DeiT-Ti [75] with joint spatiotemporal attention. All the input
frames are sized to 224$\times$224\. The testing is conducted on an NVIDIA
A100-80G GPU, utilizing PyTorch 2.1 and CUDA 11.8, with a batch size of 128.
Our VideoMamba is better, faster and cheaper for both short-term and long-term
video understanding.

The emergence of low-cost operators such as S4 [26], RWKV [73], and RetNet
[70] in the NLP domain, has carved a novel pathway for the vision model. Mamba
[25] stands out with its selective state space model (SSM), striking a balance
between maintaining linear complexity and facilitating long-term dynamic
modeling. This innovation has spurred its adoption in vision tasks, as
evidenced by Vision Mamba [91] and VMamba [50], which leverage multi-
directional SSMs for enhanced 2D image processing. These models rival
attention-based architectures in performance while offering a significant
reduction in memory usage. Given the inherently longer sequences produced by
video, a natural question arises: Can Mamba work well for video understanding?

Inspired by this, we introduce VideoMamba, a purely SSM-based model tailored
for video understanding. VideoMamba harmoniously merges the strengths of
convolution and attention in vanilla ViT [15] style. It offers a linear-
complexity method for dynamic spatiotemporal context modeling, ideal for high-
resolution long videos. The related evaluation focuses on VideoMamba’s four
key abilities:

(1) Scalability in the Visual Domain: We examine VideoMamba’s scalability and
find that, while the pure Mamba model tends to overfit as it scales, our
introduction of a simple yet effective self-distillation strategy allows
VideoMamba to achieve remarkable performance enhancements as the model and
input sizes increase, without the need for large-scale dataset pretraining.

(2) Sensitivity for Short-term Action Recognition: Our analysis extends to
assessing VideoMamba’s capability to accurately distinguish short-term
actions, especially those with fine-grained motion differences, e.g., opening
and closing. The findings reveal VideoMamba’s superior performance over
existing attention-based models [4, 2, 52]. More importantly, it is also
suitable for masked modeling, which further enhances its temporal sensitivity.

(3) Superiority in Long-term Video Understanding: We then assess VideoMamba’s
prowess in interpreting long videos. It showcases remarkable superiority over
conventional feature-based methods [35, 47] through end-to-end training.
Notably, VideoMamba operates 6$\times$ faster than TimeSformer [4] and demands
40$\times$ less GPU memory for 64-frame videos (see Fig. 1).

(4) Compatibility with Other Modalities: Lastly, we assess VideoMamba’s
adaptability with other modalities. Results in video-text retrievals show its
improved performance than ViT, particularly in long videos with complex
scenarios. This underscores its robustness and multi-modal integration
capacity.

In conclusion, our in-depth experiments reveal VideoMamba’s immense potential
in understanding both short-term (K400 [36] and SthSthV2 [24]) and long-term
(Breakfast [37], COIN [71], and LVU [84]) video contents. Given its efficiency
and effectiveness, VideoMamba is poised to become a cornerstone in the realm
of long-video comprehension. All the code and models are open-sourced to
foster future research endeavors.

##  2 Related Works

###  2.1 State Space Models

Recently, the State Space Models (SSMs) have shown significant effectiveness
of state space transformation in capturing the dynamics and dependencies of
language sequences. [26] introduces a structured state-space sequence model
(S4), specifically designed to model long-range dependencies, boasting the
advantage of linear complexity. Based on it, various models have been
developed (e.g., S5 [66], H3 [20] and GSS [56]), and Mamba [25] distinguishes
itself by introducing a data-dependent SSM layer and a selection mechanism
using parallel scan (S6). Compared to transformers [54, 6] based on quadratic-
complexity attention, Mamba excels at processing long sequences with linear
complexity.

In the vision domain, [26] first applies SSM in pixel-level image
classification, and [35] uses S4 to handle the long-range temporal
dependencies for movie clip classification. Besides, the great potential of
Mamba motivates a series of works [91, 50, 46, 87, 28, 30, 78], which
demonstrates Mamba’s better performances and higher GPU efficiency than
Transformer on visual downstream tasks like object detection and semantic
segmentation. Different from the previous works, our VideoMamba is a purely
SSM-based video model, showcasing great efficiency and effectiveness for both
short-term and long-term video understanding.

###  2.2 Video Understanding

Video understanding stands as a cornerstone in the domain of computer vision,
whose significance is further amplified by the burgeoning growth of short
video platforms. To bolster this field, numerous datasets equipped with
extensive data and meticulous human annotations have been developed, aiming to
enhance human action recognition capabilities. Notable examples include UCF101
[67] and Kinetics dataset [36, 7, 8], which have played pivotal roles in
benchmarking progress. Furthermore, other datasets [31, 34, 22, 27, 62, 49]
provide annotated activity videos tailored for action localization, fostering
deeper research into human activities. Beyond action recognition, the advent
of large-scale video-text datasets [86, 10, 88, 12, 57, 82] extends the
utility of video understanding into the realm of multi-modality tasks, such as
video captioning, retrieval and question answering, thereby broadening the
application spectrum.

As for the architecture, it has evolved from using CNN which extracts features
from video frames, to more advanced techniques. Initially, 3D CNNs [76, 9, 17,
77] expanded the traditional 2D CNN architecture to capture videos’ spatio-
temporal information. Two-Stream [65], which combines spatial and temporal
streams, TSN [80], which proposes sparse sampling, and SlowFast [19], which
uses parallel networks to capture semantics and rapid movements, further
enhance action recognition capacity. The introduction of attention-based
models [4, 2, 59, 63, 89], like TimeSformer [4] and ViViT [2], marked a
significant advancement by effectively capturing long-range dependencies
within video sequences, enhancing temporal relationship understanding. Recent
developments [44, 42, 83, 52] have focused on accurate video transformer, with
innovations like the VideoSwin’s window attention [52] and the UniFormer’s
integration of convolution and self-attention mechanisms [44], aiming to
balance computational efficiency with performance. Despite these models’
achievements in various tasks, they often come with high computational costs
for long sequences. In contrast, our VideoMamba introduces a linear-complexity
operator for efficient long-term modeling, outperforming existing methods with
faster speed and lower GPU consumption.

##  3 Method

###  3.1 Preliminaries

SSM for 1D sequence. State Space Models (SSMs) are conceptualized based on
continuous systems that map a 1D function or sequence,
$x(t)\in\mathbb{R}^{L}\rightarrow y(t)\in\mathbb{R}^{L}$ through a hidden
state $h(t)\in\mathbb{R}^{N}$. Formally, SSMs employ the following ordinary
differential equation (ODE) to model the input data:

$\displaystyle h^{\prime}(t)$
$\displaystyle={\mathbf{A}}h(t)+{\mathbf{B}}x(t),$ (1) $\displaystyle y(t)$
$\displaystyle={\mathbf{C}}h(t),$ (2)

where ${\mathbf{A}}\in\mathbb{R}^{N\times N}$ represents the system’s
evolution matrix, and ${\mathbf{B}}\in\mathbb{R}^{N\times
1},{\mathbf{C}}\in\mathbb{R}^{N\times 1}$ are the projection matrices. This
continuous ODE is approximated through discretization in modern SSMs. Mamba
[25] is one of the discrete versions of the continuous system, which includes
a timescale parameter ${\mathbf{\Delta}}$ to transform the continuous
parameters ${\mathbf{A}},{\mathbf{B}}$ to their discrete counterparts
$\overline{{\mathbf{A}}},\overline{{\mathbf{B}}}$. The transformation
typically employs the zero-order hold (ZOH) method, defined by:

$\displaystyle\overline{{\mathbf{A}}}$
$\displaystyle=\exp({\mathbf{\Delta}\mathbf{A}}),$ (3)
$\displaystyle\overline{{\mathbf{B}}}$
$\displaystyle=({\mathbf{\Delta}\mathbf{A}})^{-1}(\exp({\mathbf{\Delta}\mathbf{A}})-{\mathbf{I}})\cdot{\mathbf{\Delta}\mathbf{B}}$
(4) $\displaystyle h_{t}$
$\displaystyle=\overline{{\mathbf{A}}}h_{t-1}+\overline{{\mathbf{B}}}x_{t},$
(5) $\displaystyle y_{t}$ $\displaystyle={\mathbf{C}}h_{t}.$ (6)

Contrary to traditional models that primarily rely on linear time-invariant
SSMs, Mamba distinguishes itself by implementing a Selective Scan Mechanism
(S6) as its core SSM operator. Within S6, the parameters
${\mathbf{B}}\in\mathbb{R}^{B\times L\times N}$,
${\mathbf{C}}\in\mathbb{R}^{B\times L\times N}$, and
${\mathbf{\Delta}}\in\mathbb{R}^{B\times L\times D}$ are directly derived from
the input data $x\in\mathbb{R}^{B\times L\times D}$, indicating an intrinsic
capacity for contextual sensitivity and adaptive weight modulation. Fig. 2a
shows the details of the Mamba block.

Bidirectional SSM for Vision. The original Mamba block, designed for 1D
sequences, falls short for visual tasks requiring spatial awareness. Building
on this, Vision Mamba introduces a bidirectional Mamba (B-Mamba) block in Fig.
2b, which adapts bidirectional sequence modeling for vision-specific
applications. This block processes flattened visual sequences through
simultaneous forward and backward SSMs, enhancing its capacity for spatially-
aware processing. In this work, we extend the B-Mamba block for 3D video
understanding.

Figure 2:  Mamba blocks for 1D [25] and 2D [91] sequence. We omit the initial
normalization and the final residual for simplification.  Figure 3:  Framework
of VideoMamba. We strictly follow the architecture of vanilla ViT [15], and
adapt the bidirectional mamba block[91] for 3D video sequences.

###  3.2 VideoMamba

Overview. Fig. 3 illustrates the overall framework of VideoMamba.
Specifically, we first use 3D convolution (i.e., 1$\times$16$\times$16) to
project the input videos $\mathbf{X}^{v}\in\mathbb{R}^{3\times T\times H\times
W}$ into $L$ non-overlapping spatiotemporal patches
$\mathbf{X}^{p}\in\mathbb{R}^{L\times C}$, where
$L$$=$$t$$\times$$h$$\times$$w$ ($t$$=$$T$, $h$$=$$\frac{H}{16}$, and
$w$$=$$\frac{W}{16}$). The sequence of tokens input to the following
VideoMamba encoder is

$\displaystyle\mathbf{X}={}$
$\displaystyle\left[\mathbf{X}_{cls},\mathbf{X}\right]+\mathbf{p}_{s}+\mathbf{p}_{t},$
(7)

where $\mathbf{X}_{cls}$ is a learnable classification token that is prepended
to the start of the sequence. Following previous works [15, 2, 4], we added a
learnable spatial position embedding
$\mathbf{p}_{s}\in\mathbb{R}^{(hw+1)\times C}$ and the extra temporal one
$\mathbf{p}_{t}\in\mathbb{R}^{t\times C}$ to retain the spatiotemporal
position information, since the SSM modeling is sensitive to token position.
The tokens $\mathbf{X}$ are then passed through by $L$ stacked B-Mamba blocks,
and the representation of ${\rm[CLS]}$ token at the final layer is processed
by normalization and linear layer for classification.

Spatiotemporal Scan. To apply the B-Mamba layer for spatiotemporal input, we
extend the original 2D scan into different bidirectional 3D scans in Fig. 4:
(a) Spatial-First, organizing spatial tokens by location then stacking them
frame by frame; (b) Temporal-First, arranging temporal tokens based on the
frame then stacks along the spatial dimension; (c) Spatiotemporal, a hybrid of
both Spatial-First and Temporal-First, with v1 conducting half of them and v2
conducting full of them ($2\times$ computation). Moreover, our experiments in
Fig. 7 demonstrate that the Spatial-First bidirectional scan is the most
effective yet simple. Thanks to the linear complexity of Mamba, our VideoMamba
is capable of handling long videos of high resolution efficiently.

Figure 4:  Different scan methods. We omit the [CLS] token for simplification.

Comparison to Vim [91] and VMamba [50]. Our VideoMamba builds upon Vim, yet
streamlines its architecture by omitting features such as the middle [CLS]
token and Rotary Position Embedding (RoPE [68]), resulting in superior
performance on ImageNet-1K with gains of +0.8% and +0.7% for Vim-Ti and Vim-S,
respectively. Unlike VMamba, which incorporates additional depthwise
convolution, VideoMamba strictly follows the ViT design without downsampling
layers. To counter the overfitting issues observed in VMamba, we introduce an
effective self-distillation technique outlined in Section 3.3, demonstrate the
isotropic VideoMamba’s great scalability for image and video tasks.

Comparison to TimeSformer [4] and ViViT [2]. Traditional attention-based
models like TimeSformer and ViViT have addressed the self-attention
mechanism’s quadratic complexity by adopting divided spatiotemporal attention.
Despite being more efficient, it introduces additional parameters and
underperforms compared to joint attention, particularly in scenarios involving
masked pretraining [74, 43]. In contrast, VideoMamba processes spatiotemporal
tokens with linear complexity, outperforming TimeSformer on Kinetics-400 by
+2.6% and making significant strides on SthSthV2 with a +5.9% improvement (see
Table 3 and 4). Furthermore, VideoMamba achieves a 6$\times$ increase in
processing speed and requires 40$\times$ less GPU memory for long videos, as
detailed in Fig. 1, demonstrating its efficiency and effectiveness in handling
long-video tasks.

###  3.3 Architecture

Model | #Depth | #Dim | #Param.
---|---|---|---
Tiny | 24 | 192 | 7M
Small | 24 | 384 | 26M
Middle | 32 | 576 | 74M
Base | 24 | 768 | 98M
|  |  |

Table 1: Different model sizes. Base model is finally excluded due to its
suboptimization.

For SSM in the B-Mamba layer, we adopt the default hyperparameters as in Mamba
[25]. setting the state dimension and expansion ratio to 16 and 2,
respectively. Following ViT [15], we adjust the depth and embedding dimensions
to create models of comparable sizes in Table 1, including VideoMamba-Ti,
VideoMamba-S and VideoMamba-M. However, we observe that larger VideoMamba
tends to overfit during our experiments, leading to suboptimal performance as
illustrated in Fig. 6a. This overfitting issue is not unique to our models but
is also found in VMamba [50], where the optimal performance of VMamba-B was
achieved at three-quarters of the total training epochs. To counteract the
overfitting in larger Mamba models, we introduce an effective Self-
Distillation strategy, which uses a smaller and well-trained model as the
“teacher” to guide the training of the larger “student” model. The results,
depicted in Fig. 6a, show that this strategy leads to expected better
convergence.

###  3.4 Masked Modeling

Figure 5:  Different masking strategies. Row masking, tailored for VideoMamba
in light of the 1D convolution preceding SSM, enhances performance with
continuous tokens. The difference between clip-row and frame-row masking is
that the former masks the entire video clip, while the latter masks each frame
individually.

Recently, VideoMAE and ST-MAE [74, 18] have showcased the significant benefits
of masked modeling in enhancing a model’s capability for FINE-GRAINED temporal
understanding. UMT [43] takes this further by introducing an efficient masked
alignment technique that yields robust results across single and multi-modal
video tasks. To augment VideoMamba’s temporal sensitivity and verify its
adaptability with text modalities, we adopt a masked alignment approach
inspired by UMT. Firstly, VideoMamba is trained from scratch on video data
alone, aligning unmasked tokens with those from CLIP-ViT. Subsequently, it is
integrated with a text encoder and a cross-modal decoder (i.e., BERT [14]),
for pretraining on both image-text and video-text datasets.

It’s important to note the distinction from UMT, which employs multi-layer
alignment between the student and teacher models. In contrast, due to
VideoMamba’s unique architecture (SSM vs. Transformer), we align only the
final outputs. Regarding our masking strategy, we propose different row
masking techniques, depicted in Fig. 5, tailored to the B-Mamba block’s
preference for continuous tokens. Additionally, we explore attention masking
to preserve meaningful adjacency among tokens, leveraging the inherent
strengths of the 1D convolution within the B-Mamba block for improved
performance.

##  4 Experiments

###  4.1 Scaling Up

Figure 6: Ablation studies of Self-Distillation and Early Stopping.

Dataset and Settings. We first conduct experiments on ImageNet-1K [13], which
includes 1.28M training images and 50K validation images across 1,000
categories. For fair comparisons, we follow most of the training strategies
proposed in DeiT [75], but adopt weaker data augmentation for the tiny model
variant. Furthermore, we adjust the stochastic depth ratio to 0/0.15/0.5 for
VideoMamba-Ti/S/M. Our models are trained using the AdamW optimizer paired
with a cosine learning rate schedule over 300 epochs. The initial 5 epochs
serve as a period for linear warm-up. Default settings for the learning rate,
weight decay, and batch size are 1e-3, 0.05, and 1024, respectively. Moreover,
we use BFloat16 precision during training to enhance stability without relying
on EMA. For the VideoMamba-M model, we employ a pretrained VideoMamba-S model
as a “teacher” to guide the training process by aligning the final feature
maps through L2 loss. For large resolution ($>$224) fine-tuning, we use a
reduced learning rate (5e-6) and minimal weight decay (1e-8) for 30 epochs.

Effect of Self-Distillation. Fig. 6a reveals that when trained from scratch,
VideoMamba-B tends to overfit more easily and underperforms compared to
VideoMamba-S, whereas VideoMamba-M achieves similar performances. Fortunately,
our self-distillation has shown to be effective in achieving the desired
optimization with marginal additional computational cost. To mitigate
teacher’s potential overdirection, we experimented with early stopping [11] in
Fig. 6b, although it did not yield beneficial outcomes. These findings
indicate that self-distillation offers a viable strategy for enhancing the
scalability of the Mamba architecture without significant computational
overhead.

Arch. | Model | iso. | Input | #Param | FLOPs | IN-1K
---|---|---|---|---|---|---
Size | (M) | (G) | Top-1
CNN | ConvNeXt-T [53] | ✗ | 2242 | 29 | 4.5 | 82.1
ConvNeXt-S [53] | ✗ | 2242 | 50 | 8.7 | 83.1
ConvNeXt-B [53] | ✗ | 2242 | 89 | 15.4 | 83.8
Trans. | SwinT-T [51] | ✗ | 2242 | 28 | 4.5 | 81.3
Swin-S [51] | ✗ | 2242 | 50 | 8.7 | 83.0
Swin-B [51] | ✗ | 2242 | 88 | 15.4 | 83.5
CNN+ SSM | VMamba-T [50] | ✗ | 2242 | 22 | 5.6 | 82.2
VMamba-S [50] | ✗ | 2242 | 44 | 11.2 | 83.5
VMamba-B [50] | ✗ | 2242 | 75 | 18.0 | 83.7
CNN | ConvNeXt-S [53] | ✓ | 2242 | 22 | 4.3 | 79.7
ConvNeXt-B [53] | ✓ | 2242 | 87 | 16.9 | 82.0
Trans. | DeiT-Ti [75] | ✓ | 2242 | 6 | 1.3 | 72.2
DeiT-S [75] | ✓ | 2242 | 22 | 4.6 | 79.8
DeiT-B [75] | ✓ | 2242 | 87 | 17.6 | 81.8
DeiT-B [75] | ✓ | 3842 | 87 | 55.5 | 83.1
SSM | S4ND-ViT-B [58] | ✓ | 2242 | 89 | - | 80.4
Vim-Ti [91] | ✓ | 2242 | 7 | 1.1 | 76.1
Vim-S [91] | ✓ | 2242 | 26 | 4.3 | 80.5
VideoMamba-Ti | ✓ | 2242 | 7 | 1.1 | 76.9
VideoMamba-Ti | ✓ | 4482 | 7 | 4.3 | 79.3
VideoMamba-Ti | ✓ | 5762 | 7 | 7.1 | 79.6
VideoMamba-S | ✓ | 2242 | 26 | 4.3 | 81.2
VideoMamba-S | ✓ | 4482 | 26 | 16.9 | 83.2
VideoMamba-S | ✓ | 5762 | 26 | 28.0 | 83.5
VideoMamba-M | ✓ | 2242 | 74 | 12.7 | 82.8
VideoMamba-M | ✓ | 4482 | 75 | 50.4 | 83.8
VideoMamba-M | ✓ | 5762 | 75 | 83.1 | 84.0
|  |  |  |  |  |

Table 2: Comparison with the state-of-the-art on ImageNet. “iso.” means
isotropic architecture without downsampling layers.

Results. Table 2 showcases the results on the ImageNet-1K dataset. Notably,
VideoMamba-M outperforms other isotropic architectures by significant margins,
achieving a +0.8% improvement over ConvNeXt-B [53] and a +2.0% increase
compared to DeiT-B [75], while utilizing fewer parameters. Additionally,
VideoMamba-M holds its ground against non-isotropic backbones that leverage
hierarchical features for enhanced performance. Given Mamba’s efficiency in
processing long sequences, we further enhance performance by increasing the
resolution, achieving a top-1 accuracy of 84.0% with only 74M parameters. This
remarkable improvement extends to video tasks, as detailed in Section 4.2,
underscoring VideoMamba’s effectiveness and scalability.

Arch. | Model | iso. | Extra | Input | #Param | FLOPs | K400
---|---|---|---|---|---|---|---
Data | Size | (M) | (G) | Top-1 | Top-5
Supervised: Those models with extra data are under supervised training.
CNN | SlowFastR101+NL [19] | ✗ |  | 80$\times$2242 | 60 | 234$\times$3$\times$10 | 79.8 | 93.9
X3D-M [17] | ✗ |  | 16$\times$2242 | 4 | 6$\times$3$\times$10 | 76.0 | 92.3
X3D-XL [17] | ✗ |  | 16$\times$3122 | 20 | 194$\times$3$\times$10 | 80.4 | 94.6
Trans. | Swin-T [52] | ✗ | IN-1K | 32$\times$2242 | 28 | 88$\times$3$\times$4 | 78.8 | 93.6
Swin-B [52] | ✗ | IN-1K | 32$\times$2242 | 88 | 88$\times$3$\times$4 | 80.6 | 94.5
Swin-B [52] | ✗ | IN-21K | 32$\times$2242 | 88 | 282$\times$3$\times$4 | 82.7 | 95.5
CNN+ Trans. | MViTv1-B [16] | ✗ |  | 32$\times$2242 | 37 | 70$\times$1$\times$5 | 80.2 | 94.4
MViTv2-S [45] | ✗ |  | 16$\times$2242 | 35 | 64$\times$1$\times$5 | 81.0 | 94.6
UniFormer-S [44] | ✗ | IN-1K | 16$\times$2242 | 21 | 42$\times$1$\times$4 | 80.8 | 94.7
UniFormer-B [44] | ✗ | IN-1K | 16$\times$2242 | 50 | 97$\times$1$\times$4 | 82.0 | 95.1
UniFormer-B [44] | ✗ | IN-1K | 32$\times$2242 | 50 | 259$\times$3$\times$4 | 83.0 | 95.4
Trans. | STAM [63] | ✓ | IN-21K | 64$\times$2242 | 121 | 1040$\times$1$\times$1 | 79.2 | -
TimeSformer-L [4] | ✓ | IN-21K | 96$\times$2242 | 121 | 2380$\times$3$\times$1 | 80.7 | 94.7
ViViT-L [2] | ✓ | IN-21K | 16$\times$2242 | 311 | 3992$\times$3$\times$4 | 81.3 | 94.7
Mformer-HR [59] | ✓ | IN-21K | 16$\times$3362 | 311 | 959$\times$3$\times$10 | 81.1 | 95.2
SSM | VideoMamba-Ti | ✓ | IN-1K | 16$\times$2242 | 7 | 17$\times$3$\times$4 | 78.1 | 93.5
VideoMamba-Ti | ✓ | IN-1K | 32$\times$2242 | 7 | 34$\times$3$\times$4 | 78.8 | 93.9
VideoMamba-Ti | ✓ | IN-1K | 64$\times$3842 | 7 | 202$\times$3$\times$4 | 80.3 | 94.8
VideoMamba-S | ✓ | IN-1K | 16$\times$2242 | 26 | 68$\times$3$\times$4 | 80.8 | 94.8
VideoMamba-S | ✓ | IN-1K | 32$\times$2242 | 26 | 135$\times$3$\times$4 | 81.5 | 95.2
VideoMamba-S | ✓ | IN-1K | 64$\times$3842 | 26 | 395$\times$3$\times$4 | 82.7 | 95.6
VideoMamba-M | ✓ | IN-1K | 16$\times$2242 | 74 | 202$\times$3$\times$4 | 81.9 | 95.4
VideoMamba-M | ✓ | IN-1K | 32$\times$2242 | 74 | 403$\times$3$\times$4 | 82.4 | 95.7
VideoMamba-M | ✓ | IN-1K | 64$\times$3842 | 74 | 2368$\times$3$\times$4 | 83.3 | 96.1
Self-supervised: For UMT, the CLIP-400M is used in pretrained teacher.
Trans. | BEVT-B800e [81] | ✗ | IN-1K | 32$\times$2242 | 88 | 282$\times$3$\times$4 | 81.1 | -
ST-MAE-B1600e [18] | ✓ |  | 16$\times$2242 | 87 | 180$\times$3$\times$7 | 81.3 | 94.9
VideoMAE-S2400e [74] | ✓ |  | 16$\times$2242 | 22 | 57$\times$3$\times$5 | 79.0 | 93.8
VideoMAE-B1600e [74] | ✓ |  | 16$\times$2242 | 87 | 180$\times$3$\times$5 | 81.5 | 95.1
UMT-B800e [43] | ✓ | CLIP-400M | 8$\times$2242 | 87 | 180$\times$3$\times$5 | 85.7 | 97.0
SSM | VideoMamba-M800e | ✓ | CLIP-400M | 8$\times$2242 | 74 | 101$\times$3$\times$4 | 82.0 | 95.4
VideoMamba-M800e | ✓ | CLIP-400M | 16$\times$2242 | 74 | 202$\times$3$\times$4 | 83.4 | 95.9
VideoMamba-M800e | ✓ | CLIP-400M | 32$\times$2242 | 74 | 403$\times$3$\times$4 | 83.9 | 96.2
VideoMamba-M800e | ✓ | CLIP-400M | 64$\times$3842 | 74 | 2368$\times$3$\times$4 | 85.0 | 96.9
|  |  |  |  |  |  |  |

Table 3: Comparison with the state-of-the-art on scene-related Kinetics-400.
“iso.” means isotropic architecture without downsampling layers. Masked
modeling [43] also works for Mamba, but the inconsistent architecture leads to
inferior alignment.

Arch. | Model | iso. | Extra | Input | #Param | FLOPs | SSV2
---|---|---|---|---|---|---|---
Data | Size | (M) | (G) | Top-1 | Top-5
Supervised: Those models with extra data are under supervised training.
CNN | SlowFastR101 [19] | ✗ | K400 | 32$\times$2242 | 53 | 106$\times$3$\times$1 | 63.1 | 87.6
CT-NetR50 [41] | ✗ | IN-1K | 16$\times$2242 | 21 | 75$\times$1$\times$1 | 64.5 | 89.3
TDNR50 [79] | ✗ | IN-1K | 16$\times$2242 | 26 | 75$\times$1$\times$1 | 65.3 | 91.6
Trans. | Swin-B [52] | ✗ | K400 | 32$\times$2242 | 89 | 88$\times$3$\times$1 | 69.6 | 92.7
CNN+ Trans. | MViTv1-B [16] | ✗ | K400 | 16$\times$2242 | 37 | 71$\times$3$\times$1 | 64.7 | 89.2
MViTv1-B [16] | ✗ | K400 | 32$\times$2242 | 37 | 170$\times$3$\times$1 | 67.1 | 90.8
MViTv2-S [45] | ✗ | K400 | 16$\times$2242 | 35 | 65$\times$3$\times$1 | 68.2 | 91.4
MViTv2-B [45] | ✗ | K400 | 32$\times$2242 | 51 | 225$\times$3$\times$1 | 70.5 | 92.7
UniFormer-S [44] | ✗ | IN-1K+K400 | 16$\times$2242 | 21 | 42$\times$3$\times$1 | 67.7 | 91.4
UniFormer-B [44] | ✗ | IN-1K+K400 | 16$\times$2242 | 50 | 97$\times$3$\times$1 | 70.4 | 92.8
Trans. | TimeSformer-HR [4] | ✓ | IN-21K | 16$\times$2242 | 121 | 1703$\times$3$\times$1 | 62.5 | -
ViViT-L [2] | ✓ | IN-21K+K400 | 16$\times$2242 | 311 | 3992$\times$3$\times$4 | 65.4 | 89.8
Mformer-HR [59] | ✓ | IN-21K+K400 | 16$\times$3362 | 311 | 1185$\times$3$\times$1 | 68.1 | 91.2
SSM | VideoMamba-Ti | ✓ | IN-1K | 8$\times$2242 | 7 | 9$\times$3$\times$2 | 65.1 | 89.1
VideoMamba-Ti | ✓ | IN-1K | 16$\times$2242 | 7 | 17$\times$3$\times$2 | 66.0 | 89.6
VideoMamba-Ti | ✓ | IN-1K | 16$\times$2882 | 7 | 28$\times$3$\times$2 | 66.2 | 90.0
VideoMamba-S | ✓ | IN-1K | 8$\times$2242 | 26 | 34$\times$3$\times$2 | 66.6 | 90.4
VideoMamba-S | ✓ | IN-1K | 16$\times$2242 | 26 | 68$\times$3$\times$2 | 67.6 | 90.9
VideoMamba-S | ✓ | IN-1K | 16$\times$2882 | 26 | 112$\times$3$\times$2 | 68.1 | 91.2
VideoMamba-M | ✓ | IN-1K | 8$\times$2242 | 74 | 101$\times$3$\times$4 | 67.3 | 91.0
VideoMamba-M | ✓ | IN-1K | 16$\times$2242 | 74 | 202$\times$3$\times$4 | 68.3 | 91.4
VideoMamba-M | ✓ | IN-1K | 16$\times$2882 | 74 | 333$\times$3$\times$4 | 68.4 | 91.6
Self-supervised: For UMT, the CLIP-400M is used in pretrained teacher.
Trans. | BEVT-B800e [81] | ✗ | IN-1K+K400 | 32$\times$2242 | 88 | 321$\times$3$\times$1 | 70.6 | -
VideoMAE-S2400e [74] | ✓ |  | 16$\times$2242 | 22 | 57$\times$3$\times$2 | 66.8 | 90.3
VideoMAE-B2400e [74] | ✓ |  | 16$\times$2242 | 87 | 180$\times$3$\times$2 | 70.8 | 92.4
UMT-B800e [43] | ✓ | CLIP-400M | 8$\times$2242 | 87 | 180$\times$3$\times$2 | 70.8 | 92.6
SSM | VideoMamba-M800e | ✓ | CLIP-400M | 8$\times$2242 | 74 | 101$\times$3$\times$2 | 70.2 | 92.6
VideoMamba-M800e | ✓ | CLIP-400M | 16$\times$2242 | 74 | 202$\times$3$\times$2 | 71.0 | 92.7
VideoMamba-M800e | ✓ | CLIP-400M | 16$\times$2882 | 74 | 333$\times$3$\times$2 | 71.4 | 92.9
|  |  |  |  |  |  |  |

Table 4: Comparison with the state-of-the-art on temporal-related SthSth V2.
“iso.” means isotropic architecture without downsampling layers. Masked
modeling [43] also works for Mamba, and it performs better than VideoMAE.

###  4.2 Short-term Video Understanding

Datasets and Settings. We evaluate our VideoMamba on the popular scene-related
Kinetics-400 [36] and temporal-related Something-Something V2 [24], the
average video lengths of which are 10s and 4s. For supervised pretraining, we
fine-tune those models pretrained on ImageNet-1K with the same training
strategy as VideoMAE [74]. Specifically, for VideoMamba-M, the warmup epoch,
total epoch, stochastic depth rate, weight decay are set to 5, 50, 0.8, 0.05
for K400, and 5, 30, 0.8, 0.05 for SthSth. For the smaller models, all the
hyper-parameters are the same unless we decrease the stochastic depth rate and
increase the training epochs. Moreover, we linearly scale the base learning
rates according to the batch size, which are
$2e^{-4}\cdot\frac{batchsize}{256}$ for K400 and
$4e^{-4}\cdot\frac{batchsize}{256}$ for SthSth. As for self-supervised
pretraining, we adopt the training recipe as in UMT [43], employing CLIP-ViT-B
[60] to distill VideoMamba-M over 800 epochs. During fine-tuning, we use
similar hyperparameters as mentioned but opt for a small stochastic depth rate
and learning rate for both datasets.

Results. Table 3 and 4 list the results on short-term video datasets. (a)
Supervised: Compared with the purely attention-based methods [4, 2], our SSM-
based VideoMamba-M secures a notable advantage, outperforming ViViT-L [2] by
+2.0% and +3.0% on the scene-related K400 and the temporally-related SthSthV2
datasets, respectively. This improvement comes with significantly reduced
computational demands and less pretraining data. Furthermore, VideoMamba-M
delivers results that are on par with the SOTA UniFormer [44], which
skillfully integrates convolution with attention in a non-isotropic structure.
(b) Self-supervised: The performance of VideoMamba under masked pretraining
surpasses that of the VideoMAE [74], known for its proficiency in fine-grained
action. This achievement underscores the potential of our purely SSM-based
model in efficiently and effectively understanding short-term videos,
highlighting its suitability for both supervised and self-supervised learning
paradigms.

Ablation Studies. Through comprehensive ablation studies detailed in Fig. 7
and Table 5, we explore various aspects of our model. (a) Scan Type: Among all
the methods, the spatial-first approach emerges as the most effective, in
contrast, the temporal-first strategy is the worst. The superiority of the
spatial-first method is attributed to its ability to seamlessly leverage 2D
pretrained knowledge by scanning frame by frame. (b) Frame and Resolution:
Contrary to findings from ImageNet (see Table 2), higher resolution does not
uniformly lead to better performance. Increasing the number of frames
consistently enhances results on the K400 dataset. However, this is not the
case with SthSthV2, possibly due to the brief duration of its videos, which
may not accommodate longer inputs effectively. (c) Masked Pretraining: Our
findings reveal that row masking, being particularly compatible with 1D
convolution, outperforms commonly used random and tube masking. Clip-row
masking excels owing to its higher degree of randomness. Moreover, attention
masking stands out as the most efficient by favoring the preservation of
adjacent meaningful content. Aligning solely the model’s final output proves
most effective, likely due to architectural differences. Lastly, an optimal
masking ratio (80%) combined with stronger regularization significantly
benefits VideoMamba during masked pretraining.

Type | SSV2
---|---
SF-Bidirectional | 65.1
TF-Bidirectional | 62.4
ST-Bidirectional v1 | 63.9
ST-Bidirectional v2 | 64.2
Half-SF + Half-TF | 64.0
Half-TF + Half-SF | 64.1
Alternative SF&TF | 65.1

(a) Scan Type. Spatial-First scan is simple yet effective.

(b) Frame & Resolution for K400 and SSV2.

Figure 7: Ablation studies of scan type, frame and resolution. All the models
are fine-tuned from VideoMamba-Ti pretrained on ImageNet.

Type | SSV2
---|---
Random | 67.4
Tube | 66.3
Clip-Row | 68.2
Frame-Row | 67.8
Attention | 68.5

(a) Mask Type.

Layer | SSV2
---|---
Last 1 | 68.5
Last 2 | 68.4
Last 6 | 68.2
Last 6$\times$2 | 67.7

(b) Alignment Layer.

Ratio | SSV2
---|---
50% | 68.1
65% | 68.4
80% | 68.5
90% | 68.2

(c) Mask Ratio.

DP | SSV2
---|---
0.1 | 68.0
0.2 | 68.2
0.3 | 68.4
0.4 | 68.5

(d) Droppath.

Table 5:  Ablation studies of masked pretraining. We adopt CLIP-ViT-B [60] as
a teacher to distill VideoMamba-M for 200 epochs.

###  4.3 Long-term Video Understanding

Method | e2e | Backbone | Neck Type | Pretraining | BF | COIN
---|---|---|---|---|---|---
Dataset | Top-1 | Top-1
Timeception [32] | ✗ | 3D-ResNet | Conv. | IN-1K+K400 | 71.3 | -
VideoGraph [33] | ✗ | I3D | Conv.+Atten. | IN-1K+K400 | 69.5 | -
GHRM [90] | ✗ | I3D | Graph Conv.. | IN-1K+K400 | 75.5 | -
Distant Supervision [47] | ✗ | TimeSformer | Atten. w/ KB | IN-21K+HTM | 89.9 | 90.0
ViS4mer [35] | ✗ | Swin-B | SSM | IN-21K+K600 | 88.2 | 88.4
Turbof32 [29] | ✓ | VideoMAE-B |  | K400 | 86.8 | 82.3
Turbof32 [29] | ✓ | VideoMAE-B |  | K400+HTM-AA | 91.3 | 87.5
VideoMambaf32 | ✓ | VideoMamba-Ti |  | K400 | 94.3 | 86.2
VideoMambaf64 | ✓ | VideoMamba-Ti |  | K400 | 94.3 | 87.0
VideoMambaf32 | ✓ | VideoMamba-S |  | K400 | 95.3 | 88.4
VideoMambaf64 | ✓ | VideoMamba-S |  | K400 | 97.4 | 88.7
VideoMambaf32 | ✓ | VideoMamba-M |  | K400 | 94.8 | 88.3
VideoMambaf64 | ✓ | VideoMamba-M |  | K400 | 95.8 | 89.5
VideoMambaf32 | ✓ | VideoMamba-M${\dagger}$ |  | K400 | 97.9 | 89.6
VideoMambaf64 | ✓ | VideoMamba-M${\dagger}$ |  | K400 | 96.9 | 90.4
|  |  |  |  |  |

Table 6: Comparison with the state-of-the-art on Breakfast and COIN. “e2e”
means end-to-end methods without exhausting feature extraction. “${\dagger}$”
marks the backbone with masked pretraining.

Method | e2e | Backbone | Content($\uparrow$) | Metadata($\uparrow$) | User($\downarrow$)
---|---|---|---|---|---
Rel. | Speak | Scene | Dir. | Genre | Wtr. | Year | Like | View
VideoBERT [69] | ✗ | S3D | 52.80 | 37.90 | 54.90 | 47.30 | 51.90 | 38.50 | 36.10 | 0.32 | 4.46
Object Trans.[84] | ✗ | ResNet | 53.10 | 39.40 | 56.90 | 51.20 | 54.60 | 34.50 | 39.10 | 0.23 | 3.55
LST [35] | ✗ | ViT-L | 52.38 | 37.31 | 62.79 | 56.07 | 52.70 | 42.26 | 39.16 | 0.31 | 3.83
Performer [35] | ✗ | ViT-L | 50.00 | 38.80 | 60.46 | 58.87 | 49.45 | 48.21 | 41.25 | 0.31 | 3.93
Orthoformer [35] | ✗ | ViT-L | 50.00 | 39.30 | 66.27 | 55.14 | 55.79 | 47.02 | 43.35 | 0.29 | 3.86
ViS4mer [35] | ✗ | ViT-L | 57.14 | 40.79 | 67.44 | 62.61 | 54.71 | 48.80 | 44.75 | 0.26 | 3.63
VideoMambaf32 | ✓ | VM-Ti | 62.50 | 40.43 | 70.37 | 67.29 | 65.24 | 52.98 | 48.23 | 0.26 | 2.90
|  |  |  |  |  |  |  |  |  |  |

Table 7: Comparison with the state-of-the-art on LVU. “e2e” means end-to-end
methods without exhausting feature extraction. “Rel.”, “Dir.” and “Wtr.”
refers to “Relation”, “Director” and “Writer”, respectively.

Datasets and Settings. We rigorously assess VideoMamba’s proficiency in
processing long-term videos by leveraging three comprehensive datasets, i.e.,
Breakfast [37], COIN [71] and Long-form Video Understanding (LVU [84])
benchmark. Specifically, Breakfast comprises 1,712 videos, encapsulating 10
intricate cooking activities over 77 hours. COIN features 11,827 videos across
180 unique procedural tasks, with an average duration of 2.36 minutes. The LVU
benchmark includes approximately 30K movie clips, lasting between 1 to 3
minutes, and encompasses nine tasks across 3 primary categories: content
understanding, metadata prediction, and user engagement. For the regression
task among these, we evaluate using mean-squared error, while for the
classification tasks, accuracy is the metric of choice. In contrast to prior
studies [47, 35] that rely on features derived from pretrained video models,
such as Swin-B [51] trained on Kinetics-600, our method employs end-to-end
training as detailed in Section 4.2. Additionally, for fair comparisons, we
fine-tune our models pretrained on K400.

Results. As illustrated in Figure 1, the linear complexity of VideoMamba makes
it well-suited for end-to-end training with long-duration videos. The
comparisons in Tables 6 and 7 highlight VideoMamba’s simplicity and
effectiveness against traditional feature-based methods [35, 47] on these
tasks. It yields significant performance improvements, achieving SOTA results
even with smaller model sizes. For example, VideoMamba-Ti shows a notable
increase of +6.1% over ViS4mer using Swin-B features and a +3.0% uplift
against Turbo’s multi-modality alignment approach [29]. Notably, the results
underscore the positive impact of the scaling model and frame numbers for
long-term tasks. In the diverse and challenging set of nine tasks presented by
LVU, our VideoMamba-Ti, fine-tuned in an end-to-end manner, delivers
outstanding or comparable results to current SOTA methods. These outcomes not
only highlight VideoMamba’s effectiveness but also its great potential for
future long-video comprehension.

###  4.4 Multi-modality Video Understanding

Datasets and Settings. Following UMT [43], we utilize WebVid-2M [3] video-text
pairs and CC3M [64] image-text pairs for joint pretraining with four
objectives: vision-text contrastive learning [3], vision-text matching [40],
masked language modeling [14] and unmasked token alignment [43]. Initially, we
mask 50% image tokens and 80% video tokens, conducting pretraining across 8
frames for 10 epochs. Given Mamba’s sensitivity to positional information, an
additional unmasked tuning phase is carried out for one epoch to refine its
comprehension further. For evaluation, we undertake zero-shot video-text
retrieval tasks across five prominent benchmarks, including MSRVTT [86],
DiDeMo [1], ActivityNet [31], LSMDC [61], and MSVD [10].

Method | BB | #P | MSRVTT | DiDeMo | ANet | LSMDC | MSVD
---|---|---|---|---|---|---|---
@1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10 | @1 | @5 | @10
Singularity [38] | Swin | 5M | 28.4 | 50.2 | 59.5 | 36.9 | 61.1 | 69.3 | 30.8 | 55.9 | 66.3 | - | - | - | - | - | -
Frozen [3] | ViT | 5M | 18.7 | 39.5 | 51.6 | 20.2 | 46.4 | 58.5 | - | - | - | - | - | - | - | - | -
ALPRO [39] | ViT | 5M | 24.1 | 44.7 | 55.4 | 23.8 | 47.3 | 57.9 | - | - | - | - | - | - | - | - | -
BridgeFormer [23] | ViT | 5M | 26.0 | 46.4 | 56.4 | 25.6 | 50.6 | 61.1 | - | - | - | 12.2 | 25.9 | 32.2 | 43.6 | 74.9 | 84.9
UMT [43] | ViT | 5M | 29.6 | 52.8 | 61.9 | 33.4 | 58.3 | 67.0 | 28.3 | 53.0 | 64.2 | 16.8 | 30.5 | 37.6 | 36.2 | 65.7 | 76.1
VideoMamba | VM | 5M | 32.0 | 53.0 | 63.8 | 36.6 | 61.7 | 70.3 | 35.9 | 61.1 | 72.3 | 18.0 | 36.1 | 43.4 | 38.0 | 68.6 | 79.0
VideoCLIP [85] | S3D | 136M | 10.4 | 22.2 | 30.0 | 16.6 | 46.9 | - | - | - | - | - | - | - | - | - | -
VIOLET [21] | Swin | 138M | 25.9 | 49.5 | 59.7 | 23.5 | 49.8 | 59.8 | - | - | - | - | - | - | - | - | -
Singularity [38] | Swin | 17M | 34.0 | 56.7 | 66.7 | 37.1 | 61.7 | 69.9 | 30.6 | 55.6 | 66.9 | - | - | - | - | - | -
OmniVL [38] | ViT | 17M | 34.6 | 58.4 | 66.6 | 33.3 | 58.7 | 68.5 | - | - | - | - | - | - | - | - | -
UMT [43] | ViT | 17M | 35.5 | 59.3 | 68.6 | 41.9 | 66.7 | 75.0 | 33.8 | 59.1 | 70.4 | 18.1 | 33.1 | 42.2 | 41.4 | 70.6 | 80.1
UMT [43] | ViT | 25M | 35.2 | 57.8 | 66.0 | 41.2 | 65.4 | 74.9 | 35.5 | 60.6 | 71.8 | 19.1 | 33.4 | 42.2 | 42.3 | 71.7 | 80.8
CLIP4Clip [55] | ViT | 400M | 30.6 | 54.4 | 64.3 | - | - | - | - | - | - | 13.6 | 27.9 | 35.5 | 36.2 | 63.8 | 73.5
InternVideo [83] | ViT | 640M | 40.0 | 65.3 | 74.1 | 31.5 | 57.6 | 68.2 | 30.7 | 57.4 | 70.2 | 17.6 | 32.4 | 40.2 | 43.4 | 69.9 | 79.1
VideoMamba | VM | 17M | 34.7 | 58.9 | 68.0 | 42.0 | 67.3 | 76.8 | 40.1 | 65.7 | 76.1 | 18.4 | 35.3 | 43.0 | 40.3 | 70.0 | 79.7
VideoMamba | VM | 25M | 35.6 | 58.1 | 69.5 | 43.1 | 68.1 | 77.7 | 41.0 | 67.5 | 77.8 | 20.4 | 37.1 | 45.7 | 42.6 | 71.6 | 81.2
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 8: Zero-shot text-to-video retrieval on MSRVTT, DiDeMo, AcitivityNet,
LSMDC, and MSVD. “BB” means the visual backbone. “#P” refers to the number of
pretraining pairs. Models pretrained with large-scale pairs are noted in gray.

Results. As indicated in Table 8, under the same pretraining corpus and
similar training strategies, our VideoMamba achieves superior zero-shot video
retrieval performances to UMT [43] based on ViT [15]. It underscores Mamba’s
comparable efficiency and scalability to the ViT in handling multi-modal video
tasks. Notably, for datasets featuring longer video lengths (e.g., ANet and
DiDeMo) and more complex scenarios (e.g., LSMDC), VideoMamba demonstrates a
significant improvement. This demonstrates Mamba’s aptitude for the demands of
cross-modality alignment even in challenging multimodal contexts.

##  5 Conclusion

In this paper, we propose VideoMamba, a purely SSM-based model for efficient
video understanding. Our extensive experiments demonstrate its scalability in
the visual domain, sensitivity for short-term action recognition, superiority
in long-term video understanding and compatibility with other modalities. We
hope it can pave the way for future model design for long-video comprehension.

Limitations. Due to resource constraints, we have not yet fully validated the
scalability of VideoMamba, such as extending VideoMamba to larger sizes (e.g.,
VideoMamba-g), incorporating additional modalities (e.g., audio), and
integrating with large language models for hour-level video understanding.
Despite these limitations, our findings confirm VideoMamba’s promising
potential and we plan to conduct thorough explorations of its capabilities in
the future.

## References

  * [1] Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with natural language. In: ICCV (2017)
  * [2] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: ICCV (2021)
  * [3] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: ICCV (2021)
  * [4] Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML (2021)
  * [5] Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., Ramesh, A.: Video generation models as world simulators (2024), https://openai.com/research/video-generation-models-as-world-simulators
  * [6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: NeurIPS (2020)
  * [7] Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A short note about kinetics-600. ArXiv abs/1808.01340 (2018)
  * [8] Carreira, J., Noland, E., Hillier, C., Zisserman, A.: A short note on the kinetics-700 human action dataset. ArXiv abs/1907.06987 (2019)
  * [9] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR (2017)
  * [10] Chen, D.L., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation. In: ACL (2011)
  * [11] Cho, J.H., Hariharan, B.: On the efficacy of knowledge distillation. In: ICCV (2019)
  * [12] Das, P., Xu, C., Doell, R.F., Corso, J.J.: A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In: CVPR (2013)
  * [13] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)
  * [14] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv abs/1810.04805 (2018)
  * [15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [16] Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In: ICCV (2021)
  * [17] Feichtenhofer, C.: X3d: Expanding architectures for efficient video recognition. In: CVPR (2020)
  * [18] Feichtenhofer, C., Fan, H., Li, Y., He, K.: Masked autoencoders as spatiotemporal learners. NeurIPS (2022)
  * [19] Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: ICCV (2019)
  * [20] Fu, D.Y., Dao, T., Saab, K.K., Thomas, A.W., Rudra, A., Ré, C.: Hungry hungry hippos: Towards language modeling with state space models. In: ICLR (2023)
  * [21] Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: Violet: End-to-end video-language transformers with masked visual-token modeling. ArXiv abs/2111.12681 (2021)
  * [22] Gao, J., Sun, C., Yang, Z., Nevatia, R.: Tall: Temporal activity localization via language query. In: ICCV (2017)
  * [23] Ge, Y., Ge, Y., Liu, X., Li, D., Shan, Y., Qie, X., Luo, P.: Bridging video-text retrieval with multiple choice questions. In: CVP (2022)
  * [24] Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fründ, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., Memisevic, R.: The “something something” video database for learning and evaluating visual common sense. In: ICCV (2017)
  * [25] Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. ArXiv abs/2312.00752 (2023)
  * [26] Gu, A., Goel, K., Ré, C.: Efficiently modeling long sequences with structured state spaces. In: ICLR (2022)
  * [27] Gu, C., Sun, C., Vijayanarasimhan, S., Pantofaru, C., Ross, D.A., Toderici, G., Li, Y., Ricco, S., Sukthankar, R., Schmid, C., Malik, J.: Ava: A video dataset of spatio-temporally localized atomic visual actions. CVPR (2017)
  * [28] Guo, H., Li, J., Dai, T., Ouyang, Z., Ren, X., Xia, S.T.: Mambair: A simple baseline for image restoration with state-space model. ArXiv abs/2402.15648 (2024)
  * [29] Han, T., Xie, W., Zisserman, A.: Turbo training with token dropout. In: BMVC (2022)
  * [30] He, X., Cao, K., Yan, K., Li, R., Xie, C., Zhang, J., Zhou, M.: Pan-mamba: Effective pan-sharpening with state space model. ArXiv abs/2402.12192 (2024)
  * [31] Heilbron, F.C., Escorcia, V., Ghanem, B., Niebles, J.C.: Activitynet: A large-scale video benchmark for human activity understanding. In: CVPR (2015)
  * [32] Hussein, N., Gavves, E., Smeulders, A.W.M.: Timeception for complex action recognition. In: CVPR (2019)
  * [33] Hussein, N., Gavves, E., Smeulders, A.W.M.: Videograph: Recognizing minutes-long human activities in videos. ArXiv abs/1905.05143 (2019)
  * [34] Idrees, H., Zamir, A.R., Jiang, Y.G., Gorban, A., Laptev, I., Sukthankar, R., Shah, M.: The thumos challenge on action recognition for videos “in the wild”. Computer Vision and Image Understanding (2017)
  * [35] Islam, M.M., Bertasius, G.: Long movie clip classification with state-space video models. In: ECCV (2022)
  * [36] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, A., Suleyman, M., Zisserman, A.: The kinetics human action video dataset. ArXiv abs/1705.06950 (2017)
  * [37] Kuehne, H., Arslan, A., Serre, T.: The language of actions: Recovering the syntax and semantics of goal-directed human activities. In: CVPR (2014)
  * [38] Lei, J., Berg, T.L., Bansal, M.: Revealing single frame bias for video-and-language learning. ArXiv abs/2206.03428 (2022)
  * [39] Li, D., Li, J., Li, H., Niebles, J.C., Hoi, S.C.: Align and prompt: Video-and-language pre-training with entity prompts. In: CVPR (2022)
  * [40] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. In: NeurIPS (2021)
  * [41] Li, K., Li, X., Wang, Y., Wang, J., Qiao, Y.: Ct-net: Channel tensorization network for video classification. In: ICLR (2020)
  * [42] Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Wang, L., Qiao, Y.: Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. ArXiv abs/2211.09552 (2022)
  * [43] Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., Qiao, Y.: Unmasked teacher: Towards training-efficient video foundation models. In: ICCV (2023)
  * [44] Li, K., Wang, Y., Peng, G., Song, G., Liu, Y., Li, H., Qiao, Y.: Uniformer: Unified transformer for efficient spatial-temporal representation learning. In: ICLR (2022)
  * [45] Li, Y., Wu, C., Fan, H., Mangalam, K., Xiong, B., Malik, J., Feichtenhofer, C.: Improved multiscale vision transformers for classification and detection. ArXiv abs/2112.01526 (2021)
  * [46] Liang, D., Zhou, X., Wang, X., Zhu, X., Xu, W., Zou, Z., Ye, X., Bai, X.: Pointmamba: A simple state space model for point cloud analysis. ArXiv abs/2402.10739 (2024)
  * [47] Lin, X., Petroni, F., Bertasius, G., Rohrbach, M., Chang, S.F., Torresani, L.: Learning to recognize procedural activities with distant supervision. CVPR (2022)
  * [48] Liu, H., Yan, W., Zaharia, M., Abbeel, P.: World model on million-length video and language with ringattention. ArXiv abs/2402.08268 (2024)
  * [49] Liu, Y., Wang, L., Wang, Y., Ma, X., Qiao, Y.: Fineaction: A fine-grained video dataset for temporal action localization. TIP (2022)
  * [50] Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., Liu, Y.: Vmamba: Visual state space model. ArXiv abs/2401.10166 (2024)
  * [51] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [52] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: CVPR (2022)
  * [53] Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: CVPR (2022)
  * [54] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. NeurIPS (2019)
  * [55] Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., Li, T.: Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing (2022)
  * [56] Mehta, H., Gupta, A., Cutkosky, A., Neyshabur, B.: Long range language modeling via gated state spaces. ArXiv abs/2206.13947 (2022)
  * [57] Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In: ICCV (2019)
  * [58] Nguyen, E., Goel, K., Gu, A., Downs, G.W., Shah, P., Dao, T., Baccus, S.A., Ré, C.: S4nd: Modeling images and videos as multidimensional signals with state spaces. In: NeurIPS (2022)
  * [59] Patrick, M., Campbell, D., Asano, Y., Misra, I., Metze, F., Feichtenhofer, C., Vedaldi, A., Henriques, J.F.: Keeping your eye on the ball: Trajectory attention in video transformers. In: NeurIPS (2021)
  * [60] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)
  * [61] Rohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C.J., Larochelle, H., Courville, A.C., Schiele, B.: Movie description. International Journal of Computer Vision (2016)
  * [62] Shao, D., Zhao, Y., Dai, B., Lin, D.: Finegym: A hierarchical video dataset for fine-grained action understanding. CVPR (2020)
  * [63] Sharir, G., Noy, A., Zelnik-Manor, L.: An image is worth 16x16 words, what is a video worth? ArXiv abs/2103.13915 (2021)
  * [64] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL (2018)
  * [65] Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. NeurIPS (2014)
  * [66] Smith, J.T., Warrington, A., Linderman, S.W.: Simplified state space layers for sequence modeling. In: ICLR (2023)
  * [67] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  * [68] Su, J., Lu, Y., Pan, S., Wen, B., Liu, Y.: Roformer: Enhanced transformer with rotary position embedding. ArXiv abs/2104.09864 (2021)
  * [69] Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video and language representation learning. In: ICCV (2019)
  * [70] Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., Wei, F.: Retentive network: A successor to transformer for large language models. ArXiv abs/2307.08621 (2023)
  * [71] Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., Zhou, J.: Coin: A large-scale dataset for comprehensive instructional video analysis. In: CVPR (2019)
  * [72] Team, G.: Gemini: A family of highly capable multimodal models. ArXiv abs/2312.11805 (2023)
  * [73] Team, R.: Rwkv: Reinventing rnns for the transformer era. In: EMNLP (2023)
  * [74] Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In: NeurIPS (2022)
  * [75] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., J’egou, H.: Training data-efficient image transformers & distillation through attention. In: ICML (2021)
  * [76] Tran, D., Bourdev, L.D., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: IEEE International Conference on Computer Vision (2015)
  * [77] Tran, D., xiu Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: CVPR (2018)
  * [78] Wang, C., Tsepa, O., Ma, J., Wang, B.: Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. ArXiv abs/2402.00789 (2024)
  * [79] Wang, L., Tong, Z., Ji, B., Wu, G.: TDN: Temporal difference networks for efficient action recognition. In: CVPR (2021)
  * [80] Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Gool, L.V.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV (2016)
  * [81] Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.G., Zhou, L., Yuan, L.: Bevt: Bert pretraining of video transformers. CVPR (2022)
  * [82] Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X.J., Chen, X., Wang, Y., Luo, P., Liu, Z., Wang, Y., Wang, L., Qiao, Y.: Internvid: A large-scale video-text dataset for multimodal understanding and generation. ArXiv (2023)
  * [83] Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., Xing, S., Chen, G., Pan, J., Yu, J., Wang, Y., Wang, L., Qiao, Y.: Internvideo: General video foundation models via generative and discriminative learning. ArXiv abs/2212.03191 (2022)
  * [84] Wu, C.Y., Krahenbuhl, P.: Towards long-form video understanding. In: CVPR (2021)
  * [85] Xu, H., Ghosh, G., Huang, P.Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., Feichtenhofer, C.: Videoclip: Contrastive pre-training for zero-shot video-text understanding. ArXiv abs/2109.14084 (2021)
  * [86] Xu, J., Mei, T., Yao, T., Rui, Y.: Msr-vtt: A large video description dataset for bridging video and language. In: CVPR (2016)
  * [87] Yang, Y., Xing, Z., Zhu, L.: Vivim: a video vision mamba for medical video object segmentation. ArXiv abs/2401.14168 (2024)
  * [88] Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering. In: AAAI (2019)
  * [89] Zhang, D.J., Li, K., Wang, Y., Chen, Y., Chandra, S., Qiao, Y., Liu, L., Shou, M.Z.: Morphmlp: An efficient mlp-like backbone for spatial-temporal representation learning. In: ECCV (2022)
  * [90] Zhou, J., Lin, K.Y., Li, H., Zheng, W.: Graph-based high-order relation modeling for long-term action recognition. CVPR (2021)
  * [91] Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X.: Vision mamba: Efficient visual representation learning with bidirectional state space model. ArXiv abs/2401.09417 (2024)
  * [92] Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., Wang, Y.: Vlogger: Make your dream a vlog. ArXiv abs/2401.09414 (2024)

VideoMamba: State Space Model for Efficient Video Understanding Appendix

Arch. | Model | iso. | Extra | Input | #Param | FLOPs | K400
---|---|---|---|---|---|---|---
Data | Size | (M) | (G) | Top-1 | Top-5
Supervised: Those models with extra data are under supervised training.
Trans. | STAM [63] | ✓ | IN-21K | 64$\times$2242 | 121 | 1040$\times$1$\times$1 | 79.2 | -
TimeSformer-L [4] | ✓ | IN-21K | 96$\times$2242 | 121 | 2380$\times$3$\times$1 | 80.7 | 94.7
ViViT-L [2] | ✓ | IN-21K | 16$\times$2242 | 311 | 3992$\times$3$\times$4 | 81.3 | 94.7
Mformer-HR [59] | ✓ | IN-21K | 16$\times$3362 | 311 | 959$\times$3$\times$10 | 81.1 | 95.2
SSM | VideoMamba-Ti | ✓ | IN-1K | 8$\times$2242 | 7 | 9$\times$3$\times$4 | 76.9 | 92.9
VideoMamba-Ti | ✓ | IN-1K | 16$\times$2242 | 7 | 17$\times$3$\times$4 | 78.1 | 93.5
VideoMamba-Ti | ✓ | IN-1K | 32$\times$2242 | 7 | 34$\times$3$\times$4 | 78.8 | 93.9
VideoMamba-Ti | ✓ | IN-1K | 64$\times$2242 | 7 | 69$\times$3$\times$4 | 79.6 | 94.2
VideoMamba-Ti | ✓ | IN-1K | 64$\times$3842 | 7 | 202$\times$3$\times$4 | 80.3 | 94.8
VideoMamba-S | ✓ | IN-1K | 8$\times$2242 | 26 | 34$\times$3$\times$4 | 79.3 | 94.2
VideoMamba-S | ✓ | IN-1K | 16$\times$2242 | 26 | 68$\times$3$\times$4 | 80.8 | 94.8
VideoMamba-S | ✓ | IN-1K | 32$\times$2242 | 26 | 135$\times$3$\times$4 | 81.5 | 95.2
VideoMamba-S | ✓ | IN-1K | 64$\times$2242 | 26 | 271$\times$3$\times$4 | 81.8 | 95.3
VideoMamba-S | ✓ | IN-1K | 64$\times$3842 | 26 | 395$\times$3$\times$4 | 82.7 | 95.6
VideoMamba-M | ✓ | IN-1K | 8$\times$2242 | 74 | 101$\times$3$\times$4 | 80.6 | 94.6
VideoMamba-M | ✓ | IN-1K | 16$\times$2242 | 74 | 202$\times$3$\times$4 | 81.9 | 95.4
VideoMamba-M | ✓ | IN-1K | 32$\times$2242 | 74 | 403$\times$3$\times$4 | 82.4 | 95.7
VideoMamba-M | ✓ | IN-1K | 64$\times$2242 | 74 | 806$\times$3$\times$4 | 82.8 | 96.0
VideoMamba-M | ✓ | IN-1K | 64$\times$3842 | 74 | 2368$\times$3$\times$4 | 83.3 | 96.1
Self-supervised: For UMT, the CLIP-400M is used in pretrained teacher.
Trans. | ST-MAE-B1600e [18] | ✓ |  | 16$\times$2242 | 87 | 180$\times$3$\times$7 | 81.3 | 94.9
VideoMAE-S2400e [74] | ✓ |  | 16$\times$2242 | 22 | 57$\times$3$\times$5 | 79.0 | 93.8
VideoMAE-B1600e [74] | ✓ |  | 16$\times$2242 | 87 | 180$\times$3$\times$5 | 81.5 | 95.1
UMT-B800e [43] | ✓ | CLIP-400M | 8$\times$2242 | 87 | 180$\times$3$\times$5 | 85.7 | 97.0
SSM | VideoMamba-M800e | ✓ | CLIP-400M | 8$\times$2242 | 74 | 101$\times$3$\times$4 | 82.0 | 95.4
VideoMamba-M800e | ✓ | CLIP-400M | 16$\times$2242 | 74 | 202$\times$3$\times$4 | 83.4 | 95.9
VideoMamba-M800e | ✓ | CLIP-400M | 32$\times$2242 | 74 | 403$\times$3$\times$4 | 83.9 | 96.2
VideoMamba-M800e | ✓ | CLIP-400M | 64$\times$2242 | 74 | 806$\times$3$\times$4 | 84.3 | 96.6
VideoMamba-M800e | ✓ | CLIP-400M | 64$\times$3842 | 74 | 2368$\times$3$\times$4 | 85.0 | 96.9
|  |  |  |  |  |  |  |

Table IX: More results on scene-related Kinetics-400. “iso.” means isotropic
architecture without downsampling layers.

##  Appendix 0.A More Results

In Table IX, we present additional results on the Kinetics-400 dataset. These
results clearly demonstrate that our SSM-based model outperforms all previous
attention-based methods. We observe consistent performance improvements with
increasing resolution and frame count.

config | Single-Modality | Multi-Modality
---|---|---
SthSthV2 | K400 | 5M & 17M & 25M
optimizer | AdamW | AdamW
optimizer momentum | $\beta_{1},\beta_{2}{=}0.9,0.95$ | $\beta_{1},\beta_{2}{=}0.9,0.999$
weight decay | 0.05 | 0.05
learning rate schedule | cosine decay | cosine decay
learning rate | 1.2e-3 | 4e-4
minimal learning rate | 1e-5 | 4e-6
batch size | 2048 | 2048 $\mathbf{I}$, 2048 $\mathbf{V}$
warmup epochs | 40 | 1
total epochs | 800 | 10
mask ratio | 80% | 50% $\mathbf{I}$, 80% $\mathbf{V}$, 50% $\mathbf{T}$
input frame | 8 | 8
drop path | 0.4 | 0.25
flip augmentation | no | yes | yes
augmentation |  MultiScaleCrop[0.66, 0.75, 0.875, 1] |  MultiScaleCrop[0.5, 1]

Table X:  Masked pre-training settings. $\mathbf{I}$-image,
$\mathbf{V}$-video, $\mathbf{T}$-text.

config |  224$\times$224 |  448$\times$448 |  512$\times$512
---|---|---|---
optimizer |  AdamW
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.999$
weight decay |  0.1(Ti), 0.05(S,M) |  1e-8 |  1e-8
learning rate schedule |  cosine decay
base learning rate |  5e-4 |  5e-6 |  5e-6
minimal learning rate |  1e-5 |  5e-6 |  5e-6
base batch size |  512
repeated augmentation |  no(Ti), yes(S,M)
warmup epochs |  5(Ti,S), 30(M) |  5 |  2
total epochs |  300 |  30 |  10
drop path |  0(Ti), 0.15(S), 0.5(M)
label smoothing |  0.1
cutmix |  1.0
augmentation |  RandAug(7, 0.25)(Ti), RandAug(9, 0.5)(S,M)

Table XI:  Training settings for ImageNet-1K.

config |  224$\times$224 |  384$\times$384
---|---|---
optimizer |  AdamW
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.999$
weight decay |  0.1(Ti), 0.05(S,M,M${\dagger}$) |  1e-8
learning rate schedule |  cosine decay
base learning rate |  4e-4(Ti,S), 2e-4(M), 1e-4(M${\dagger}$) |  5e-6
minimal learning rate |  1e-6
base batch size |  256
repeated augmentation |  2
warmup epochs |  5 |  2
total epochs |  70(Ti), 50(S,M), 45(M${\dagger}$) |  10
drop path |  0.1(Ti), 0.35(S), 0.8(M), 0.4(M${\dagger}$)
layer-wise lr decay |  0.75(S,M,M${\dagger}$), 0.8(M${\dagger}$)
flip augmentation |  yes
label smoothing |  0.1
cutmix |  1.0
augmentation |  RandAug(7, 0.25)(Ti), RandAug(9, 0.5)(S,M,M${\dagger}$)

Table XII:  Training settings for Kinetics-400. “${\dagger}$” means masked
pretraining.

config |  224$\times$224 |  288$\times$288
---|---|---
optimizer |  AdamW
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.999$
weight decay |  0.1(Ti), 0.05(S,M,M${\dagger}$) |  1e-8
learning rate schedule |  cosine decay
base learning rate |  4e-4(Ti,S,M) 1e-4(M${\dagger}$) |  5e-6
minimal learning rate |  1e-6
base batch size |  256
repeated augmentation |  2
warmup epochs |  5 |  2
total epochs |  35(Ti), 30(S,M,M${\dagger}$) |  10
drop path |  0.1(Ti), 0.35(S), 0.8(M), 0.4(M${\dagger}$)
layer-wise lr decay |  0.75(S,M,M${\dagger}$), 0.8(M${\dagger}$)
flip augmentation |  no
label smoothing |  0.1
cutmix |  1.0
augmentation |  RandAug(7, 0.25)(Ti), RandAug(9, 0.5)(S,M,M${\dagger}$)

Table XIII:  Training settings for SthSthV2. “${\dagger}$” means masked
pretraining.

config |  BreakFast & LVU |  COIN
---|---|---
optimizer |  AdamW
optimizer momentum |  $\beta_{1},\beta_{2}{=}0.9,0.999$
weight decay |  0.1(Ti), 0.05(S,M,M${\dagger}$)
learning rate schedule |  cosine decay
base learning rate |  2e-4
minimal learning rate |  1e-6
base batch size |  256
repeated augmentation |  2
warmup epochs |  5
total epochs |  70(Ti), 50(S,M), 45(M${\dagger}$) |  40(Ti), 35(S), 30(M,M${\dagger}$)
drop path |  0.1(Ti), 0.35(S), 0.8(M), 0.4(M${\dagger}$)
layer-wise lr decay |  0.75(S,M,M${\dagger}$), 0.8(M${\dagger}$)
flip augmentation |  yes
label smoothing |  0.1
cutmix |  1.0
augmentation |  RandAug(7, 0.25)(Ti), RandAug(9, 0.5)(S,M,M${\dagger}$)

Table XIV:  Training settings for Breaskfast, COIN and LVU. “${\dagger}$”
means masked pretraining. We directly sample the frames from the raw video
sparsely.

##  Appendix 0.B More Implementation Details

###  0.B.1 Training Details

We sparsely sample frames from the raw videos as in TSN [80] for all the
datasets. Table X details the masked pretraining hyperparameters. For the
unmasked multi-modality pretraining, we load the pretrained model and train it
for an additional epoch with a learning rate of 8e-5. Moreover, Tables XI,
XII, XIII, and XIV show the training details for the different datasets used
for fine-tuning.

###  0.B.2 Dataset Descriptions

Dataset | #image/video | #text | Type
---|---|---|---
COCO | 113K | 567K | image
Visual Genome | 100K | 768K | image
SBU Captions | 860K | 860K | image
CC3M | 2.88M | 2.88M | image
CC12M | 11.00M | 11.00M | image
WebVid-2M | 2.49M | 2.49M | video
WebVid-10M | 10.73M | 10.73M | video
5M corpus = CC3M$+$WebVid-2M | 5.37M | 5.37M | video+image
17M corpus = 5M$+$COCO$+$VG$+$SBU$+$CC12M | 17.44M | 18.57M | video+image
25M corpus = 17M$+$WebVid-10M$-$WebVid-2M | 25.68M | 26.81M | video+image

Table XV: Statistics of multi-modality datasets.

Dataset | #video | #text | Avg Video
---|---|---|---
Train | Val | Test | Train | Val | Test | Length (s)
Image Classification |  |  |  |  |
ImageNet-1K | 1,281,167 | 50,000 | 100,000 | - | - | - | -
Short-term Action Recognition |  |  |  |  |
Kinetics-400 | 240,436 | 19,787 | - | - | - | - | 10
Something-Something V2 | 168,913 | 24,777 | - | - | - | - | 4
Long-term Action Recognition |  |  |  |  |
Breakfast | 1,577 | - | 410 | - | - | - | 137
COIN | 9,026 | - | 2,796 | - | - | - | 142
LVU | 7,619 | 1,666 | 1,551 | - | - | - | 134
Relation | 138 | 49 | 41 | - | - | - | 127
Speak | 871 | 196 | 188 | - | - | - | 133
Scene | 514 | 107 | 81 | - | - | - | 132
Director | 680 | 163 | 107 | - | - | - | 137
Genre | 2807 | 569 | 584 | - | - | - | 130
Writer | 748 | 174 | 168 | - | - | - | 142
Year | 725 | 163 | 141 | - | - | - | 133
Like | 658 | 142 | 139 | - | - | - | 159
View | 478 | 103 | 102 | - | - | - | 112
Video-Text Retrieval |  |  |  |  |
MSRVTT | 7,010 | - | 1,000 | 140,200 | - | 1,000 | 15
DiDeMo | 8,496 | 1,094 | 1,036 | 8,496 | 1,094 | 1,036 | 29
ActivityNet | 10,009 | 4,917 | - | 10,009 | 4,917 | - | 180
LSMDC | 101,055 | - | 1,000 | 101,055 | - | 1,000 | 5
MSVD | 1,200 | 100 | 670 | 1,200 | 100 | 670 | 15

Table XVI: Statistics of single-modality datasets.

We show the statistics of multi-modality datasets in Table XV, and single-
modality datasets in Table XVI.