# MAViL: Masked Audio-Video Learners

Po-Yao Huang  Vasu Sharma  Hu Xu  Chaitanya Ryali  Haoqi Fan  Yanghao Li
Shang-Wen Li  Gargi Ghosh  Jitendra Malik  Christoph Feichtenhofer

###### Abstract

We present Masked Audio-Video Learners (MAViL) to train audio-visual
representations. Our approach learns with three complementary forms of self-
supervision: (1) reconstruction of masked audio and video input data, (2)
intra- and inter-modal contrastive learning with masking, and (3) self-
training by reconstructing joint audio-video contextualized features learned
from the first two objectives. Pre-training with MAViL not only enables the
model to perform well in audio-visual classification and retrieval tasks but
also improves representations of each modality in isolation, without using
information from the other modality for fine-tuning or inference. Empirically,
MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1%
accuracy). For the first time, a self-supervised audio-visual model
outperforms ones that use external supervision on these benchmarks.

Machine Learning, ICML

Meta AI, FAIR

##  1 Introduction

We study self-supervised representation learning from audio and video, two
rich, heterogeneous, yet closely related modalities of human perception. Self-
supervised learning (SSL) with Transformers (Vaswani et al., 2017; Paulus et
al., 2017) has revolutionized many research areas. There are mainly two types
self-supervision: reconstruction and contrastive learning. In natural language
processing (NLP), BERT (Devlin et al., 2019) pre-training by performing masked
prediction on large-scale language corpora has refreshed the leaderboards of
many NLP tasks. For other modalities, masked autoencoders (MAE) have
effectively learned uni-modality representations by reconstructing images (He
et al., 2022), video (Feichtenhofer et al., 2022), and audio (Huang et al.,
2022a). On the other hand, contrastive learning is popular to learn cross-
modality alignment in language-image (Radford et al., 2021) or audio-visual
(Owens & Efros, 2018) tasks.

Figure 1: Masked Audio-Video Learners (MAViL). We introduce three multimodal
objectives for learning complementary audio-video representations: (1) Masked
audio-video reconstruction; (2) Masked inter-modal and intra-modal contrastive
learning; (3) Masked self-training on contextualized multimodal features.
MAViL not only learns strong joint audio-video representations, but can also
improve single modality encoders, without using the other modality during
fine-tuning or inference.

In this work, we investigate a unifying methodology where masked autoencoding
meets contrastive objectives for learning audio-video representations. Besides
targeting strong joint representations for audio-video tasks, we are also
interested in uni-modal advances; _e.g_., for audio-only tasks, we envision
that an audio encoder would work better if it learns also from video over only
from audio.

We present Masked Audio-Video Learners (MAViL). Our model consists of an
audio-visual encoder pair, a shallow fusion-encoder layer to connect and
exchange multimodal information, and separate decoders for reconstructing raw
inputs and contextualized features, as illustrated in Fig. 1. We train MAViL
using three objectives outlined next.

First, we reconstruct information that has been removed by applying a high
masking ratio (_e.g_. 80%) to the input (He et al., 2022; Feichtenhofer et
al., 2022; Huang et al., 2022a). With this, MAViL learns complementary audio-
video representations by reconstructing a single modality, with the
supplementary context from the other.

Second, to utilize natural audio-visual correspondences in videos, we employ
contrastive objectives (Gutmann & Hyvärinen, 2010; van den Oord et al., 2018)
to learn an aligned latent space where an audio and a video clip are close to
each other if they are spatiotemporally/audiovisually corresponding (_e.g_.,
from the same instance) and enforce distance in embeddings for non-
corresponding pairs. MAViL uses two forms of contrastive alignment. (i) A
cross-modal objective that pulls together video and audio from the same sample
and contrasts it with other samples (_i.e_. inter-modal contrast). (ii) A
within-modal objective, that generates two complementary masks per modality
and then pulls together encoder embeddings from the same instance, while
contrasting it with embeddings from other instances of the same modality
(_i.e_. intra-modal contrast). Our experiments show that both inter- and
intra-modal contrastive learning contribute to stronger representations.

Third, to learn both _complementary_ and _aligned_ audio-video
representations, we go beyond reconstructing _heterogeneous_ , uni-modal raw
inputs (audio or video) and introduce a new pre-training objective that
reconstructs _homogeneous_ contextualized features in a joint audio-video
latent space.

We are motivated by the recent successes in visual SSL that use disparate
teacher models to generate contextualized features as reconstruction targets.
For example, BEiT (Bao et al., 2021) uses features by DALL-E (Ramesh et al.,
2021) as its target. MaskFeat (Wei et al., 2022a) predicts masked target
features such as HOG (Dalal & Triggs, 2005) or DINO (Caron et al., 2021).
Inspired by these uni-modal teachers, we take a step further to propose a new
masked _multimodal contextualized reconstruction_ pretext task. Our core idea
is: instead of predicting raw inputs in heterogeneous modalities, the model
with masked-view inputs jointly predicts contextualized representations in the
homogeneous _audio-video latent space_ generated by a teacher (_i.e_. an
identical model pre-trained with the masked reconstruction and contrastive
objectives above) with complete-view inputs in all modalities. This approach
ensures the students learn well-aligned and contextualized audio-video
representations and thus improves its performance on downstream tasks. To
achieve this without relying on external teacher models, we use a simple two-
stage self-training framework (illustrated in Fig. 2). In the 1st stage, we
train the teacher MAViL and use raw inputs as its reconstruction targets. In
the 2nd stage, the student MAViL (the final model for later fine-tuning)
learns to reconstruct the contextualized features output by the fusion encoder
of the teacher.

Experiments on audio-video classification and retrieval tasks validate the
effectiveness of our methods. We show that audio-video pre-training
significantly outperforms audio-only pre-training. MAViL surpasses self-
supervised benchmarks and is on par with the best supervised pre-trained
model. Specifically, MAViL achieves a new state-of-the-art (audio-only and
audio-video) for classification on AudioSet (48.4 and 53.1 mAP) and VGGSound
(60.8 and 67.1 Acc), as well as on audio-to-video retrieval in VTT (23.5 R@1)
and YouCook (33.0 R@1).

In summary, this paper presents MAViL to learn a joint audio-visual
representation by reconstructing masked audio and video inputs, contrasting
inter-modal and intra-modal embeddings, as well contextualized self-training
on joint audio-video latents from a previously learned model. MAViL
outperforms the prior state-of-the-art on AudioSet and VGGSound classification
as well as audio-to-video retrieval on VTT and YouCook.

##  2 Related Work

#### Supervised Audio-Video Models.

Inspired by the fact that infants pick up a language and learn their
understanding about the world by relying on visual signals and co-occurring
audio events (Roy, 1999), research on audio-visual learning has a long history
(Ramachandram & Taylor, 2017; Chrupała, 2022). Audio-video automatic speech
recognition (ASR) (Petajan, 1984; Chen & Rao, 1998; Potamianos et al., 2003)
and person identification (Aleksic & Katsaggelos, 2006) have been studied
before the deep learning era. Advances in deep learning enabled progress in
supervised audio and audio-video tasks (Ngiam et al., 2011; Kim et al., 2013;
Ephrat et al., 2018; Kazakos et al., 2019; Xiao et al., 2020; Chen et al.,
2020; Nagrani et al., 2021; Koutini et al., 2021; Chen et al., 2022) However,
deep learning techniques are data and annotation hungry. Although there are
efforts to collect large-scale, labeled datasets in certain audio-video tasks
(Harwath et al., 2018), the dependency on heavy investment in annotation slows
down the application of advancements in modeling to more tasks. In comparison,
MAViL relies on no labeled data.

#### Self-Supervised Audio-Video Representation Learning.

To alleviate annotation effort and better exploit unlabeled audio-video
content abundant on the Internet, a considerable body of researches explore
self-supervised techniques for learning audio-video representations (Aytar et
al., 2016; Arandjelovic & Zisserman, 2017, 2018; Korbar et al., 2018; Shi et
al., 2022). Similar to the popular CLIP (Radford et al., 2021) model for
image-text, inter-modal contrastive learning has been studied to associate
views from paired audio-video instances as self-supervision (Ma et al., 2020;
Owens & Efros, 2018; Morgado et al., 2021b; Patrick et al., 2021b). Data
augmentation (Patrick et al., 2021a; Wang et al., 2021) and methods for mining
harder negatives (Zeng et al., 2021; Recasens et al., 2021; Morgado et al.,
2021a) have been studied to improve performance. MAViL integrates masked auto-
encoding and contrastive learning. We noticed another independent and
concurrent study CAV-MAE (Gong et al., 2022b) also leverages MAE (He et al.,
2022) and inter-modal contrastive objective (Huang et al., 2022b). Compared to
theirs, MAViL additionally facilitates intra-modal contrastive and promotes
multimodal contextualized reconstruction that deliver superior performance.

#### Knowledge Distillation

KD (Hinton et al., 2015; Tian et al., 2019; Park et al., 2019b) or student-
teacher learning was initially been designed for model compression (Romero et
al., 2015; Mishra & Marr, 2018; Cho & Hariharan, 2019). Recently, KD is
gaining attention in the context of SSL, including learning better
distribution of SSL representation (Wei et al., 2022c) by
distilling/reconstructing teacher’s contextualized features (Grill et al.,
2020; Bao et al., 2021; Wei et al., 2022b). MoCo (He et al., 2020), DINO
(Caron et al., 2021), and dBOT (Liu et al., 2022) leverage self-distillation
to bootstrap targets from model snapshots in previous pre-training steps.
Data2vec (Baevski et al., 2022) performs self-distillation in disparate
modalities, where each one bootstraps targets separately. MaskFeat (Wei et
al., 2022a) pre-trains by prediction of masked target features such as HOG
(Dalal & Triggs, 2005) or DINO. In contrast, MAViL is the fist work that
employs multimodal self-distillation with MAE. It uniquely incorporates
complete-view multimodal fusion in the teacher network while the students take
masked-view inputs.

##  3 Masked Audio-Video Learners

We introduce Masked Audio-Video Learners (MAViL), a self-supervised audio-
video representation learning framework illustrated in Fig. 1. Our framework
consists of two stages: In the 1st stage, MAViL learns to reconstruct
spectrogram and pixels simultaneously by exploiting complementary information
from each modality (§3.1), and couples this with contrastive learning to
encourage intermediate (inter-/intra-)modal alignments of semantically similar
audio and video pairs (§3.2). In the 2nd stage, as illustrated in Fig. 2, we
use the first-stage model trained with raw targets as the teacher. We use its
contextualized features to guide the student in self-training (§3.3). In the
following, we describe these three types of supervision in detail and start by
discussing MAViL trained with raw inputs.

###  3.1 Masked Uni-Modal Reconstruction

Since perception naturally involves processing visual and acoustic information
jointly, MAViL uses a fusion encoder to learn joint audio-visual latents in a
multimodal Transformer and uses the complementary information from each
modality to reconstruct raw audio and video simultaneously. This is different
from uni-modal MAE approaches (_e.g_. MAE (He et al., 2022), Audio-MAE (Huang
et al., 2022a), or Video-MAE (Feichtenhofer et al., 2022)).

Specifically, given an audio-video pair
$(\mathrm{a},\mathrm{v})\in\mathcal{D}$, we first patchify/tokenize
spectrogram and video frame pixels into audio and visual tokens with uni-modal
patch-embedding layers into $\mathbf{a}=[a_{1}\dots a_{N}]$ for spectrogram
and $\mathbf{v}=[v_{1}\dots v_{M}]$, where $a_{i},v_{j}\in\mathbb{R}^{H}$ for
RGB patches. This is achieved via applying (FFT/video) transform(s),
2D-convolutional layer, and flattening. Similar to MAE (He et al., 2022), we
add fixed 2-D sinusoidal positional embeddings to the embedded patches for
each modality. For masked audio-video reconstruction, we randomly mask the
majority (_i.e_. 80%) of audio and video tokens and only feed the $20\%$
unmasked audio tokens $\mathbf{a}^{\prime}$ and video tokens
$\mathbf{v}^{\prime}$ to the audio encoder $f_{\text{a}}(.)$ and the video
encoder $f_{\text{v}}(.)$, respectively. The resulting uni-modal embedding is
denoted as $\mathbf{a}_{\text{um}}=f_{\text{a}}(\mathbf{a}^{\prime})$ and
$\mathbf{v}_{\text{um}}=f_{\text{v}}(\mathbf{v}^{\prime})$.

We follow our uni-modal encoders with a multimodal fusion encoder. We
investigate two Transformer variants as the joint fusion encoder: Vanilla
Transformers (Vaswani et al., 2017) and Multimodal Bottleneck Transformers
(MBT (Nagrani et al., 2021)). The vanilla Transformer $g_{\text{av}}(.)$
jointly encodes audio and video tokens as
$(\mathbf{a}_{\text{um}}^{l+1}\|\mathbf{v}_{\text{um}}^{l+1})=\text{Transformer}^{l}(\mathbf{a}_{\text{um}}^{l}\|\mathbf{v_{\text{um}}}^{l})$,
where “$\|$” means concatenation. The bottleneck variant tries to encourage
the model to more selectively collate and condense relevant information in
each modality by forcing information exchange between modalities to pass
through a small number of learnable bottleneck features
$\mathbf{b}^{0}=[b_{1}\dots b_{4}],b_{i}\in\mathbb{R}^{H}$, as originally
proposed in MBT for supervised audio-video learning. Precisely,
$\mathbf{a}_{\text{um}}^{l+1}\|\mathbf{b}_{\text{a}}^{l+1}=g_{\text{av}}^{l}(\mathbf{a}_{\text{um}}^{l}\|\mathbf{b}^{l})$
and
$\mathbf{v}_{\text{um}}^{l+1}\|\mathbf{b}_{\text{v}}^{l+1}=g_{\text{av}}^{l}(\mathbf{v}_{\text{um}}^{l}\|\mathbf{b}^{l})$,
where
$\mathbf{b}^{l+1}=(\mathbf{b}_{\text{a}}^{l+1}+\mathbf{b}_{\text{v}}^{l+1})/2$.
In each variant, we stack $L$ Transformer blocks to produce the audio-video
jointly encoded output $\mathbf{a}_{\text{mm}}$ and $\mathbf{v}_{\text{mm}}$.

We use standard Transformer blocks as the audio and video decoders. The
encoded tokens from the fusion encoder are then projected and padded with
trainable [MASK] tokens. After restoring the original order (time-frequency
for audio tokens and space-time for video tokens), we add the decoder’s (fixed
2-D sinusoidal) positional embeddings and feed the restored sequence into the
decoder $f_{\text{a}}^{-1}(.)$ and $f_{\text{v}}^{-1}(.)$. At the top of the
decoder stack of each modality, we add a linear head to reconstruct the input
spectrogram and pixels. Formally, the decoder output for spectrogram
reconstruction is
$\mathbf{\hat{a}}=f_{\text{a}}^{-1}(g_{\text{av}}(f_{\text{a}}(\mathbf{a}^{\prime})))$
and the pixel reconstruction is
$\mathbf{\hat{v}}=f_{\text{v}}^{-1}(g_{\text{av}}(f_{\text{v}}(\mathbf{v}^{\prime})))$.
For notation clarity, we omit the [MASK] tokens and linear projection head.

Let $\hat{\mathbf{a}}_{i},i=1\dots n$ be the reconstruction of $i$-th masked
spectrogram patch generated by the decoder’s reconstruction head. Similarly,
$\hat{\mathbf{v}}_{j},j=1\dots m$ for video patches. In masked audio-video
reconstruction, MAViL is self-supervised by minimizing the reconstruction
objective $\mathcal{L}_{r}^{\text{raw}}$ defined as:

$\mathcal{L}_{r}^{\text{raw}}=\frac{1}{n}\sum_{i=1}^{n}(\hat{\mathbf{a}}_{i}-\mathbf{a}_{i})^{2}+\frac{1}{m}\sum_{j=1}^{m}(\hat{\mathbf{v}}_{j}-\mathbf{v}_{j})^{2}.$
(1)

Figure 2: Masked multimodal contextualized reconstruction in the aligned
multimodal latent space. Stage-1 (left): Training MAViL teacher with raw
inputs as the targets. Stage-2 (right): Distilling to MAViL student with
teacher’s multimodal contextualized features (complementary and aligned with
masked autoencoding and contrastive learning after Stage-1) as the targets.

###  3.2 Contrastive Audio-Video Learning with Masking

With the natural audio-visual alignment found in videos, we aim to learn from
spatiotemporal correspondences within and across modalities in a self-
supervised fashion. We use contrastive learning to enforce alignment among
multiple “views” from the same training instance. The “views” to be aligned
can be within modality observations of the instance itself or its semantic
observations in different modalities (_e.g_., a video and its corresponding
audio track). We use the InfoNCE (van den Oord et al., 2018) loss for
contrastive learning. Let
$\mathbf{x}=[\mathbf{x}_{1}\dots\mathbf{x}_{B}],\mathbf{y}=[\mathbf{y}_{1}\dots\mathbf{y}_{B}];\mathbf{x}_{i},\mathbf{y}_{j}\in\mathbb{R}^{H}$
be the instance-level representations of audio/video in a size $B$ sampled
batch. The contrastive loss $\mathcal{L}_{\text{c}}(\mathbf{x},\mathbf{y})$ is
defined as:

$\mathcal{L}_{\text{c}}(\mathbf{x},\mathbf{y})=-\frac{1}{B}\sum_{i=1}^{B}{\rm
log}\frac{{\rm
exp}(\text{S}(\mathbf{x}_{i},\mathbf{y}_{i})/\tau)}{\sum_{j=1}^{B}{\rm
exp}(\text{S}(\mathbf{x}_{i},\mathbf{y}_{j})/\tau))},$ (2)

where
$\text{S}(\mathbf{x}_{i},\mathbf{y}_{j})=\frac{\mathbf{x}_{i}^{T}\mathbf{y}_{j}}{\|\mathbf{x}_{i}\|\|\mathbf{y}_{j}\|}$
is the cosine similarity between embeddings $\mathbf{x}_{i}$, $\mathbf{y}_{j}$
and $\tau$ is the softmax temperature. We design two types of contrastive
objectives for self-supervised learning from audio-video pairs: 1) _inter-
modal_ contrastive learning $\mathcal{L}_{\text{c}}^{\text{inter}}$ and 2)
_intra-modal_ contrastive learning $\mathcal{L}_{\text{c}}^{\text{intra}}$.

Inter-modal contrastive learning promotes _cross-modal alignment_
(intuitively, the audio and visual embeddings of a video should be close to
each other). MAViL performs contrastive learning over masked instances for
better computation efficiency and to jointly train with the masked
reconstructing loss above. We first average embeddings from uni-modal encoders
over the sequence length, namely,
$\mathbf{a}_{\text{emb}}=\text{Avg}(\mathbf{a}_{\text{um}})$ and
$\mathbf{v}_{\text{emb}}=\text{Avg}(\mathbf{v}_{\text{um}})$. The positive
pairs are the video and audio clips from the same video, while all the other
combinations of audio-video pairs are considered negatives.
$\mathcal{L}_{\text{c}}^{\text{inter}}$ draws closer the representations of
paired audio and video and meanwhile pushes away the mismatched ones.
Minimizing it effectively learns an audio-video latent space where
semantically similar instances in different views (_e.g_. paired audio-video)
are close to each other and instances are distant if they are uncorrelated. We
do not use the fusion output $\mathbf{a}_{\text{mm}}$ and
$\mathbf{v}_{\text{mm}}$ for contrastive learning since the joint encoding
layer provides a shortcut of cross-modal information exchange for every paired
audio-video clip, resulting in sub-optimal performance.

Intra-modal contrastive learning promotes _within-modality alignment_ , where
the representations of different views, _i.e_., different masked samples of an
audio (or video) stream should be close to each other. For each modality, we
sample a second random view containing 20% of tokens that are mutually
exclusive from the first view. We then use encoded representations from these
views as the positive pairs to contrast against other instance-level
representations of the same modality in the training batch as negatives.

Formally, the inter- and intra-modal contrastive objectives in MAViL are
defined as:

$\displaystyle\mathcal{L}_{\text{c}}^{\text{inter}}$
$\displaystyle=\mathcal{L}_{c}(\mathbf{a}_{\text{emb}},\mathbf{v}_{\text{emb}}),$
(3) $\displaystyle\mathcal{L}_{\text{c}}^{\text{intra}}$
$\displaystyle=\frac{1}{2}\left[\mathcal{L}_{c}(\mathbf{a}_{\text{emb}},\mathbf{\ddot{a}}_{\text{emb}})+\mathcal{L}_{c}(\mathbf{v}_{\text{emb}},\mathbf{\ddot{v}}_{\text{emb}})\right],$

where $\mathbf{\ddot{a}}_{\text{emb}}$ and $\mathbf{\ddot{v}}_{\text{emb}}$
represent the second view sampled from different time-frequency and space-time
locations for audio and video modalities, respectively. Overall, MAViL is
self-supervised by minimizing the following objective:

$\mathcal{L}_{\text{{MAViL}}}=\mathcal{L}_{r}^{\text{raw}}+\alpha\mathcal{L}_{\text{c}}^{\text{inter}}+\beta\mathcal{L}_{\text{c}}^{\text{intra}}.$
(4)

where $\alpha$ and $\beta$ are the weights that balance loss functions.

###  3.3 Masked Multimodal Contextualized Reconstruction

To learn multimodal representations that capture joint audio-visual
information, we go beyond reconstructing raw uni-modal (audio or video) inputs
as in MAE and design a new pretext task that reconstructs multimodal
contextualized representations in the joint latent space. To achieve this, we
simply employ a 2-stage training framework illustrated in Fig. 2. In the 1st
stage, we follow the training paradigm introduced in §3 to train a MAViL
teacher model by reconstructing spectrograms and pixels. In the 2nd stage, we
extend the training framework into a teacher-student framework. We freeze
MAViL trained in the 1st stage and treat it as the teacher to generate
contextualized multimodal features as reconstruction targets. These targets
then guide the learning of the re-initialized _student_ MAViL.

We forward complete-view audio and video to the audio, video, and fusion
encoders of the frozen teacher network
$\mathbf{a}^{\text{Teacher}}\|\mathbf{v}^{\text{Teacher}}=g_{\text{av}}^{\text{Teacher}}(f_{\text{a}}^{\text{Teacher}}(\mathbf{a})\|f_{\text{v}}^{\text{Teacher}}(\mathbf{v}))$.
Masked-view audio and video tokens are passed through student MAViL encoders
and decoders for reconstruction (_i.e_.,
$\mathbf{\tilde{a}}=f_{\text{a}}^{-1}(g_{\text{av}}(f_{\text{a}}(\mathbf{a}^{\prime})))$
as the audio feature prediction; and
$\mathbf{\tilde{v}}=f_{\text{v}}^{-1}(g_{\text{av}}(f_{\text{v}}(\mathbf{v}^{\prime})))$
as the video feature prediction). The contextualized reconstruction objective
is defined as:

$\mathcal{L}_{r}^{\text{context}}=\frac{1}{n}\sum_{i=1}^{n}(\tilde{\mathbf{a}}_{i}-\mathbf{a}_{i}^{\text{Teacher}})^{2}+\frac{1}{m}\sum_{j=1}^{m}(\tilde{\mathbf{v}}_{j}-\mathbf{v}_{j}^{\text{Teacher}})^{2}.$
(5)

In the 2nd stage training, we jointly minimize the masked contextualized
reconstruction objective and the contrastive objectives111We empirically
observe that adding the raw autoencoding loss on pixel/spectrogram information
as in the 1st stage of training does not provide any further gains.. The
student MAViL’s objective is:

$\mathcal{L}_{\text{{MAViL}}}=\mathcal{L}_{r}^{\text{context}}+\alpha\mathcal{L}_{\text{c}}^{\text{inter}}+\beta\mathcal{L}_{\text{c}}^{\text{intra}}.$
(6)

Note that this objective contains pure latent targets. The teacher multimodal
encoder encourages the student network to predict contextualized joint
audiovisual representations with masked view inputs. We use the audio/video
encoders in the final student model for fine-tuning downstream evaluation
tasks.

##  4 Experiments

We conduct extensive evaluations including audio-video classification on
AudioSet (Gemmeke et al., 2017) (AS-2M, AS-20K) and VGGSound (Chen et al.,
2020); and audio-to-video retrieval on MSR-VTT (Xu et al., 2016); and YouCook
(Zhou et al., 2018). We use AS-20K for model analysis and ablation studies.

###  4.1 Datasets

#### AudioSet

(Gemmeke et al., 2017) (AS-2M, AS-20K) contains 2 million 10-second YouTube
clips for audio event detection. 527 types of audio events are weakly
annotated (Li et al., 2022; Hershey et al., 2017, 2021) for each clip, and
multiple events can occur in one clip. The _full_ training set has two
subsets: A class-wise _balanced_ (22K clips) and an _unbalanced_ (2 million
clips) set. The _eval_ set has 20K clips. We downloaded around 1.97M
unbalanced training, 20K balanced training, and 19K evaluation clips. For the
AS-2M experiments, we use the union of unbalanced and balanced sets for pre-
training and fine-tuning. For the AS-20K experiments, we use AS-2M for pre-
training and the 20K balanced set for fine-tuning. We report the testing mAP
on the 19K _eval_ set used by AST (Gong et al., 2021).

#### VGGSound

(Chen et al., 2020) contains around 200K video clips of 10-second length,
annotated with 309 sound classes consisting of human actions, sound-emitting
objects, and human-object interactions. The dataset is split into $\sim$183K
train and $\sim$15k test samples. Unlike AudioSet, the audio event in each
clip is guaranteed to be “visually present” in the video. We report top-1
classification accuracy.

###  4.2 Implementation Details

We use different temporal footprints for audio and video. For audio, following
(Nagrani et al., 2021; Gong et al., 2021), we transform raw waveform (pre-
processed into mono-channel under 16K sampling rate) into 128 Kaldi (Povey et
al., 2011)-compatible Mel-frequency bands with a 25ms Hanning window that
shifts every 10 ms. For a 10-second recording in AudioSet, the resulting
spectrogram is of 1$\times$1024$\times$128 dimension. We tokenize it as non-
overlapped $16\times 16$ patches (_i.e_., kernel and stride are 16 for both
time and frequency). The flattened audio token length $N$ is $512$. For video,
MAViL takes 4-second clips (8 frames, 2 frames per second). The frame size is
$224\times 224$. The spatial kernel and stride are $16\times 16$ and the
temporal kernel and stride is 2. The flattened audio token length $N$ is
$784$.

We follow the main design choices of Audio-MAE (Huang et al., 2022a) and
original MAE (He et al., 2022) for images. We use 12-layer Transformers (with
768-dimensions and 12 attention heads) as the encoders for each modality. The
embedding dimension $H$ is $768$. The audio-video fusion encoder layer is a
two-layer ($L$=2) (vanilla or MBT) Transformer on top of uni-modal encoders.
Similarly, the audio and video decoders are 8-layer Transformers with an
embedding dimension of 512 and 16 attention heads. The encoder and decoder in
MAViL have 164M and 27M parameters, respectively. For balancing the losses in
(4), we set $\alpha=0.1,\tau_{\text{c}}^{\text{inter}}=0.1$ and
$\beta=0.01,\tau_{\text{c}}^{\text{intra}}=1.0$ to scale the gradients from
three objectives into a comparable range to improve training stability.

For the audio branch, following (Huang et al., 2022a), we pre-train the audio
Transformers from scratch. For the visual branch, we initialized the visual
Transformer with the self-supervised MAE (He et al., 2022) pre-trained on
ImageNet. MAViL operates under a fully self-supervised setup.

###  4.3 Experimental Setup

We pre-train MAViL on AudioSet-2M audio-video clips without using any labels.
By default, we pre-train with 64 GPU with a 512 accumulated batch size and a
0.0002 learning rate (please refer to Appendix for detailed hyperparameters in
all experiments). We pre-train for 20 epochs in each stage. Each stage takes
$\sim$20 hours.

We then evaluate the learned audio and video representations by fine-tuning on
audio-video event classification tasks (AudioSet-20K, AudioSet-20M, VGGSound).
We follow the setup in MAE to only keep the pre-trained audio and video
encoders. We apply average pooling on top of the encoded tokens followed by a
linear layer for classification. We use the standard fine-tuning pipeline in
MAE and adopt hyperparameters and augmentation in prior audio-based and audio-
video event classification works (Gong et al., 2021; Huang et al., 2022a;
Nagrani et al., 2021). Specifically, we use mixup (Zhang et al., 2018) and
balanced sampling (Li et al., 2022) and fine-tuning masking (Huang et al.,
2022a) We apply random time shifts and magnitude augmentation for audio and
standard video augmentations in MAE for video (Feichtenhofer et al., 2022). We
fine-tune the model under 3 scenarios: 1) audio-only, 2) video-only, and 3)
audio+video. Empirically we observe a discrepancy in the convergence rate
between audio and video. We circumvent this by applying a 50% learning rate
reduction for the weights of the video encoder when performing audio+video
fusion fine-tuning. Fine-tuning for 100 epochs on AS-2M takes $\sim$10 hours.
More details are in Appendix.

In the following, we first analyze MAViL on AS-20K (§4.4) and then present the
main results compared to prior works in §4.5. The gray entries represent the
default setup.

###  4.4 Model Analysis

#### Masked Audio-Video Learning.

We start by examining the masking ratio, the key hyperparameter in MAE. We
report the classification accuracy in AS-20K fine-tuning (after AS-2M pre-
training) for both audio and video encoders. We use the same masking ratio for
both audio and video and ablate different values in Table 1. The results
suggest that 80% masking yields the best performance (and works slightly
better than 70% or 90%). This is similar to the optimal values of 80% in
Audio-MAE (Huang et al., 2022a), 90% in Video-MAE (Feichtenhofer et al.,
2022), encoding only non-masked tokens significantly reduces the sequence
length and therefore computation (which scales quadratically to the length in
self-attention layers of Transformers).

Ratio |  50% |  60% |  70% |  80% |  90%
---|---|---|---|---|---
Audio |  35.3 |  36.5 |  36.7 |  36.8 |  36.8
Video |  17.0 |  17.3 |  17.5 |  17.7 |  17.5
Table 1: Masking Ratio. Performance is measured as mAP in AS-20K for
individual fine-tuning as Audio or Video classification.

We next investigate the improvements by learning with complementary multimodal
information in masked autoencoding. We ablate either using a vanilla
Transformer stack or MBT to fuse multimodal information for reconstructing raw
input. Table 2 summarizes the accuracy on AS-20K for using either 2 or 4
layers of fusion encoder. The first column (None) reports the baselines
without the joint audio-video encoder layer. MAE and video-MAE baselines
without the fusion audio-video encoder layer. Using fusion leads to +0.4 mAP
gain for audio and video. Although MBT layers achieve better results, the
difference is not significant over using vanilla Transformers for masked
audio-video learning. Two MBT(2) Transformer blocks are sufficient and perform
as well as four MBT(4) blocks. We choose to use vanilla Transformers by
default for simplicity.

Fusion Enc. | None | Vanilla(2) |  Vanilla(4) |  MBT(2) |  MBT(4)
---|---|---|---|---|---
Audio | 36.4 | 36.8 | 36.7 | 36.9 | 36.8
Video | 17.4 | 17.8 | 17.6 | 17.8 | 17.7
Table 2: Fusion Encoder: Transformer type(# layers). The “None” are the uni-
modal MAE baselines (Audio-MAE and Video-MAE).

#### Contrastive Audio-Video Learning with MAE.

Next, we investigate improvements when contrastive learning is used in
conjunction with MAE. We employ an intra-contrastive loss on only 20% of
visible patches in each modality. As shown in Table 3, our contrastive losses
are shown to be important for learning audio-video representations. Both
inter-modal and intra-modal contrastive learning improve the results, and
combining them yields the best performance. Note that while audio and video
are used jointly in pre-training, the fine-tuning is done with uni-modal
encoders. Therefore the improvements for audio and video classification
demonstrate that learning from both modalities can improve an individual
modality.

Contastive objs. |  None |  inter |  intra |  inter+intra
---|---|---|---|---
Audio |  36.8 |  38.4 |  38.1 |  39.0
Video |  17.8 |  21.0 |  19.8 |  22.2
Table 3: Masked Contrastive Audio-Video Learning.

We also experimented with alternative pooling functions to represent each
instance in contrastive learning. Empirically, we observe a simple average
pooling over uni-modal encoder output (_i.e_.,
$\mathbf{a}_{\text{emb}}=\text{Avg}(\mathbf{a}_{\text{um}})$ for visible audio
tokens, and similar for video) yields the best result, compared to using the
[CLS] token (-0.3 mAP) or adding additional MLP layers (-0.3 mAP).

#### Multimodal Contextualized Reconstruction.

As illustrated in Fig. 2, MAViL employs a two-stage training. In the 1st
stage, we train a model to reconstruct raw inputs with masked and contrastive
audio-video learning analyzed in the previous experiments. In the 2nd stage,
we re-initialize the student MAViL model and train it to reconstruct the
contextualized features generated by the frozen teacher from the 1st stage. We
compare reconstruction targets in Table 4.

Target | Raw Inputs | A/V-MAE | MAViL-Uni | MAViL-Fusion
---|---|---|---|---
Audio | 39.0 | 39.5 | 40.0 | 40.7
Video | 22.2 | 23.1 | 23.8 | 24.1
Table 4: Reconstruction Target for the 2nd stage MAViL. Raw: spectrogram/pixel
(column 1). Contextualized: A-MAE/V-MAE (uni-modal output, column 2); MAViL-
Uni (uni-modal output, column 3); MAViL-Fusion (multimodal fusion output,
column 4).

Our multimodal contextualized reconstruction predicts pre-trained MAViL
teacher’s multimodal fusion encoder output (_i.e_.,
$g_{\text{av}}^{\text{Teacher}}(f_{\text{a}}^{\text{Teacher}}(\mathbf{a})\|f_{\text{v}}^{\text{Teacher}}(\mathbf{v}))$,
MAViL-Fusion). First we compare it with raw input reconstruction and single
modal contextualized reconstruction (_i.e_., predicting separately pre-trained
(uni-modal) Audio-MAE and Video-MAE teachers’ encoder output). The results
show that predicting MAViL teacher’s fusion encoder output outperforms
predicting uni-modal MAEs output or raw spectrogram/pixels (identical to one-
staged MAViL).

Next we compare with reconstructing (uni-modal) MAViL-Uni Teacher features
(_i.e_., $f_{\text{a}}^{\text{Teacher}}(\mathbf{a})$ and
$f_{\text{v}}^{\text{Teacher}}(\mathbf{v})$ which is before the fusion encoder
in Fig. 2). The results imply that stronger audio-video representations (+0.7
for audio mAP) can be learned by masked prediction of contextualized and
synchronized multimodal features, even though they are generated by a pre-
trained MAViL teacher without external data or model. The gain with multimodal
contextualized reconstruction is (+1.7 audio and +1.9 video mAP).

#### Additional Ablations.

Additional ablations are summarized in Table 5. We draw the following
observations:

(a) Larger data size is critical for self-supervised learning (Tab. 5a ablates
using 10%, 50% or 100% of AS-2M for pre-training of MAViL).

(b) Longer pre-training is beneficial and 20 epochs are sufficient (Table 5b)
for good performance.

(c) Tuning contrastive loss weights is helpful (Table 5c).

(d) ImageNet pre-training is useful for the video classification stream (Table
5d). Initializing from an ImageNet fine-tuned model that uses labels (IN-SL)
is beneficial yet we avoid using it to keep MAViL fully self-supervised.

e) Increasing the encoder sizes (ViT-S/B/L) improves mAP (Table 5e); we use
ViT-B for efficiency as default.

AS-2M 10% 50% 100% Audio 33.1 39.2 40.7 Video 19.3 22.5 24.1

(a) Dataset size

epoch 10 15 20 25 Audio 39.1 40.1 40.7 40.7 Video 20.4 22.5 24.1 24.2

(b) pre-training epoch

$\alpha$ 0.3 0.1 0.05 Audio 40.5 40.7 40.2 Video 19.6 24.1 23.6

(c) Contrastive weight

Init. Scratch IN-SSL IN-SL Audio 40.5 40.7 40.7 Video 22.6 24.1 24.7

(d) Visual backbone initialization

Backbone ViT-S ViT-B ViT-L Audio 36.2 40.7 41.8 Video 20.5 24.1 25.2

(e) Model Size

Table 5: Additional ablations on AS-20K.

In summary, MAViL boost the uni-modal MAE accuracy from 36.4$\rightarrow$40.7
mAP for audio and 17.4$\rightarrow$24.1 mAP for video encoders, clearly
showing that learning from audio and video can improve each modality
individually.

|  | AS-20K | AS-2M
---|---|---|---
Method | PT | A | V | A+V | A | V | A+V
Audio-only Models
PANNs (Kong et al., 2020) | - | 27.8 | - | - | 43.9 | - | -
AST (Gong et al., 2021) | SL | 34.7 | - | - | 45.9 | - | -
HTS-AT (Chen et al., 2022) | SL | - | - | - | 47.1 | - | -
PaSST (Koutini et al., 2021) | SL | - | - | - | 47.1 | - | -
Data2vec (Baevski et al., 2022) | SSL | 34.5 | - | - | - | - | -
SSAST (Gong et al., 2022a) | SSL | 31.0 | - | - | - | - | -
MAE-AST (Baade et al., 2022) | SSL | 30.6 | - | - | - | - | -
Aud-MAE(Huang et al., 2022a) | SSL | 37.0 | - | - | 47.3 | - | -
Audio-Video Models
G-Blend (Wang et al., 2020) | - | 29.1 | 22.1 | 37.8 | 32.4 | 18.8 | 41.8
Perceiver (Jaegle et al., 2021) | - | - | - | - | 38.4 | 25.8 | 44.2
CAV-MAE$\dagger$ (Gong et al., 2022b) | SSL | 37.7 | 19.8 | 42.0 | 46.6 | 26.2 | 51.5
Attn AV (Fayek & Kumar, 2020) | SL | - | - | - | 38.4 | 25.7 | 46.2
MBT *(Nagrani et al., 2021) | SL | 31.3 | 27.7 | 43.9 | 44.3 | 32.3 | 52.1
Our Audio-Video Models
MAViL-Stage1 | SSL | 39.0 | 22.2 | 42.5 | 47.4 | 28.1 | 51.8
MAViL | SSL | 40.7 | 24.1 | 44.3 | 48.4 | 30.2 | 53.1
Table 6:  Comparison to prior works on AudioSet (AS). mAP$\uparrow$ is
reported for audio (A), video (V) and audio+video (A+V) classification. MAViL
uses ViT-B encoders for each modality. Models are pre-trained on AS-2M and
fine-tuned on either AS-20K or AS-2M. PT: pre-training type; SL: (ImageNet)
supervised learning; SSL: self-supervised learning; $\dagger$:concurrent work.
*We de-emphasize the model with non-standard training/testing split. We
underline the best model and bold the best self-supervised model. Table 7-9
follow the same notations.

###  4.5 Main Results

We compare MAViL with other previous models on the standard benchmarks. We
denote the first-stage MAViL (trained with Eq. 4 as MAViL-Stage1. The full
model MAViL is the second-stage student trained with Eq. 6.

#### Audio-Video Event Classification.

Table 6 summarizes the performance comparison on AudioSet (AS). We report
accuracy for fine-tuning the audio (A), video (V) or audio-video encoders
(A+V). Our MAViL sets new state-of-the-art audio-only, audio+video
classification accuracy on both the unbalanced AS-2M and balanced AS-20K.
Compared to all the self-supervised audio-only or audio-video models under
AudioSet, MAViL outperforms them by a large margin. MAViL outperforms the
concurrent CAV-MAE (Gong et al., 2022b). We attribute this to our multimodal
contextualized reconstruction and improved contrastive learning with both
inter-modal and intra-modal contrast. MAViL also outperforms data2vec (Baevski
et al., 2022), demonstrating the superiority of using aligned multimodal
contexts (Eq. 5) as the reconstruction targets over uni-modal contexts.

For the first time, our fully self-supervised MAViL outperforms supervised
audio-video models (_e.g_., MBT (Nagrani et al., 2021)) on audio and
audio+video classification tasks. However, compared to MBT, the fully self-
supervised trained video backbone still needs to catch up in performance,
likely due to the fact that the visual part in AudioSet is noisy and is not
necessarily relevant to the audio events (Mohamed et al., 2022). Such dataset
bias could make the visual pre-training sub-optimal. This is a constraint of
pre-training on AudioSet.

Notably, comparing MAViL-Stage1 and (teacher-guided) MAViL (both are with
(inter-/intra-)modal contrastive learning), there is consistent and
significant improvement of the later with multimodal contextualized
reconstruction. This implies that multimodal contextualized reconstruction is
complementary to contrastive learning, on top of the baselines using
autoencoding of raw inputs (cf. Table 4).

The comparison on VGGSound is shown in Table 7. MAViL again sets a new state-
of-the-art for audio-only and audio+video classification tasks where it
greatly outperforms MBT and the concurrent CAV-MAE. In this out-of-pre-
training-domain task, we observe similar improvements for multimodal
contextualized reconstruction shown by the gain of MAViL for audio-visual
classification (A+V) over the 1st stage (67.1% vs. 63.8% accuracy) of
training.

#### Transfer to Audio/Speech-only tasks.

To test the generalizability of the learned audio representations, we further
evaluate them by transferring AS-pre-trained MAViL to other out-of-domain
speech-only or audio-only tasks. We conduct experiments on the Environmental
Sound Classification (ESC-50) (Piczak, 2015) and Speech Commands (SPC-V1)
(Warden, 2018) datasets. We fine-tune only the audio branch of AudioSet pre-
trained MAViL. The results are summarized in Table 8. MAViL outperforms all
recent supervised and self-supervised models and achieves a new state-of-the-
art. MAViL demonstrates desirable transferability from audio-video self-
supervised pre-training to audio-only downstream tasks.

Method | PT | A | V | A+V
---|---|---|---|---
CAV-MAE$\dagger$(Gong et al., 2022b) | SSL | 59.5 | 47.0 | 65.5
AV-SlowFast (Xiao et al., 2020) | SL | 50.1 | - | -
VGGSound (Chen et al., 2020) | SL | 48.8 | - | -
MBT (Nagrani et al., 2021) | SL | 52.3 | 51.2 | 64.1
MAViL-Stage1 | SSL | 59.9 | 48.3 | 63.8
MAViL | SSL | 60.8 | 50.9 | 67.1
Table 7: VGGSound results (Accuracy$\uparrow$).  Method | PT | ESC-50 | SPC-1
---|---|---|---
AST (Gong et al., 2021) | SL | 88.7 | 95.5
SS-AST (Gong et al., 2022a) | SSL | 88.8 | 96.0
MAE-AST (Baade et al., 2022) | SSL | 90.0 | 95.8
Aud-MAE (Huang et al., 2022a) | SSL | 94.1 | 96.9
MAViL | SSL | 94.3 | 97.4
Table 8: Comparison on audio-only tasks (Accuracy$\uparrow$).

#### Audio-to-Video Retrieval.

Beyond classification tasks, MAViL learns synchronized audio-video
representations which are feasible for textless cross-modal retrieval. To
evaluate the quality of the learned representations, we conduct audio-to-video
retrieval experiments on MSR-VTT and YouCook. In this task, a model encodes
the audio track of a video as the query and performs a search over the testing
video collection by measuring and ranking the similarity between the query
embedding and the video embeddings of testing videos. We fine-tune MAViL with
the audio-video pairs in the training sets and report the retrieval
performance under the standard recall@1 metric on the testing sets.

Following AVLNet (Rouditchenko et al., 2021) and TVLT (Tang et al., 2022), we
also investigate an additional setup to pre-train on HowTo100M (Miech et al.,
2019), a 1.3 million instructional video collection, instead of pre-training
on AudioSet. Note that there are clear domain gaps between pre-training
datasets (AudioSet or HowTo100M) and the target datasets (MSR-VTT or YouCook).
As shown in Table 9, our model outperforms supervised pre-trained models
(AVLNet (Rouditchenko et al., 2021) and self-supervised pre-trained model
(TVLT (Tang et al., 2022) and set new SOTA performance on this task.

Method | PT data | MSR-VTT | YouCook
---|---|---|---
AVLNet  (Rouditchenko et al., 2021) | HT100M | 20.1 | 30.7
TVLT  (Tang et al., 2022) | HT100M | 22.6 | 31.8
MAViL | AS-2M | 22.2 | 31.3
MAViL | HT-100M | 23.5 | 33.0
Table 9: Comparison on audio-to-video retrieval tasks in VTT and YouCook.
(Recall@1$\uparrow$)

#### Qualitative Visualizations

of MAViL-stage1 audio and video decoders can be found in the Appendix.

##  5 Conclusion

We have presented MAViL, a self-supervised audio-video representation learning
framework where masked autoencoding meets contrastive learning. MAViL includes
a set of three objectives for learning complementary and aligned audio-video
representations. First, we use multimodal masked autoencoding to reconstruct
raw inputs in each modality with an encoder-fusion-decoder architecture.
Second, we apply contrastive learning to associate masked audio-video
embeddings, both within and across modalities. Third, we show that the trained
model can serve as a multimodal teacher, to guide new MAViL students in
learning stronger audio-video representations. By aligning and reconstructing
contextualized audio-video representations, the fully self-supervised MAViL
sets a new state-of-the-art compared to recent work in seven audio and audio-
video classification and retrieval tasks outperforming also models with
external supervision.

## Appendix

The Appendix is organized as follows: In §A, we first demonstrate qualitative
audio and video reconstruction results. In §B, we provide the complete
experimental details and hyper-parameters for pre-training and fine-tuning on
each dataset.

Figure 3:  Video clip and spectrogram reconstruction on the AudioSet eval set.
We sample 4 paired (video, audio spectrogram) as follows: Top left: a puppy
video; Top right: a recording from an ambulance’s dash camera; Bottom left: a
person dialing a phone in a dark room; Bottom right: a singer dancing. Input
masking ratio: 70%. In each 3-row group, we show the original video and its
audio (spectrogram) (top), masked input (middle), and MAViL output (bottom).
The spectrogram shape is 1024$\times$128; patch size is 16$\times$16\. Each
spectrogram has 64$\times$8=512 patches with 154 (70% masked) patches being
visible to MAViL. The 8-frame video clip size is $8\times 3\times 224\times
224$; patch size is $16\times 16$. Each video has $4\times 196=784$ patches
after patch embedding (temporal kernel/stride=2) with 235 (70% masked) patches
being visible to MAViL.

##  Appendix A Audio-Video Reconstructions by MAViL-stage1 Decoders

For visualization, we employ MAViL with ViT-B as the audio and video backbone.
We train it with 80% masking ratio on the AudioSet-2M (AS-2M) training set
with raw spectrograms and video frames as its reconstruction targets (Eq. 4).
We then visualize the reconstruction by the MAViL decoder with 70% input
tokens masked to its encoder on AudioSet eval set. The results are shown in
Fig. 3.

As can be observed, MAViL successfully reconstructs both the audio spectrogram
and video frames of a video clip from their extremely corrupted versions. For
videos, the generated reconstructions are of high fidelity and maintain
spatial and temporal consistency of visual objects across a variety of input
domains, scenes, and lighting conditions. For audio reconstruction, MAViL
respects the positions and arrangements of time-frequency components in the
spectrogram, which are crucial for humans to understand and perceive sound.
The audio and video are consistent and well-aligned in time.

| Pre-training | Fine-tuning
---|---|---
Configuration | AS-2M PT | AS-2M | AS-20K | VGGSound | ESC | SPC
Optimizer |  AdamW (Loshchilov & Hutter, 2019)
Optimizer momentum |  $\beta_{1}=0.9$, $\beta_{2}=0.95$
Weight decay | 0.00001
Base learning rate | 0.0002 |  0.0002$\dagger$ | 0.001 | 0.0002 | 0.001 | 0.001
Learning rate schedule |  half-cycle cosine decay (Loshchilov & Hutter, 2017)
Minimum learning rate | 0.000001
Gradient clipping | None
Warm-up epochs | 4 | 20 | 4 | 4 | 4 | 1
Epochs | 20 | 100 | 60 | 60 | 60 | 10
Batch size | 512 | 512 | 64 | 256 | 64 | 256
GPUs | 64 | 64 | 8 | 32 | 4 | 4
Weighted sampling | False | True | False | True | False |  False*
Weighted sampling size | - | 200,000 | - | 200,000 | - | -
Augmentation | R | R | R | R+N | R | R+N
SpecAug (Park et al., 2019a) (time/frequency) | - | 192/48 | 192/48 | 192/48 | 96/24 | 48/48
Drop path (Larsson et al., 2017) | 0.0 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Mixup (Zhang et al., 2018) | 0.0 | 0.5 | 0.5 | 0.5 | 0.0 | 0.5
Multilabel | n/a | True | True | False | False | False
Loss Function | MSE | BCE | BCE | BCE | CE | BCE
Dataset Mean for Normalization | -4.268 | -4.268 | -4.268 | -5.189 | -6.627 | -6.702
Dataset Std for Normalization | 4.569 | 4.569 | 4.569 | 3.260 | 5.359 | 5.448
Table 10: Pre-training (PT) and Fine-tuning (FT) hyperparameters. For
augmentation, R: sampling random starting points with cyclic rolling in time;
N: adding random noise (signal-to-noise ratio (SNR): 20dB) to spectrograms.
For loss functions, BCE: binary cross entropy loss (for multi-label datasets
or when using mixup); CE: cross-entropy loss, MSE: mean square error loss. *:
We repeat and balance each class to 50% of the size of the unknown class.
$\dagger$: For ViT-S, We use a learning rate of 0.0005 on AS-2M FT and 0.002
on AS-20K FT as we find larger learning rates work better for ViT-S encoder in
the ablation study.

##  Appendix B Experimental Details & Hyperparameters

In this section, we provide additional experimental details for data
preprocessing, implementation, pre-training, fine-tuning, and inference. The
hyperparameters are summarized in Table 10. The code and pre-trained models to
reproduce the results will be available soon.

#### Data Preprocessing.

We downloaded and preprocessed 2.01 million AudioSet videos (with both video
and audio track) and 198K VGGSound videos. We resize the video track with 360p
(keep the aspect ratio and longer dimension is 260 and re-sample audio into
16,000 sampling rate. We use different temporal footprints for audio and
video.

For the audio part, following (Nagrani et al., 2021; Gong et al., 2021), we
transform raw waveform (pre-processed into mono-channel under 16K sampling
rate) into 128 Kaldi (Povey et al., 2011)-compatible Mel-frequency bands with
a 25ms Hanning window that shifts every 10 ms. We normalize the spectrogram
with the mean and variance calculated on the AudioSet training set. For a
10-second recording in AudioSet, the resulting (1-channel) spectrogram is of
1$\times$1024$\times$128 dimension.

For the video part, we use 4-second clips (8 frames under 2 frames per second
(fps)). The frame size is $224\times 224$ with random horizontal flip
(probability is 0.5) and multi-scale random crop (scale=(0.2, 1.0)) as data
augmentations in pre-training. We apply no augmentation in the
testing/inference phase. For a 10-second video clip in AudioSet, we randomly
sample the starting point and take the consecutive 4 seconds (cyclically loop
to the beginning if it is shorter than 4 seconds) to construct the 4-second
video clip input. The resulting (3-channel) video clip is of $8\times 3\times
224\times 224$ dimension.

#### Implementation.

We follow the main design choices of Audio-MAE (Huang et al., 2022a) and
original MAE (He et al., 2022) for images. We use separated 12-layer
Transformers (with 768-dimensions and 12 attention heads) as the encoders for
each modality. The embedding dimension $H$ is $768$ in all Transformers (ViT-B
as default).

The patch embedding layers are also separated. For each $1\times 1024\times
128$ input spectrogram of a 10-second audio, we tokenize it into non-
overlapping $16\times 16$ spectrogram patches with an audio patch embedding
layer. The kernel and stride in the audio patch embedding is 16 for both time
and frequency. There are $64\times 8$ spectrogram patches/tokens. The length
$N$ of flattened audio token sequence is $512$. Each audio token is a
$768$-dimensional vector. After appending the [CLS] token, adding positional
embedding, and applying 80% of masking, the final input audio token sequence
$\mathbf{a^{\prime}}\in\mathbb{R}^{103\times 768}$. For each $8\times 3\times
224\times 224$ video clip ($4$-second long), we tokenize it into non-
overlapped $2\times 16\times 16$cells/patches with a video patch embedding
layer. The spatial kernel and stride is $16\times 16$ and the temporal kernel
and stride is $2$, resulting in $4\times 14\times 14$ video patches/tokens.
The length $M$ of flattened video token sequence is $784$. Each video token is
a $768$-dimensional vector. After appending the [CLS] token, adding positional
embedding, and applying 80% of masking, the final input audio token sequence
$\mathbf{v^{\prime}}\in\mathbb{R}^{158\times 768}$.

We also investigated alternative design choices such as sharing audio-video
encoder weights with separated inputs or concatenated as in Multi-MAE
(Bachmann et al., 2022). However these alternative architectures delivers
worse performance compare to the proposed architecture (separated encoder with
separated inputs).

The audio-video fusion encoder layer is a two-layer ($L$=2) (vanilla or MBT
(Nagrani et al., 2021)) Transformer on top of uni-modal encoders. In the MBT
setup, we append additional 4 trainable MBT tokens for each modality after the
uni-modal encoder output. The audio and video decoders are 8-layer
Transformers with an embedding dimension of 512 and 16 attention heads. The
encoder and decoder in MAViL have 164M and 27M parameters, respectively.

The softmax temperature in NCE (Eq. 2) is set to
$\tau_{\text{c}}^{\text{inter}}=0.1$ (more tolerant) for inter-modal
contrastive learning and $\tau_{\text{c}}^{\text{intra}}=1.0$ (stricter) for
intra-modal contrastive learning. For balancing the losses in (4), we set
$\alpha=0.1$ and $\beta=0.01$ to scale the gradients from three objectives
into a comparable range to improve training stability.

#### Pre-training.

We pre-train MAViL on the union of unbalanced and balanced training sets of
AS-2M. By default, we pre-train with 64 A100 GPU with a 512 accumulated batch
size and a 0.0002 base learning rate. We pre-train for 20 epochs in each
stage. Each stage takes $\sim$20 hours. Note that the effective learning rate
($lr_{\text{eff}}$) depends on the base learning rate ($lr_{\text{base}}$) and
the batch size. Precisely, $lr_{\text{eff}}=lr_{\text{base}}*\frac{\text{batch
size}}{256}$. We also experimented using strong data augmentations (_e.g_.,
mixup (Zhang et al., 2018), SpecAug (Zhang et al., 2018), and CutMix (Yun et
al., 2019)) for audio and video pre-training but found the resulting
performance similar or worse. Therefore we discard these strong data
augmentations for audio and video in the pre-training phase by default.

Besides the proposed two-stage training framework, we also experimented with a
one-stage variant that exploits the exponential move average (EMA) of MAViL
(with raw input as the targets) as the teacher (similar to data2vec (Baevski
et al., 2022)) to generate contextualized targets to be reconstructed by a
separated decoder head in each modality. This setup results in less stable
training behavior in the audio-video scenario of our concern. This is likely
due to asynchronous convergence behavior in each modality, which is also
observed in (Wang et al., 2020). The final one-stage performance with EMA is
worse than the proposed two-stage training.

#### Fine-tuning.

For fine-tuning on audio-video (_e.g_., AudioSet-20K, AudioSet-20M, VGGSound)
and audio/speech-only (_e.g_. ESC, SPC) classification tasks, we follow the
setup in MAE to only keep the pre-trained audio and video encoders (both are
ViT-B). We average pool the encoded tokens and apply a linear layer for
classification. We use the standard fine-tuning pipeline in MAE and adopt
hyperparameters and augmentation in prior audio-based and audio-video event
classification works (Gong et al., 2021; Huang et al., 2022a; Nagrani et al.,
2021). Specifically, we use mixup (Zhang et al., 2018) and balanced sampling
(Li et al., 2022) and fine-tuning masking (Huang et al., 2022a) (a 20% random
masking rate for time and frequency in audio spectrograms; 20% for space and
time in video clips). We apply random time shifts and magnitude augmentation
for audio and apply standard video augmentations such as random horizontal
flip and random multi-scale cropping for video (Feichtenhofer et al., 2022).

To perform importance sampling that balance the fine-tuning scheme on the
unbalanced AS-2M (and VGGSound), we apply a distributed weighted sampler as
prior works (Li et al., 2022; Gong et al., 2021; Chen et al., 2022; Koutini et
al., 2021). We set the probability of sampling a sample proportional to the
inverse frequency of its labels, where the label frequency is estimated over
the training set. Specifically, for a instance $i$ in a dataset $\mathcal{D}$
with a label pool $\mathbf{C}$, its sampling weight is proportional to
$\sum_{c_{i}\in\mathbf{C}}{w_{c}}$, where
$w_{c}=\frac{1000}{\sum_{i\in\mathbf{D}}{c_{i}}+\epsilon}$ and $\epsilon=0.01$
is set to avoid underflow in majority classes. In each fine-tuning epoch on
AS-2M, we sample 200K instances ($\sim$10% of AS-2M) with replacement. We
fine-tune for 100 epochs, which aggregate to $\sim$10 full epochs of AS-2M and
takes $\sim$10 hours.

In the case where the task concerns multi-label classification (_e.g_. AS-2M)
or the mixup augmentation is enabled (_e.g_., AS-20K and VGGSound), we use
binary cross-entropy loss (BCE) as the fine-tuning objective without label
smoothing (Müller et al., 2019) otherwise we use cross-entropy (CE) (_e.g_.
the experiments on ESC).

We fine-tune the model under 3 scenarios: 1) audio-only, 2) video-only, and 3)
audio+video. In audio-only and video-only scenario, we fine-tune the audio or
video encoder in the (second-stage) MAViL. We apply the hyperparameter setups
specified in Table 10. In the audio+video setup, we append a 3-layer vanilla
Transformer on top of the audio and video encoder in the (second-stage) MAViL
and fine-tune it with audio and video input. Empirically we observe a
discrepancy in convergence rate between audio and video. We circumvent this by
applying a 50% learning rate reduction for the weights of the video encoder
when performing audio+video fusion fine-tuning.

#### Inference.

After finetuning, we simply pick the last checkpoint for inference. For video
and audio+video tasks, we follow the common practice in video action
recognition (Feichtenhofer et al., 2019; Fan et al., 2021; Li et al., 2021) to
uniformly sample 10 4-second video clips over the time domain of a video; and
we feed forward the sampled 10 video clips to generate the prediction for each
of them. Note that for audio+video classification, the audio input is the same
10-sec audio recording.

We average the 10 predictions as the instance-level prediction and report the
classification performance in Table 6-7. Note that these are still single-
modal results instead of ensembling multiple models. Table 11 compares the
1-clip prediction and 10-clip prediction results (mAPs on AS-2M).

# Clips (AS-2M) |  1 |  10
---|---|---
Audio |  48.4 |  48.4
Video |  29.0 |  30.2
Audio+Video |  52.6 |  53.1
Table 11: Number of video clips in the inference time.

## References

  * Aleksic & Katsaggelos (2006) Aleksic, P. S. and Katsaggelos, A. K.  Audio-visual biometrics.  _Proceedings of the IEEE_ , 94(11):2025–2044, 2006.
  * Arandjelovic & Zisserman (2017) Arandjelovic, R. and Zisserman, A.  Look, listen and learn.  In _ICCV_ , 2017.
  * Arandjelovic & Zisserman (2018) Arandjelovic, R. and Zisserman, A.  Objects that sound.  In _ECCV_ , 2018.
  * Aytar et al. (2016) Aytar, Y., Vondrick, C., and Torralba, A.  Soundnet: Learning sound representations from unlabeled video.  In _NeurIPS_ , 2016.
  * Baade et al. (2022) Baade, A., Peng, P., and Harwath, D.  MAE-AST: Masked autoencoding audio spectrogram transformer.  In _Interspeech_ , 2022.
  * Bachmann et al. (2022) Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A.  MultiMAE: Multi-modal multi-task masked autoencoders.  _arXiv preprint arXiv:2204.01678_ , 2022.
  * Baevski et al. (2022) Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  In _ICML_ , 2022.
  * Bao et al. (2021) Bao, H., Dong, L., and Wei, F.  BEiT: BERT pre-training of image transformers.  In _ICLR_ , 2021.
  * Caron et al. (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chen et al. (2020) Chen, H., Xie, W., Vedaldi, A., and Zisserman, A.  VGGSound: A large-scale audio-visual dataset.  In _ICASSP_ , 2020.
  * Chen et al. (2022) Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S.  HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection.  In _ICASSP_ , 2022.
  * Chen & Rao (1998) Chen, T. and Rao, R. R.  Audio-visual integration in multimodal communication.  _Proceedings of the IEEE_ , 86(5):837–852, 1998\.
  * Cho & Hariharan (2019) Cho, J. H. and Hariharan, B.  On the efficacy of knowledge distillation.  In _ICCV_ , 2019.
  * Chrupała (2022) Chrupała, G.  Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques.  _Journal of Artificial Intelligence Research_ , 73:673–707, 2022.
  * Dalal & Triggs (2005) Dalal, N. and Triggs, B.  Histograms of oriented gradients for human detection.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA_ , pp. 886–893. IEEE Computer Society, 2005.  doi: 10.1109/CVPR.2005.177.  URL https://doi.org/10.1109/CVPR.2005.177.
  * Devlin et al. (2019) Devlin, J., Chang, M., Lee, K., and Toutanova, K.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * Ephrat et al. (2018) Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., and Rubinstein, M.  Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–11, 2018.
  * Fan et al. (2021) Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C.  Multiscale vision transformers.  In _ICCV_ , 2021.
  * Fayek & Kumar (2020) Fayek, H. M. and Kumar, A.  Large scale audiovisual learning of sounds with weakly labeled data.  In _IJCAI_ , 2020.
  * Feichtenhofer et al. (2019) Feichtenhofer, C., Fan, H., Malik, J., and He, K.  Slowfast networks for video recognition.  In _ICCV_ , 2019.
  * Feichtenhofer et al. (2022) Feichtenhofer, C., Fan, H., Li, Y., and He, K.  Masked autoencoders as spatiotemporal learners.  In _NeurIPS_ , 2022.
  * Gemmeke et al. (2017) Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M.  Audio set: An ontology and human-labeled dataset for audio events.  In _ICASSP_ , 2017.
  * Gong et al. (2021) Gong, Y., Chung, Y.-A., and Glass, J.  AST: Audio Spectrogram Transformer.  In _Interspeech_ , 2021.
  * Gong et al. (2022a) Gong, Y., Lai, C.-I., Chung, Y.-A., and Glass, J. R.  SSAST: Self-Supervised Audio Spectrogram Transformer.  In _AAAI_ , 2022a.
  * Gong et al. (2022b) Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D., Karlinsky, L., Kuehne, H., and Glass, J.  Contrastive audio-visual masked autoencoder.  _arXiv preprint arXiv:2210.07839_ , 2022b.
  * Grill et al. (2020) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M.  Bootstrap your own latent: A new approach to self-supervised learning.  In _NeruIPS_ , 2020.
  * Gutmann & Hyvärinen (2010) Gutmann, M. and Hyvärinen, A.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Teh, Y. W. and Titterington, D. M. (eds.), _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010_ , volume 9 of _JMLR Proceedings_ , pp. 297–304. JMLR.org, 2010.  URL http://proceedings.mlr.press/v9/gutmann10a.html.
  * Harwath et al. (2018) Harwath, D., Recasens, A., Surís, D., Chuang, G., Torralba, A., and Glass, J.  Jointly discovering visual objects and spoken words from raw sensory input.  In _ECCV_ , 2018.
  * He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , 2020.
  * He et al. (2022) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners.  In _CVPR_ , 2022.
  * Hershey et al. (2017) Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., Slaney, M., Weiss, R., and Wilson, K.  CNN architectures for large-scale audio classification.  In _ICASSP_ , 2017.  URL https://arxiv.org/abs/1609.09430.
  * Hershey et al. (2021) Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M.  The benefit of temporally-strong labels in audio event classification.  In _ICASSP_ , 2021.
  * Hinton et al. (2015) Hinton, G., Vinyals, O., and Dean, J.  Distilling the knowledge in a neural network.  _NeurIPS Deep Learning and Representation Learning Workshop_ , 2015\.
  * Huang et al. (2022a) Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., Feichtenhofer, C., et al.  Masked autoencoders that listen.  In _NeurIPS_ , 2022a.
  * Huang et al. (2022b) Huang, Z., Jin, X., Lu, C., Hou, Q., Cheng, M.-M., Fu, D., Shen, X., and Feng, J.  Contrastive masked autoencoders are stronger vision learners.  _arXiv preprint arXiv:2207.13532_ , 2022b.
  * Jaegle et al. (2021) Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J.  Perceiver: General perception with iterative attention.  In _ICML_ , 2021.
  * Kazakos et al. (2019) Kazakos, E., Nagrani, A., Zisserman, A., and Damen, D.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In _ICCV_ , 2019.
  * Kim et al. (2013) Kim, Y., Lee, H., and Provost, E. M.  Deep learning for robust feature generation in audiovisual emotion recognition.  In _ICASSP_ , 2013.
  * Kong et al. (2020) Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D.  PANNs: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894, 2020.
  * Korbar et al. (2018) Korbar, B., Tran, D., and Torresani, L.  Cooperative learning of audio and video models from self-supervised synchronization.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Koutini et al. (2021) Koutini, K., Schlüter, J., Eghbal-zadeh, H., and Widmer, G.  Efficient training of audio transformers with patchout.  _arXiv preprint arXiv:2110.05069_ , 2021.
  * Larsson et al. (2017) Larsson, G., Maire, M., and Shakhnarovich, G.  FractalNet: Ultra-deep neural networks without residuals.  In _ICLR_ , 2017.
  * Li et al. (2022) Li, J. B., Qu, S., Huang, P., and Metze, F.  AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification.  _CoRR_ , abs/2203.13448, 2022.  doi: 10.48550/arXiv.2203.13448.  URL https://doi.org/10.48550/arXiv.2203.13448.
  * Li et al. (2021) Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., and Feichtenhofer, C.  Improved multiscale vision transformers for classification and detection.  _arXiv preprint arXiv:2112.01526_ , 2021.
  * Liu et al. (2022) Liu, X., Zhou, J., Kong, T., Lin, X., and Ji, R.  Exploring target representations for masked autoencoders.  _arXiv preprint arXiv:2209.03917_ , 2022.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  SGDR: Stochastic gradient descent with warm restarts.  In _ICLR_ , 2017.
  * Loshchilov & Hutter (2019) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Ma et al. (2020) Ma, S., Zeng, Z., McDuff, D., and Song, Y.  Active contrastive learning of audio-visual video representations.  In _ICLR_ , 2020.
  * Miech et al. (2019) Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., and Sivic, J.  HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips.  In _ICCV_ , 2019.
  * Mishra & Marr (2018) Mishra, A. and Marr, D.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy.  In _ICLR_ , 2018.
  * Mohamed et al. (2022) Mohamed, A., Lee, H.-y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., Sainath, T. N., and Watanabe, S.  Self-supervised speech representation learning: A review.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1179–1210, 2022.
  * Morgado et al. (2021a) Morgado, P., Misra, I., and Vasconcelos, N.  Robust audio-visual instance discrimination.  In _CVPR_ , 2021a.
  * Morgado et al. (2021b) Morgado, P., Vasconcelos, N., and Misra, I.  Audio-visual instance discrimination with cross-modal agreement.  In _CVPR_ , 2021b.
  * Müller et al. (2019) Müller, R., Kornblith, S., and Hinton, G. E.  When does label smoothing help?  In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pp. 4696–4705, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/f1748d6b0fd9d439f71450117eba2725-Abstract.html.
  * Nagrani et al. (2021) Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C.  Attention bottlenecks for multimodal fusion.  In _NeurIPS_ , 2021.
  * Ngiam et al. (2011) Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y.  Multimodal deep learning.  In _ICML_ , 2011.
  * Owens & Efros (2018) Owens, A. and Efros, A. A.  Audio-visual scene analysis with self-supervised multisensory features.  In _ECCV_ , 2018.
  * Park et al. (2019a) Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V.  Specaugment: A simple data augmentation method for automatic speech recognition.  _ArXiv_ , abs/1904.08779, 2019a.
  * Park et al. (2019b) Park, W., Kim, D., Lu, Y., and Cho, M.  Relational knowledge distillation.  In _CVPR_ , 2019b.
  * Patrick et al. (2021a) Patrick, M., Asano, Y. M., Kuznetsova, P., Fong, R., Henriques, J. F., Zweig, G., and Vedaldi, A.  On compositions of transformations in contrastive self-supervised learning.  In _ICCV_ , 2021a.
  * Patrick et al. (2021b) Patrick, M., Huang, P., Misra, I., Metze, F., Vedaldi, A., Asano, Y. M., and Henriques, J. F.  Space-time crop & attend: Improving cross-modal video representation learning.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pp. 10540–10552. IEEE, 2021b.  doi: 10.1109/ICCV48922.2021.01039.  URL https://doi.org/10.1109/ICCV48922.2021.01039.
  * Paulus et al. (2017) Paulus, R., Xiong, C., and Socher, R.  A deep reinforced model for abstractive summarization.  _arXiv_ , abs/1705.04304, 2017.
  * Petajan (1984) Petajan, E. D.  _Automatic Lipreading to Enhance Speech Recognition (Speech Reading)_.  PhD thesis, University of Illinois at Urbana-Champaign, 1984.
  * Piczak (2015) Piczak, K. J.  ESC: Dataset for Environmental Sound Classification.  In _Proceedings of the 23rd Annual ACM Conference on Multimedia_ , pp. 1015–1018. ACM Press, 2015.  ISBN 978-1-4503-3459-4.  doi: 10.1145/2733373.2806390.  URL http://dl.acm.org/citation.cfm?doid=2733373.2806390.
  * Potamianos et al. (2003) Potamianos, G., Neti, C., Gravier, G., Garg, A., and Senior, A. W.  Recent advances in the automatic recognition of audiovisual speech.  _Proceedings of the IEEE_ , 91(9):1306–1326, 2003\.
  * Povey et al. (2011) Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., et al.  The kaldi speech recognition toolkit.  In _IEEE 2011 workshop on automatic speech recognition and understanding_ , number CONF. IEEE Signal Processing Society, 2011.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramachandram & Taylor (2017) Ramachandram, D. and Taylor, G. W.  Deep multimodal learning: A survey on recent advances and trends.  _IEEE signal processing magazine_ , 34(6):96–108, 2017.
  * Ramesh et al. (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I.  Zero-shot text-to-image generation.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8821–8831. PMLR, 2021.  URL http://proceedings.mlr.press/v139/ramesh21a.html.
  * Recasens et al. (2021) Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Pătrăucean, V., Altché, F., Valko, M., et al.  Broaden your views for self-supervised video learning.  In _ICCV_ , 2021.
  * Romero et al. (2015) Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y.  FitNets: Hints for thin deep nets.  _ICLR_ , 2015.
  * Rouditchenko et al. (2021) Rouditchenko, A., Boggust, A., Harwath, D., Chen, B., Joshi, D., Thomas, S., Audhkhasi, K., Kuehne, H., Panda, R., Feris, R., et al.  AVLnet: Learning audio-visual language representations from instructional videos.  In _Interspeech_ , 2021.
  * Roy (1999) Roy, D.  Learning from sights and sounds: a computational model.  _PhD Thesis, MIT Media Laboratory_ , 1999.
  * Shi et al. (2022) Shi, B., Hsu, W.-N., Lakhotia, K., and Mohamed, A.  Learning audio-visual speech representation by masked multimodal cluster prediction.  In _ICLR_ , 2022.
  * Tang et al. (2022) Tang, Z., Cho, J., Nie, Y., and Bansal, M.  TVLT: Textless Vision-Language Transformer.  In _NeurIPS_ , 2022.
  * Tian et al. (2019) Tian, Y., Krishnan, D., and Isola, P.  Contrastive representation distillation.  In _ICLR_ , 2019.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.  URL http://arxiv.org/abs/1807.03748.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _NeurIPS_ , 2017.
  * Wang et al. (2021) Wang, L., Luc, P., Recasens, A., Alayrac, J.-B., and Oord, A. v. d.  Multimodal self-supervised learning of general audio representations.  _arXiv preprint arXiv:2104.12807_ , 2021.
  * Wang et al. (2020) Wang, W., Tran, D., and Feiszli, M.  What makes training multi-modal classification networks hard?  In _CVPR_ , 2020.
  * Warden (2018) Warden, P.  Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.  _ArXiv e-prints_ , April 2018.  URL https://arxiv.org/abs/1804.03209.
  * Wei et al. (2022a) Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., and Feichtenhofer, C.  Masked feature prediction for self-supervised visual pre-training.  In _CVPR_ , 2022a.
  * Wei et al. (2022b) Wei, L., Xie, L., Zhou, W., Li, H., and Tian, Q.  MVP: Multimodality-guided visual pre-training.  In _ECCV_ , 2022b.
  * Wei et al. (2022c) Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D., and Guo, B.  Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.  _arXiv preprint arXiv:2205.14141_ , 2022c.
  * Xiao et al. (2020) Xiao, F., Lee, Y. J., Grauman, K., Malik, J., and Feichtenhofer, C.  Audiovisual slowfast networks for video recognition.  _arXiv preprint arXiv:2001.08740_ , 2020.
  * Xu et al. (2016) Xu, J., Mei, T., Yao, T., and Rui, Y.  MSR-VTT: A large video description dataset for bridging video and language.  In _CVPR_ , 2016.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _ICCV_ , 2019.
  * Zeng et al. (2021) Zeng, Z., McDuff, D., Song, Y., et al.  Contrastive learning of global and local video representations.  In _NeurIPS_ , 2021.
  * Zhang et al. (2018) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  In _ICLR_ , 2018.
  * Zhou et al. (2018) Zhou, L., Xu, C., and Corso, J. J.  Towards automatic learning of procedures from web instructional videos.  In _AAAI_ , 2018.