# Exploring the Role of Audio in Video Captioning

Yuhan Shen† , Linjie Yang‡, Longyin Wen‡, Haichao Yu‡, Ehsan Elhamifar†, Heng
Wang‡
† Northeastern University ‡ ByteDance
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done during
YS’s internship at ByteDance.

###### Abstract

Recent focus in video captioning has been on designing architectures that can
consume both video and text modalities, and using large-scale video datasets
with text transcripts for pre-training, such as HowTo100M. Though these
approaches have achieved significant improvement, the audio modality is often
ignored in video captioning. In this work, we present an audio-visual
framework, which aims to fully exploit the potential of the audio modality for
captioning. Instead of relying on text transcripts extracted via automatic
speech recognition (ASR), we argue that learning with raw audio signals can be
more beneficial, as audio has additional information including acoustic
events, speaker identity, _etc_. Our contributions are twofold. First, we
observed that the model overspecializes to the audio modality when pre-
training with both video and audio modality, since the ground truth (_i.e_.,
text transcripts) can be solely predicted using audio. We proposed a Modality
Balanced Pre-training (MBP) loss to mitigate this issue and significantly
improve the performance on downstream tasks. Second, we slice and dice
different design choices of the cross-modal module, which may become an
information bottleneck and generate inferior results. We proposed new local-
global fusion mechanisms to improve information exchange across audio and
video. We demonstrate significant improvements by leveraging the audio
modality on four datasets, and even outperform the state of the art on some
metrics without relying on the text modality as the input.

##  1 Introduction

Large-scale pre-training [3, 20, 33, 43, 50, 59, 66] plays a key role in
boosting the performance of modern deep learning models. It is even more so
for vision and language tasks, such as video captioning[1, 12, 36, 41, 58,
65], where leveraging large video datasets with text supervision for pre-
training is essential to achieve competitive results. However, manually
annotating captions for video datasets is costly and not scalable. Thus
existing video captioning datasets [49, 61, 63, 69] are often limited in size.

To address this challenge, recent work collected datasets from instructional
videos, where ASR generated transcripts can be used as text supervision,
_e.g_., How2 [46], CrossTask [72], HowTo100M [34], HD-VILA-100M [64], _etc_.
This has established a new trend of pre-training on large-scale video datasets
with text transcripts for video captioning [16, 32, 47]. We argue that text
transcripts from ASR only includes partial information from audio, and
hypothesize end-to-end learning using the audio modality can potentially lead
to better performance, since audio can provide additional information (shown
in Fig. 2) including acoustic events, speaker identity, _etc_.

More specifically, our paper seeks to better understand the following
questions:

  * •

_To what extent, can the audio modality improve video captioning?_

  * •

_How can the potential of the audio modality be fully realized in an audio-
visual framework for captioning?_

To this end, we start with a simple multi-modal pre-training framework for
video captioning with ASR transcripts as supervision (shown in Fig. 2), and
look into different components that may hinder the performance of the pre-
trained audio-visual model on the downstream datasets.

First, we observed that simply jointly training of the audio and video
modalities may result in degenerated models that overspecialize to the audio
modality and underfit on the video modality. As text transcripts are used as
video captions during pre-training, the model essentially learns to cheat and
solve the ASR problem instead of extracting information from both visual and
audio signals. To mitigate this issue, we proposed the Modality Balanced Pre-
training (MBP) loss, which takes into account both the unimodal losses and
cross-modal loss. A weighting mechanism is introduced to automatically balance
different modalities during training. Fig. 4 shows that our MBP loss enforces
the model to pay more attention to the underfitted video modality and drives
the final loss much smaller.

Figure 1: Audio provides critical complementary information in multi-modal
video captioning. We show two examples of generated captions when we input 1)
only video, 2) video and ASR text, and 3) video and audio. Audio can provide
additional information that cannot be obtained from visual modality or ASR
text, _e.g_., sound of crying, laughter, and speaker gender.222 YouTube ID of
the two videos: a50yimv7Lqg and BFYBAg9fSL4.

Second, we did a thorough study on the design of the cross-modal fusion
module, which is responsible for the information exchange between the audio
and video modality. An improperly designed cross-modal fusion module may
become an information bottleneck and result in inferior performance for video
captioning. We proposed new local-global fusion mechanisms to encourage the
information flow cross different modalities. We analyzed the relevance of the
annotated captions to the audio modality on downstream datasets, and observed
that the local fusion mechanisms are more beneficial to the flow of fine-
grained information like single words in speech, while the global fusion
mechanisms are more effective on holistic information like acoustic events or
scenes. The local-global design enables the fusion module to capture
information at different granularities, and mingle audio and video information
at different levels. Compared with existing designs, our local-global fusion
has shown empirically better results.

By combining the two contributions, we demonstrate that audio is crucial to
video captioning and provides both speech and non-speech information. Fig. 2
shows a few examples on how our model effectively integrates the information
from both the audio and video modality, and generates better captions than
video-only and video-text variants.

We summarize our contributions as follows:

  * •

Proposed to pre-train video captioning models based on video and audio
modalities, explored the role of audio in video captioning, and demonstrated
the benefits of audio on four benchmarks.

  * •

Proposed the MBP loss to balance different modalities automatically during
training, and ease the issue of overspecialization to the audio modality.

  * •

Did an extensive evaluation on the effects of different cross-modal fusion
modules on audio-visual video captioning, and proposed a novel local-global
fusion module to effectively integrate audio and video information for video
captioning.

Figure 2: Overview of our audio-visual video captioning framework. We design
two tasks for caption generation during pre-training: Predict Current Caption
(PCC) and Predict Next Caption (PNC). For downstream fine-tuning, we only
adopt PCC because the goal is to predict current caption given the input
frames and audio.

##  2 Related Work

Video Captioning. Most works in video captioning [1, 12, 36, 41, 58, 65] focus
on designing a better model to generate text descriptions given precomputed
video features via an encoder-decoder framework. SwinBert [29] attempted to
train the encoder-decoder framework directly from raw video pixels instead of
refining precomputed features. In addition to visual modality, some works
studied video captioning from visual data and ASR texts [15, 32, 47, 48, 49,
53]. A few prior works also studied audio-visual video captioning [7, 17, 45,
54], but they are often limited to small-scale video captioning datasets and
precomputed input features. To the best of our knowledge, we propose the first
end-to-end audio-visual video captioning framework.

Multi-Modal Pre-training. A growing number of works are investigating multi-
modal pre-training in videos, _e.g_., video-text pre-training [24, 32, 33, 34,
40, 50, 67] and video-text-audio pre-training [3, 4, 66], which mostly adopt
contrastive learning and/or masked language modeling to learn better
representations for downstream tasks. As only encoders are trained for
multiple modalities, a separate decoder needs to be trained on top of the
encoders for generative tasks such as video captioning. A recent work MV-GPT
[47] shows the benefits of pre-training an end-to-end encoder-decoder
framework to video captioning. Unlike MV-GPT that relies on ASR text as input,
our framework directly uses video and audio. A Textless Vision-Language
Transformer (TVLT) [52] was recently proposed to take visual and audio inputs
for multi-modal representation learning without ASR inputs. However, the pre-
trained TVLT is a discriminative model that cannot be directly applied to
generative tasks. While a multi-modal network receives more information and is
expected to boost performance, recent works [18, 37, 42, 60] have identified a
key challenge in training a multi-modal network that one modality may converge
faster than other modalities and undermine the representation learning of
other modalities. We propose a Modality Balanced Pre-training objective to
mitigate this issue and facilitate a powerful audio-visual video captioning
model.

Cross-Modal Fusion. Given the representations of multiple modalities, a cross-
modal fusion module [19, 33, 34, 43] will fuse these representations into a
shared embedding space to generate cross-modal representations. In order to
fuse a sequence of representations generated by Transformers [56], there are
two major types of cross-modal fusion modules: merged fusion, and cross
fusion. In merged fusion, the two modalities are concatenated and then fed
into a Transformer block [25, 27, 32, 59]. In cross fusion, the two modalities
are fed into different Transformer blocks separately, where cross attention
Transformers are used to allow cross-modal interaction [31, 47, 48, 51, 55].
Besides, some recent works propose variants of cross-modal fusion modules that
use bottleneck tokens [35] or prune single-modal units [62] to control the
flow of cross-modal interaction.

##  3 Audio-Visual Video Captioning

In this section, we propose our methods for audio-visual video captioning. In
Sec. 3.1, we present an overview of our framework. In Sec. 3.2, we describe
our MBP loss to balance different modalities during pre-training. In Sec. 3.3,
we investigate different cross-modal fusion modules and propose a local-global
fusion module to improve information flow between audio and video at different
granularities.

###  3.1 Framework Overview

As shown in Fig. 2, we follow the common practice in video captioning [32, 47]
and use an encoder-decoder framework including four main modules: a video
encoder, an audio encoder, a cross-modal encoder, and a caption decoder, all
of which are Transformer architectures [56].

Given a video, the video encoder extracts a sequence of video embeddings
${\varphi}^{v}\in\mathbb{R}^{N_{v}\times D}$ from the raw image frames, and
the audio encoder extracts a sequence of audio embeddings
${\varphi}^{a}\in\mathbb{R}^{N_{a}\times D}$ from the associated audio
spectrogram, where $N_{v}$ and $N_{a}$ are the numbers of video tokens and
audio tokens, and $D$ is feature dimension. Then, we employ a cross-modal
encoder $f_{\Theta}(\cdot)$ to generate multi-modal embeddings
${\varphi}^{c}\in\mathbb{R}^{(N_{a}+N_{v})\times D}$ for cross-modal
interaction. Finally, we use a decoder $g_{\Theta^{\prime}}(\cdot)$
conditioned on ${\varphi}^{c}$ to output the captions auto-regressively. By
default, we use Video Swin Transformer [30] as the video encoder, and Audio
Spectrogram Transformer [14] as the audio encoder.

Inspired by [47], we design two tasks for caption generation during pre-
training: Predict Current Caption (PCC) and Predict Next Caption (PNC). Given
the audio-visual embeddings as the context of the caption decoder, we feed two
distinct BOS (Beginning of Sentence) tokens to initiate caption generation,
namely BOS1 and BOS2, which initiate the prediction of the current and next
caption respectively. The PNC task enforces the model to anticipate future
events, which is more challenging and requires higher level of semantic
understanding. Note that this multi-task training is only used for pre-
training. For downstream fine-tuning, we only feed BOS1 token to the decoder
as the goal is to predict the current caption.

###  3.2 Modality Balanced Pre-training

With captions as supervision, a commonly used objective is to minimize the
negative log-likelihood of generating the ground-truth caption:

$L=\mathcal{L}(g_{\Theta^{\prime}}(f_{\Theta}({\varphi}^{a},{\varphi}^{v})),y),$
(1)

where $\mathcal{L}$ is the cross entropy loss and $y$ is the ground-truth
caption. We refer to this loss as audio-video decoder loss, as both audio and
video features are input into the decoder.

Prior works [3, 33, 47, 59, 66, 67] have proved that large-scale pre-training
is essential to multi-modal learning. Thus, we pre-train our audio-visual
video captioning model on a large-scale video dataset before fine-tuning it on
the downstream captioning datasets. Although there are some large-scale video
datasets for video-language pre-training (_e.g_., HowTo100M [34] and HD-
VILA-100M [64]), they do not contain manually annotated captions and use ASR
transcripts as text supervision. One significant issue based on our
experiments (see Fig. 4) is that the model tends to learn only from the audio
and ignore the video modality, since the ASR caption can be solely derived
from human speech in the audio. Although we include two tasks for current and
future caption predictions, the model still favors the audio features when
predicting the current caption.

To address this issue, we propose the MBP loss to balance different modalities
for multi-modal training. In essence, we add losses for audio-only and video-
only predictions to help improve the representation learning of the two
modalities. To measure how well the model exploits a certain modality, we
first define the following two losses, _i.e_., audio-only decoder loss and
video-only decoder loss:

$\displaystyle L_{a}$
$\displaystyle=\mathcal{L}(g_{\Theta^{\prime}}(f_{\Theta}({\varphi}^{a},\mathbf{0})),y),$
(2) $\displaystyle L_{v}$
$\displaystyle=\mathcal{L}(g_{\Theta^{\prime}}(f_{\Theta}(\mathbf{0},{\varphi}^{v})),y),$

where we set video embeddings to all-zeros to get the audio-only decoder loss
$L_{a}$, and set audio embeddings to all-zeros to get the video-only decoder
loss $L_{v}$ based on Eq. (1). We refer to $L_{a}$ or $L_{v}$ as mono-modal
losses. If a mono-modal loss is small, it means that the corresponding
modality is well utilized by the model. We then measure the gap between the
multi-modal loss and the mono-modal losses by a Mono-to-Multi Discrepancy
(MMD) index:

$G_{a}=(L_{a}-L)^{2};G_{v}=(L_{v}-L)^{2},$ (3)

where $G_{a}$ and $G_{v}$ measure the discrepancy between audio-only/video-
only decoder loss and audio-video decoder loss, respectively. Inspired by
G-Blend [60] that uses Overfitting-to-Generalization Ratio (OGR) to
iteratively update training weights for different modalities, we guide the
weights of mono-modal losses based on whether the modality is well utilized by
the model:

$L_{pretrain}=L+w_{a}L_{a}+w_{v}L_{v},$ (4)

where $w_{a}$ and $w_{v}$ are weights updated over iterations:

$w_{m}^{(t)}=\beta w_{m}^{(t-1)}+(1-\beta)\tilde{w}_{m}^{(t)},m\in\\{a,v\\},$
(5)

where $\beta\in(0,1)$ is a smoothing hyperparameter, $t$ is the iteration
number, and $\tilde{w}_{m}^{(t)}$ is obtained using a softmax function over
the gap between each mono-modal loss and the multi-modal loss of the current
iteration:

$\tilde{w}_{m}^{(t)}=\frac{\exp{(\alpha
G_{m}^{(t)})}}{\sum_{m^{\prime}}\exp{(\alpha
G_{m^{\prime}}^{(t)})}},m\in\\{a,v\\},$ (6)

where $\alpha>0$ is a temperature hyperparameter. If $G_{m}$ is large for a
certain modality on the pre-training dataset, we will assign a higher weight
$w_{m}$ for modality $m$ in Eq. (4). By doing this, we enforce the model to
give more attention to modality $m$ and mitigate its overspecialization to the
other modality. When the optimization progresses and the gaps $G_{a}$ and
$G_{v}$ change over time, we dynamically adjust the coefficients $w_{a}$ and
$w_{v}$ according to Eq. (5) to increase the weight of the underfitted
modality. Our strategy can be easily extended to more than two modalities,
which is out of the scope of this work. Empirically, we found the MBP loss not
sensitive to the two hyperparameters $\beta$ and $\alpha$. We set $\beta=0.99$
and $\alpha=10$ for all the experiments.

(a) Merged Fusion

(b) Cross Fusion

(c) Global Cross Fusion

Figure 3: Three different cross-modal fusion designs. Superscripts $a,v$
denote audio and video modalities. $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$
represent query, key, and value in multi-head attention models. For clarity,
we use dot lines to denote the flow of global tokens in (c).

###  3.3 Cross-Modal Fusion: Beyond Cross Attention

In addition to the MBP objective, the cross-modal encoder is another component
where different modalities interact with each other. In this section, we
explore different design choices for the cross-modal fusion module to better
leverage the audio modality for video captioning.

####  3.3.1 Background

We first briefly introduce some relevant concepts of cross-modal fusion
modules. Given $N_{q}$ query vectors of dimension $d$,
$\mathbf{Q}\in\mathbb{R}^{N_{q}\times d}$, and $N_{v}$ key-value pairs,
$\mathbf{K}\in\mathbb{R}^{N_{v}\times d}$ and
$\mathbf{V}\in\mathbb{R}^{N_{v}\times d}$, an attention function maps queries
to output vectors with a scaled dot product:

$\vspace{-2mm}\operatorname{Att}(\mathbf{Q},\mathbf{K},\mathbf{V})=\operatorname{Softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V}.$
(7)

Multi-Head Attention (MHA) is a module with multiple heads of attention
functions, and a Transformer layer consists of an MHA and a Feed-Forward Block
(FFB). The keys and values are usually the same features in the Transformer
layer. For simplicity, we denote a Transformer layer by:

$\operatorname{Transformer}(\mathbf{X},\mathbf{Y})=\operatorname{FFB}(\operatorname{MHA}(\mathbf{X},\mathbf{Y},\mathbf{Y})).\vspace{-2mm}$
(8)

####  3.3.2 Cross-Modal Fusion

As mentioned in Sec. 3.1, we get a sequence of audio tokens
${\varphi}^{a}\in\mathbb{R}^{N_{a}\times D}$ from the audio encoder, and a
sequence of video tokens ${\varphi}^{v}\in\mathbb{R}^{N_{v}\times D}$ from the
video encoder. Then we feed audio and video tokens into a cross-modal encoder
for cross-modal fusion. Below we introduce two popular fusion methods: merged
fusion and cross fusion.

In merged fusion (Fig. 3(a)), the tokens of two modalities are concatenated
and then passed into Transformer blocks:

$\varphi^{a}_{i+1},\varphi^{v}_{i+1}=\operatorname{Transformer}_{i}([\varphi^{a}_{i};\varphi^{v}_{i}],[\varphi^{a}_{i};\varphi^{v}_{i}]).$
(9)

Audio tokens $\varphi^{a}_{i}$ and video tokens $\varphi^{v}_{i}$ are the
inputs of the $i$-th Transformer layer. Each audio token can attend to all
audio and video tokens, and it is the same for video tokens.

In cross fusion (Fig. 3(b)), each modality has its own Transformer layers, and
different modalities exchange information via cross attention, _i.e_., one
modality is used as the context (keys and values) of the other modality:

$\displaystyle\varphi^{a}_{i+1}=\operatorname{Transformer}_{i}^{a}(\varphi^{a}_{i},\varphi^{v}_{i}),$
(10)
$\displaystyle\varphi^{v}_{i+1}=\operatorname{Transformer}_{i}^{v}(\varphi^{v}_{i},\varphi^{a}_{i}).$

Each layer has two modality-specific Transformers.

####  3.3.3 Global Cross Fusion

We use the term “local fusion” to denote the two fusion modules introduced in
Sec. 3.3.2 as the interaction is among local tokens, regardless of intra-
modality or inter-modality. However, only using local interaction can be sub-
optimal, as local tokens are often noisy and may be less informative for
cross-modal interaction. To select more salient features for cross-modal
fusion, we introduce global tokens $G^{a}$ and $G^{v}$ for each modality as
shown in Fig. 3(c). Global tokens serve as a global representation of the
audio clip or the video clip. In global cross fusion, instead of using all
video tokens as the context for audio tokens, we use the global video token as
the context:

$\displaystyle\varphi^{a}_{i+1},G^{a}_{i+1}=\operatorname{Transformer}_{i}^{a}([\varphi^{a}_{i};G^{a}_{i}],[\varphi^{a}_{i};G^{v}_{i}]),$
(11)
$\displaystyle\varphi^{v}_{i+1},G^{v}_{i+1}=\operatorname{Transformer}_{i}^{v}([\varphi^{v}_{i};G^{v}_{i}],[\varphi^{v}_{i};G^{a}_{i}]).$

Here, we restrict all cross-modal attention flow to be via these global
tokens, and local tokens will only be used for intra-modal attention. The
global token of the first cross layer is learnable and is initialized with a
Gaussian distribution, the same as the CLS token in BERT [20].

####  3.3.4 Local-Global Fusion

Local and global tokens capture information in different granularities. Local
audio and video tokens capture local features such as words in the speech or
objects in a video frame, while global audio and video tokens capture high-
level concepts like sounds of laughter or people gathering on a street. To
take advantage of both local and global information, we propose to combine
local fusion and global cross fusion. Let $\varphi^{a(G)}_{i+1}$ and
$\varphi^{v(G)}_{i+1}$ denote the embeddings from global cross fusion in Eq.
(11), and $\varphi^{a(L)}_{i+1}$ and $\varphi^{v(L)}_{i+1}$ denote the
embeddings from local fusion in Eq. (9) or Eq. (10), we compute an average of
these two embeddings before feeding them into the next fusion layer, _i.e_.,

$\varphi^{m}_{i+1}=(\varphi^{m(G)}_{i+1}+\varphi^{m(L)}_{i+1})/2,\;m\in\\{a,v\\}.$
(12)

This unified fusion module is named as “local-global fusion”, and is able to
progressively refine the tokens using both local and global guidance. We name
this variant with merged fusion as “local-global merged fusion”, and the
variant with cross fusion as “local-global cross fusion”.

##  4 Experiments

###  4.1 Pre-training and Downstream Datasets

We use the HowTo100M dataset [34] for pre-training, and evaluate the proposed
method on four video captioning datasets, including YouCook2 [69], MSRVTT
[63], VATEX [61] and ActivityNet-Captions [49]. We use four metrics, _i.e_.,
BLEU-4 (B) [38], METEOR (M) [5], ROUGE-L (R) [28], and CIDEr (C) [57], for
evaluation.

HowTo100M [34] consists of $1.2M$ instructional videos from YouTube. We
download videos with ASR transcripts and audio from YouTube, and remove
currently unavailable videos, resulting in $1.08M$ videos in total. Following
[32], we start with a single ASR sentence and iteratively expand the length of
video clip forward or backward by adding next or previous sentences until the
clip is longer than 5 seconds.

YouCook2 [69] contains $2,000$ cooking videos with $15.4k$ video clips. Each
video clip is annotated with a single sentence. MSRVTT [63] contains $10k$
open-domain video clips for video captioning. Each video clip is annotated
with $20$ captions. VATEX [61] consists of $41,250$ videos. Each video is
annotated with $10$ English captions and $10$ Chinese captions, and we use the
English captions. ActivityNet-Captions [23] is a large-scale video paragraph
captioning dataset consisting of $100k$ temporally localized captions for
$20k$ long videos. To be consistent with the other datasets, we train our
model on sentence-level captions. Following [70], we compose paragraph-level
captions by simply concatenating sentence-level captions and evaluate the
captioning performance at paragraph-level in testing. Please refer to the
appendix for more details.

Audio Relevance Analysis. The performance of video captioning on downstream
datasets is also related to the language styles of the annotations, so we
design two metrics, Speech Coverage Rate (SCR) and Audio Relevance Score
(ARS), to analyze how related the annotated captions are to audio. Concisely,
SCR measures the percentage of words in ground-truth captions that also occur
in speech transcripts; ARS measures the relevance of the captions to audio.
ARS is empirically calculated based on the word frequencies on audio
captioning [11, 21] and image captioning [9] datasets. If the frequency of a
word is much higher in audio captions than that in image captions, we assume
that this word is highly related to audio modality and assign a high score to
a sentence with the word. We include more details in the appendix. Examples of
the words with a high ARS are “loudly”, “snore”, “footstep”, _etc_. Those
words are usually related to acoustic events, scenes, or sound patterns, and
require more holistic or long-term information from the audio for the model to
understand the context. Tab. 1 lists the SCR and ARS on downstream datasets.
We notice that a large portion of the captions on YouCook2 are mentioned in
speech. Conversely, though only a small amount of captions are covered by
speech, the captions on VATEX are most relevant to audio modality.

| YouCook2 | MSRVTT | VATEX | ActivityNet
---|---|---|---|---
SCA | 48.67 | 14.7 | 5.53 | 12.63
ARS | 1.202 | 2.903 | 3.692 | 3.487

Table 1: The Speech Coverage Rate (SCA, %) and Audio Relevance Score (ARS) on
the downstream datasets.

MBP | PCC | PNC | YouCook2 | MSRVTT | VATEX | ActivityNet-Captions
---|---|---|---|---|---|---
B | M | R | C | B | M | R | C | B | M | R | C | B | M | R | C
G-Blend [60] | ✓ | ✓ | 19.4 | 23.4 | 48.6 | 208.5 | 44.0 | 29.6 | 62.4 | 55.1 | 38.0 | 24.9 | 52.1 | 68.7 | 11.6 | 16.1 | 30.6 | 25.3
| ✓ |  | 15.9 | 20.2 | 43.0 | 166.8 | 41.4 | 28.5 | 60.7 | 47.6 | 31.2 | 21.9 | 47.7 | 50.7 | 10.0 | 14.6 | 28.4 | 20.1
✓ | ✓ |  | 18.5 | 22.7 | 47.0 | 192.4 | 44.7 | 29.9 | 62.7 | 53.5 | 36.9 | 24.7 | 51.7 | 67.5 | 11.3 | 15.5 | 30.2 | 24.7
| ✓ | ✓ | 17.2 | 21.7 | 45.8 | 184.2 | 42.0 | 28.4 | 60.8 | 48.4 | 31.2 | 22.0 | 48.0 | 51.4 | 10.4 | 14.9 | 28.9 | 20.2
✓ | ✓ | ✓ | 20.6 | 24.2 | 49.6 | 217.0 | 46.0 | 30.6 | 64.0 | 57.0 | 38.8 | 25.9 | 52.9 | 73.5 | 11.7 | 16.1 | 30.7 | 26.1

Table 2:  Ablation studies on multi-modal pre-training with our audio-visual
captioning framework from Fig. 2. MBP: Modality Balanced Pre-training; PCC:
Predict Current Caption; PNC: Predict Next Caption.

Fusion | YouCook2 | VATEX
---|---|---
B | M | R | C | B | M | R | C
cross | 19.5 | 23.4 | 48.9 | 211.2 | 37.8 | 24.7 | 52.0 | 68.9
merged | 19.9 | 23.6 | 49.1 | 210.7 | 38.4 | 25.1 | 52.3 | 71.1
global | 18.6 | 22.9 | 48.0 | 202.3 | 39.2 | 25.5 | 52.8 | 72.6
local-global cross | 19.9 | 23.9 | 49.2 | 213.9 | 38.6 | 25.8 | 53.0 | 73.4
local-global merged | 20.6 | 24.2 | 49.6 | 217.0 | 38.8 | 25.9 | 52.9 | 73.5

Table 3: Ablation studies on cross-modal fusion modules.

###  4.2 Experimental Setup

Video Encoder: We sample 16 frames from each video clip. The frames are
resized and cropped into images of size $224\times 224$. Each video clip with
the size of $16\times 224\times 224\times 3$ is fed into the Video Swin
Transformer [30] initialized with the weights pre-trained on Kinetics 600 [6]
and tokenized into $N_{v}=8\times 7\times 7=392$ video tokens. Then we add a
linear layer to project the dimension of each video token to $D=768$, to be
consistent with the other modules.

Audio Encoder: We first extract log mel spectrogram of the raw audio.
Following [66], each audio is resampled to 22,050Hz and divided into frames of
1536 samples with hop length of $588$. Then we apply 64 mel-scale filters. We
use a 12-layer Transformer on audio spectrogram to output $N_{a}=64$ audio
tokens with feature dimension of 768.

Cross Encoder: We use a 3-layer Transformer as the cross-modal encoder. In our
experiments, we comprehensively compare the results of the cross-modal fusion
methods introduced in Sec. 3.3. Overall, merged fusion outperforms cross
fusion, and local-global merged fusion achieves the best performance. For
other experiments, we use local-global merged fusion unless explicitly stated
otherwise.

Caption Decoder: We use a 3-layer Transformer as the caption decoder. In
training, we use causal masking to ensure that only history inputs are used.
In testing, we use beam-search with beam width of 5 for caption generation.

Training Details: We pre-train the model on HowTo100M for 100 epochs using
Adam optimizer [22]. For each video, we sample three video-caption pairs from
a long video in one iteration. We pre-train the model on 64 Nvidia Tesla A100
GPUs and it takes around five days. The base learning rate is $10^{-4}$ and we
use a linear decay learning rate schedule with a warm-up of $10\%$ training
epochs as in [32]. For fine-tuning, we set the initial learning rate as
$10^{-5}$. We train the model for 5 epochs on YouCook2 and MSRVTT, 10 epochs
on VATEX, and 30 epochs on ActivityNet-Captions. It is noted that we employ
modality balancing during pre-training but not during downstream fine-tuning,
to allow the model to adapt to the more informative modality for caption
generation. We have included more details on the model architecture and
implementation in the appendix.

Figure 4: The pre-training losses without and with MBP. (Solid lines: losses
without MBP. Dotted lines: losses with MBP.)

###  4.3 Results and Discussions

Multi-modal pre-training objectives. We evaluate the effects of our pre-
training objectives in Tab. 2. Our proposed MBP improves the performance by a
large margin on four datasets. The addition of Predicting Next Caption leads
to a remarkable boost as well. We also compare MBP with G-Blend [60]: G-Blend
aims to reduce overfitting, which happens when the model performs well on the
training set but fails to generalize to new data, whereas MBP aims to reduce
overspecialization, which happens when the model performs well on a single
modality but fails to learn from the other modalities. Hence, G-Blend updates
mono-modal weights by computing Overfitting-to-Generation Ratio while MBP uses
Mono-to-Multi Discrepancy. According to the results in Tab. 2, G-Blend also
improves the performance, but MBP consistently outperforms G-Blend, showing
that MBP is more suitable to avoid the issue of overspecialization in audio-
visual captioning pre-training. In Fig. 4, we show the trend of training
losses with and without MBP, including multi-modal loss (audio-video loss) and
mono-modal losses (audio-only loss and video-only loss). Note that for the
experiments without MBP, audio-only loss and video-only loss are only computed
for analysis and not back-propagated. Although the two audio-video losses have
similar scales, both audio-only loss and video-only loss are reduced with MBP,
especially for video-only loss. We conjecture that the model reduces
overspecialization to the audio modality and learns to better utilize the
video modality with MBP. Though there is only a small decrease in the audio-
video training loss, the performance on downstream tasks are significantly
improved (see Tab. 2), which demonstrates the effectiveness of our pre-
training objective.

Inputs | YouCook2 | MSRVTT | VATEX | ActivityNet
---|---|---|---|---
B | M | R | C | B | M | R | C | B | M | R | C | B | M | R | C
A | 13.4 | 17.6 | 38.5 | 138.9 | 32.3 | 23.9 | 54.3 | 24.8 | 15.4 | 15.6 | 37.6 | 14.2 | 6.3 | 11.2 | 24.5 | 6.6
V | 11.7 | 18.0 | 41.5 | 139.2 | 42.6 | 28.5 | 61.0 | 52.0 | 36.4 | 24.6 | 51.2 | 67.3 | 9.7 | 14.4 | 27.8 | 21.3
V+A | 20.6 | 24.2 | 49.6 |  217.0 (77.8$\uparrow$) | 46.0 | 30.6 | 64.0 |  57.0 (5.0$\uparrow$) | 38.8 | 25.9 | 52.9 |  73.5 (6.2$\uparrow$) | 11.7 | 16.1 | 30.7 |  26.1 (4.8$\uparrow$)
V+T | 19.1 | 23.3 | 48.7 | 205.6 (66.4$\uparrow$) | 43.3 | 29.1 | 61.7 | 53.4 (1.4$\uparrow$) | 37.7 | 25.2 | 52.0 | 69.0 (1.7$\uparrow$) | 11.2 | 15.9 | 29.4 | 24.9 (3.6$\uparrow$)
V+A+T | 20.9 | 24.4 | 49.9 |  221.6 (82.4$\uparrow$) | 46.4 | 30.2 | 64.1 |  57.3 (5.3$\uparrow$) | 39.1 | 26.3 | 53.4 |  73.7 (6.4$\uparrow$) | 11.8 | 16.1 | 30.9 |  26.4 (5.1$\uparrow$)

Table 4: The performance when we input different modalities (V: video, A:
audio, T: text). We show the improvement of multi-modal video captioning over
the video-only method in terms of CIDEr in the parenthesis.

Method | Pre-training dataset | Inputs | B | M | R | C
---|---|---|---|---|---|---
GIT2 [59] † | 12.9B image-text pairs | V | 9.4 | 15.6 | 37.5 | 131.2
MART [41] | - | V | 8.0 | 15.9 | - | 36.0
SwinBert [29] | - | V | 9.0 | 15.6 | 37.3 | 109.0
VideoBert [50] | YouTube | V | 4.0 | 11.0 | 27.5 | 49.0
M-MASS [16] | YT8M-cook+Recipe1M | V+T | 12.0 | 18.3 | 39.0 | 123.0
ActBert [71] | HT100M | V | 5.4 | 13.3 | 30.6 | 65.0
Value [26] | HT100M+TV | V+T | 12.4 | 18.8 | 40.4 | 130.4
UniVL [32] | HT100M | V+T | 17.4 | 22.4 | 46.5 | 181.0
MV-GPT [47] | HT100M | V+T | 21.9∗ | 27.1∗ | 49.4 | 221.0
Ours | HT100M | V+A | 20.6 | 24.2 | 49.6 | 217.0
Ours | HT100M | V+T | 19.1 | 23.3 | 48.7 | 205.6
Ours | HT100M | V+A+T | 20.9 | 24.4 | 49.9 | 221.6

(a) YouCook2

Method | Pre-training dataset | Inputs | B | M | R | C
---|---|---|---|---|---|---
GIT2 [59] † | 12.9B image-text pairs | V | 54.8 | 33.1 | 68.2 | 75.9
Open-Book [68] | - | V | 33.9 | 23.7 | 50.2 | 52.9
SwinBert [29] | - | V | 45.4 | 30.6 | 64.1 | 55.9
MMCNN [54] | - | V+A | 42.7 | 28.5 | 61.5 | 47.2
MGSA [7] | - | V+A | 45.4 | 28.6 | - | 50.1
Decembert [53] | HT100M | V+T | 45.2 | 29.7 | 64.7 | 52.0
UniVL [32] | HT100M | V+T | 41.8 | 28.9 | 60.8 | 50.0
MV-GPT [47] | HT100M | V+T | 48.9∗ | 38.7∗ | 64.0 | 60.0
Ours | HT100M | V+A | 46.0 | 30.6 | 64.0 | 57.0
Ours | HT100M | V+T | 43.3 | 29.1 | 61.7 | 53.4
Ours | HT100M | V+A+T | 46.4 | 30.2 | 64.1 | 57.3

(b) MSRVTT

Method | Pre-training dataset | Inputs | B | M | R | C
---|---|---|---|---|---|---
GIT2 [59] † | 12.9B image-text pairs | V | 42.7 | 28.8 | 56.5 | 94.5
SwinBert [29] | - | V | 38.7 | 26.2 | 53.2 | 73.0
MGRMP [8] | - | V | 34.2 | 23.5 | 50.3 | 57.6
Value [26] | HT100M+TV | V+T | 32.9 | 24.1 | 50.1 | 58.1
Ours | HT100M | V+A | 38.8 | 25.9 | 52.9 | 73.5
Ours | HT100M | V+T | 37.7 | 25.2 | 52.0 | 69.0
Ours | HT100M | V+A+T | 39.1 | 26.3 | 53.4 | 73.7

(c) VATEX

Method | Pre-training dataset | Inputs | B | M | R | C
---|---|---|---|---|---|---
VTransformer [70] | ActivityNet | V | 9.31 | 15.54 | - | 21.33
Transformer-XL [10] | ActivityNet | V | 10.25 | 14.91 | - | 21.71
MART [41] ‡ | ActivityNet | V | 9.78 | 15.57 | 30.63 | 22.16
COOT [13] ‡ | HT100M | V | 10.85 | 15.99 | 31.45 | 28.19
Ours | HT100M | V+A | 11.73 | 16.14 | 30.68 | 26.11
Ours | HT100M | V+T | 11.22 | 15.94 | 29.40 | 24.92
Ours | HT100M | V+A+T | 11.83 | 16.14 | 30.93 | 26.38

(d) ActivityNet-Captions

Table 5: Comparison to SOTA. The top two results are in bold and italic.
$\dagger$ use a large-scale annotated image captioning dataset. $\ast$ The
evaluation library in MV-GPT [47] has discrepancy with cococaption [9] in
terms of BLEU4 and METEOR, so the scores are not directly comparable.
${\ddagger}$ use relationships between sentences to generate paragraphs.

Cross-modal fusion modules. We conduct ablation studies on cross-modal fusion
in Tab. 3. In terms of the two local fusion modules, merged fusion performs
favorably against cross fusion on both datasets. Compared with local fusion
modules, global cross fusion performs better on VATEX while it performs worse
on YouCook2. As shown in Tab. 1, the annotated captions are very related to
audio events and scenes on VATEX while the captions are largely covered by
speech on YouCook2. The results show that global cross fusion helps the flow
of holistic information, _e.g_., acoustic events, while local fusion is more
focused on fine-grained information, _e.g_., single words in speech. Moreover,
adding global cross fusion greatly improves on both local fusion modules when
comparing the results from the first two rows with the last two rows in Tab.
3.

Overall, local-global merged fusion performs best, showing the advantages of
using both local and global tokens for cross-modal fusion. Since there are
only a few global tokens, they do not bring much extra computational cost.
Compared with merged fusion, local-global merged fusion increases FLOPs by
only 6.96% (from 316G to 338G), but produces substantially better results.

Visualization of attention maps. Fig. C shows the maps of the attention from
audio modality to the RGB space for global and local fusion modules using
Attention Rollout [2]. We visualize the attention maps summed over all the
frames in the video clip as in [35]. We note that global fusion focuses on
salient regions related to acoustic events (piano), while local fusion is
better at leveraging the semantically key words in speech (“put” and
“spinach”).

Figure 5: Attention maps from audio modality to the RGB space for global cross
fusion and local fusion modules on VATEX (top) and YouCook2 (bottom).

Results with different modalities. Tab. 4 shows the results when we input
different modalities. For audio-only and video-only methods, we zero-mask the
other modality at the cross-modal encoder. For V+T setting, we use BERT [20]
to replace the audio encoder as in [32]. For V+A+T setting, we use global
cross fusion between video and audio, and use merged fusion between video and
text, to leverage the salient information in audio and text. On all four
datasets, integrating audio (V+A) leads to a significant improvement compared
to the video-only method. Besides, comparing V+A with V+T, audio consistently
outperforms text. In particular, on VATEX, compared with the video-only
method, V+A improves the CIDEr by 6.2%, while V+T only improves by 1.7%,
showing that audio conveys more information than ASR texts. The comparisons
between V+A+T and V+T also demonstrate the benefits of incorporating audio for
video captioning. In addition, V+A+T further improves the CIDEr score by 4.6%
on YouCook2 on the basis of the V+A method, probably because the ASR text
contains more accurate semantic words than audio. On the other datasets where
the speech is not a dominant component in the annotated captions, the
performance of V+A+T is very close to that of V+A, showing that audio can
cover the major information in speech while bypassing the time-consuming
process of ASR [52]. In Fig. 2, we show two qualitative results on VATEX. In
the first video, audio provides more information about laughter and crying,
which the ASR text does not contain. In the second video, the V+T method
mistakenly recognizes the little girl as a boy and misses the information that
a man talks to her, while the V+A method corrects the error and adds the
missing description by leveraging the information from audio.

Comparison with the SOTA methods. In Tab. 5, we compare the performance of our
methods with state-of-the-art methods on four downstream captioning
benchmarks. MMCNN [54] and MGSA [7] are two works that reported audio-visual
video captioning results on MSRVTT. Our V+A method clearly outperforms both of
them by a large margin. Our proposed V+A framework achieves comparable or even
better performance than prior works that rely on information from the text
modality, strong pre-trained text encoder/decoder, or additional data for pre-
training. Compared with MV-GPT [47] that requires ASR transcripts during
inference, our audio-visual model generates captions directly from video
pixels and audio, bypassing the need of ASR before generating captions, which
is time-consuming [52]. Contrary to MV-GPT that initializes their model with
the weights of BERT [20] and GPT-2 [44] pre-trained on large-scale document
datasets, we train our audio encoder and caption decoder from scratch and do
not use any pre-trained language models in our V+A framework. Additionally,
the number of parameters of our model (267M) is much smaller than MV-GPT
($>$360M). We prove that directly learning from raw audio and video frames is
sufficient for our model to capture key information in the data for generating
high quality captions. The V+A+T method further improves the performance and
achieves the state-of-the-art on YouCook2 and VATEX. On ActivityNet-Captions,
without using any relationships between sentences during training or inference
to generate paragraphs, our method can achieve better performance than the
methods that exploits relationships across sentences [13, 41].

##  5 Conclusion

We present an end-to-end pre-training framework for audio-visual video
captioning. A novel modality balanced pre-training loss is proposed to balance
the learning of different modalities during pre-training, which demonstrates
the effectiveness for audio-visual video captioning. We also comprehensively
investigate different cross-modal fusion modules for audio-visual fusion and
propose a new local-global fusion module. Our model can capture different
types of information in human speech and background sounds, achieving
comparable or even better results against the models using ASR text or complex
language models.

## References

  * [1] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian.  Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12487–12496, 2019.
  * [2] Samira Abnar and Willem Zuidema.  Quantifying attention flow in transformers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197, Online, July 2020. Association for Computational Linguistics.
  * [3] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  Advances in Neural Information Processing Systems, 34:24206–24221, 2021.
  * [4] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.  Self-supervised multimodal versatile networks.  Advances in Neural Information Processing Systems, 33:25–37, 2020\.
  * [5] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [6] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman.  A short note about kinetics-600.  arXiv preprint arXiv:1808.01340, 2018.
  * [7] Shaoxiang Chen and Yu-Gang Jiang.  Motion guided spatial attention for video captioning.  In Proceedings of the AAAI conference on artificial intelligence, pages 8191–8198, 2019.
  * [8] Shaoxiang Chen and Yu-Gang Jiang.  Motion guided region message passing for video captioning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1543–1552, 2021.
  * [9] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [10] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019.
  * [11] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.  Clotho: An audio captioning dataset.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. IEEE, 2020.
  * [12] Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen.  Video captioning with attention-based lstm and semantic consistency.  IEEE Transactions on Multimedia, 19(9):2045–2055, 2017.
  * [13] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox.  Coot: Cooperative hierarchical transformer for video-text representation learning.  Advances in neural information processing systems, 33:22605–22618, 2020.
  * [14] Yuan Gong, Yu-An Chung, and James Glass.  AST: Audio Spectrogram Transformer.  In Proc. Interspeech 2021, pages 571–575, 2021.
  * [15] Jack Hessel, Bo Pang, Zhenhai Zhu, and Radu Soricut.  A case study on combining asr and visual features for generating instructional video captions.  In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 419–429, 2019.
  * [16] Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut.  Multimodal pretraining for dense video captioning.  In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 470–490, 2020.
  * [17] Vladimir Iashin and Esa Rahtu.  A better use of audio-visual cues: Dense video captioning with bi-modal transformer.  In British Machine Vision Conference (BMVC), 2020.
  * [18] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [20] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [21] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim.  Audiocaps: Generating captions for audios in the wild.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 119–132, 2019.
  * [22] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 2015.
  * [23] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In Proceedings of the IEEE international conference on computer vision, pages 706–715, 2017.
  * [24] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331–7341, 2021.
  * [25] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11336–11344, 2020.
  * [26] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al.  Value: A multi-task benchmark for video-and-language understanding evaluation.  In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021.
  * [27] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [28] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81, 2004.
  * [29] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Swinbert: End-to-end transformers with sparse attention for video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17949–17958, 2022.
  * [30] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3202–3211, 2022.
  * [31] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [32] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [33] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889, 2020.
  * [34] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640, 2019.
  * [35] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  Advances in Neural Information Processing Systems, 34:14200–14213, 2021.
  * [36] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles.  Spatio-temporal graph for video captioning with knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870–10879, 2020.
  * [37] Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris.  Adamml: Adaptive multi-modal learning for efficient video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7576–7585, 2021.
  * [38] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [39] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  Proc. Interspeech 2019, pages 2613–2617, 2019.
  * [40] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F Henriques, and Andrea Vedaldi.  Support-set bottlenecks for video-text representation learning.  In International Conference on Learning Representations, 2020.
  * [41] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai.  Memory-attended recurrent network for video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8347–8356, 2019.
  * [42] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu.  Balanced multimodal learning via on-the-fly gradient modulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8238–8247, 2022.
  * [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [44] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [45] Tanzila Rahman, Bicheng Xu, and Leonid Sigal.  Watch, listen and tell: Multi-modal weakly supervised dense event captioning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 8908–8917, 2019.
  * [46] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze.  How2: a large-scale dataset for multimodal language understanding.  arXiv preprint arXiv:1811.00347, 2018.
  * [47] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.  End-to-end generative pretraining for multimodal video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17959–17968, 2022.
  * [48] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid.  Look before you speak: Visually contextualized utterances.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16877–16887, 2021.
  * [49] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou.  Dense procedure captioning in narrated instructional videos.  In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 6382–6391, 2019.
  * [50] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464–7473, 2019.
  * [51] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, 2019.
  * [52] Zineng Tang, Jaemin Cho, Yixin Nie, and Mohit Bansal.  Tvlt: Textless vision-language transformer.  Advances in neural information processing systems, 2022.
  * [53] Zineng Tang, Jie Lei, and Mohit Bansal.  Decembert: Learning from noisy instructional videos via dense captions and entropy minimization.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415–2426, 2021.
  * [54] Yapeng Tian, Chenxiao Guan, Justin Goodman, Marc Moore, and Chenliang Xu.  Audio-visual interpretable and controllable video captioning.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops, 2019.
  * [55] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
  * [56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [57] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [58] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu.  Reconstruction network for video captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7622–7631, 2018.
  * [59] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [60] Weiyao Wang, Du Tran, and Matt Feiszli.  What makes training multi-modal classification networks hard?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020.
  * [61] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581–4591, 2019.
  * [62] Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang.  Multimodal token fusion for vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12186–12195, 2022.
  * [63] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  * [64] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo.  Advancing high-resolution video-language representation with large-scale video transcriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036–5045, 2022.
  * [65] Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai.  Stat: Spatial-temporal attention mechanism for video captioning.  IEEE transactions on multimedia, 22(1):229–241, 2019.
  * [66] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi.  Merlot reserve: Neural script knowledge through vision and language and sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375–16387, 2022.
  * [67] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  Advances in Neural Information Processing Systems, 34:23634–23651, 2021.
  * [68] Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu.  Open-book video captioning with retrieve-copy-generate network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9837–9846, 2021.
  * [69] Luowei Zhou, Chenliang Xu, and Jason J Corso.  Towards automatic learning of procedures from web instructional videos.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [70] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong.  End-to-end dense video captioning with masked transformer.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8739–8748, 2018.
  * [71] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746–8755, 2020.
  * [72] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic.  Cross-task weakly supervised learning from instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537–3545, 2019.

In this appendix, we include:

  * •

Additional implementation details.

  * •

Details on audio relevance analysis.

  * •

Additional qualitative results.

  * •

Our discussions on potential societal impact.

  * •

Our discussions on limitation.

##  Appendix A Additional Implementation Details

Paragraph Video Captioning on ActivityNet-Captions: ActivityNet-Captions
contains one reference for each video in the training set and two references
for each video in the validation/testing set. Following [70], we use ground-
truth segments and sentences for training. For validation/testing, we use the
ground-truth segments from the first reference in validation/testing set and
evaluate against the two references.

Video Encoder: We equally divide each video clip into $16$ segments and
randomly sample $1$ frame from each segment during training. For testing, we
uniformly sample 16 frames. Following prior works [29, 30], given the raw
video frames of the size $T\times H\times W\times 3$, where $T$ is the number
of frames, $H\times W$ is the image height and width, and $3$ is the RGB
channels, the size of the output features of Video Swin Transformer is
$\frac{T}{2}\times\frac{H}{32}\times\frac{W}{32}\times 8C$, where $C$ is the
channel dimension. In our experiments, the input size of the video encoder is
$16\times 224\times 224\times 3$ and the channel dimension is $C=128$, so the
output size is $8\times 7\times 7\times 1024$. To be consistent with the token
dimension of the other modules, we add a linear layer to map the feature
dimension into $768$. Hence, the number of video tokens is $N_{v}=8\times
7\times 7=392$ and the token dimension is $D=768$.

Audio Encoder: Our implementation of the audio encoder is similar to that in
[66]. Each audio is resampled to 22,050Hz and divided into frames of 1536
samples with hop length of $588$. Then we apply 64 mel-scale filters and take
logarithm on the amplitude to get the log mel spectrogram. To tackle the
variable length of audios, we set a maximum length of frames as 256 ($\sim$6.8
sec.). The audio clips shorter than 256 frames will be zero-padded and longer
clips will be down-sampled to be 256 frames. As a result, the dimension of
input mel-spectrogram is $256\times 64$ for each audio. We use a 12-layer
Transformer with 12 attention heads on audio spectrogram. We apply the linear
1-dimensional layout on the spectrogram as in [66] rather than a two-
dimensional (image-like) one in [14], as we notice 1-dimensional layout is
more suitable for fine-grained audio events like speech. The Transformer
produces audio features of dimension $256\times 768$ and then, following [66],
we apply an average pooling by a factor of four to resize the sequence to a
length of $N_{a}=64$ audio tokens. During training, we use time and frequency
masking as in SpecAugment [39] for data augmentation. The time mask parameter
is set as $32$ and the frequency mask parameter is set as $128$.

Cross Encoder: We use a 3-layer Transformer with 12 attention heads as the
cross-modal encoder. The feature embeddings of different modalities will be
added with the position embedding and token type embedding to distinguish the
position and modality of the tokens.

Figure A: The ARS of top 100 words. Figure B: Some qualitative examples when
we input different modalities to generate captions.

##  Appendix B Audio Relevance Analysis

We design two metrics, _i.e_. Speech Coverage Rate (SCR) and Audio Relevance
Score (ARS), to measure how relevant the captions are to speech and audio.

For SCR, we tokenize the annotated captions and speech transcripts, and
compute the percentage of tokens in captions that are covered by the
associated speech transcripts.

For ARS, we collect the captions from audio captioning and image captioning
datasets. We calculate the word frequencies in audio captioning and image
captioning datasets, and assign a higher score to the words that occur more
frequently in audio captions than image captions. We use the annotated
captions from AudioCaps [21] and Clotho [11] as audio captions and the
annotated captions from CoCo-Captions [9] as image captions. We tokenize and
lemmatize all captions, and remove all punctuation and stop words. Then we
compute the word frequency in audio captions and image captions. Let
$f^{a}(\mathbf{w})$ and $f^{i}(\mathbf{w})$ denote the frequency of word
$\mathbf{w}$ in audio captions and image captions respectively, then ARS is
computed as:

$\text{ARS}(\mathbf{w})=\max(\log\frac{f^{a}(\mathbf{w})}{f^{i}(\mathbf{w})},0).$
(13)

Hence, a higher ARS will be assigned to those words whose frequency is much
higher in audio captions than that in image captions. On the contrary, if a
word is more frequent in image captions, then the log of the division is
negative, and the ARS will be set as zero. The ARS of a sentence is the sum of
the ARS of all non-stop words in it, and we compute the average ARS of all
captions on each dataset.

Figure A shows the ARS of the top 100 words. We can see that most of those
words are highly relevant to audio modality, including verbs that describe a
certain type of sound, _e.g_. “snore”, “gurgle”, “bleat”, adjectives or
adverbs that describe the pattern of sound, _e.g_. “loudly”, “muffled”, or
nouns that are usually associated with a type of sound, _e.g_. “vibration”,
“gunshot”, _etc_.

##  Appendix C Additional Qualitative Results

Qualitative results. We show some qualitative examples in Fig. B by comparing
the ground truth, and predictions of our model with video-only, video+text,
video+audio, and video+audio+text inputs. In the first example, when only
visual modality is available, the model misclassifies the peppercorns as cumin
seeds as they are hardly visible in the video. By adding audio and/or ASR text
as input, the model correctly generates the caption because the ingredients
are clearly introduced in the speech, which shows that we can infer the
ingredient from raw audio without relying on an off-the-shelf ASR system. In
the second example, the caption is not explicitly described in the audio, but
the model is able to correct its prediction from “rock, paper, scissors” to
“patty cake” by adding audio or text modality as it can infer the type of game
from the speech. In particular, when we add audio as the input, the model not
only predicts the correct type of game, but also provides more information
about the audio, _i.e_. recognizing that the people are laughing. The third
example shows that audio is also helpful when there is no speech in the video.
Using video-only or video+text as the inputs, the model generates incorrect
captions, _e.g_. “smoking a cigarette” or “smoking a pipe” as the microphone
is not quite visible. However, if we add audio as the input, the model will
correctly detect that the man is singing and music is playing in the
background. The last example is a failure case, where the audio-visual model
mistakenly recognizes the sounds of camel bell ringing and child crying as the
sounds of the bagpipes and the drums. It indicates that the model gives too
much attention to the audio modality and ignores the visual appearance of
people riding a camel.

Figure C: Additional visualization on the attention maps from audio modality
to the RGB space for global cross fusion and local fusion modules on YouCook2,
MSRVTT, VATEX, and ActivityNet, respectively.

Attention maps. We show additional visualization on the attention maps on the
four downstream datasets respectively. Similar to what we observed in the main
paper, in the first and the fourth example, where the items in the videos are
mentioned in the speech, _i.e_., “bacon” and “hair”, the local attention will
focus on the regions of those items. In the second example, where there is
sound of singing in the video, both local and global modules will give
attention to the performers on the stage, and we notice that the attention of
the global module will be more concentrated. In the third example, where there
is sound of speaking and laughter, the global module focuses more on the mouth
region, where the sound comes from.

##  Appendix D Discussions on Societal Impact

Video captioning makes videos more accessible to all users, including users
with accessibility issues, _e.g_. lowvision and blind users. Furthermore,
audio-visual video captioning also benefits users with hearing disability by
including text descriptions of the audio modality. However, our framework is a
data-driven system, so the quality of generated captions may be biased to the
distribution of training data. As our pre-training data are obtained from
online YouTube videos, our system may produce harmful captions that contains
toxic contents or social biases present in the training data. To avoid
undesirable effects, careful examination is required before adopting the
outputs of our framework for real-world applications. Another ethical concern
on using YouTube videos is how to protect user privacy. In our experiments, we
download the videos currently available on YouTube. Therefore, if a user
deletes a video from their YouTube channel, we will not be able to use the
videos.

##  Appendix E Discussions on Limitations

The failure case in Fig. B shows that our approach is not always successful in
the fusion of visual and audio modalities. A potential direction is to design
a more intelligent cross-modal fusion module to dynamically update the weights
of different modalities for caption generation. Besides, the dominance of the
modality is dataset-dependent, and we pre-trained our audio-visual video
captioning model on the HowTo100M dataset, where audio dominates due to the
use of ASR transcripts as text supervision. We did not verify the
effectiveness of our proposed modality balancing pre-training strategy on
vision-dominant datasets due to the lack of such large-scale pre-training
datasets with video-text pairs. We believe that our proposed approach has the
potential to be advantageous in those settings, and it could be explored in
future research.