Vladimir<EMAIL_ADDRESS>Esa<EMAIL_ADDRESS>Computing
Sciences
Tampere University
Tampere, Finland A Better Use of Audio-Visual Cues

# A Better Use of Audio-Visual Cues:
Dense Video Captioning with Bi-modal Transformer

###### Abstract

Dense video captioning aims to localize and describe important events in
untrimmed videos. Existing methods mainly tackle this task by exploiting only
visual features, while completely neglecting the audio track. Only a few prior
works have utilized both modalities, yet they show poor results or demonstrate
the importance on a dataset with a specific domain. In this paper, we
introduce Bi-modal Transformer which generalizes the Transformer architecture
for a bi-modal input. We show the effectiveness of the proposed model with
audio and visual modalities on the dense video captioning task, yet the module
is capable of digesting any two modalities in a sequence-to-sequence task. We
also show that the pre-trained bi-modal encoder as a part of the bi-modal
transformer can be used as a feature extractor for a simple proposal
generation module. The performance is demonstrated on a challenging
ActivityNet Captions dataset where our model achieves outstanding performance.
The code is available:  v-iashin.github.io/bmt

##  1 Introduction

Current video sharing platforms contain a large amount of video material. The
ability to generate descriptions of this content would be highly valuable for
many tasks, such as content-based retrieval or recommendation [Tran et
al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri, Miech et
al.(2019)Miech, Zhukov, Alayrac, Tapaswi, Laptev, and Sivic]. Moreover, they
would enable visually-impaired people to consume video material and improve
their quality of life [Rohrbach et al.(2017)Rohrbach, Torabi, Rohrbach,
Tandon, Pal, Larochelle, Courville, and Schiele].

This kind of video descriptions are usually provided as natural language
sentences or captions, a compact and intuitive format and, most importantly,
can be digested by humans. Early works [Yao et al.(2015)Yao, Torabi, Cho,
Ballas, Pal, Larochelle, and Courville, Venugopalan et al.(2015a)Venugopalan,
Rohrbach, Donahue, Mooney, Darrell, and Saenko, Venugopalan et
al.(2015b)Venugopalan, Xu, Donahue, Rohrbach, Mooney, and Saenko, Yu et
al.(2016)Yu, Wang, Huang, Yang, and Xu] described the video content with only
one sentence, which might be too “sparse” for long videos – one might try to
think up a relatively short sentence which describes the whole film. To
mitigate this issue, [Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and
Carlos Niebles] proposed dense video captioning which requires a model to,
first, localize “events”, and, then, to produce one-sentence description for
each of them instead of generating one caption for the entire film (see Fig.
1).

The task is usually formulated as a sequence-to-sequence (video to caption)
task. Therefore, the progress in the field is significantly influenced by
advances in machine translation. Hence, many models rely on an encoder-decoder
architecture which consists of two recurrent neural networks (RNNs) or,
recently-proposed Transformer-like model [Vaswani et al.(2017)Vaswani,
Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]. An event
localization module usually utilizes an RNN structure which first encodes the
input to produce a hidden representation and, then, makes predictions using
this representation.

Considering the natural co-occurrence of visual and audio tracks in a video
and the fact that human perception is multi-modal, recent advances in deep
learning practice audio-visual training [Owens and Efros(2018), Zhao et
al.(2018)Zhao, Gan, Rouditchenko, Vondrick, McDermott, and Torralba, Long et
al.(2018)Long, Gan, De Melo, Wu, Liu, and Wen, Zhu and Rahtu(2020b), Zhu and
Rahtu(2020a)]. Yet, most of the existing works on dense video captioning
employ only visual inputs. In this work, we address this issue by introducing
a novel bi-modal transformer with the multi-headed proposal generator. Our
captioning module is inspired by the transformer architecture and, more
precisely, how the attention module fuses the information from both sequences.
While an efficient object detector YOLO [Redmon and Farhadi(2018)] inspires
the design of each proposal head in the bi-modal multi-headed proposal
generator.

The proposed method effectively utilizes audio and visual cues. We demonstrate
the performance of our model on the challenging open-domain ActivityNet
Captions dataset [Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Carlos
Niebles]. The results show the state-of-the-art performance of our bi-modal
dense video captioning module as well as our bi-modal proposal generator on
BLEU@3–4 and F1 metrics.

Figure 1: Example video with the predictions of our model alongside the ground
truth.

##  2 Related Work

The dense video captioning task requires a model to, first, localize events
within a video and, then, to produce a textual one-sentence description of
what is happening during the event. The dense video captioning task branches
out from the video captioning which task is to caption a video without
localizing the event. The video captioning field evolved from hand-crafted
rule models [Kojima et al.(2002)Kojima, Tamura, and Fukunaga, Lee et
al.(2008)Lee, Hakeem, Haering, and Zhu, Das et al.(2013)Das, Xu, Doell, and
Corso] to encoder-decoder architectures [Yao et al.(2015)Yao, Torabi, Cho,
Ballas, Pal, Larochelle, and Courville, Venugopalan et al.(2015a)Venugopalan,
Rohrbach, Donahue, Mooney, Darrell, and Saenko, Venugopalan et
al.(2015b)Venugopalan, Xu, Donahue, Rohrbach, Mooney, and Saenko, Yu et
al.(2016)Yu, Wang, Huang, Yang, and Xu] inspired by advances in machine
translation [Rohrbach et al.(2013)Rohrbach, Qiu, Titov, Thater, Pinkal, and
Schiele]. Later, the captioning models were further enhanced by semantic
tagging [Gan et al.(2017)Gan, Gan, He, Pu, Tran, Gao, Carin, and Deng, Pan et
al.(2017)Pan, Yao, Li, and Mei], reinforcement learning [Wang et
al.(2018c)Wang, Chen, Wu, Wang, and Yang Wang], attention [Yan et
al.(2019)Yan, Tu, Wang, Zhang, Hao, Zhang, and Dai], extended memory [Wang et
al.(2018b)Wang, Wang, Huang, Wang, and Tan, Pei et al.(2019)Pei, Zhang, Wang,
Ke, Shen, and Tai], and other modalities [Xu et al.(2017)Xu, Yao, Zhang, and
Mei, Hori et al.(2017)Hori, Hori, Lee, Zhang, Harsham, Hershey, Marks, and
Sumi, Wang et al.(2018d)Wang, Wang, and Wang, Hao et al.(2018)Hao, Zhang, and
Guan].

###  2.1 Dense Video Captioning

The task of dense video captioning, as well as a test-bed, ActivityNet
Captions dataset, were introduced by Krishna _et al_ [Krishna et
al.(2017)Krishna, Hata, Ren, Fei-Fei, and Carlos Niebles] who utilized the
idea of the Deep Action Proposals network [Escorcia et al.(2016)Escorcia,
Heilbron, Niebles, and Ghanem] to generate event proposals and an LSTM network
to encode the context and generate captions. The idea of context-awareness was
further developed in [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] who
employed a bi-directional variant of Single-Stream Temporal Action proposal
network (SST) [Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos
Niebles] which makes better use of the video context, an LSTM network with
attentive fusion and context gating was used to generate context-aware
captions. Zhou _et al_ [Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and
Xiong] adapted Transformer architecture [Vaswani et al.(2017)Vaswani, Shazeer,
Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] to tackle the task
and used transformer encoder’s output as input to a modification of ProcNets
[Zhou et al.(2018a)Zhou, Xu, and Corso] to generate proposals.

Recently, the idea of reinforcement learning was found to be beneficial for
image captioning (Self-critical Sequence Training (SCST)) [Rennie et
al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel] and, hence, applied in
dense video captioning as well. More precisely, the SCST was used in a
captioning module to optimize the non-differentiable target metric, _e.g_
METEOR [Denkowski and Lavie(2014)]. Specifically, Li _et al_ [Li et
al.(2018)Li, Yao, Pan, Chao, and Mei] integrated the reward system and
enriched Single-Shot-Detector-like structure [Liu et al.(2016)Liu, Anguelov,
Erhan, Szegedy, Reed, Fu, and Berg] with descriptiveness regression for
proposal generation. Similarly, Xiong _et al_ [Xiong et al.(2018)Xiong, Dai,
and Lin] used an LSTM network trained with the sentence- and paragraph-level
rewards for maintaining coherent and concise story-telling, while the event
proposal module was adopted from Structured Segment Networks [Zhao et
al.(2017)Zhao, Xiong, Wang, Wu, Tang, and Lin]. Mun _et al_ [Mun et
al.(2019)Mun, Yang, Ren, Xu, and Han] further developed the idea of coherent
captioning by observing the overall context and optimizing two-level rewards,
an SST module is used for proposal generation, and a Pointer Network [Vinyals
et al.(2015)Vinyals, Fortunato, and Jaitly] to distill proposal candidates.

Another direction of research relies on weak supervision which is designed to
mitigate the problem of laborious annotation of the datasets. To this end,
Duan _et al_ [Duan et al.(2018)Duan, Huang, Gan, Wang, Zhu, and Huang]
proposed an autoencoder architecture which generates proposals and, then,
captions them while being supervised only with a set of non-localized captions
in a cycle-consistency manner. However, the results appeared to be far from
the supervised methods.

###  2.2 Multi-modal Dense Video Captioning

It is natural to assume that, besides visual information, a video
understanding system might benefit from the cues contained in other modalities
like audio [Rahman et al.(2019)Rahman, Xu, and Sigal], speech (subtitles) [Shi
et al.(2019)Shi, Ji, Liang, Duan, Chen, Niu, and Zhou], or both [Iashin and
Rahtu(2020)]. Specifically, Rahman _et al_ [Rahman et al.(2019)Rahman, Xu, and
Sigal] were the first to include audio modality into the dense video
captioning set up. They borrowed the idea of cycle-consistency from [Duan et
al.(2018)Duan, Huang, Gan, Wang, Zhu, and Huang] and employed multi-modal
Tucker decomposition [Ben-Younes et al.(2017)Ben-Younes, Cadene, Cord, and
Thome] to combine information from both modalities and pass it to a GRU-based
[Chung et al.(2014)Chung, Gulcehre, Cho, and Bengio] caption decoder. However,
since the model is trained in a weakly supervised setting, the results do not
reach the performance of the supervised models.

Shi _et al_ [Shi et al.(2019)Shi, Ji, Liang, Duan, Chen, Niu, and Zhou]
proposed to utilize the corresponding speech along with frame features to
further improve captioning performance on cooking videos. They suggested
employing a transformer’s encoder to encode video frames and subtitle
embeddings produced by a pre-trained BERT model [Devlin et al.(2019)Devlin,
Chang, Lee, and Toutanova]. Next, an LSTM generates proposals, and the other
two LSTMs were used for the encoder-decoder captioning module. Despite the
significant gains in captioning performance, we believe these findings are not
conclusive as instructional videos is an ill-suited domain to show the
benefits of the speech modality for a captioning task since subtitles alone
can be a very accurate proxy for captions in such videos (see [Miech et
al.(2019)Miech, Zhukov, Alayrac, Tapaswi, Laptev, and Sivic]).

In contrast, Iashin _et al_ [Iashin and Rahtu(2020)] showed the importance of
the speech modality on a free-domain dataset. They proposed to train three
transformers for each modality individually and fuse features by concatenation
before predicting the next caption word while borrowing the proposal generator
from [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu]. However, the suggested
approach for feature fusion is rather straightforward and inefficient.
Moreover, the adopted proposal generator is based solely on video features
which contrasts with the idea of the dense video captioning task.

Our method is mostly similar to [Iashin and Rahtu(2020)], yet we show
significantly better results on the task while utilizing only visual and audio
cues. Besides, our proposal generator does employ both modalities and
significantly outperforms the state-of-the-art. Furthermore, we present a
single model which utilizes bi-modal encoder for both: the proposal generator
and captioning module, making it an elegant approach for the dense video
captioning task.

Figure 2: The design of Bi-modal Transformer with Multi-headed Proposal
Generator. The proposed model inputs features extracted by VGGish, I3D, and
GloVe pre-trained models (bottom left). Then, the bi-modal encoder with N
layers processes the audio and visual features and passes its bi-modal
representation to the proposal generator (top). After, the generated proposals
are used to clip the input features (left). The clipped features are passed
through the encoder again. The output of the encoder, then, is used at every
layer (N) of the bi-modal decoder (bottom). The decoder attends to the bi-
modal encoder’s representation as well as the previous caption words and
produces its internal representation of the context. This representation is
passed to the generator (right) to generate the next word. Residual
connections are removed for clarity. Best viewed in color.

##  3 Our Framework

Our approach consists of two parts: the bi-modal transformer and multi-headed
proposal generator (see Fig. 2). The model expects the input to be a set of
continuous features stacked together in a sequence. To represent a visual
stream, we use a pre-trained Inflated 3D (I3D) network [Carreira and
Zisserman(2017)] while for the audio stream we employ pre-trained VGGish
[Hershey et al.(2017)Hershey, Chaudhuri, Ellis, Gemmeke, Jansen, Moore,
Plakal, Platt, Saurous, Seybold, et al.], the tokens (roughly, words) are
embedded with pre-trained GloVe [Pennington et al.(2014)Pennington, Socher,
and Manning] (see Sec. 6.2 for implementation details). Also, since the
transformer is permutation invariant it has no sense of recurrence. Thus, the
order of features within a sequence is preserved by adding the positional
encoding to the output of the embedding layers. Following [Vaswani et
al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and
Polosukhin], we use cosine and sine functions.

Next, the audio and visual sequences, are passed through the transformer’s bi-
modal N-layered encoder to produce bi-modal sequence representations utilizing
novel bi-modal multi-headed attention blocks to fuse the features from both
sequences. Then, the novel proposal generator utilizes these features to
generate proposals and their confidence scores. After, a pre-defined number of
most confident proposals are selected to clip the input feature sequences.
Next, the clipped features are processed with the encoder to re-represent the
features considering only the features which are left after clipping.

The bi-modal encoder’s representation is used at every layer in the bi-modal
decoder. Concretely, the encoder’s outputs are passed to the corresponding bi-
modal attention blocks in the decoder layer along with the representation of
the previously generated caption words. The last-layer representation of the
decoder is used in the generator where the next caption word is produced. To
avoid an empty input to the decoder in the beginning, a special start-token is
used. The caption is generated word-by-word until a special end-token is
sampled.

This section, first, presents the design of the captioning module (Sec. 3.1)
and, second, the proposal generator (Sec. 3.2) while the training procedure is
explained in Sec. 3.3.

###  3.1 Captioning Module

The task of dense video captioning requires to produce a caption for each
proposal. Therefore, bi-modal encoder inputs audio $A$ and visual $V$ feature
sequences which temporally correspond to the proposal and outputs two
sequences: audio-attended visual features $V^{a}$ and visual-attended audio
features $A^{v}$. These features are used by the bi-modal decoder which
attends to these features and the previous caption words
$(c_{1},\,c_{2},\,\dots,\,c_{t})$. Finally, the bi-modal decoder outputs the
representation which is employed to model a distribution of the next caption
word $(c_{t+1})$ over the vocabulary. The proposal index is omitted for
clarity.

##### Bi-modal Encoder

In contrast to the encoder in [Vaswani et al.(2017)Vaswani, Shazeer, Parmar,
Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin], our bi-modal encoder inputs
two streams: audio $(A\in\mathrm{R}^{T_{a}\times d_{a}})$ and visual
$(V\in\mathrm{R}^{T_{v}\times d_{v}})$ features corresponding to the proposal.
Then, the features are passed in a stack of $N$ encoder layers. Instead of
two, each layer has three sub-layers: self-attention, bi-modal attention
(new), and position-wise fully-connected layers. Specifically, given
$A^{\text{fc}}_{0}=A$ and $V^{\text{fc}}_{0}=V$, an $n^{\text{th}}$ encoder
layer is defined as

$\displaystyle A^{\text{self}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(A^{\text{fc}}_{n-1},A^{\text{fc}}_{n-1},A^{\text{fc}}_{n-1}),$
// audio self-attention (1) $\displaystyle V^{\text{self}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(V^{\text{fc}}_{n-1},V^{\text{fc}}_{n-1},V^{\text{fc}}_{n-1}),$
// visual self-attention (2) $\displaystyle A^{\text{mm}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(A^{\text{self}}_{n},V^{\text{self}}_{n},V^{\text{self}}_{n}),$
// visual-attended audio feats. (3) $\displaystyle V^{\text{mm}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(V^{\text{self}}_{n},A^{\text{self}}_{n},A^{\text{self}}_{n}),$
// audio-attended visual feats. (4) $\displaystyle A^{\text{fc}}_{n}$
$\displaystyle=\text{TwoFullyConnected}(A^{\text{mm}}_{n}),$ //
$\mathrm{R}^{T_{a}\times d_{a}}\leftarrow\mathrm{R}^{T_{a}\times
4d_{a}}\leftarrow\mathrm{R}^{T_{a}\times d_{a}}$ (5) $\displaystyle
V^{\text{fc}}_{n}$
$\displaystyle=\text{TwoFullyConnected}(V^{\text{mm}}_{n}),$ //
$\mathrm{R}^{T_{v}\times d_{v}}\leftarrow\mathrm{R}^{T_{v}\times
4d_{v}}\leftarrow\mathrm{R}^{T_{v}\times d_{v}}$ (6)

where all sub-layers have distinct sets of trainable weights and mostly
resemble the blocks of Transformer [Vaswani et al.(2017)Vaswani, Shazeer,
Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin], yet we allow the
dimension of the weights in multi-headed attention in (3) & (4) to be
different for both modalities because we expect them to have a different size.
We define the multi-headed attention in Sec. 6.1. The encoder outputs visual-
attended audio features $(A^{v}=A_{N}^{\text{fc}})$ and audio-attended visual
features $(V^{a}=V_{N}^{\text{fc}})$, which are used the decoder.

##### Bi-modal Decoder

The bi-modal decoder inputs the previous sequence of caption words
$C_{t}=(c_{1},\,c_{2},\,\dots,\,c_{t})\in\mathrm{R}^{t\times d_{c}}$ and,
opposed to the original Transformer’s decoder [Vaswani et al.(2017)Vaswani,
Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin], ours gets
the output from the bi-modal encoder $(A^{v}\in\mathrm{R}^{T_{a}\times
d_{a}},\,V^{a}\in\mathrm{R}^{T_{v}\times d_{v}})$. Thus, instead of three, it
has four sub-layers: self-attention, bi-modal encoder-decoder attention (new),
bridge (new), & position-wise fully-connected layers. For
$C^{\text{fc}}_{0}=C_{t}$, an $n^{\text{th}}$ decoder layer is defined as

$\displaystyle C^{\text{self}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(C^{\text{fc}}_{n-1},C^{\text{fc}}_{n-1},C^{\text{fc}}_{n-1}),$
// caption self-attention (7) $\displaystyle C^{A^{v}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(C^{\text{self}}_{n},A^{v},A^{v}),$ //
audio-visual attended prev. caps. (8) $\displaystyle C^{V^{a}}_{n}$
$\displaystyle=\text{MultiHeadAttention}(C^{\text{self}}_{n},V^{a},V^{a}),$ //
visual-audio attended prev. caps. (9) $\displaystyle C^{\text{mm}}_{n}$
$\displaystyle=\text{OneFullyConnected}\big{(}[C^{A^{v}}_{n},\,C^{V^{a}}_{n}]\big{)},$
// $\mathrm{R}^{t\times d_{c}}\leftarrow\mathrm{R}^{t\times 2d_{c}}$;
$[\cdot,\cdot]$ — concat. (10) $\displaystyle C^{\text{fc}}_{n}$
$\displaystyle=\text{TwoFullyConnected}(C^{\text{mm}}_{n}),$ //
$\mathrm{R}^{t\times d_{c}}\leftarrow\mathrm{R}^{t\times
4d_{c}}\leftarrow\mathrm{R}^{t\times d_{c}}$ (11)

where, as in the encoder, trainable weights have distinct dimensions depending
on a modality and are not shared across sub-layers. The decoder outputs
caption features $(C^{\text{av}}_{t}=C^{\text{fc}}_{N})$.

##### Generator

The purpose of the generator is to model the distribution for the next caption
word $c_{t+1}$ given the output of the decoder
$C^{\text{av}}_{t}\in\mathrm{R}^{t\times d_{c}}$. Therefore, the generator is,
usually, a fully-connected layer with the softmax activation which maps the
caption features of size $d_{c}$ into a dimension corresponding to the size of
the vocabulary in the training set.

##### Residual Connection

Following the original Transformer architecture, we employ the residual
connection [He et al.(2016)He, Zhang, Ren, and Sun] surrounding each sub-layer
of the encoder and decoder except for the bridge layer since in- and out-
dimensions are different. Additionally, we adopt Layer Normalization [Ba et
al.(2016)Ba, Kiros, and Hinton] before applying a sub-layer: $x+\text{sub-
layer}\big{(}\text{LayerNorm}(x)\big{)}$.

##### Dropout

We also regularize our model with dropout [Srivastava et al.(2014)Srivastava,
Hinton, Krizhevsky, Sutskever, and Salakhutdinov] which is applied: a) before
adding the residual in the residual connection, b) before the activation in
the bridge layer, c) on outputs of the positional encoding, d) between layers
in the position-wise fully-connected network, and e) after the softmax
operation in the scaled dot-product attention (see Sec. 6.1).

Figure 3: The Bi-modal Multi-headed Proposal Generator inputs the two-stream
output from the bi-modal encoder, processes it with two stacks of proposal
generation heads. The predictions from all heads form a common pool of
predictions. Thus, the pool consists of $T_{v}\cdot
K_{v}\cdot|\Psi_{v}|+T_{a}\cdot K_{a}\cdot|\Psi_{a}|$ proposals, which are
sorted on the confidence score and passed back to clip input features to the
captioning module.

###  3.2 Event Proposal Generation Module

The proposal generator generates a set of proposals for a given video. It
consists of two blocks: a bi-modal encoder and bi-modal multi-headed proposal
generator (not related to multi-headed attention). The bi-modal encoder in
this module inputs the whole sequence opposed to the bi-modal encoder in the
captioning module, which inputs a sequence of features corresponding to a
proposal. Specifically, it inputs both: visual-attended audio features
$A^{v}\in\mathrm{R}^{T_{a}\times d_{a}}$ and audio-attended visual features
$V^{a}\in\mathrm{R}^{T_{v}\times d_{v}}$. Since the sequence lengths
$(T_{a},\,T_{v})$ might be distinct, the fusion of predictions cannot be done
at each time-step. To this end, we propose the module which makes predictions
for each modality at every timestamp individually forming a common pool of
cross-modal predictions (see Fig. 3).

##### Proposal Generation Head

The proposal generation head inputs a sequence of $T$ features, and makes
predictions at each timestamp on the interval $[1,\,T]$, and for every prior
segment length anchor in the set $\Psi$. The design of the proposal generation
head is partly inspired by YOLO object detector [Redmon et al.(2016)Redmon,
Divvala, Girshick, and Farhadi, Redmon and Farhadi(2017), Redmon and
Farhadi(2018)]. Specifically, it is a fully-convolutional network which, in
our case, consists of only three layers. Opposed to YOLO, we preserve the
sequence length across all layers using padding and identity stride. Moreover,
YOLO utilizes predictions from three different scales to predict different-
scale objects. Hence, only three sizes of receptive fields are used. Instead,
our model makes predictions at a single scale while controlling the receptive
field with a kernel size $k$ which is distinct in each proposal generation
head. More precisely, the $1^{\text{st}}$ convolutional layer has a kernel
size $k$ while in the $2^{\text{nd}}$ and the $3^{\text{rd}}$ the kernel size
is 1. The layers are separated with ReLU activations and dropout.

##### Predictions

Temporal boundaries and confidence for a proposal are obtained using three
values which were predicted by the proposal generation head: a location of a
segment center $\sigma(c)$ relative to a position $p$ in the sequence while
$\sigma(\cdot)$ is a sigmoid function which bounds the values into $[0,\,1]$
interval, a coefficient $\exp(l)$ for an anchor, and objectness score
$\sigma(o)$

$\displaystyle\text{center}=p+\sigma(c);$
$\displaystyle\text{length}=\text{anchor}\cdot\exp{(l)};$
$\displaystyle\text{confidence}=\sigma(o).$ (12)

The prediction of the center and length are in grid-cells (not in seconds). To
obtain seconds, both are multiplied by a cell size which corresponds to a
temporal span of the feature.

##### Bi-modal Multi-headed Proposal Generator

The common pool of predictions is formed with predictions made by each of the
proposal generation heads. Specifically, our model has $K_{a}$ and $K_{v}$
heads for audio and visual modalities with distinct sets of kernel sizes.
Overall, our model generates $3\cdot\big{(}T_{a}\cdot
K_{a}\cdot|\Psi_{a}|+T_{v}\cdot K_{v}\cdot|\Psi_{v}|\big{)}$ proposals. For
the final predictions, we select top-100 proposals out of the common pool
based on the confidence score.

##### Segment Length Priors & Kernel Sizes

To select a set of anchors, we use K-Means clustering algorithm with the
Euclidean distance metric, as opposed to intersection over the union in YOLO.
Due to granularity of feature extractors, feature lengths $(T_{a},\,T_{v})$
might not necessarily equal. Thus, we obtain distinct numbers of anchors for
audio and visual modalities $\big{(}|\Psi_{a}|,\,|\Psi_{v}|\big{)}$ to keep
$T_{a}\cdot|\Psi_{a}|$ close to $T_{v}\cdot|\Psi_{v}|$ to balance the impact
of each modality to the common pool of predictions. Similarly, the kernel
sizes are determined by K-Means. We motivate it with an expectation that the
receptive field will correspond to an event with a higher probability. We
scale the resulting cluster centroids (in secs) by the feature time span to
obtain values in grid-cell coordinates. Next, we round the values to the next
odd integer for more elegant padding. Again, to preserve the balance in the
share of predictions from each modality, we obtain an equal number of kernel
sizes $K_{a}=K_{v}$ both modalities.

###  3.3 Training Procedure

Our model is trained in two stages: first, the captioning module is trained
with ground truth proposals and, then, the proposal generator is trained using
the pre-trained bi-modal encoder from the captioning model. Similar to
[Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin] and [Iashin and Rahtu(2020)], we optimize KL-
divergence loss and apply Label Smoothing [Szegedy et al.(2016)Szegedy,
Vanhoucke, Ioffe, Shlens, and Wojna] to force a model to be less confident
about predictions anticipating noisy annotations. Also, masking is used to
ignore padding and prevent the model from attending to the next positions in
the ground truth caption. During training of the event proposal generation
module, all proposal generation heads for each modality are trained
simultaneously summing up losses from all heads and both modalities. Each head
uses YOLO-like loss: MSE for the localization losses (no square root) and
cross-entropy for (no)objectness losses. The NMS is avoided for efficiency and
to preserve the possibility of dense events. For the implementation details, a
reader is referred to supplementary material (Sec. 6.3).

##  4 Experiments

We employ ActivityNet Captions dataset [Krishna et al.(2017)Krishna, Hata,
Ren, Fei-Fei, and Carlos Niebles], which consists of 100k temporally localized
sentences for 20k YouTube videos. The dataset is split into 50/25/25 % parts
for training, validation, and testing. The validation set of videos is
annotated by two different annotators. We report the results on the validation
subsets as ground truth is not available for the testing set. Since the
dataset is distributed as a set of links to YouTube videos, it is not possible
to collect the whole dataset as some videos became unavailable. The authors
also provide C3D features which are not suitable for our experimentation as
they are missing audio information. In total, we had 91 % of the videos. We
omit the unavailable videos from the validation sets. We compared the results
of other methods on the 91 % and 100 % of videos in Sec. 6.4.1 and observed
similar performance suggesting the videos to be missing completely at random.

To evaluate the event proposal generation module we employ precision, recall,
and mainly rely on F1-score (harmonic mean of precision and recall). While
METEOR [Denkowski and Lavie(2014)] and BLEU@3–4 [Papineni et
al.(2002)Papineni, Roukos, Ward, and Zhu] were used for captioning as they are
highly correlated with human judgement. All metrics are averaged for every
video and temporal Intersection over Union thresholds: $[0.3,0.5,0.7,0.9]$. As
it has been noted in [Mun et al.(2019)Mun, Yang, Ren, Xu, and Han], the
original evaluation script had a critical issue which resulted in an incorrect
evaluation of previous models. Therefore, we re-implement [Wang et
al.(2018a)Wang, Jiang, Ma, Liu, and Xu, Zhou et al.(2018b)Zhou, Zhou, Corso,
Socher, and Xiong] and compare with the results obtained with the corrected
script.

###  4.1 Comparison to the State-of-the-art

|  | Full Dataset |  GT Proposals |  Learned Proposals
---|---|---|---|---
| RL | was Available | B@3 | B@4 | M | B@3 | B@4 | M
Li _et al_[Li et al.(2018)Li, Yao, Pan, Chao, and Mei] | yes | yes | 4.55 | 1.62 | 10.33 | 2.27 | 0.73 | 6.93
Xiong _et al_[Xiong et al.(2018)Xiong, Dai, and Lin] | yes | yes | – | – | – | 2.84 | 1.24 | 7.08
Mun _et al_[Mun et al.(2019)Mun, Yang, Ren, Xu, and Han] | yes | yes | 4.41 | 1.28 | 13.07 | 2.94 | 0.93 | 8.82
Krishna _et al_[Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Carlos Niebles] | no | yes | 4.09 | 1.60 | 8.88 | 1.90 | 0.71 | 5.69
Li _et al_[Li et al.(2018)Li, Yao, Pan, Chao, and Mei] | no | yes | 4.51 | 1.71 | 9.31 | 2.05 | 0.74 | 6.14
Zhou _et al_[Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and Xiong] | no | yes | 5.76 | 2.71 | 11.16 | 2.91 | 1.44 | 6.91
Wang _et al_[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | no | yes | – | – | 10.89 | 2.27 | 1.13 | 6.10
Mun _et al_[Mun et al.(2019)Mun, Yang, Ren, Xu, and Han] | no | yes | – | – | – | – | – | 6.92
Iashin _et al_[Iashin and Rahtu(2020)] | no | no | 4.52 | 1.98 | 11.07 | 2.53 | 1.01 | 7.46
Rahman _et al_[Rahman et al.(2019)Rahman, Xu, and Sigal] | no | no | 3.04 | 1.46 | 7.23 | 1.85 | 0.90 | 4.93
Ours | no | no | 4.63 | 1.99 | 10.90 | 3.84 | 1.88 | 8.44
Table 1: Comparison with state-of-the-art results on the dense video
captioning task. The results are reported on the validation subset of
ActivityNet Captions in both settings: captioning ground truth (GT) and
learned proposals on BLEU@3–4 (B@3–4) and METEOR (M) metrics. For a fair
comparison on METEOR, we additionally report the results of models without the
reward (METEOR) maximization (RL) and indicate whether full dataset was
available for training. The best and the 2${}^{\text{nd}}$ best results are
highlighted.

We present the comparison between the bi-modal transformer with multi-headed
proposal generator (Ours) and other methods in the existing literature
[Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Carlos Niebles, Li et
al.(2018)Li, Yao, Pan, Chao, and Mei, Zhou et al.(2018b)Zhou, Zhou, Corso,
Socher, and Xiong, Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu, Xiong et
al.(2018)Xiong, Dai, and Lin, Mun et al.(2019)Mun, Yang, Ren, Xu, and Han,
Iashin and Rahtu(2020), Rahman et al.(2019)Rahman, Xu, and Sigal] on the dense
video captioning task. The results of the comparison for captioning both
ground truth (GT) and learned proposals are shown in Tab. 1. Since evaluating
captioning is still challenging and METEOR is probably the best among other
options, yet it only provides a proxy for how good a caption is. Therefore we
believe that the direct optimization of METEOR using a reinforcement learning
technique (RL) might not necessarily result in a better caption. To this end,
we also include the results of [Li et al.(2018)Li, Yao, Pan, Chao, and Mei,
Mun et al.(2019)Mun, Yang, Ren, Xu, and Han] without the RL module. Moreover,
we obtained the results of [Iashin and Rahtu(2020)] on the same subset of
videos as we have since they additionally removed the videos with no speech
modality from the evaluation.

According to the results, in the learned proposals setup, our dense video
captioning model outperforms all of the models, which have no reward
maximization on METEOR (no RL) while being on par when captioning ground truth
proposals. Notably, our model has the highest BLEU metrics in the learned
proposal setup yet lies far away from [Zhou et al.(2018b)Zhou, Zhou, Corso,
Socher, and Xiong] when captioning ground truth proposals on BLEU and performs
on par with this model on METEOR.

Comparing to the RL methods, our model still outperforms them on BLEU metrics
in both setups but loses in METEOR due to the absence of reward-maximization
module. We draw the attention of a reader to the performance of [Li et
al.(2018)Li, Yao, Pan, Chao, and Mei] with and without the RL module — METEOR
has dropped significantly yet other metrics remained on the same level.

Interestingly, we also outperform [Iashin and Rahtu(2020)] who also use the
transformer in multi-modal setup yet has more parameters (149M vs 51M). We
note again that the results are not fair to neither of [Rahman et
al.(2019)Rahman, Xu, and Sigal, Iashin and Rahtu(2020)] and ours since models
have been trained on fewer videos.

| Full Dataset |  |  |
---|---|---|---|---
| was Available |  Prec. |  Rec. | F1
Xiong _et al_[Xiong et al.(2018)Xiong, Dai, and Lin] | yes | 51.41 | 24.31 | 33.01
Wang _et al_[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | yes | 44.80 | 57.60 | 50.40
Zhou _et al_[Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and Xiong] | yes | 38.57 | 86.33 | 53.31
Mun _et al_[Mun et al.(2019)Mun, Yang, Ren, Xu, and Han] | yes | 57.57 | 55.58 | 56.56
Ours | no | 48.23 | 80.31 | 60.27

Table 2: Comparison with state-of-the-art proposal generation methods on dense
video captioning task. Results are reported on the validation set of
ActivityNet Captions. Metrics: Precision, Recall, & F1-measure. The top-2 is
highlighted.

Next, we compare our bi-modal multi-headed proposal generation module with
other proposal generation modules from other dense video captioning models.
The results for [Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and Xiong] and
[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] are reported for 100
proposals per video. The results of the comparison are presented in Tab. 2.
Despite our model being trained on fewer videos, our proposal generation model
achieves state-of-the-art performance on the F1 metric. Specifically, our
model provides impressive ground truth segment coverage while being accurate
in its predictions.

###  4.2 Ablation Study

In this section, we show how the training procedure and modality impact the
final results. The results are presented in Tab. 3 for both settings:
captioning ground truth (performance of the captioning module) and leaned
proposal (full dense video captioning model).

##### Training Procedures

Our final model is trained in the following way. First, we train the
captioning model on the ground truth proposal. Second, we freeze the weights
on the encoder and train the proposal generator using the frozen encoder. The
final results are obtained by captioning the proposals obtained from the
trained proposal generator. Hence, the acronym “Cap $\rightarrow$ Prop” which
reads as: “the proposal generator is trained using the pre-trained encoder
from the captioning module”. We compare this training procedure to other two
methods: a) when both captioning and proposal generator modules are trained
separately and b) when, first, the proposal module is trained and, then, the
captioning module uses the pre-trained encoder with frozen weights during
training. This is the opposite of the training procedure used for the final
model, thus, abbreviated to “Prop $\rightarrow$ Cap”.

##### Different Sets of Modalities

The final model uses both audio and visual modalities to make predictions. We
compare the performance of a bi-modal model with uni-modal ones. Specifically,
for uni-modal settings, we employ the uni-modal transformer architecture
similar to one in [Iashin and Rahtu(2020)]. The difference between the hyper-
parameters used for the final model and the uni-modal transformer is in the
input dimension. For the uni-modal transformer, we follow the original paper
where the input is first embedded into $D_{q}$ dimension (see (14)) and
remains the same everywhere later. We select 1024 for visual-only and 128 for
audio-only transformers; the size of the pre-trained GloVe is projected with a
FC layer to match the size.

##### Results

We report every combination of the settings in Tab. 3. Specifically, we
observed that the captioning module does not benefit from the pre-training for
the proposal generation (“Prop $\rightarrow$ Cap” vs “Cap $\rightarrow$ Prop”
& “Separate”). The results of the learned proposal setting show the importance
of the pre-training but only in the “Cap $\rightarrow$ Prop” setting. Overall,
we claim that the captioning training does not benefit from utilizing the pre-
trained proposal generator’s encoder and, even, performs worse with it. While,
the proposal generator ends up with better performance if pre-trained
captioning module’s encoder is used.

The comparison of the cross-modal performance shows that using both modalities
(audio and visual) gives the best result in nearly all cases in both settings.
However, it is shown that the audio modality is the weakest among the three
implying that visual modality might contain a stronger signal for video
understanding. Nevertheless, the gap between the visual-only and bi-modal case
is consistent in all settings. This suggests that the audio still provides
essential cues for dense video captioning. More ablations studies can be found
in Sec. 6.4.

Training |  | GT Proposals | Learned proposals
---|---|---|---
Procedure | Modality | B@3 | B@4 | M | B@3 | B@4 | M
Separately | Audio | 2.85 | 1.14 | 8.81 | 2.50 | 1.11 | 6.89
Visual | 3.77 | 1.66 | 10.29 | 2.94 | 1.36 | 7.69
Bi-modal | 4.62 | 1.99 | 10.89 | 3.47 | 1.65 | 8.05
Prop $\rightarrow$ Cap | Audio | 2.59 | 0.99 | 8.81 | 2.23 | 0.93 | 6.88
Visual | 3.62 | 1.56 | 10.16 | 3.08 | 1.45 | 7.81
Bi-modal | 4.10 | 1.78 | 10.48 | 3.07 | 1.47 | 7.67
Cap $\rightarrow$ Prop | Audio | 2.85 | 1.14 | 8.81 | 2.58 | 1.15 | 6.98
Visual | 3.77 | 1.66 | 10.29 | 2.85 | 1.30 | 7.47
Bi-modal | 4.62 | 1.99 | 10.89 | 3.84 | 1.88 | 8.44
Table 3: The impact of training procedures and input modalities. We compare
the training procedure of the final model when the proposal generator uses the
pre-trained encoder on the captioning task (“Cap $\rightarrow$ Prop”) to an
opposite scenario (“Prop $\rightarrow$ Cap”), and the situation when both of
them are trained separately. The results are shown on validation sets of
ActivityNet Captions when captioning ground truth (GT) and learned proposals.

##  5 Conclusion

We believe that the handling of multiple modalities is under-explored in the
computer vision community. In this paper, we present a novel bi-modal
transformer with a bi-modal multi-headed proposal generation module showing
how audio might facilitate the performance of dense video captioning. We
perform our experimentation on the ActivityNet Captions dataset and achieve
state-of-the-art results on F1 and BLEU metrics. The the ablation study
results show that the proposed model provides an effective and elegant way of
fusing audio and visual features while outperforming the uni-modal
configurations in all settings.

Acknowledgments Funding for this research was provided by the Academy of
Finland projects 327910 & 324346\. We also acknowledge CSC – IT Center for
Science, Finland, for computational resources.

## References

  * [Ba et al.(2016)Ba, Kiros, and Hinton] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * [Ben-Younes et al.(2017)Ben-Younes, Cadene, Cord, and Thome] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome.  MUTAN: Multimodal tucker fusion for visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2612–2620, 2017.
  * [Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos Niebles] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles.  SST: Single-stream temporal action proposals.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2911–2920, 2017.
  * [Carreira and Zisserman(2017)] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6299–6308, 2017.
  * [Chung et al.(2014)Chung, Gulcehre, Cho, and Bengio] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In _NIPS 2014 Workshop on Deep Learning, December 2014_ , 2014.
  * [Das et al.(2013)Das, Xu, Doell, and Corso] Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso.  A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2634–2641, 2013.
  * [Denkowski and Lavie(2014)] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the ninth workshop on statistical machine translation_ , pages 376–380, 2014.
  * [Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  10.18653/v1/N19-1423.
  * [Duan et al.(2018)Duan, Huang, Gan, Wang, Zhu, and Huang] Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, and Junzhou Huang.  Weakly supervised dense event captioning in videos.  In _Advances in Neural Information Processing Systems_ , pages 3059–3069, 2018.
  * [Escorcia et al.(2016)Escorcia, Heilbron, Niebles, and Ghanem] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  DAPs: Deep action proposals for action understanding.  In _European Conference on Computer Vision_ , pages 768–784, 2016\.
  * [Gan et al.(2017)Gan, Gan, He, Pu, Tran, Gao, Carin, and Deng] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng.  Semantic compositional networks for visual captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5630–5639, 2017.
  * [Gemmeke et al.(2017)Gemmeke, Ellis, Freedman, Jansen, Lawrence, Moore, Plakal, and Ritter] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 776–780, 2017.
  * [Hao et al.(2018)Hao, Zhang, and Guan] Wangli Hao, Zhaoxiang Zhang, and He Guan.  Integrating both visual and audio cues for enhanced video caption.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Hershey et al.(2017)Hershey, Chaudhuri, Ellis, Gemmeke, Jansen, Moore, Plakal, Platt, Saurous, Seybold, et al.] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  Cnn architectures for large-scale audio classification.  In _2017 ieee international conference on acoustics, speech and signal processing (icassp)_ , pages 131–135, 2017.
  * [Hori et al.(2017)Hori, Hori, Lee, Zhang, Harsham, Hershey, Marks, and Sumi] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi.  Attention-based multimodal fusion for video description.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4193–4202, 2017.
  * [Iashin and Rahtu(2020)] Vladimir Iashin and Esa Rahtu.  Multi-modal dense video captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 958–959, 2020.
  * [Kingma and Ba(2014)] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * [Kojima et al.(2002)Kojima, Tamura, and Fukunaga] Atsuhiro Kojima, Takeshi Tamura, and Kunio Fukunaga.  Natural language description of human activities from video images based on concept hierarchy of actions.  _International Journal of Computer Vision_ , 50(2):171–184, 2002.
  * [Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Carlos Niebles] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In _Proceedings of the IEEE international conference on computer vision_ , pages 706–715, 2017.
  * [Lee et al.(2008)Lee, Hakeem, Haering, and Zhu] Mun Wai Lee, Asaad Hakeem, Niels Haering, and Song-Chun Zhu.  Save: A framework for semantic annotation of visual events.  In _2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1–8, 2008.
  * [Li et al.(2018)Li, Yao, Pan, Chao, and Mei] Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei.  Jointly localizing and describing events for dense video captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7492–7500, 2018.
  * [Liu et al.(2016)Liu, Anguelov, Erhan, Szegedy, Reed, Fu, and Berg] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  SSD: Single shot multibox detector.  In _European conference on computer vision_ , pages 21–37, 2016.
  * [Long et al.(2018)Long, Gan, De Melo, Wu, Liu, and Wen] Xiang Long, Chuang Gan, Gerard De Melo, Jiajun Wu, Xiao Liu, and Shilei Wen.  Attention clusters: Purely attention based local feature integration for video classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7834–7843, 2018.
  * [Miech et al.(2019)Miech, Zhukov, Alayrac, Tapaswi, Laptev, and Sivic] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2630–2640, 2019.
  * [Mun et al.(2019)Mun, Yang, Ren, Xu, and Han] Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bohyung Han.  Streamlined dense video captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6588–6597, 2019.
  * [Owens and Efros(2018)] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 631–648, 2018.
  * [Pan et al.(2017)Pan, Yao, Li, and Mei] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei.  Video captioning with transferred semantic attributes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6504–6512, 2017.
  * [Papineni et al.(2002)Papineni, Roukos, Ward, and Zhu] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, et al.] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  In _Advances in neural information processing systems_ , pages 8026–8037, 2019.
  * [Pei et al.(2019)Pei, Zhang, Wang, Ke, Shen, and Tai] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai.  Memory-attended recurrent network for video captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8347–8356, 2019.
  * [Pennington et al.(2014)Pennington, Socher, and Manning] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543, 2014.
  * [Rahman et al.(2019)Rahman, Xu, and Sigal] Tanzila Rahman, Bicheng Xu, and Leonid Sigal.  Watch, listen and tell: Multi-modal weakly supervised dense event captioning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 8908–8917, 2019.
  * [Redmon and Farhadi(2017)] Joseph Redmon and Ali Farhadi.  YOLO9000: better, faster, stronger.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7263–7271, 2017.
  * [Redmon and Farhadi(2018)] Joseph Redmon and Ali Farhadi.  YOLOv3: An incremental improvement.  _arXiv_ , 2018.
  * [Redmon et al.(2016)Redmon, Divvala, Girshick, and Farhadi] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * [Rennie et al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7008–7024, 2017.
  * [Rohrbach et al.(2017)Rohrbach, Torabi, Rohrbach, Tandon, Pal, Larochelle, Courville, and Schiele] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele.  Movie description.  _International Journal of Computer Vision_ , 123(1):94–120, 2017.
  * [Rohrbach et al.(2013)Rohrbach, Qiu, Titov, Thater, Pinkal, and Schiele] Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal, and Bernt Schiele.  Translating video content to natural language descriptions.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 433–440, 2013.
  * [Shi et al.(2019)Shi, Ji, Liang, Duan, Chen, Niu, and Zhou] Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou.  Dense procedure captioning in narrated instructional videos.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6382–6391, 2019.
  * [Srivastava et al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * [Sun et al.(2018)Sun, Yang, Liu, and Kautz] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8934–8943, 2018.
  * [Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4489–4497, 2015.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * [Venugopalan et al.(2015a)Venugopalan, Rohrbach, Donahue, Mooney, Darrell, and Saenko] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko.  Sequence to sequence-video to text.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4534–4542, 2015a.
  * [Venugopalan et al.(2015b)Venugopalan, Xu, Donahue, Rohrbach, Mooney, and Saenko] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko.  Translating videos to natural language using deep recurrent neural networks.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1494–1504, Denver, Colorado, May–June 2015b. Association for Computational Linguistics.  10.3115/v1/N15-1173.
  * [Vinyals et al.(2015)Vinyals, Fortunato, and Jaitly] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.  Pointer networks.  In _Advances in neural information processing systems_ , pages 2692–2700, 2015.
  * [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu.  Bidirectional attentive fusion with context gating for dense video captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7190–7198, 2018a.
  * [Wang et al.(2018b)Wang, Wang, Huang, Wang, and Tan] Junbo Wang, Wei Wang, Yan Huang, Liang Wang, and Tieniu Tan.  M3: Multimodal memory modelling for video captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7512–7520, 2018b.
  * [Wang et al.(2018c)Wang, Chen, Wu, Wang, and Yang Wang] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang.  Video captioning via hierarchical reinforcement learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4213–4222, 2018c.
  * [Wang et al.(2018d)Wang, Wang, and Wang] Xin Wang, Yuan-Fang Wang, and William Yang Wang.  Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 795–801, New Orleans, Louisiana, June 2018d. Association for Computational Linguistics.  10.18653/v1/N18-2125.
  * [Xiong et al.(2018)Xiong, Dai, and Lin] Yilei Xiong, Bo Dai, and Dahua Lin.  Move forward and tell: A progressive generator of video descriptions.  In _Proceedings of the European Conference on Computer Vision_ , pages 468–483, 2018.
  * [Xu et al.(2017)Xu, Yao, Zhang, and Mei] Jun Xu, Ting Yao, Yongdong Zhang, and Tao Mei.  Learning multimodal attention lstm networks for video captioning.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 537–545, 2017.
  * [Yan et al.(2019)Yan, Tu, Wang, Zhang, Hao, Zhang, and Dai] Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai.  STAT: spatial-temporal attention mechanism for video captioning.  _IEEE Transactions on Multimedia_ , 22(1):229–241, 2019.
  * [Yao et al.(2015)Yao, Torabi, Cho, Ballas, Pal, Larochelle, and Courville] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville.  Describing videos by exploiting temporal structure.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4507–4515, 2015.
  * [YouTube Data API()] YouTube Data API.  Video Categories.  https://developers.google.com/youtube/v3/docs/videoCategories, [Accessed 1 November 2019].
  * [Yu et al.(2016)Yu, Wang, Huang, Yang, and Xu] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu.  Video paragraph captioning using hierarchical recurrent neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4584–4593, 2016.
  * [Zhao et al.(2018)Zhao, Gan, Rouditchenko, Vondrick, McDermott, and Torralba] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 570–586, 2018.
  * [Zhao et al.(2017)Zhao, Xiong, Wang, Wu, Tang, and Lin] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2914–2923, 2017.
  * [Zhou et al.(2018a)Zhou, Xu, and Corso] Luowei Zhou, Chenliang Xu, and Jason J Corso.  Towards automatic learning of procedures from web instructional videos.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018a.
  * [Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and Xiong] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong.  End-to-end dense video captioning with masked transformer.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8739–8748, 2018b.
  * [Zhu and Rahtu(2020a)] Lingyu Zhu and Esa Rahtu.  Separating sounds from a single image.  _arXiv preprint arXiv:2007.07984_ , 2020a.
  * [Zhu and Rahtu(2020b)] Lingyu Zhu and Esa Rahtu.  Visually guided sound source separation using cascaded opponent filter network.  _arXiv preprint arXiv:2006.03028_ , 2020b.

##  6 Supplementary Material

###  6.1 Multi-headed Attention

##### Scaled dot-product attention

The notion of multi-headed attention is based on the idea of scaled dot-
product attention which is defined as follows

$\displaystyle\text{Attention}(Q,\,K,\,V)=\text{Softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d}}\bigg{)}V,$
(13)

where $\sqrt{d}$ is a scaling factor designed to keep the Softmax gradients in
a sufficient range, $Q,\,K,\,V$ are sequences of queries, keys, and values,
Softmax is applied row-wise.

##### Attention with Many Heads

The concept of multiple heads was introduced in [Vaswani et al.(2017)Vaswani,
Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] to allow a
model to learn $H$ distinct representation sub-spaces at each position while
preserving the same computation efficiency. An attention head is usually
presented as (13) with parametrized inputs

$\displaystyle\text{head}_{h}(q,k,v)=\text{Attention}(qW^{q}_{h},\,kW^{k}_{h},\,vW^{v}_{h}),\quad
h\in[1,H]$ (14)

where $q\in\mathrm{R}^{T_{q}\times D_{q}},\,k\in\mathrm{R}^{T_{k}\times
D_{k}},\,v\in\mathrm{R}^{T_{k}\times D_{k}}$ and
$W^{*}_{h}\in\mathrm{R}^{D_{*}\times D_{\text{in}}}$. Note that the inputs $k$
and $v$ are expected to have the same dimension $(T_{k}\times D_{k})$ while
$q$ might have a different one. The weights $W^{*}_{h}$ are mapping the
corresponding inputs into an internal space $D_{\text{in}}=\frac{D_{q}}{H}$
such that $D_{q}$ is a multiple of $H$. The mapping into $D_{\text{in}}$ space
allows the attention to be calculated between the features which originally
were of distinct dimensions $(D_{q}\neq D_{k})$. The multi-headed attention
is, then, defined as the concatenation of $H$ attention heads mapped back to
sub-space of queries $(D_{q})$ with $W^{\text{out}}\in\mathrm{R}^{H\cdot
D_{\text{in}}\times D_{q}}$

$\displaystyle\text{MultiHeadAttention}(q,k,v)=\big{[}\text{head}_{1}(q,k,v),\,\text{head}_{2}(q,k,v),\,\dots,\,\text{head}_{H}(q,k,v)\big{]}W^{\text{out}}.$
(15)

###  6.2 Feature Extraction

Both audio and visual features are pre-calculated before training. The audio
features are extracted with the VGGish network [Hershey et al.(2017)Hershey,
Chaudhuri, Ellis, Gemmeke, Jansen, Moore, Plakal, Platt, Saurous, Seybold, et
al.], which was pre-trained on AudioSet [Gemmeke et al.(2017)Gemmeke, Ellis,
Freedman, Jansen, Lawrence, Moore, Plakal, and Ritter]. More specifically, the
VGGish model processes 0.96 seconds long segments. The audio segments, in
turn, are represented as log mel-scaled spectrograms of size $96\times 64$
which are obtained via Short-time Fourier Transform. The STFT utilizes a 25 ms
Hann window with 15 ms step applied to the 16 kHz mono audio track. The pre-
classification layer of VGGish outputs a 128-d embedding for each spectrogram.
Therefore, the audio track of an $i^{\text{th}}$ video in the dataset is
represented with a sequence of 128-d features of length $T^{i}_{a}$, each
feature in the stack represents 0.96 seconds of the original audio track.

To extract features from the visual stream, we employ the I3D network
[Carreira and Zisserman(2017)] pre-trained on Kinetics dataset. Specifically,
I3D inputs 64 RGB and 64 optical flow frames of size $224^{2}$ extracted at 25
fps. Similar to [Iashin and Rahtu(2020)], we extract the flow frames using
PWCNet [Sun et al.(2018)Sun, Yang, Liu, and Kautz]. Both sets of frames are,
first, resized such that $\min(\text{Height},\,\text{Width})=256$, and, then,
the central region of size $224^{2}$ is cropped. After, both stacks of frames
are passed through the corresponding streams of I3D. It outputs a 1024-d
representation for RGB and flow 64-frame stacks from the second-to-the-last
layer. Following the authors of I3D, we sum the representations from both
streams. It results in a single 1024-d representation for every stack of 64
frames. Therefore, the visual track of $i\text{th}$ video is represented with
a sequence of 1024-d features of length $T^{i}_{v}$ where every features spans
2.56 seconds (64 frames) of the original video.

The tokens (or, roughly, words) from captions are embedded with Global Vector
(GloVe) representations pre-trained on the Common Crawl dataset (2.2M
vocabulary) [Pennington et al.(2014)Pennington, Socher, and Manning]. The pre-
trained model is represented as a lookup table which maps a token to a 300-d
embedding. If a token is missing in the vocabulary, an average vector among
all vocabulary words is returned. Therefore, each previous token of a caption
is represented with a 300-d vector.

Therefore, the bi-modal encoder’s in and out dimensions are $d_{a}=128$ and
$d_{v}=1024$ for audio and visual streams while the decoder inputs and outputs
$d_{c}=300$.

###  6.3 Implementation Details

The batch of size 32 and 16 were used during training of captioning and
proposal generation modules, respectively. To form a batch, in the captioning
module, the features are padded up to the longest sequence in the batch. For
the proposal generator, the features are extracted from entire videos and
padded up to 300 for visual and 800 for audio to form a batch. These number
were selected to cover all possible lengths of the features in the training
set. The padding is masked out as it is done for the next caption tokens in
the decoder (see Sec. 3.3). Each head in the bi-modal multi-headed generator
predicts $\Psi_{a}=48$ and $\Psi_{v}=128$ anchors for audio and visual
modalities. We used the following lists each of size $K_{a}=K_{v}=10$ for
kernel sizes (given in cell-coordinates): $[5,13,23,35,51,69,91,121,161,211]$
for audio and $[1,5,9,13,19,25,35,45,61,79]$ for visual modalities which are
determined by K-Means algorithm. The size of both intermediate layers in
proposal generation head is 512. Note that
$\frac{128}{48}=\frac{800}{300}=\frac{2.56}{0.96}$ which preserves the balance
between predictions from both modalities (see Sec. 3.2).

Since the modality features might have a different size, we also need to map
them into an internal space inside of the bi-modal attention modules
$(D_{\text{in}})$, see Eq. (14) for more details. We select the internal space
to be of size $D_{\text{in}}=1024$. Both the encoder and decoder of the bi-
modal transformer have $N=2$ layers and $H=4$ heads in each of the multi-
headed attention modules. The caption vocabulary size and, hence, the
generator’s output dimension is 10 172. We use $\gamma=0.7$ in the label
smoothing and the probability of dropout $p=0.1$. The localization and
objectness loss coefficients are 1, and the noobjectness coefficient is 100.
Adam optimizer with default hyper-parameters [Kingma and Ba(2014)] and
learning rate $5\cdot 10^{-5}$ is used to train both caption and proposal
generator. The hyper-parameters are selected on the validation set.

We highlight that the whole process of training both parts of the model was
designed to keep a unified training procedure avoiding using different
techniques such as reduce-on-plateau, weight decay, different learning rate,
optimizer when training the proposal generator, sometimes, favoring elegance
at the cost of performance. We encourage others to try different combinations
when training both stages to achieve better results.

The captioning module was trained until METEOR on the validation set has not
improved for 30 epochs while the proposal generator is trained for 70 epochs
at most. In our experiments, the training of the final captioning module
reaches the peak performance at 26${}^{\text{th}}$ epoch while the proposal
generator achieves the highest F1-score on the validation set at
17${}^{\text{th}}$ epoch. We select the proposals on the epoch with the
highest metric and caption them with the best captioning model. The training
of the captioning module until the best performance takes 10 hours and 3.5
hours to train the proposal generator on one Nvidia GeForce RTX 2080Ti. We use
PyTorch [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, et al.] as our primary library for the
implementation.

| Validation | B@3 | B@4 | M | Recall |  Prec. | F1
---|---|---|---|---|---|---|---
Wang _et al_[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | Full | 2.27 | 1.13 | 6.10 | 57.60 | 44.80 | 50.40
As ours | 2.29 | 1.15 | 6.14 | 57.86 | 44.88 | 50.55
Zhou _et al_[Zhou et al.(2018b)Zhou, Zhou, Corso, Socher, and Xiong] | Full | 2.91 | 1.44 | 6.91 | 86.33 | 38.57 | 53.31
As ours | 2.92 | 1.45 | 6.92 | 86.33 | 38.55 | 53.30
Ours | Full | 3.50 | 1.72 | 7.69 | 73.22 | 43.97 | 54.95
As ours | 3.84 | 1.88 | 8.44 | 80.31 | 48.23 | 60.27
Table 4: The performance of other methods on the filtered ActivityNet Captions
validation set for videos which are no longer available (around 91 % (as
ours)). The results are reported in the learned proposal setting. As expected,
the performance of other models remains at the same level while ours gains the
missing 9 %. Metrics are BLEU3–4, METEOR, recall, precision, and F1-measure.

###  6.4 More Ablation Studies

####  6.4.1 Why Do You Exclude Videos from the Validation Set?

In our experimentation, we exclude videos which are no longer available on
YouTube (9 %) from the ground truth validation datasets as it would be unfair
to compare our model to the methods which could make a prediction based on the
video content while our model gets zero scores on a missing video. Therefore,
we evaluate the predictions made by other models [Zhou et al.(2018b)Zhou,
Zhou, Corso, Socher, and Xiong, Wang et al.(2018a)Wang, Jiang, Ma, Liu, and
Xu] on the same validation set as we have. We selected only these two methods
as they made either a code or evaluation results publicly available.

In other words, we hypothesise that the performance of other methods will not
change after excluding videos from both predictions and ground truth while the
performance of our method will be higher by around 9 % (a portion of the
missing videos). The results of the comparison are shown in Tab. 4 and,
indeed, imply that the performance of other methods remains on the same level
(less than 2 % change). We remind a reader that the compared methods were
trained on the full training dataset while ours was trained on only 91 %.

Caption | Proposal |  |  |  |  |  |
---|---|---|---|---|---|---|---
Module | Generator | B@3 | B@4 | M | Recall | Prec. | F1
Wang _et al_[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | 2.29 | 1.15 | 6.14 | 57.86 | 44.88 | 50.55
Ours | [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] | 2.87 | 1.41 | 7.03 | 57.86 | 44.88 | 50.55
Ours | Ours | 3.84 | 1.88 | 8.44 | 80.31 | 48.23 | 60.27
Table 5: The comparison of the captioning performance between our model and
[Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] on the learned proposals
provided in [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu]. The results are
reported on the filtered ActivityNet Caption validation datasets.

####  6.4.2 Might Your Model Improve Results of Other Methods?

Since [Wang et al.(2018a)Wang, Jiang, Ma, Liu, and Xu] have not made the
results publicly available for captioning ground truth (see Tab. 1), we cannot
compare it with our model directly. To this end, we apply our final captioning
model on the generated proposals from [Wang et al.(2018a)Wang, Jiang, Ma, Liu,
and Xu] to eliminate the effect caused by different proposal generator
modules. The results of the comparison are reported on the filtered
ActivityNet Caption validation datasets (see Sec. 6.4.1) and shown in Tab. 5.
The results suggest that our model has a better captioning performance on this
set of metrics.

Figure 4: The performance comparison between different modalities (Audio-only,
Visual-only, and Bi-modal) in two settings (ground truth and learned
proposals) across different YouTube video categories. The video categories are
sorted according to the performance of the Audio-Visual model in the learned
proposal setup. The number of videos in a category is shown in brackets.
ActivityNet Captions validation subset is used for the comparison.

####  6.4.3 What is the Impact of Audio and Visual Cues across Different Video
Types?

Following [Iashin and Rahtu(2020)], we inspect if the final model’s
performance consistently improves across different types of videos. To form a
list of video types, we retrieve a YouTube video category for each video in
the validation dataset. The YouTube category is annotated by the author when
they upload a video to the service. YouTubeAPI [YouTube Data API()] was used
to retrieve the categories automatically. Since there was a time gap between
downloading videos and their categories, 67 were no longer available. Such
videos were removed from the comparison. Also, we removed one video category
with less than 20 videos.

Fig. 4 shows the performance comparison between bi-modal (final), audio-only,
and visual-only models across different video categories in two settings:
captioning ground truth and learned proposals. The results suggest the
consistent gain in performance when both modalities are used compared to the
uni-modal models. This pattern holds across both settings and all categories.
In addition, it appears that the visual modality provides more cues to the
model than the audio modality in nearly all cases. Moreover, the dataset seems
to be biased to “Sports” and “People & Blogs” videos, which hold almost half
of the dataset. Yet, the results show no evidence of over-fitting to these
categories. Among all categories, “Music” appears to be the “easiest” one,
which might be explained by a small variety of ways to describe the content of
this kind. Meanwhile, the models perform the worst on “Gaming” and “Nonprofits
& Activism” categories, which might occur because of the lack of such videos
in the dataset.

2${}^{\text{nd}}$ Encoder’s Sub-layer | B3 | B4 | M
---|---|---|---
Self-Attention | 3.60 | 1.74 | 8.14
Bi-modal Attention | 3.84 | 1.88 | 8.44

Table 6: The effect of replacing the bi-modal attention with a self-attention
module in encoder layers. The comparison in shown on validation subsets of
ActivityNet Captions in the learned proposal setting. The metrics are BLEU3–4
and METEOR.

####  6.4.4 What Happens if the Bi-Modal Attention Block is Replaced by Uni-
modal Self-Attention?

In this ablation study, we would like to estimate the influence of the bi-
modal attention blocks on the model performance (see middle blocks of Encoder
and Decoder layers in Fig. 2). Yet, we can ablate only the encoder as the bi-
modal attention is essential for the decoder since it inputs two streams. One
solution would be to fuse the outputs of the encoder. This would, in turn,
allow us to replace two bi-modal attention blocks in the decoder with one.
However, it is not possible since the temporal spans of the encoder’s output
streams, in general, are distinct ($A^{v}\in\mathrm{R}^{T_{a}\times d_{a}}$
and $V^{a}\in\mathrm{R}^{T_{v}\times d_{v}}$). Therefore, in this setting,
each encoder layer has two pairs of self-attention blocks which preserves the
final model’s number of parameters.

The results are presented in Tab. 6. We observed a substantial decline in
performance among all metrics when the bi-modal attention block is replaced by
self-attention. This further suggests the importance of the proposed approach.
Besides, the results of the model with self-attention in encoder layers are
still much stronger than any V- and A-only models (see Tab 3), which proves
the importance of an audio-visual approach to the task.

###  6.5 Qualitative Analysis

Figure 5: The results of the qualitative analysis for a video from ActivityNet
Caption validation dataset. The predictions of our bi-modal model are compared
to the uni-modal model predictions and ground truth (GT) annotations. The
video shows a man who explains how to do a martial art movement—the YouTube
video id EIibo7aTpys.

Fig. 5 provides the qualitative analysis of the final captioning model
compared to ground truth captions. Additionally, we provide captions produced
by uni-modal captioning models (audio- and visual-only). The results show that
the caption, produced by a bi-modal captioning module as well as the audio-
only model managed to grasp the concept of talking when captioning the largest
segment (the top one) while video-only model neglects it. The video, by
itself, consists of an explanation of how to do a martial art movement and
highly verbose. Therefore, even though the ground truth does not mention that
the person talks during the video, the predictions of our final model are not
entirely erroneous. However, the colour of the man’s shirt is incorrectly
guessed, which might be explained by the presence of the black punching bag on
the screen. Finally, the caption produced by the visual-only model also makes
sense.

Moreover, if we consider the results of the audio-only model, we may notice
that it mostly gets the signal of “talking” and exploits it in a prediction.
Indeed, it might be challenging even for a non-English human to understand
what the video is about given only the audio track. We also notice that
captions provided by an annotator are significantly more detailed compared to
the predictions of the captioning model, which are somewhat more general. This
is the issue which needs more attention in future research as it seems to be a
problem for any dense video captioning system.