#  TeachText: CrossModal Generalized Distillation for Text-Video Retrieval

Ioana Croitoru 1,2,* Simion-Vlad Bogolin1,2,* Marius Leordeanu2,3
Hailin Jin4 Andrew Zisserman1 Samuel Albanie1,5,$\dagger$ Yang
Liu1,6,$\dagger$
1Visual Geometry Group, Univ. of Oxford 2Inst. of Mathematics of the Romanian
Academy 3Univ. Politehnica of Bucharest
4Adobe Research 5Dept. of Engineering, Univ. of Cambridge 6Wangxuan Inst. of
Computer Technology, Peking Univ.

###### Abstract

In recent years, considerable progress on the task of text-video retrieval has
been achieved by leveraging large-scale pretraining on visual and audio
datasets to construct powerful video encoders. By contrast, despite the
natural symmetry, the design of effective algorithms for exploiting large-
scale language pretraining remains under-explored. In this work, we are the
first to investigate the design of such algorithms and propose a novel
generalized distillation method, TeachText, which leverages complementary cues
from multiple text encoders to provide an enhanced supervisory signal to the
retrieval model. Moreover, we extend our method to video side modalities and
show that we can effectively reduce the number of used modalities at test time
without compromising performance. Our approach advances the state of the art
on several video retrieval benchmarks by a significant margin and adds no
computational overhead at test time. Last but not least, we show an effective
application of our method for eliminating noise from retrieval datasets. Code
and data can be found at https://www.robots.ox.ac.uk/~vgg/research/teachtext/.

##  1 Introduction

00footnotetext:  *Equal contribution. $\dagger$Corresponding authors.

The focus of this work is text-video retrieval—the task of identifying which
video among a pool of candidates best matches a natural language query
describing its content. Video search has a broad range of applications across
domains such as wildlife monitoring, security, industrial process monitoring
and entertainment. Moreover, as humanity continues to produce video at ever-
increasing scale, the ability to perform such searches effectively and
efficiently takes on critical commercial significance to video hosting
platforms such as YouTube.

Figure 1: Distilling the knowledge from multiple text encoders for stronger
text-video retrieval. Prior works [42, 36, 20] have shown the considerable
benefit of transitioning from video encoders that ingest a single modality
(left) to multi-modal video encoders (centre). In this work, we show that
retrieval performance can be further significantly enhanced by learning from
multiple text encoders through the TeachText algorithm which imposes no
additional cost during inference. Text-to-video retrieval performance gain
(geometric mean of R1-R5-R10) is reported for a [36] model as well as for our
method on the MSR-VTT [72] dataset.

A central theme of recently proposed retrieval methods has been the
investigation of how to best use multiple video modalities to improve
performance. In particular, architectures based on mixtures-of-experts [42,
36] and multi-modal transformers [20] have shown the benefit of making use of
diverse sets of pre-trained models for related tasks (such as image
classification, action recognition and ambient sound classification) as a
basis for video encoding during training and testing.

In this work, we explore whether commensurate gains could be achieved by
leveraging multiple text embeddings learned on large-scale written corpora.
Different from video embeddings using multiple modalities and pretraining
tasks, it is less obvious that there is sufficient diversity among collections
of text embeddings to achieve a meaningful boost in performance. In fact, our
inspiration stems from a careful investigation of the performance of different
text embeddings across a range of retrieval benchmarks (Fig. 2). Strikingly,
we observe not only that there is considerable variance in performance across
text embeddings, but also that their ranking is not consistent, strongly
supporting the idea of using multiple text embeddings.

Motivated by this finding, we propose a simple algorithm, TeachText, to
effectively exploit the knowledge captured by collections of text embeddings.
Our approach requires a student model to learn from a single or multiple
teacher retrieval models with access to different text embeddings by
distilling their text-video similarity matrices into an enhanced supervisory
signal. As shown in Fig. 1, TeachText is capable of delivering a significant
performance gain. Moreover, this gain is complementary to that of adding more
video modalities to the video encoder but importantly, unlike the addition of
video modalities, does not incur additional computational cost during
inference.

Our main contributions can be summarised as follows: (1) We propose the
TeachText algorithm, which leverages the additional information given by the
use of multiple text encoders; (2) We show that directly learning the
retrieval similarity matrix between the joint query video embeddings, which to
the best of our knowledge is novel, is an effective generalized distillation
technique for this task (and we compare our approach to alternatives among
prior work such as uni-modal relationship distillation [47]); (3) We show an
application of our approach in eliminating noise from modern training datasets
for the text-video retrieval task; (4) We demonstrate the effectiveness of our
approach empirically, achieving state of the art performance on six text-video
retrieval benchmarks.

##  2 Related Work

Video retrieval methods. The task of indexing video content to enable
retrieval has a rich history in computer vision—sophisticated systems have
been developed to find specific objects [56], actions [34], predefined
semantic categories [28], irregularities [4] and near-duplicates [15, 54]. In
this work, we focus on the task of retrieving content that matches a given
natural language description. For this particular task, there has been
considerable interest in developing cross-modal methods that employ a joint-
embedding space for text queries and video content [2, 73, 17, 45, 74, 70, 3].
These joint video-text embeddings, which aim to map videos and text
descriptions into a common space such that matching video and text pairs are
close together, form an attractive computational model for tackling this
problem, since they allow for efficient indexing (although hierarchical
embeddings have also been investigated [14]). Recently, two key themes have
emerged towards improving the quality of these embeddings. First, large-scale
weakly supervised pretraining methods [43, 41, 31] have sought to expand their
training data by exploiting the speech contained in the videos themselves as a
supervisory signal. Second, the integration of multiple modalities (which has
long been considered important for semantic indexing [57]) has been shown to
yield significant gains in performance [42, 36, 20, 49]. We focus on
candidates from this latter theme as a basis for investigating our approach.

Text embeddings. The representation of language through learned embeddings has
been widely studied [44, 50, 51] and applied in a variety of natural language
processing applications. Several works have demonstrated that even with large-
scale pretraining, there still are benefits to finetuning the models on the
target task [50, 16] and that larger models (often employing multiple
attention heads) yield higher performance [16]. Recently, [8] provided a
detailed comparisons on the importance of language features for vision
applications and proposes a word embedding that is specifically designed for
vision tasks. In this work, we first study how various pretrained language
embeddings affect the performance for text-video retrieval and then propose a
method to take advantage of the benefits of combining multiple text
embeddings.

Knowledge Distillation/Privileged Information. The purpose of knowledge
distillation is to transfer knowledge from one model (teacher) to another
model (student). This idea was originally introduced in the context of
decision tree simplification [6] and model compression [7], and later extended
by [24] who formalised this knowledge transfer as the temperature-
parameterised process of knowledge distillation. The concept was further
generalised in the unifying framework of generalized distillation [39] for
learning with privileged information [64] (via similarity control and
knowledge transfer [63]), together with knowledge distillation [24]. Our
approach distills knowledge of the similarities between video and text samples
into the student and therefore represents a form of generalized distillation.
While most knowledge distillation methods train the student with the teacher’s
outputs as targets, more recent methods propose different approaches [53, 75,
27]. Of most relevance to our approach, [47] transfer mutual relations of data
examples and propose distance-wise and angle-wise distillation losses that
penalize structural differences in relations instead of training the student
to mimic the output of the teacher—we compare to their approach in Sec. 5.

##  3 Motivation and intuition

Figure 2: Influence of varying the text embedding. Different text embeddings
are presented on the x axis: w2v [44], mt_grovle [8], openai-gpt [50], roberta
[37], albert [33], gpt2-large [51], gpt2-xl [51], gpt2-xl-F along with their
performance in geometric mean of $R1$-$R5$-$R10$ on five datasets. For each
experiment, we report the mean (diamond) and standard deviation (error bar) of
three randomly seeded runs.This study is performed using the CE retrieval
architecture [36]: each model differs only in its use of pre-trained text
embedding at input. We observe a significant variance in performance when
changing the text embedding, both across and within datasets. The difference
in rankings across datasets suggests the presence of additional information
among different text embeddings.

Figure 3: Share of correctly retrieved samples based on the used pre-trained
text embedding on MSR-VTT. We observe that each embedding has a considerable
share of sample retrieved correctly only by itself (in terms of R1 left and R5
right), further justifying our approach. Best viewed in color.

Recently, [51] points out that even though language representation learning
systems (such as [37, 33, 50]) are pre-trained on vast amounts of data, they
are still sensitive to slight changes in the data distribution and task
specification. In this way, most systems can be viewed as narrow experts
rather than competent generalists.

Consequently, in Fig. 2 we investigate how the usage of different off-the-
shelf pre-trained text embeddings affects the retrieval performance. We
observe that there is significant variance both within and across datasets,
suggesting that each embedding captures different types of information. Our
intuition is that this information comes from the diversity of architectures,
pretraining datasets and pretraining objectives, which differs across the text
embeddings.

Next, we give details about the used text embeddings and summarise the key
differences between them in relationship with our findings. Word2vec (w2v)
[44] is a lightweight text embedding that is widely used for vision tasks [35,
11, 68]. Multi-task GrOVLE (mt_grovle) [8], is an extension of w2v that is
specially designed for vision-language tasks (in our experiments, however, we
find that it slightly under-performs w2v). The finetuned transformer language
model (openai-gpt) [50] embedding is trained on a book corpus containing long
stretches of contiguous text. We observe that it performs well on datasets
that have longer text queries such as ActivityNet. RoBERTa and ALBERT [37, 33]
are based on the BERT architecture [16] and are trained on the same data which
consists of unpublished books and Wikipedia articles. RoBERTa [37] focuses on
hyperparameter optimization and shows that greater model capacity leads to
better performance while ALBERT[33] proposes some parameter-reduction
techniques to reduce memory consumption and increase training speed. In our
experiments, we observe a high variation in performance when comparing the
two. In contrast to the other embeddings, gpt2[51] is trained on a crawled
dataset that was designed to be as diverse as possible. We observe that gpt2
performs most robustly in our experiments, especially on smaller datasets such
as MSR-VTT and MSVD. However, it nevertheless exhibits a domain gap to each
corpus (highlighted by the fact that performance increases when fine-tuning
gpt2-xl, termed gpt2-xl-F throughout the paper, on queries from the text-video
retrieval datasets).

Additionally, in Fig. 3 we show how many correctly retrieved queries are
shared between three text embeddings on MSR-VTT: gpt2-xl, gpt2-xl-F and w2v.
Only around 19% (R1), respectively 42% (R5) queries are correctly retrieved by
all the three considered text embeddings. This means that a significantly
number of queries are sensitive to the used text embedding, consolidating our
intuition.

##  4 Method

Figure 4: TeachText teacher-student framework overview. Given a batch of input
videos and queries in natural language during training, the student model, $M$
(left) and teacher models $T_{1},\dots,T_{N}$ (right) each produce similarity
matrices (visualised as square grids). The similarity matrix produced by $M$
is encouraged to match the aggregated matrices of the teachers through the
distillation loss $\mathcal{L}_{d}$ in addition to the retrieval loss
$\mathcal{L}_{r}$. Note that both the student and teachers ingest the same
video embeddings (VE), but employ different text embeddings (TES for the
student, TE${}_{1},\dots,$TEN for the teachers). At test time, the teacher
models are discarded.

Motivated by the findings from Sec. 3, our work aims to study the influence of
using multiple text embeddings for text-video retrieval.

###  4.1 Problem description and learning setup

Let $D=\\{(v_{i},c_{i})\\}_{i=1}^{n}$ be a dataset of paired videos and
captions. Following the multi-modal experts approach of [42, 36, 20], for each
video we have access to a collection of video embeddings (sometimes referred
to as experts) $x_{i}$ extracted from the various modalities of video $v_{i}$
using a pretrained video encoder (VE) in addition to a text embedding $t_{i}$
(extracted using a text encoder, TE) for each caption/query $c_{i}$111These
embeddings are produced by models that have been trained on relevant tasks
(such as action recognition for the video encoder and language modelling for
the text encoder). The objective of the text-video retrieval task is to learn
a model $M(x_{i},t_{j})$ which assigns a high similarity value to pairings
$(x_{i},t_{j})$ of video and text embeddings that are in correspondence (i.e.
$i=j$) and a low similarity otherwise. As is common in the literature [5, 42],
we parameterise the model as a dual-encoder that produces joint-embeddings in
a shared space such that they can be compared directly
$M(x_{i},t_{j})=F(x_{i})^{T}Q(t_{j})\in\mathbb{R}$ where $F$ and $Q$ represent
the learnt video and text encoder respectively. To train the video and text
encoder for the task of retrieval, we adopt a contrastive ranking loss [58]:

$\begin{aligned} \mathcal{L}_{r}=\frac{1}{B}\sum_{i=1}^{B}\sum_{i\neq
j}\left[max(0,s_{ij}-s_{ii}+m)+\right.\\\
\left.max(0,s_{ji}-s_{ii}+m)\right]\end{aligned}\mbox{}\vspace{-0.15cm}\\\ $
(1)

where $B$ represents the batch size used during training,
$s_{ij}=F(x_{i})^{T}Q(t_{j})$ is the similarity score between the encoded
video $F(x_{i})$ and query $Q(t_{j})$ while $m$ is the margin.

The key idea behind our approach is to learn a retrieval model, $M$, that, in
addition to the loss described above, also has access to information provided
by a collection of pre-trained teacher retrieval models which are trained on
the same task but ingest different text embeddings.

###  4.2 TeachText algorithm

To enhance the retrieval performance of model $M$, we propose the TeachText
algorithm which aims to exploit cues from multiple text embeddings. An
overview of our approach is provided in Fig. 4. Initially, we train a
collection of teacher models $\\{T_{k}:k\in\\{1,\dots,N\\}\\}$ for the text-
video retrieval task using the approach described in Sec. 4.1. The teachers
share the same architecture but each model $T_{k}$ uses a different text
embedding as input (extracted using a pre-trained text encoder TEk). In the
second phase the parameters of the teachers are frozen. We then proceed by
sampling a batch of $B$ pairs of videos and captions and computing a
corresponding similarity matrix $S_{k}\in\mathbb{R}^{B\times B}$ for each
teacher $T_{k}$ (Fig. 4 right). These $N$ similarity matrices are then
combined with an aggregation function, $\Phi:\mathbb{R}^{N\times B\times
B}\rightarrow\mathbb{R}^{B\times B}$, to form a single supervisory similarity
matrix (Fig. 4, centre-right). Concurrently, the batch of videos and captions
are likewise processed by the student model, $M$, which produces another
similarity matrix, $S_{s}\in\mathbb{R}^{B\times B}$. Finally, in addition to
the standard retrieval loss (Eq. 1), a distillation loss, $\mathcal{L}_{d}$,
encourages the $S_{s}$ to lie close to the aggregate
$\Phi(S_{1},\dots,S_{N})$. The algorithm is summarized in Alg. 1. During
inference, the teacher models are discarded and the student model $M$ requires
only a single text embedding. Next, we give details of the distillation loss
used for the similarity matrix learning.

Algorithm 1 TeachText algorithm

1: Phase 1: Learn teacher models

2: Train $N$ teacher models $T_{k}=(F_{k},Q_{k})$, $k\in\\{1,\dots,N\\}$ using
the training pairs $(x_{i},t_{j}^{k})$ where $t_{j}^{k}$ represents the text
modality used by teacher $T_{k}$ in a standard retrieval training setup (Sec.
4.1).

3: Phase 2: Learn the student model, $M=(F,Q)$

4: for minibatch of $B$ paired samples $\\{(v_{i},c_{i})\\}$ do

5: For each pair $(v_{i},c_{i})$ extract video experts and text embedding
pairs $(x_{i},t_{i})$ using VE and $TE_{S}$.

6: Compute student similarity matrix $S_{s}$ where
$S_{s}(i,j)=F(x_{i})^{T}Q(t_{j})$ for $i,j\in\\{1,\dots,B\\}$

7: Compute the loss $\mathcal{L}_{r}$ via Eqn. 1 using $S_{s}$.

8: for teacher $T_{k}$, $k=1,\dots,N$ do

9: For each pair $(v_{i},c_{i})$ extract the video experts and text embedding
pairs $(x_{i},t_{i}^{k})$ using $VE$ and $TE_{k}$.

10: Compute the similarity matrix $S_{k}$ where
$S_{k}(i,j)=F_{k}(x_{i})^{T}Q_{k}(t_{i}^{k})$ for $i,j\in\\{1,\dots,B\\}$.

11: end for

12: Compute aggregate teacher matrix $\Phi(S_{1},\dots,S_{N})$.

13: Compute the loss $\mathcal{L}_{d}$ between $S_{s}$ and
$\Phi(S_{1},\dots,S_{N})$ via Eqn. 2.

14: Update $M$ with gradients computed from the composite loss
$\mathcal{L}=\mathcal{L}_{r}+\mathcal{L}_{d}$.

15: end for

###  4.3 Learning the similarity matrix

As noted in Sec. 4.1, the essence of the retrieval task is to create a model
that is able to establish cross-modal correspondences between videos and
texts/queries, assigning a high similarity value to a pairing in which a query
accurately describes a video, and a low similarity otherwise. This renders the
similarity matrix a rich source of information about the knowledge held by the
model. In order to be able to transfer knowledge from the teachers to the
student, we encourage the student to produce a similarity matrix that matches
an aggregate of those produced by the teachers. In this way, we convey
information about texts and video correspondences without strictly forcing the
student to produce exactly the same embeddings as the teachers. To this end,
we define the similarity matrix distillation loss as:

$\mathcal{L}_{d}=\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B}\left[\textit{l}(\Phi(i,j),S_{s}(i,j))\right]\mbox{}\vspace{-0.2cm}\\\
$ (2)

where $B$ represents the batch size, $\Phi=\Phi(S_{1},\dots,S_{N})$ represents
the aggregate of the teacher similarity matrices and $S_{s}$ represents the
similarity matrix of the student. Finally, inspired from other distillation
works such as [47], l represents the Huber loss and is defined as

$\textit{l}(x,y)=\begin{cases}\frac{1}{2}(x-y)^{2}\quad\quad\text{if
}|x-y|\leq 1,\\\ |x-y|-\frac{1}{2}\quad\quad
otherwise\end{cases}\mbox{}\vspace{-0.2cm}\\\ $ (3)

We explored several forms of aggregation function and found that a simple
element-wise mean, $\Phi(S_{1},\dots,S_{N})=\frac{1}{N}\sum_{k=1}^{N}S_{k}$,
worked well in practice.

The idea of learning directly the cross-modal similarity matrix is, to the
best of our knowledge novel. It draws inspiration from the work of relational
knowledge distillation [47] which considered the idea of learning from
relationships and introduced two algorithms to implement this concept in a
uni-modal setting through pairwise and triplet distance sampling. We compare
our matrix learning approach with theirs in Sec. 5.

###  4.4 Student model

A key advantage of our approach is that it is agnostic to the architectural
form of the student and teachers, and thus the student (and teachers) can
employ any method from the current literature. We test our TeachText algorithm
using three different recent works MoEE [42], CE [36], MMT [20] as the student
and teacher base architectures. All these works employ multi-modal video
encoders for the text-video retrieval task. For more details, please consult
the original paper of each method.

Establishing a stronger baseline. In addition to these models, we also
investigate our approach on a model which shares the CE architecture of [36]
but includes a series of small technical improvements to provide a stronger
baseline against which we also test the TeachText algorithm. Starting from
this base architecture, we refine the input embedding selection, finding that
the face and OCR video modalities employed by [36] do not consistently produce
improvement so we remove them as inputs to the video encoder. We update the
model to use the more powerful gpt2-xl text embedding of [51] and following
[20], we finetune this text embedding on captions from the target dataset to
bring additional improvement. Combining all of these changes (ablations
provided in Sec. 5.3 and Fig. 5a) results in the CE+ model which we include as
an additional baseline. Thus, in summary we use four ([42, 36, 20] and CE+)
different base architectures for the student model.

###  4.5 Teacher models

The teacher models use the same architecture as the student model. Concretely,
for each of the four base architectures described in Sec. 4.4, we create a
pool of multiple teachers, each using a different pre-trained text embedding
as input. The candidate text embeddings we consider are: mt_grovle [8],
openai-gpt [50], gpt2-large [51], gpt2-xl [51], w2v [44]. So, we obtain a set
of up to five models that form the teachers $T_{k}$, $k=1..5$ used by
TeachText.

###  4.6 Training and implementation details

In order to train our final student, we combine the retrieval loss and the
proposed distillation loss $\mathcal{L}=\mathcal{L}_{r}+\mathcal{L}_{d}$. Our
model is trained in Pytorch [48] using the Adam [29] optimizer. TeachText does
not add any additional trainable parameters or modalities to the final model.
Moreover, when training the student using TeachText, only the additional loss
term $\mathcal{L}_{d}$ is added, all other hyper-parameters remaining the
same.

##  5 Experimental setup

Figure 5: (a) Baseline improvements. The y-axes (scaled for clarity) denotes
retrieval performance on MSR-VTT. We begin by presenting the performance of
the original CE [36]. Firstly, we correct compression artefacts in the pre-
processing used for embedding extraction (CE HQ, more details in Suppl. Mat.).
Secondly, we refine the used video modalities and text modalities (Mod ref and
Text ref). Finally, we finetune the text embedding (F) and change the
optimizer to Adam [29], thus obtaining the CE+ baseline. (b) Use additional
text embeddings at inference time. All experiments were performed with the
same architecture [36], but with different text embeddings: gpt2-xl (first
bullet), gpt2-xl-F (second bullet), the concatenation of gpt2-xl and gpt2-xl-F
(third bullet), the mean of gpt2-xl and gpt2-xl-F (fourth bullet) and using
TeachText (last bullet). By using multiple text embeddings at test time, which
introduces an overhead, a boost in performance is obtained. However, by using
TeachText there is no additional overhead at test time and the performance is
superior.

###  5.1 Datasets description

To provide an extensive comparison we test our approach on seven video
datasets that have been explored in recent works as benchmarks for the task of
text-video retrieval: LSMDC [52], DiDeMo [1], MSVD [13], MSRVTT [72],
ActivityNet [9], VaTeX [69] and QuerYD [46]. We follow the same experimental
setup as prior works [36, 20, 14, 49].

###  5.2 Metrics

To assess performance, we follow prior work (e.g [17, 42, 45, 74, 43, 36, 20])
and report standard retrieval metrics, including R@K (recall at rank K, where
higher is better) and MdR (median rank where lower is better). For certain
analyses, to maintain conciseness we report the geometric mean of R@1, R@5 and
R@10 rather than individual metrics (this statistic aims to be representative
of overall retrieval performance). The numbers are reported for the task of
retrieving a video given text queries t2v which is more common in real world
applications. The numbers for the reverse task v2t and the number of
parameters for each model are reported in the Suppl. Mat. For each experiment,
we report the mean and standard deviation of three randomly seeded runs.

###  5.3 Ablations

In this section we present an extensive study of our proposed approach.
Following the setup used in prior works [36, 20] we conduct ablations on the
MSR-VTT dataset [72], except where otherwise stated.

Model | MSRVTT | MSRVTT 1k-A | MSVD | DiDeMo | LSMDC | ActivityNet
---|---|---|---|---|---|---
Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText
MoEE | $24.4_{\pm 0.1}$ | $\mathbf{25.8}_{\pm 0.1}$ | $41.6_{\pm 0.4}$ | $\mathbf{43.4}_{\pm 0.6}$ | $41.8_{\pm 0.3}$ | $\mathbf{43.2}_{\pm 0.5}$ | $33.2_{\pm 1.4}$ | $\mathbf{40.2}_{\pm 0.7}$ | $23.8_{\pm 0.4}$ | $\mathbf{26.0}_{\pm 0.5}$ | $40.1_{\pm 0.3}$ | $\mathbf{45.2}_{\pm 0.1}$
CE | $24.4_{\pm 0.1}$ | $\mathbf{25.9}_{\pm 0.1}$ | $42.0_{\pm 0.8}$ | $\mathbf{43.8}_{\pm 0.3}$ | $42.3_{\pm 0.6}$ | $\mathbf{42.6}_{\pm 0.4}$ | $34.2_{\pm 0.4}$ | $\mathbf{39.5}_{\pm 0.5}$ | $23.7_{\pm 0.3}$ | $\mathbf{25.5}_{\pm 0.5}$ | $40.4_{\pm 0.3}$ | $\mathbf{45.0}_{\pm 0.6}$
MMT | - | - | $44.7_{\pm 0.4}$ | $\mathbf{45.6}_{\pm 0.7}$ | - | - | - | - | $24.6_{\pm 0.7}$ | $\mathbf{25.9}_{\pm 0.6}$ | $44.0_{\pm 0.4}$ | $\mathbf{47.9}_{\pm 0.4}$
CE+ | $29.2_{\pm 0.2}$ | $\mathbf{30.4}_{\pm 0.0}$ | $50.3_{\pm 0.2}$ | $\mathbf{50.9}_{\pm 0.4}$ | $46.5_{\pm 1.0}$ | $\mathbf{46.6}_{\pm 0.5}$ | $35.8_{\pm 0.4}$ | $\mathbf{40.4}_{\pm 0.4}$ | $28.1_{\pm 0.3}$ | $\mathbf{30.7}_{\pm 0.3}$ | $39.7_{\pm 0.0}$ | $\mathbf{46.3}_{\pm 0.2}$

Table 1: Method generality. Retrieval performance (geometric mean of
R1-R5-R10) on various datasets when applying TeachText on top of different
base models: MoEE[42], CE[36], MMT[20] (on available datasets) and CE+. We
present in bold cases where TeachText brings an improvement over the base
architecture. We observe that our method improves the performance for all
underlying base models and on all datasets.

Baseline improvements. We propose CE+ as an additional baseline which consists
of a series of technical improvements to the model of [36]. As seen in Fig. 5a
each modification described in Sec. 4.4 brings additional gain over the base
architecture. We observe in particular that finetuning the text embedding on
the target dataset has a high influence, further highlighting the critical
role played by text embeddings and justifying their study. In addition to
other changes we found that certain video embedding expert features were
highly sensitive to compression choices used in video pre-processing, which we
correct accordingly (more details in Suppl. Mat.). Please note that for a fair
comparison, in Sec. 5.4 we report the numbers of re-training the methods [42,
36] using these embeddings extracted with the updated pre-processing which
yields a higher performance than the ones reported in the original papers.
Using multiple text embeddings during inference. TeachText makes no use of
additional information at test time. However, it is natural to ask whether the
additional text embeddings can be trivially included as part of the model
architecture. In Fig. 5(b) we compare our approach with some relatively simple
text embedding aggregation techniques, which require access to multiple text
embeddings during both training and inference. We observe that TeachText
outperforms these aggregation techniques such as direct concatenation or mean
of the text embeddings, suggesting that the proposed method is effective in
capturing the additional information given by multiple text embeddings.
Moreover, the text encoder of existing systems [42, 36, 20] typically employs
many parameters, so adding multiple text embeddings to the architecture adds a
significant number of parameters (100M+). For example, the concatenation of
two text embeddings (provided that they have the same size) almost doubles the
total number of parameters for CE+. In contrast, when employing TeachText, no
parameters are added.
Teacher variation.

Figure 6: (a) Teacher study. We show the influence of learning from different
number of teachers on the MSR-VTT dataset (all students share the same CE+
model, y-axes scaled for clarity). The teachers were added in the following
order: gpt2-xl, w2v, gpt2-xl-F, mt_grovle, openai-gpt, gpt2-large. The
performance of the combined teachers grows as more teachers are added, however
it reaches a plateau after the first 3 teachers. The trend is similar for
student performance. (b) Distillation type. Presenting various alternatives
for distilling the information from the teacher: relational distillation [47]
which preserves intra-text and intra-video relationships, pairwise distance
distillation (Pdist \- adapting [47] for cross modal relationships), ranking
distillation inspired by [59] at Rank 1 and Rank 10 and TeachText. The first
bullet represents the student without distillation.

The teacher models share the same architecture with the student, but use a
different text embedding. We next conduct an ablation on the influence of the
number of used teachers. We observe in Fig. 6a that performance increases with
the addition of more teachers. Since the combined performance of the teachers
after adding more than 3 remains about the same, we do not obtain a further
improvement. Thus, for our final experiments presented in Sec. 5.4 we use a
combination of three teachers, having the following text embeddings: w2v [44],
gpt2-xl [51] and gpt2-xl-F (gpt2-xl finetuned on the captions from the target
dataset). A study of how each individual text embedding affects the final
performance can be found in the Suppl. Mat. section Teacher study, where we
observe that even when using a teacher with lower performance (w2v), the
student has a significant boost in performance.
Distillation ablation.

Figure 7: (a) Denoising. We present the effect of denoising on retrieval
performance on MSR-VTT (y-axes scaled for clarity). Some of the captions
available in datasets with multiple captions per video may be noisy and
actively harm the training process. We estimate the degree of noise present in
a caption by looking at the teacher rank and drop the caption if necessary. We
observe the effectiveness of denoising when applied in isolation (CE+ vs CE+
Denoise) and in conjunction with the full TeachText method. (b) TeachVideo \-
Extension to video side modalities. We observe that our method can be
effective in taking advantage of the additional information brought by using
multiple video side modalities, without incurring computational overhead at
test time.

We compare the proposed learning of the similarity matrix with other
distillation alternatives. As seen in Fig.6b, our proposed approach is
effective in capturing the relationships between video and text. We first
provide comparisons between TeachText and several possible instantiations of
relational distillation [47]. Indeed, given the highly general nature of [47],
TeachText can be interpreted within this framework as a particular relational
configuration that employs cross-modal distillation through batches of
similarity matrices. Since the original work of [47] considered single-
modality applications, we explore two variations of [47] as baselines for the
text-video retrieval task. The first one (Relational), preserves the same
intra-text and intra-video relationships independently. We use the same cost
function as in [47] and enforce it on both video and text embeddings. The
second approach (Pdist), uses the cross modal pairwise distances as a relation
measure between text and video as opposed to the similarity matrix. While
these methods indeed bring a gain, we observe that TeachText is more
effective.
We also provide a baseline inspired by the work of [59] which highlights the
importance of looking only at the top K predictions given by the teacher. To
do so, we enforce the same similarities using TeachText only for the top K
ranks given by the teacher rather than for the whole mini-batch. We show the
performance for K=1 and K=10 (Rank 1 and Rank 10 presented in Fig.6b).
Restricting to only top K predictions when distilling the similarity matrix
results in a slight drop in performance.
Method generality. To demonstrate the generality of TeachText, we test it
against three state of the art methods [42, 36, 20] in addition to the
proposed CE+ baseline. In Tab. 1 we observe a consistent gain in performance,
independent of the base architecture. Moreover, a gain is achieved across all
the datasets that we tested, having over 5% absolute gain on DiDeMo and
ActivityNet datasets for MoEE, CE and CE+ models. Note that for MMT [20] we
report results on the datasets included in the public implementation provided
by the authors222https://github.com/gabeur/mmt.
Method application – Denoising. One immediate application of our method is
data denoising. Existing real-world text-video datasets for the retrieval task
suffer from label noise which can harm training. More concretely, in crowd-
sourced datasets such as MSR-VTT there are some captions that are highly
ambiguous/generic (e.g ”A tutorial is presented”, ”Clip showing different
colours”, ”A man is writing”) and can describe multiple videos from the
dataset. We therefore propose to use TeachText teachers to filter out such
cases. For this scenario, we simply remove low-ranked predictions given by
teachers and re-train the student using only the new samples. Specifically, we
remove all sentences for which the correct video is not ranked in top 40 from
the training set. This method is best-suited for datasets where multiple
captions per video are available, ensuring that we can remove noisy captions
without removing the video itself from training. Following this, we apply the
denoising on MSR-VTT and MSVD datasets with the CE+ model. As seen in Fig. 7a,
this can be an effective way of further improving the results. Please note,
denoising is not used in any other ablations.
TeachVideo – Extension to video modalities. While the focus of this work is
the use of multiple text embeddings, it is natural to consider whether this
approach can be extended to the video encoder modalities. Thus, we introduce
the TeachVideo algorithm which follows the same setup as the original
TeachText, but now the teacher has access to multiple video modalities instead
of multiple text modalities. In this study, all students and all teachers use
the same text embedding, so we can assess the gains due to TeachVideo. By
employing TeachVideo we retain the computational advantage of requiring fewer
video modalities during inference. As it can be seen from our experiments
presented in Fig. 7b, the method is effective and brings a boost over the
original student. We believe this extension may be useful in scenarios in
which limited computational resources are available during inference.

Qualitative examples and other ablation studies are presented in Suppl. Mat.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
Dual[18] | $7.7$ | $22.0$ | $31.8$ | $32.0$
HGR[14] | $9.2$ | $26.2$ | $36.5$ | $24.0$
MoEE[42]333Please note that the numbers reported are higher than in the original paper due to compression artefacts correction. | $11.1_{\pm 0.1}$ | $30.7_{\pm 0.1}$ | $42.9_{\pm 0.1}$ | $15.0_{\pm 0.0}$
CE[36]3 | $11.0_{\pm 0.0}$ | $30.8_{\pm 0.1}$ | $43.3_{\pm 0.3}$ | $15.0_{\pm 0.0}$
TT-CE | $11.8_{\pm 0.1}$ | $32.7_{\pm 0.1}$ | $45.3_{\pm 0.1}$ | $13.0_{\pm 0.0}$
TT-CE+ | $\textbf{15.0}_{\pm 0.1}$ | $\textbf{38.5}_{\pm 0.1}$ | $\textbf{51.7}_{\pm 0.1}$ | $\textbf{10.0}_{\pm 0.0}$

Table 2: MSR-VTT full split: Comparison to state of the art.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
MoEE[42]3 | $21.6_{\pm 1.0}$ | $50.8_{\pm 1.1}$ | $65.6_{\pm 0.7}$ | $5.3_{\pm 0.6}$
CE[36]3 | $21.7_{\pm 1.3}$ | $51.8_{\pm 0.5}$ | $65.7_{\pm 0.6}$ | $5.0_{\pm 0.0}$
MMT[20] | $24.6_{\pm 0.4}$ | $54.0_{\pm 0.2}$ | $67.1_{\pm 0.5}$ | $4.0_{\pm 0.0}$
SSB[49] | $27.4$ | $56.3$ | $67.7$ | $\mathbf{3.0}$
TT-MMT | $24.8_{\pm 0.2}$ | $55.9_{\pm 0.7}$ | $68.5_{\pm 1.0}$ | $4.3_{\pm 0.5}$
TT-CE+ | $\mathbf{29.6}_{\pm 0.3}$ | $\mathbf{61.6}_{\pm 0.5}$ | $\mathbf{74.2}_{\pm 0.3}$ | $\mathbf{3.0}_{\pm 0.0}$

Table 3: MSR-VTT 1k-A split[74]: Comparison with others.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
VSE++[19] | $15.4$ | $39.6$ | $53.0$ | $9.0$
M-Cues[45] | $20.3$ | $47.8$ | $61.1$ | $6.0$
MoEE[42]3 | $21.1_{\pm 0.2}$ | $52.0_{\pm 0.7}$ | $66.7_{\pm 0.2}$ | $5.0_{\pm 0.0}$
CE[36]3 | $21.5_{\pm 0.5}$ | $52.3_{\pm 0.8}$ | $67.5_{\pm 0.7}$ | $5.0_{\pm 0.0}$
TT-CE | $22.1_{\pm 0.4}$ | $52.2_{\pm 0.5}$ | $67.2_{\pm 0.6}$ | $5.0_{\pm 0.0}$
TT-CE+ | $\mathbf{25.4}_{\pm 0.3}$ | $\mathbf{56.9}_{\pm 0.4}$ | $\mathbf{71.3}_{\pm 0.2}$ | $\mathbf{4.0}_{\pm 0.0}$

Table 4: MSVD: Comparison to state of the art methods.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
S2VT[67] | $11.9$ | $33.6$ | $-$ | $13.0$
FSE[77] | $13.9_{\pm 0.7}$ | $36.0_{\pm 0.8}$ | $-$ | $11.0_{\pm 0.0}$
MoEE[42]3 | $16.1_{\pm 1.0}$ | $41.2_{\pm 1.6}$ | $55.2_{\pm 1.6}$ | $8.3_{\pm 0.5}$
CE[36]3 | $17.1_{\pm 0.9}$ | $41.9_{\pm 0.2}$ | $56.0_{\pm 0.5}$ | $8.0_{\pm 0.0}$
TT-CE | $21.0_{\pm 0.6}$ | $47.5_{\pm 0.9}$ | $61.9_{\pm 0.5}$ | $\mathbf{6.0}_{\pm 0.0}$
TT-CE+ | $\mathbf{21.6}_{\pm 0.7}$ | $\mathbf{48.6}_{\pm 0.4}$ | $\mathbf{62.9}_{\pm 0.6}$ | $\mathbf{6.0}_{\pm 0.0}$

Table 5: DiDeMo: Comparison to state of the art methods.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
JSFus[74] | $9.1$ | $21.2$ | $34.1$ | $36.0$
MoEE[42]3 | $12.1_{\pm 0.7}$ | $29.4_{\pm 0.8}$ | $37.7_{\pm 0.2}$ | $23.2_{\pm 0.8}$
CE[36]3 | $12.4_{\pm 0.7}$ | $28.5_{\pm 0.8}$ | $37.9_{\pm 0.6}$ | $21.7_{\pm 0.6}$
MMT[20] | $13.2_{\pm 0.4}$ | $29.2_{\pm 0.8}$ | $38.8_{\pm 0.9}$ | $21.0_{\pm 1.4}$
TT-MMT | $13.6_{\pm 0.5}$ | $31.2_{\pm 0.4}$ | $40.8_{\pm 0.5}$ | $17.7_{\pm 0.5}$
TT-CE+ | $\mathbf{17.2}_{\pm 0.4}$ | $\mathbf{36.5}_{\pm 0.6}$ | $\mathbf{46.3}_{\pm 0.3}$ | $\mathbf{13.7}_{\pm 0.5}$

Table 6: LSMDC: Comparison to state of the art methods.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@50\uparrow$ | $MdR\downarrow$
---|---|---|---|---
MoEE[42]3 | $19.7_{\pm 0.3}$ | $50.0_{\pm 0.5}$ | $92.0_{\pm 0.2}$ | $5.3_{\pm 0.5}$
CE[36]3 | $19.9_{\pm 0.3}$ | $50.1_{\pm 0.7}$ | $92.2_{\pm 0.6}$ | $5.3_{\pm 0.5}$
HSE[76] | $20.5$ | $49.3$ | $-$ | $-$
MMT[20] | $22.7_{\pm 0.2}$ | $54.2_{\pm 1.0}$ | $93.2_{\pm 0.4}$ | $5.0_{\pm 0.0}$
SSB[49] | $\mathbf{26.8}$ | $58.1$ | $93.5$ | $\mathbf{3.0}$
TT-MMT | $25.0_{\pm 0.3}$ | $\mathbf{58.7}_{\pm 0.4}$ | $95.6_{\pm 0.2}$ | $4.0_{\pm 0.0}$
TT-CE+ | $23.5_{\pm 0.2}$ | $57.2_{\pm 0.5}$ | $\mathbf{96.1}_{\pm 0.1}$ | $4.0_{\pm 0.0}$

Table 7: ActivityNet: Comparison to state of the art methods.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
VSE[30] | $28.0$ | $64.3$ | $76.9$ | $3.0$
Dual[18] | $31.1$ | $67.4$ | $78.9$ | $3.0$
VSE++[19] | $33.7$ | $70.1$ | $81.0$ | $2.0$
HGR[14] | $35.1$ | $73.5$ | $83.5$ | $2.0$
SSB[49] | $44.6$ | $81.8$ | $89.5$ | $\mathbf{1.0}$
CE[36] | $47.9_{\pm 0.1}$ | $84.2_{\pm 0.1}$ | $91.3_{\pm 0.1}$ | $2.0_{\pm 0.0}$
TT-CE | $49.7_{\pm 0.1}$ | $85.6_{\pm 0.1}$ | $92.4_{\pm 0.1}$ | $2.0_{\pm 0.0}$
TT-CE+ | $\mathbf{53.2}_{\pm 0.2}$ | $\mathbf{87.4}_{\pm 0.1}$ | $\mathbf{93.3}_{\pm 0.0}$ | $\mathbf{1.0}_{\pm 0.0}$

Table 8: VaTeX: Comparison to state of the art methods.

Model | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$
---|---|---|---|---
MoEE[42] | $11.6_{\pm 1.3}$ | $30.2_{\pm 3.0}$ | $43.2_{\pm 3.1}$ | $14.2_{\pm 1.6}$
CE[36] | $13.9_{\pm 0.8}$ | $37.6_{\pm 1.2}$ | $48.3_{\pm 1.4}$ | $11.3_{\pm 0.6}$
TT-CE | $14.2_{\pm 1.4}$ | $36.6_{\pm 2.0}$ | $\mathbf{51.1}_{\pm 2.1}$ | $\mathbf{9.7}_{\pm 1.2}$
TT-CE+ | $\mathbf{14.4}_{\pm 0.5}$ | $\mathbf{37.7}_{\pm 1.7}$ | $50.9_{\pm 1.6}$ | $9.8_{\pm 1.0}$

Table 9: QuerYD: Comparison to state of the art methods.

###  5.4 Comparison to prior work

As it can be seen in Tab.2,3,4,5,6,7,8,9 our approach is effective and
achieves state of the art results on six datasets. All methods are trained for
the retrieval task using only the samples from the target datasets. In order
to be as fair as possible, we included the results of our TeachText
(abbreviated TT) applied also to the best existing method for each dataset.
So, the architecture and the used features are identical during inference
(e.g. TT-CE has the same architecture and uses the same video and text
embeddings as CE). We highlight in bold the best performing method.

##  6 Conclusion

In this paper, we present a novel algorithm TeachText for the text-video
retrieval task. We use a teacher-student paradigm where a student learns to
leverage the additional information given by one or multiple teachers, sharing
the architecture, but each using a different pre-trained text embedding at
input. In this way, we achieve state of the art results on six benchmarks.
Finally, we present an application of our approach for denoising video
retrieval datasets.
Acknowledgements. This work was supported by EPSRC Programme Grants Seebibyte
EP/M013774/1 and VisualAI EP/T028572/1, and a gift from Adobe. M.L. was
supported by UEFISCDI, under project EEA-RO-2018-0496. The authors would like
to thank Gyungin Shin and Iulia Duta for assistance. S.A. would like to
acknowledge the support of Z. Novak and S. Carlson in enabling his
contribution.

## References

  * [1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017.
  * [2] Yusuf Aytar, Mubarak Shah, and Jiebo Luo.  Utilizing semantic word similarity measures for video retrieval.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  arXiv preprint arXiv:2104.00650, 2021.
  * [4] Oren Boiman and Michal Irani.  Detecting irregularities in images and in video.  International journal of computer vision, 74(1):17–31, 2007.
  * [5] Piotr Bojanowski, Rémi Lajugie, Edouard Grave, Francis Bach, Ivan Laptev, Jean Ponce, and Cordelia Schmid.  Weakly-supervised alignment of video with text.  In Proceedings of the IEEE international conference on computer vision, pages 4462–4470, 2015.
  * [6] Leo Breiman and Nong Shang.  Born again trees.  University of California, Berkeley, Berkeley, CA, Technical Report, 1:2, 1996.
  * [7] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006.
  * [8] Andrea Burns, Reuben Tan, Kate Saenko, Stan Sclaroff, and Bryan A Plummer.  Language features matter: Effective language representations for vision-language tasks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7474–7483, 2019.
  * [9] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [10] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman.  VGGFace2: A dataset for recognising faces across pose and age.  2018\.
  * [11] Yue Cao, Mingsheng Long, Jianmin Wang, and Shichen Liu.  Deep visual-semantic quantization for efficient image retrieval.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1328–1337, 2017.
  * [12] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [13] David L Chen and William B Dolan.  Collecting highly parallel data for paraphrase evaluation.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics, 2011.
  * [14] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10638–10647, 2020.
  * [15] Ondrej Chum, James Philbin, Michael Isard, and Andrew Zisserman.  Scalable near identical image and shot detection.  In CIVR, 2007.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [17] Jianfeng Dong, Xirong Li, and Cees GM Snoek.  Word2visualvec: Image and video to sentence matching by visual feature prediction.  arXiv preprint arXiv:1604.06838, 2016.
  * [18] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, and Xun Wang.  Dual dense encoding for zero-example video retrieval.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [19] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler.  Vse++: Improving visual-semantic embeddings with hard negatives.  arXiv preprint arXiv:1707.05612, 2017.
  * [20] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  European Conference on Computer Vision, 2020.
  * [21] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan.  Large-scale weakly-supervised pre-training for video action recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 12046–12055, 2019.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, 2016.
  * [23] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson.  Cnn architectures for large-scale audio classification.  In International Conference on Acoustics, Speech and Signal Processing. 2017.
  * [24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [25] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu.  Squeeze-and-excitation networks.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [26] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [27] Zehao Huang and Naiyan Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  arXiv preprint arXiv:1707.01219, 2017.
  * [28] Yu-Gang Jiang, Chong-Wah Ngo, and Jun Yang.  Towards optimal bag-of-features for object categorization and semantic video retrieval.  In Proceedings of the 6th ACM international conference on Image and video retrieval, pages 494–501, 2007.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [30] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  arXiv preprint arXiv:1411.2539, 2014.
  * [31] Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani.  Video understanding as machine translation.  arXiv preprint arXiv:2006.07203, 2020.
  * [32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [33] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  arXiv preprint arXiv:1909.11942, 2019.
  * [34] Ivan Laptev and Patrick Pérez.  Retrieving actions in movies.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [35] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu.  Hoi analysis: Integrating and decomposing human-object interaction.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5011–5022. Curran Associates, Inc., 2020.
  * [36] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  arXiv preprint arXiv:1907.13487, 2019.
  * [37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [38] Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell.  Synthetically supervised feature learning for scene text recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 435–451, 2018.
  * [39] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik.  Unifying distillation and privileged information.  ICLR, 2016.
  * [40] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining.  In European Conference on Computer Vision (ECCV), pages 181–196, 2018.
  * [41] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  arXiv preprint arXiv:1912.06430, 2019.
  * [42] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learning a text-video embedding from incomplete and heterogeneous data.  arXiv preprint arXiv:1804.02516, 2018.
  * [43] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2630–2640, 2019.
  * [44] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [45] Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval.  In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pages 19–27, 2018.
  * [46] Andreea-Maria Oncescu, Joao F. Henriques, Yang Liu, Andrew Zisserman Zisserman, and Samuel Albanie.  Queryd: a video dataset with high-quality textual and audio narrations.  arXiv preprint arXiv:2011.11071, 2020.
  * [47] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in neural information processing systems, pages 8026–8037, 2019.
  * [49] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, João Henriques, and Andrea Vedaldi.  Support-set bottlenecks for video-text representation learning.  arXiv preprint arXiv:2010.02824, 2020.
  * [50] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018.
  * [51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  preprint, 1(8):9, 2019.
  * [52] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele.  Movie description.  International Journal of Computer Vision, 123(1):94–120, 2017.
  * [53] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [54] Lifeng Shang, Linjun Yang, Fei Wang, Kwok-Ping Chan, and Xian-Sheng Hua.  Real-time large scale near-duplicate web video retrieval.  In Proceedings of the 18th ACM international conference on Multimedia, pages 531–540, 2010.
  * [55] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304, 2017.
  * [56] Josef Sivic and Andrew Zisserman.  Video google: A text retrieval approach to object matching in videos.  In ICCV, page 1470. IEEE, 2003.
  * [57] Cees GM Snoek and Marcel Worring.  Multimodal video indexing: A review of the state-of-the-art.  Multimedia tools and applications, 25(1):5–35, 2005.
  * [58] Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng.  Grounded compositional semantics for finding and describing images with sentences.  Transactions of the Association for Computational Linguistics, 2:207–218, 2014.
  * [59] Jiaxi Tang and Ke Wang.  Ranking distillation: Learning compact ranking models with high performance for recommender system.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2289–2298, 2018.
  * [60] FFmpeg team.  Ffmpeg.  https://www.ffmpeg.org/.  Accessed: 2020-04-30.
  * [61] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [62] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [63] Vladimir Vapnik and Rauf Izmailov.  Learning using privileged information: similarity control and knowledge transfer.  J. Mach. Learn. Res., 16(1):2023–2049, 2015.
  * [64] Vladimir Vapnik and Akshay Vashist.  A new learning paradigm: Learning using privileged information.  Neural networks, 22(5-6):544–557, 2009.
  * [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [66] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko.  Sequence to sequence-video to text.  In Proceedings of the IEEE international conference on computer vision, pages 4534–4542, 2015.
  * [67] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko.  Translating videos to natural language using deep recurrent neural networks.  arXiv preprint arXiv:1412.4729, 2014.
  * [68] Qingzhong Wang and Antoni B Chan.  Describing like humans: on diversity in image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4195–4203, 2019.
  * [69] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4581–4591, 2019.
  * [70] Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In Proceedings of the IEEE International Conference on Computer Vision, pages 450–459, 2019.
  * [71] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [72] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  * [73] Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso.  Jointly modeling deep video and compositional text to bridge vision and language in a unified framework.  In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015\.
  * [74] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 471–487, 2018.
  * [75] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [76] Bowen Zhang, Hexiang Hu, and Fei Sha.  Cross-modal and hierarchical modeling of video and text.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 374–390, 2018.
  * [77] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton.  Lookahead optimizer: k steps forward, 1 step back.  In Advances in Neural Information Processing Systems, pages 9597–9608, 2019.
  * [78] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017\.

Appendix

In this supplementary material, we provide additional details on the video
embeddings (Sec. A) and text embeddings (Sec. B) used in the main submission.
We provide further details of Fig. 1 from the main paper (Sec. C) as well as
details on optimization (Sec. D), modifications to the embedding pre-
processing pipeline used in prior work (Sec. E) and summaries of the datasets
used (Sec. F). Finally, we include additional ablations (Sec. G) and a more
comprehensive set of metrics for comparison with previous work, along with
qualitative results (Sec. H).

##  Appendix A Video embeddings (experts) description

In this work, we used the set of pretrained experts considered by the authors
of [36]. For completeness, we summarise here the manner in which these experts
were extracted.

  * •

Two form of action experts are used: Action(KN) and Action(IG). The former is
an I3D architecture trained on Kinetics [12], which produces 1024-dimensional
embeddings from frame clips extracted at 25fps and center cropped to 224
pixels. The Action(IG) model is a 34-layer R(2+1)D model [62] that has ben
trained on IG-65m [21]: it operates on frames extracted at 30 fps in clips of
8 at 112 $\times$ 112 pixel resolution.

  * •

Two forms of object experts are used, named Obj(IN) and Obj(IG). They are
produced from frame-level embeddings extracted at 25fps. The Obj(IN) model
consists of an SENet-154 backbone [25] which has been trained on ImageNet for
image classification. Obj(IG) is formed from a ResNext-101 [71] extractor
which was trained on Instagram data that was weakly labelled with hashtags
[40]. For both models, frames are resized to 224$\times$224 pixels.

  * •

The face expert uses a ResNet50 [22] that has been trained for task of face
classification on the VGGFace2 dataset [10], producing a 512-dimensional
embedding for each detected face following detection.

  * •

The audio expert is produced using the VGGish model, trained for audio
classification on the YouTube-8m dataset and described by [23].

  * •

The scene expert is a 2208-dim embedding that is extracted frames (at 25 fps)
for a center crop of 224$\times$224 pixels. The model, which is pretrained on
Places365 [78], uses a DenseNet-161 [26] architecture.

  * •

The speech expert is produced using the Google Cloud API (to transcribe the
speech content).

  * •

The OCR expert is a word2vec encoding [44] of text detected in frames using
[38, 55].

###  A.1 Experts refinement – Modifying the Kinetics action recognition model

Apart from dropping the OCR and face experts as described in the main paper,
one small modification we propose to the expert selection made by [36] is to
replace the Action(IG) from an I3D model [12] to an R2P1D model [62] (matching
the architecture Action(IG)) which has also been pretrained on IG-65m [21] and
then finetuned on the Kinetics dataset [12].

##  Appendix B Text embeddings description

We use several text embeddings. In addition to the Sec. 3 from the main paper,
further technical details about each of them are given below:

  * •

mt_grovle [8] is a vision-sensitive language embedding which is adapted from
w2v using WordNet and an original vision-language graph built from Visual
Genome [32]. The size of the final pre-trained embedding is 300.

  * •

OpenAI-GPT [50] is a pre-trained text embedding which uses transformers [65]
and language modeling on a large corpus (the Toronto Book Corpus) (the final
model has 110M params). The size of the final pre-trained embedding is 768.

  * •

RoBERTa [37] is a BERT-based embedding [16]. The model is trained longer with
bigger batch size on more data, having 125M params. The size of the final pre-
trained embedding is 768.

  * •

ALBERT [33] is a lightweight modification to BERT [16] which overcomes some
memory limitations, having 11M params. The size of the final pre-trained
embedding is 768.

  * •

GPT2-large [51] is a transformer-based [65] model trained on even more data
(40 Gb of text) without any supervision, having 774M params. The size of the
final pre-trained embedding is 1280.

  * •

GPT2-xl [51] is similar to GPT2-large, but has more parameters (1558M params).
The size of the final pre-trained embedding is 1600.

  * •

W2V [44] is one of the most popular text embeddings used in vision tasks. It
uses a neural network model to learn word representations. The size of the
final pre-trained embedding is 300.

##  Appendix C Further Details for Fig. 1

In Fig.1 from the main paper we highlight that the gain for a model that uses
multiple text embeddings (last bar) is comparable with the gain of a model
that uses multiple video modalities (middle bar), having as comparison a model
that uses only one video modality (first bar). The first bar represents the CE
[36] model trained with one video embedding, namely Obj(IG) (the performance
of the model is 19.8±0.1 in geometric mean of R1-R5-R10). The second bar
represents a CE model using 7 video modalities both for inference and training
(the performance of the model is 24.4±0.1 in geometric mean of R1-R5-R10). In
the third and final bar of the chart we present the performance of using three
different text embeddings with TeachText at training, while using only one
text embedding at inference time (the performance of the model is 30.4±0.0 in
geometric mean of R1-R5-R10). All the numbers are presented after the
modification of the pre-processing pipeline (please see Sec. E for further
details). All the experts used by CE [36] are described in Sec.A.

##  Appendix D Optimization setup

CE+ models are trained in Pytorch [48] using the Adam optimizer [29]. We use a
learning rate of 0.001 and weight decay of 1E-5. When using a base
architecture different to the proposed CE+, we use the same hyper-parameters
as in the public codebase for the underlying method
(CE444https://shorturl.at/ksxIS and MMT555https://github.com/gabeur/mmt). For
MoEE, we use the re-implementation provided by the authors of the CE method
[36].

##  Appendix E Modification to pre-processing pipeline

During our preliminary analysis, we found out that some pretrained expert
models produce embeddings that are fairly sensitive to jpeg compression
artifacts. To address this, we re-extracted features from video frames densely
extracted with minimal jpeg compression (corresponding to the use of ffmpeg
[60] and the -qscale:v 2 flag). In order to be fair in our comparisons, we
apply this corrections everywhere. Due to this factor, we re-train MoEE [42]
and CE [36] and report higher numbers.

##  Appendix F Dataset details

To provide an extensive comparison we test our approach on seven video
datasets that have been explored in recent works as benchmarks for the task of
text-video retrieval. Next, we give details about all the datasets used.
MSRVTT [72] contains 10k videos, each having 20 captions. In order to test the
retrieval performance, we report results on the official split which contains
2990 videos for the test split and 497 for validation, following the setup
used in [36]. We perform most of our ablations on this split. To enable
comparison with as many other methods as possible, we also report results on
the 1k-A split as used in [36, 20, 49]. For this split, we report the
performance after training 100 epochs. The split contains 1000 video
candidates for testing and 9000 for training. We use the same candidates as
defined in [36] which are used by all the other works [74, 20, 49], using each
of the 20 captions associated to each video independently during evaluation
and averaging performance across them.
MSVD [13] contains 80k English descriptions for a total of 1970 videos. We use
the standard split of 1200 (training), 100 (validation) and 670 (testing) as
used in other works [66, 73, 36]. The videos from MSVD do not have audio
streams.
DiDeMo [1] contains 10464 videos sourced from a large-scale creative commons
collection [61] and features moments of unedited, diverse content (concerts,
sports, pets etc.). The dataset comprises 3-5 pairs of descriptions per video.
We adopt the paragraph-video retrieval protocols used by [76, 36] and use
splits corresponding to 8392 train, 1065 validation and 1004 test videos.
LSMDC [52] contains 118081 short video clips extracted from 202 movies. Each
clip is described by a caption that is either extracted from the movie script
or from transcribed DVS (descriptive video services) for the visually
impaired. There are 7408 clips in the validation set and the testing is
performed on 1000 videos from movies that are disjoint from the training and
val sets as described in the Large Scale Movie Description Challenge
(LSMDC)666https://shorturl.at/cdrI6.
ActivityNet [9] contains 20k videos extracted from YouTube and has around 100K
descriptive sentences. We follow the same paragraph-video retrieval setup as
used in prior works [76, 36] and report results on the val1 split. So, we use
10009 videos for training and 4917 videos for testing.
VaTeX [69] contains 34911 videos with multilingual captions (Chinese and
English). There are 10 captions per video for each language. We follow the
same protocol as in [14, 49] and split the validation set equally (1500
validation and 1500 testing videos). In this work, we only use the English
annotations.
QuerYD [46] contains 1815 videos in the training split and 388 and 390 for
validation and testing. The videos are sourced from YouTube and cover a
diverse range of visual content. The dataset contains 31441 descriptions, from
which 13019 are precisely localized in the video content (having start time
and end time annotations) and the other 18422 are coarsely localized. For this
work, we do not use the localization annotations and report results for the
official splits.

Figure 8: (a) Batch size variation. We vary the batch size for the MSR-VTT
dataset to see how this affects the performance. We observe that batch size
influences performance. The underlying architecture used for this experiment
is CE+. (b) Similarity matrix aggregation. We present a comparison of
different similarity matrix aggregation: min, max and average. As can be seen,
the average aggregation has the best results (both when evaluating the teacher
standalone or in conjunction with our TeachText algorithm).

##  Appendix G Ablations

In this section, we present additional ablations.

###  G.1 Batch size variation

In Fig. 8a we vary the batch size for the MSR-VTT dataset in order to see how
the performance is affected. As can be seen, we obtain the best value using
the same batch size as for the method without applying TeachText algorithm (64
in this case).

###  G.2 Similarity matrix aggregation study

In Fig. 8b we present several similarity matrix aggregation possibilities:
min, max and average. We observe that using the mean of the similarity
matrices is more effective. Because of this, we use the mean as the final
aggregation technique in our TeachText algorithm.

###  G.3 Denoising

Figure 9: (a) Rank variation for denoising. The denoising involves dropping
captions that are assigned a low ranking by the teacher for the training set.
In this experiment, we vary the rank below which we drop sentences. Please
note that for a rank of 5 (on the training set) the amount of dropped
sentences is approximately 46%. Note that MSR-VTT has 20 captions per video,
so after applying this filter we keep on average 10 captions per video. (b)
Denoising. We present the effect of denoising on retrieval performance on
MSVD. Some of the captions available in datasets with multiple captions per
video may be noisy and actively harm the training process. We estimate the
degree of noise present in a caption by looking at the teacher rank and drop
the caption if necessary. We observe the effectiveness of denoising when
applied in isolation (CE+ vs CE+ Denoise) and in conjunction with the full
TeachText method. The experiment is presented for dropping sentences that rank
higher than rank 100.

In Fig. 9a we vary the threshold used to filter out sentences from the
training set. As can be seen, this denoising method is effective and it can
provide a significant gain in performance. In this experiment we have found
out that the best threshold for MSRVTT is rank 40. Additionally, we present
denoising results in Fig. 9b for the MSVD dataset using the 100 threshold.
This method turns out to be effective in reducing noise for retrieval
datasets. Denoising is not use in any other ablation studies. The final
results when comparing with other state of the art methods are presented using
denoising on MSRVTT and MSVD datasets.

###  G.4 Distillation setup

As stated in the main paper, the distillation setup admits a number of
variants. In addition to the methods presented in Fig. 6b from the main paper,
in Fig. 10a we present several additional comparisons. More exactly, we test
our approach against a more classical distillation setup where we directly
regress the embeddings given by the teacher (Embd regress). This setup does
not follow the idea of relational distillation. Additionally, we also apply
the angle distillation as introduced by [47] (Relational angle) where we use
exactly the loss as in the public code777https://github.com/lenscloth/RKD.
Please note that the drop in performance as opposed to the student without
distillation can be explained by some technical challenges that we encountered
in order to make the angle loss compatible with the ranking loss used for this
task. Last but not least, we also show that a small improvement can be
obtained by using a self learning technique, where the teacher has the exact
same architecture and inputs as the student.

###  G.5 Loss study

In the main paper, we follow recent literature [47] and use the Huber loss for
distillation. However, we wanted to see how various losses affect the
performance. We test with L1 and L2 losses. As can be seen in Fig. 10b, the
Huber loss performs better than L1 loss and a bit better than L2.

Figure 10: (a) Distillation type. The first bar represents the student
performance without distillation (CE+). In addition to the methods presented
in the main paper, here we test other distillation approaches: Embd regress
which is a classical approach where the query and video joint embeddings are
directly regressed based on the embeddings given by the teacher, Relational
angle where we apply the angle relationships as introduced by [47]. In
addition, we present results of our method in a self-learning setup where the
teacher is the student from a previous run (Self learn). The last bar
represent the performance of the TeachText approach. (b) Loss study. In this
picture, we show how various distillation losses (L1, L2, Huber) affect the
performance.

###  G.6 Mixture of architectures

Our TeachText assumes that the only difference between the teacher and the
student is the used pre-trained text embedding fed to the model. However, our
method is not limited to this constraint. In this section, we show how having
multiple teachers, that now have a different underlying architecture affect
the performance of our method. Please note that in all other ablations, the
architecture is shared between student and teacher. This is the only
exception. Our preliminary results shown in Fig. 11 suggest that there isn’t
much improvement that may be achieved by using a mixture of architectures as
teachers. This is somehow expected, since these methods usually share the same
video modalities so there isn’t much additional information that may be
captured by the combination of multiple architectures. However, we expect to
get a further boost if we diversify the set of used modalities.

Figure 11: Mixture of architectures. We perform some preliminary experiments
to see if the method may benefit from learning from teachers that do not share
the same architecture. The x axis corresponds to the models which are used as
teachers. In cases labeled with 3 text each, we used three different
variations of each architecture as teachers, accounting for a total number of
no. methods * 3 teachers. As can be seen, the results suggest that there is no
clear benefit in using multiple architectures as teachers.

###  G.7 Architecture extension

Model | MSRVTT | MSRVTT 1k-A | MSVD | DiDeMo | LSMDC | ActivityNet
---|---|---|---|---|---|---
Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText | Base | TeachText
MoEE | $24.4_{\pm 0.1}$ | $\mathbf{25.8}_{\pm 0.1}$ | $41.6_{\pm 0.4}$ | $\mathbf{43.4}_{\pm 0.6}$ | $41.8_{\pm 0.3}$ | $\mathbf{43.2}_{\pm 0.5}$ | $33.2_{\pm 1.4}$ | $\mathbf{40.2}_{\pm 0.7}$ | $23.8_{\pm 0.4}$ | $\mathbf{26.0}_{\pm 0.5}$ | $40.1_{\pm 0.3}$ | $\mathbf{45.2}_{\pm 0.1}$
CE | $24.4_{\pm 0.1}$ | $\mathbf{25.9}_{\pm 0.1}$ | $42.0_{\pm 0.8}$ | $\mathbf{43.8}_{\pm 0.3}$ | $42.3_{\pm 0.6}$ | $\mathbf{42.6}_{\pm 0.4}$ | $34.2_{\pm 0.4}$ | $\mathbf{39.5}_{\pm 0.5}$ | $23.7_{\pm 0.3}$ | $\mathbf{25.5}_{\pm 0.5}$ | $40.4_{\pm 0.3}$ | $\mathbf{45.0}_{\pm 0.6}$
MMT | - | - | $44.7_{\pm 0.4}$ | $\mathbf{45.6}_{\pm 0.7}$ | - | - | - | - | $24.6_{\pm 0.7}$ | $\mathbf{25.9}_{\pm 0.6}$ | $44.0_{\pm 0.4}$ | $\mathbf{47.9}_{\pm 0.4}$
CE+ | $29.2_{\pm 0.2}$ | $\mathbf{30.4}_{\pm 0.0}$ | $50.3_{\pm 0.2}$ | $\mathbf{50.9}_{\pm 0.4}$ | $46.5_{\pm 1.0}$ | $\mathbf{46.6}_{\pm 0.5}$ | $35.8_{\pm 0.4}$ | $\mathbf{40.4}_{\pm 0.4}$ | $28.1_{\pm 0.3}$ | $\mathbf{30.7}_{\pm 0.3}$ | $39.7_{\pm 0.0}$ | $\mathbf{46.3}_{\pm 0.2}$
CE-L | $25.5_{\pm 0.1}$ | $\mathbf{26.9}_{\pm 0.1}$ | $45.7_{\pm 0.2}$ | $\mathbf{46.5}_{\pm 0.8}$ | $41.3_{\pm 0.5}$ | $\mathbf{42.6}_{\pm 0.7}$ | $36.4_{\pm 0.5}$ | $\mathbf{41.5}_{\pm 0.4}$ | $24.1_{\pm 0.2}$ | $\mathbf{25.9}_{\pm 0.3}$ | $39.6_{\pm 0.5}$ | $\mathbf{45.7}_{\pm 0.2}$

Table 10: Method generality. Retrieval performance on various datasets when
applying TeachText on top of different base models. In addition to the main
paper, we present results on the CE-L architecture which has a significant
drop in the number of used parameters as compared to the other models. We
present in bold cases where TeachText brings an improvement over the base
architecture. As can be seen, our method is effective and brings a consistent
boost independent of the base architecture.

In addition to the main paper, we also introduce a new CE-L base architecture.
This is similar to the CE [36] and CE+, but uses w2v as the text embedding. In
this way, the number of parameters are greatly reduced, making this the most
lightweight architecture in terms of number of parameters that we can create.
In Tab. 10, you can see that our method TeachText is effective even when using
this lightweight architecture. This architecture also has the lowest numbers
of parameters when compared to other state of the art methods as can be seen
in Tab.11,12,13,14,15,16,17,18.

Figure 12: (a) Amount of training data vs performance. As it can be seen, with
the increase of training data, the improvement brought by TeachText increases.
(b) Performance vs teacher type. We study the influence of teachers with
different text embeddings at input: w2v and gpt2-xl. The first point
represents the performance of the student without using TeachText. We observe
a boost in performance independent of the nature of the teacher.

Figure 13: Share of samples correctly retrieved samples in terms of R1 when
using TeachText on the MSR-VTT test set. In sub-fig (a) we show the case where
we learn from 3 teachers, while in sub-fig (b) you can find the single teacher
case. We can see that the model with TeachText, preserves most of the
knowledge from the student without TeachText, but also acquires new
information from the teacher (yellow area). Best viewed in color.

Figure 14: Qualitative results. We present the top 3 video retrievals for each
query, given by the TeachText method used on top of a CE+ architecture.
Moreover, we show the rank and similarity for the teacher, as well as for the
student without using TeachText for the ground truth video. We mark in green
cases where the retrieval is correct in terms of R1 and with red cases where
is incorrect. For each of the cases shown, the model learns from the teacher
to correct its prediction.

###  G.8 Model complexity

Changes in the pretrained text embedding strongly affect the number of
parameters. Because of this factor, using more text embeddings at test time
may strongly affect the total number of learnable parameters available to the
model (in addition to adding the requirement to extract additional text
embeddings during inference). While the simple ’Mean’ aggregation from Fig.5b
in the main paper, does not change the number of parameters, the ’Concat’
aggregation adds a significant quantity (approx 240M learnable parameters,
yielding total model sizes of 503.98M vs 262.73M for CE+). The proposed
TeachText approach leaves the number of parameters untouched.

Since changing the text embedding to CE+ results in an increase in number of
learnable parameters, we also study a CE-L architecture as a lightweight
alternative in this Suppl. Mat. (described in Sec. 10), which demonstrates
that the gain from the proposed TeachText approach is not limited to models
with many learnable parameters. Please check Tab.11,12,13,14,15,16,17,18 for
the exact number of params for every used architecture.

###  G.9 Amount of training data vs performance.

We next study how training data quantity influences the proposed method. In
Fig. 12a we observe that by using the TeachText with more and more data, the
performance gap increases, suggesting that its benefit may prove to be useful
even in larger scale dataset scenarios.

###  G.10 Teacher study

In Fig. 12b, we study how each embedding affects the final performance. We
observe that even though the model ingesting w2v embeddings has a significant
lower performance than the student model without using TeachText, there is a
significant gain when learning from the teacher which uses w2v. This again
indicates that there is additional information captured by using a different
text embedding which can be exploited by TeachText.

###  G.11 Influence of distillation over the correctly retrieved samples

In Fig. 13 we present the shares of correctly retrieved samples in terms of R1
on the test set of the MSR-VTT dataset for the student with and without
TeachText and for the teacher. In Fig. 13a we present results when we learn
from the three teachers and in Fig. 13b we considered the case when we learn
only from one teacher (namely w2v). There is a significant share of correctly
retrieved sample between the student using TeachText and the teacher.

##  Appendix H Comparison to prior work

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
Dual[18] | t2v | $7.7$ | $22.0$ | $31.8$ | $32.0$ | v2t | $13.0$ | $30.8$ | $43.3$ | $15.0$ | -
HGR[14] | t2v | $9.2$ | $26.2$ | $36.5$ | $24.0$ | v2t | $15.0$ | $36.7$ | $48.8$ | $11.0$ | -
MoEE[42]3 | t2v | $11.1_{\pm 0.1}$ | $30.7_{\pm 0.1}$ | $42.9_{\pm 0.1}$ | $15.0_{\pm 0.0}$ | v2t | $16.5_{\pm 0.1}$ | $43.1_{\pm 0.5}$ | $57.3_{\pm 0.6}$ | $7.7_{\pm 0.5}$ | $400.41M$
CE[36]3 | t2v | $11.0_{\pm 0.0}$ | $30.8_{\pm 0.1}$ | $43.3_{\pm 0.3}$ | $15.0_{\pm 0.0}$ | v2t | $17.0_{\pm 0.5}$ | $43.5_{\pm 0.4}$ | $57.8_{\pm 0.5}$ | $7.2_{\pm 0.2}$ | $183.45M$
TT-CE | t2v | $11.8_{\pm 0.1}$ | $32.7_{\pm 0.1}$ | $45.3_{\pm 0.1}$ | $13.0_{\pm 0.0}$ | v2t | $19.3_{\pm 0.4}$ | $47.0_{\pm 0.7}$ | $60.0_{\pm 0.4}$ | $6.7_{\pm 0.5}$ | $183.45M$
TT-CE-L | t2v | $13.0_{\pm 0.0}$ | $34.6_{\pm 0.1}$ | $47.3_{\pm 0.2}$ | $12.0_{\pm 0.0}$ | v2t | $22.4_{\pm 0.3}$ | $50.4_{\pm 0.6}$ | $63.8_{\pm 0.3}$ | $5.3_{\pm 0.5}$ | $66.72M$
TT-CE+ | t2v | $\textbf{15.0}_{\pm 0.1}$ | $\textbf{38.5}_{\pm 0.1}$ | $\textbf{51.7}_{\pm 0.1}$ | $\textbf{10.0}_{\pm 0.0}$ | v2t | $\textbf{25.3}_{\pm 0.1}$ | $\textbf{55.6}_{\pm 0.0}$ | $\textbf{68.6}_{\pm 0.4}$ | $\textbf{4.0}_{\pm 0.0}$ | $262.73M$

Table 11: MSR-VTT full split: Comparison to state of the art.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
MoEE[42]3 | t2v | $21.6_{\pm 1.0}$ | $50.8_{\pm 1.1}$ | $65.6_{\pm 0.7}$ | $5.3_{\pm 0.6}$ | v2t | $22.4_{\pm 0.5}$ | $51.2_{\pm 1.0}$ | $66.1_{\pm 0.4}$ | $5.2_{\pm 0.3}$ | $400.41M$
CE[36]3 | t2v | $21.7_{\pm 1.3}$ | $51.8_{\pm 0.5}$ | $65.7_{\pm 0.6}$ | $5.0_{\pm 0.0}$ | v2t | $22.7_{\pm 0.4}$ | $51.8_{\pm 0.4}$ | $65.7_{\pm 0.2}$ | $5.0_{\pm 0.0}$ | $183.45M$
MMT[20] | t2v | $24.6_{\pm 0.4}$ | $54.0_{\pm 0.2}$ | $67.1_{\pm 0.5}$ | $4.0_{\pm 0.0}$ | v2t | $24.4_{\pm 0.5}$ | $56.0_{\pm 0.9}$ | $67.8_{\pm 0.3}$ | $4.0_{\pm 0.0}$ | $133.36M$
SSB[49] | t2v | $27.4$ | $56.3$ | $67.7$ | $\mathbf{3.0}$ | v2t | $26.6$ | $55.1$ | $67.5$ | $\mathbf{3.0}$ | $-$
TT-MMT | t2v | $24.8_{\pm 0.2}$ | $55.9_{\pm 0.7}$ | $68.5_{\pm 1.0}$ | $4.3_{\pm 0.5}$ | v2t | $25.1_{\pm 1.0}$ | $57.1_{\pm 0.8}$ | $69.9_{\pm 1.1}$ | $4.0_{\pm 0.0}$ | $133.36M$
TT-CE-L | t2v | $26.5_{\pm 0.4}$ | $58.0_{\pm 0.8}$ | $71.1_{\pm 0.4}$ | $4.0_{\pm 0.0}$ | v2t | $27.6_{\pm 0.8}$ | $58.0_{\pm 0.6}$ | $70.0_{\pm 0.5}$ | $4.0_{\pm 0.0}$ | $66.72M$
TT-CE+ | t2v | $\mathbf{29.6}_{\pm 0.3}$ | $\mathbf{61.6}_{\pm 0.5}$ | $\mathbf{74.2}_{\pm 0.3}$ | $\mathbf{3.0}_{\pm 0.0}$ | v2t | $\mathbf{32.1}_{\pm 0.5}$ | $\mathbf{62.7}_{\pm 0.5}$ | $\mathbf{75.0}_{\pm 0.2}$ | $\mathbf{3.0}_{\pm 0.0}$ | $262.73M$

Table 12: MSR-VTT 1k-A split[74]: Comparison with others.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
VSE++[19] | t2v | $15.4$ | $39.6$ | $53.0$ | $9.0$ | v2t | $21.2$ | $43.4$ | $52.2$ | $9.0$ | -
M-Cues[45] | t2v | $20.3$ | $47.8$ | $61.1$ | $6.0$ | v2t | $\mathbf{31.5}$ | $51.0$ | $61.5$ | $5.0$ | -
MoEE[42]3 | t2v | $21.1_{\pm 0.2}$ | $52.0_{\pm 0.7}$ | $66.7_{\pm 0.2}$ | $5.0_{\pm 0.0}$ | v2t | $27.3_{\pm 0.9}$ | $55.1_{\pm 1.2}$ | $65.0_{\pm 0.8}$ | $4.3_{\pm 0.5}$ | $131.37M$
CE[36]3 | t2v | $21.5_{\pm 0.5}$ | $52.3_{\pm 0.8}$ | $67.5_{\pm 0.7}$ | $5.0_{\pm 0.0}$ | v2t | $26.3_{\pm 1.4}$ | $53.7_{\pm 0.4}$ | $65.3_{\pm 1.1}$ | $4.8_{\pm 0.2}$ | $84.04M$
TT-CE | t2v | $22.1_{\pm 0.4}$ | $52.2_{\pm 0.5}$ | $67.2_{\pm 0.6}$ | $5.0_{\pm 0.0}$ | v2t | $26.0_{\pm 0.4}$ | $53.3_{\pm 0.4}$ | $63.9_{\pm 0.1}$ | $4.9_{\pm 0.1}$ | $84.04M$
TT-CE-L | t2v | $22.5_{\pm 0.0}$ | $53.7_{\pm 0.3}$ | $68.7_{\pm 0.5}$ | $5.0_{\pm 0.0}$ | v2t | $25.6_{\pm 0.2}$ | $\mathbf{55.7}_{\pm 0.9}$ | $65.9_{\pm 0.5}$ | $\mathbf{4.0}_{\pm 0.0}$ | $27.78M$
TT-CE+ | t2v | $\mathbf{25.4}_{\pm 0.3}$ | $\mathbf{56.9}_{\pm 0.4}$ | $\mathbf{71.3}_{\pm 0.2}$ | $\mathbf{4.0}_{\pm 0.0}$ | v2t | $27.1_{\pm 0.4}$ | $55.3_{\pm 1.0}$ | $\mathbf{67.1}_{\pm 0.2}$ | $\mathbf{4.0}_{\pm 0.0}$ | $87.79M$

Table 13: MSVD: Comparison to state of the art methods.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $R@50\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $R@50\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---|---|---
S2VT[67] | t2v | $11.9$ | $33.6$ | - | $76.5$ | $13.0$ | v2t | $13.2$ | $33.6$ | - | $76.5$ | $15.0$ | -
FSE[77] | t2v | $13.9_{\pm 0.7}$ | $36.0_{\pm 0.8}$ | - | $78.9_{\pm 1.6}$ | $11.0_{\pm 0.0}$ | v2t | $13.1_{\pm 0.5}$ | $33.9_{\pm 0.4}$ | - | $78.0_{\pm 0.8}$ | $12.0_{\pm 0.0}$ | -
MoEE[42]3 | t2v | $16.1_{\pm 1.0}$ | $41.2_{\pm 1.6}$ | $55.2_{\pm 1.6}$ | $81.7_{\pm 1.4}$ | $8.3_{\pm 0.5}$ | v2t | $16.0_{\pm 1.5}$ | $41.7_{\pm 1.9}$ | $54.6_{\pm 1.7}$ | $81.0_{\pm 1.4}$ | $8.7_{\pm 0.9}$ | $107.26M$
CE[36]3 | t2v | $17.1_{\pm 0.9}$ | $41.9_{\pm 0.2}$ | $56.0_{\pm 0.5}$ | $83.4_{\pm 0.7}$ | $8.0_{\pm 0.0}$ | v2t | $17.1_{\pm 0.1}$ | $41.8_{\pm 0.9}$ | $55.2_{\pm 1.0}$ | $83.0_{\pm 0.8}$ | $7.7_{\pm 0.5}$ | $79.29M$
TT-CE | t2v | $21.0_{\pm 0.6}$ | $47.5_{\pm 0.9}$ | $61.9_{\pm 0.5}$ | $86.4_{\pm 0.8}$ | $6.0_{\pm 0.0}$ | v2t | $20.3_{\pm 0.6}$ | $46.6_{\pm 0.6}$ | $59.8_{\pm 1.2}$ | $85.7_{\pm 0.6}$ | $6.7_{\pm 0.5}$ | $79.29M$
TT-CE-L | t2v | $\mathbf{22.3}_{\pm 0.2}$ | $\mathbf{50.1}_{\pm 0.9}$ | $\mathbf{64.3}_{\pm 0.5}$ | $\mathbf{86.9}_{\pm 0.4}$ | $\mathbf{5.3}_{\pm 0.5}$ | v2t | $\mathbf{21.3}_{\pm 0.4}$ | $\mathbf{48.3}_{\pm 0.5}$ | $\mathbf{62.5}_{\pm 0.3}$ | $86.6_{\pm 0.1}$ | $\mathbf{6.0}_{\pm 0.0}$ | $43.51M$
TT-CE+ | t2v | $21.6_{\pm 0.7}$ | $48.6_{\pm 0.4}$ | $62.9_{\pm 0.6}$ | $86.8_{\pm 0.3}$ | $6.0_{\pm 0.0}$ | v2t | $21.1_{\pm 0.2}$ | $47.3_{\pm 0.2}$ | $61.1_{\pm 0.4}$ | $\mathbf{86.7}_{\pm 0.2}$ | $6.3_{\pm 0.5}$ | $99.51M$

Table 14: DiDeMo: Comparison to state of the art methods.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
JSFus[74] | t2v | $9.1$ | $21.2$ | $34.1$ | $36.0$ | v2t | - | - | - | - | -
MoEE[42]3 | t2v | $12.1_{\pm 0.7}$ | $29.4_{\pm 0.8}$ | $37.7_{\pm 0.2}$ | $23.2_{\pm 0.8}$ | v2t | $11.9_{\pm 0.5}$ | $28.0_{\pm 0.5}$ | $37.4_{\pm 0.5}$ | $25.5_{\pm 1.5}$ | $159.78M$
CE[36]3 | t2v | $12.4_{\pm 0.7}$ | $28.5_{\pm 0.8}$ | $37.9_{\pm 0.6}$ | $21.7_{\pm 0.6}$ | v2t | $11.4_{\pm 0.4}$ | $28.4_{\pm 0.7}$ | $36.5_{\pm 0.5}$ | $25.0_{\pm 0.8}$ | $116.86M$
MMT[20] | t2v | $13.2_{\pm 0.4}$ | $29.2_{\pm 0.8}$ | $38.8_{\pm 0.9}$ | $21.0_{\pm 1.4}$ | v2t | $12.1_{\pm 0.1}$ | $29.3_{\pm 1.1}$ | $37.9_{\pm 1.1}$ | $22.5_{\pm 0.4}$ | $133.16M$
TT-MMT | t2v | $13.6_{\pm 0.5}$ | $31.2_{\pm 0.4}$ | $40.8_{\pm 0.5}$ | $17.7_{\pm 0.5}$ | v2t | $12.5_{\pm 0.3}$ | $31.3_{\pm 0.6}$ | $41.0_{\pm 1.1}$ | $18.7_{\pm 0.5}$ | $133.16M$
TT-CE-L | t2v | $14.2_{\pm 0.2}$ | $30.6_{\pm 0.3}$ | $40.0_{\pm 0.5}$ | $20.3_{\pm 0.5}$ | v2t | $13.6_{\pm 0.3}$ | $30.8_{\pm 0.9}$ | $38.9_{\pm 0.8}$ | $21.5_{\pm 0.4}$ | $87.22M$
TT-CE+ | t2v | $\mathbf{17.2}_{\pm 0.4}$ | $\mathbf{36.5}_{\pm 0.6}$ | $\mathbf{46.3}_{\pm 0.3}$ | $\mathbf{13.7}_{\pm 0.5}$ | v2t | $\mathbf{17.5}_{\pm 0.6}$ | $\mathbf{36.0}_{\pm 1.2}$ | $\mathbf{45.0}_{\pm 0.5}$ | $\mathbf{14.3}_{\pm 0.9}$ | $388.24M$

Table 15: LSMDC: Comparison to state of the art methods.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@50\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@50\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
MoEE[42]3 | t2v | $19.7_{\pm 0.3}$ | $50.0_{\pm 0.5}$ | $92.0_{\pm 0.2}$ | $5.3_{\pm 0.5}$ | v2t | $18.3_{\pm 0.5}$ | $48.3_{\pm 0.8}$ | $92.0_{\pm 0.2}$ | $6.0_{\pm 0.0}$ | $330.42M$
CE[36]3 | t2v | $19.9_{\pm 0.3}$ | $50.1_{\pm 0.7}$ | $92.2_{\pm 0.6}$ | $5.3_{\pm 0.5}$ | v2t | $18.6_{\pm 0.3}$ | $48.6_{\pm 0.7}$ | $92.0_{\pm 0.2}$ | $6.0_{\pm 0.0}$ | $260.68M$
HSE[76] | t2v | $20.5$ | $49.3$ | $-$ | $-$ | v2t | $18.7$ | $48.1$ | - | - | -
MMT[20] | t2v | $22.7_{\pm 0.2}$ | $54.2_{\pm 1.0}$ | $93.2_{\pm 0.4}$ | $5.0_{\pm 0.0}$ | v2t | $22.9_{\pm 0.8}$ | $54.8_{\pm 0.4}$ | $93.1_{\pm 0.2}$ | $4.3_{\pm 0.5}$ | $127.35M$
SSB[49] | t2v | $\mathbf{26.8}$ | $58.1$ | $93.5$ | $\mathbf{3.0}$ | v2t | $\mathbf{25.5}$ | $57.3$ | $93.5$ | $\mathbf{3.0}$ | -
TT-MMT | t2v | $25.0_{\pm 0.3}$ | $\mathbf{58.7}_{\pm 0.4}$ | $95.6_{\pm 0.2}$ | $4.0_{\pm 0.0}$ | v2t | $24.4_{\pm 0.1}$ | $\mathbf{58.2}_{\pm 0.3}$ | $95.7_{\pm 0.1}$ | $4.0_{\pm 0.0}$ | $127.35M$
TT-CE-L | t2v | $23.3_{\pm 0.1}$ | $56.3_{\pm 0.1}$ | $95.5_{\pm 0.1}$ | $4.0_{\pm 0.0}$ | v2t | $20.7_{\pm 0.2}$ | $52.8_{\pm 0.2}$ | $94.4_{\pm 0.0}$ | $5.0_{\pm 0.0}$ | $103M$
TT-CE+ | t2v | $23.5_{\pm 0.2}$ | $57.2_{\pm 0.5}$ | $\mathbf{96.1}_{\pm 0.1}$ | $4.0_{\pm 0.0}$ | v2t | $23.0_{\pm 0.3}$ | $56.1_{\pm 0.2}$ | $\mathbf{95.8}_{\pm 0.0}$ | $4.0_{\pm 0.0}$ | $376.02M$

Table 16: ActivityNet: Comparison to state of the art methods.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
VSE[30] | t2v | $28.0$ | $64.3$ | $76.9$ | $3.0$ | v2t | $-$ | $-$ | $-$ | $-$ | $-$
Dual[18] | t2v | $31.1$ | $67.4$ | $78.9$ | $3.0$ | v2t | $-$ | $-$ | $-$ | $-$ | $-$
VSE++[19] | t2v | $33.7$ | $70.1$ | $81.0$ | $2.0$ | v2t | $-$ | $-$ | $-$ | $-$ | $-$
HGR[14] | t2v | $35.1$ | $73.5$ | $83.5$ | $2.0$ | v2t | $-$ | $-$ | $-$ | $-$ | $-$
SSB[49] | t2v | $44.6$ | $81.8$ | $89.5$ | $\mathbf{1.0}$ | v2t | $58.1$ | $83.8$ | $90.9$ | $\mathbf{1.0}$ | $-$
CE[36] | t2v | $47.9_{\pm 0.1}$ | $84.2_{\pm 0.1}$ | $91.3_{\pm 0.1}$ | $2.0_{\pm 0.0}$ | v2t | $60.7_{\pm 1.0}$ | $89.0_{\pm 0.4}$ | $94.9_{\pm 0.2}$ | $\mathbf{1.0}_{\pm 0.0}$ | $115.56M$
TT-CE | t2v | $49.7_{\pm 0.1}$ | $85.6_{\pm 0.1}$ | $92.4_{\pm 0.1}$ | $2.0_{\pm 0.0}$ | v2t | $62.1_{\pm 0.2}$ | $90.0_{\pm 0.1}$ | $95.3_{\pm 0.1}$ | $\mathbf{1.0}_{\pm 0.0}$ | $115.56M$
TT-CE-L | t2v | $51.5_{\pm 0.1}$ | $86.5_{\pm 0.1}$ | $92.6_{\pm 0.1}$ | $\mathbf{1.0}_{\pm 0.0}$ | v2t | $\mathbf{65.0}_{\pm 0.5}$ | $90.3_{\pm 0.5}$ | $95.9_{\pm 0.2}$ | $\mathbf{1.0}_{\pm 0.0}$ | $55.07M$
TT-CE+ | t2v | $\mathbf{53.2}_{\pm 0.2}$ | $\mathbf{87.4}_{\pm 0.1}$ | $\mathbf{93.3}_{\pm 0.0}$ | $\mathbf{1.0}_{\pm 0.0}$ | v2t | $64.7_{\pm 0.3}$ | $\mathbf{91.5}_{\pm 0.3}$ | $\mathbf{96.2}_{\pm 0.1}$ | $\mathbf{1.0}_{\pm 0.0}$ | $223.1M$

Table 17: VaTeX: Comparison to state of the art methods.

Model | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Task | $R@1\uparrow$ | $R@5\uparrow$ | $R@10\uparrow$ | $MdR\downarrow$ | Params
---|---|---|---|---|---|---|---|---|---|---|---
MoEE[42] | t2v | $11.6_{\pm 1.3}$ | $30.2_{\pm 3.0}$ | $43.2_{\pm 3.1}$ | $14.2_{\pm 1.6}$ | v2t | $13.0_{\pm 3.1}$ | $30.9_{\pm 2.0}$ | $43.0_{\pm 2.8}$ | $14.5_{\pm 1.8}$ | $57.75M$
CE[36] | t2v | $13.9_{\pm 0.8}$ | $37.6_{\pm 1.2}$ | $48.3_{\pm 1.4}$ | $11.3_{\pm 0.6}$ | v2t | $13.7_{\pm 0.7}$ | $35.2_{\pm 2.7}$ | $46.9_{\pm 3.2}$ | $12.3_{\pm 1.5}$ | $30.82M$
TT-CE | t2v | $14.2_{\pm 1.4}$ | $36.6_{\pm 2.0}$ | $\mathbf{51.1}_{\pm 2.1}$ | $\mathbf{9.7}_{\pm 1.2}$ | v2t | $14.1_{\pm 0.5}$ | $34.8_{\pm 3.0}$ | $\mathbf{49.1}_{\pm 0.3}$ | $\mathbf{11.3}_{\pm 0.5}$ | $30.82M$
TT-CE+ | t2v | $\mathbf{14.4}_{\pm 0.5}$ | $\mathbf{37.7}_{\pm 1.7}$ | $50.9_{\pm 1.6}$ | $9.8_{\pm 1.0}$ | v2t | $\mathbf{14.3}_{\pm 0.6}$ | $\mathbf{36.3}_{\pm 0.9}$ | $48.3_{\pm 1.2}$ | $\mathbf{11.3}_{\pm 0.5}$ | $30.82M$

Table 18: QuerYD: Comparison to state of the art methods.

In Tab.11,12,13,14,15,16,17,18 we make an extensive comparison of our method
with other methods from the literature. In addition to the numbers reported in
the main paper, we also report results for the v2t task. Moreover, we present
the number of parameters of each method where available. As can be seen, our
TeachText algorithm brings a clear improvement and the total number of
parameters remains the same as for the base architecture. In addition to the
main paper, we also introduce a new CE-L base architecture. This is similar to
the CE [36] and CE+, but uses w2v as text embedding. In this way, the number
of parameters is greatly reduced. As can be seen from the tables, this
lightweight architecture combined with our TeachText algorithm has very good
results showcasing the effectiveness of TeachText across different parameter
regimes. Moreover, some qualitative results can be seen in Fig.14.