# Examining Correlation Between Trust and Transparency with Explainable
Artificial Intelligence

Arnav Kartikeya<EMAIL_ADDRESS>
###### Abstract

Trust between humans and artificial intelligence(AI) is an issue which has
implications in many fields of human computer interaction. The current issue
with artificial intelligence is a lack of transparency into its decision
making, and literature shows that increasing transparency increases trust.
Explainable artificial intelligence has the ability to increase transparency
of AI, which could potentially increase trust for humans. This paper attempts
to use the task of predicting yelp review star ratings with assistance from an
explainable and non explainable artificial intelligence to see if trust is
increased with increased transparency. Results show that for these tasks,
explainable artificial intelligence provided significant increase in trust as
a measure of influence.

##  1 Introduction

Trust in automation, primarily artificial intelligence, is an important issue
in the field of human computer interaction. Prior literature has shown that
with an increase in transparency, trust increases as well [1]. Artificial
Intelligence(AI) has the well known problem of a lack of justification of its
decision making, therefore decreasing transparency between a user and the AI.
Explainable artificial intelligence(XAI) provides increased transparency to
the user, which has the potential of increasing trust. The XAI used in this
paper is LIME, an algorithm developed in 2016 for model agnostic explanations
into model decision making [4]. This paper attempts to find the correlation
between XAI, increased transparency, and trust in specific tasks.

A common task for artificial intelligence is to suggest or aid a human in a
task. Recommendation algorithms used by Netflix, or systems aiding pilots in
spatially disorienting situations are prime examples of this task. Literature
examining the relation between trust and transparency in these tasks exist, as
well as literature on the impact of XAI on human interaction [5]. Our work
makes the following contributions: It examines trust through the lens of XAI’s
influence of human decision making, and uses metrics of quantitative data as
oppose to subjective opinion-based surveys. Influence in this case is defined
as how often a user changes their decision with additional aid from an
artificial intelligence model. Rather than using metrics of opinion-based
surveys, this paper examines how exactly the model’s decision impacts human
decisions, which will show how trust and transparency correlate in realistic
scenarios rather than subjective opinions. I also examine the effectiveness of
opinion-based surveys such as the Trust in Automation Questionnaire in how it
represents actual influence of a model in a realistic scenario.

I hypothesize the following:

  1. 1.

the LIME XAI model will influence human decisions significantly more than a
normal artificial intelligence,

  2. 2.

trust measured through the Trust in Automation opinion-based survey [2] will
correspond with the actual influence from artificial intelligence.

##  2 Experimental Design

In order to experimentally examine the relation between trust and transparency
in the task described previously, a method of two separate surveys given to
two separate groups was used. The surveys both asked the respondent to
complete the task of predicting the star rating an individual gave to a
restaurant with only the information of the text in the review, and a machine
learning model’s output. The surveys only vary in the amount of information
given by the model, meaning the exact texts given and questions asked were the
exact same. The following section elaborates on the differences and
similarities. The machine learning model in both cases were the same, they
both were pre-trained facebook fasttext models [3], and both were trained on
the same official yelp dataset.

###  2.1 Similarities between surveys

Both surveys follow the same two sequence set of questions, and these
sequences repeat 15 times to create 30 questions. The sequence is as follows.
The first question in the sequence asks the user to predict, on a scale of 1
to 5 stars, what they believe the Yelp review associated with the question
rated the restaurant. The second question gives a model output in the form of
an image, which is either the label of the class of yelp review (in the form
of ___label__X.0_ , where X represents the number 1 through 5 for stars).
Depending on the survey, either explanation is shown. After all the 15 two-
sequence questions are answered by the respondent, they will answer the Trust
In Automation Questionnaire on a 1 to 7 likert scale.

###  2.2 Differences between surveys

The previous section mentions that either class label or Figure 1 is used as
the model’s output, and this depends on the survey being answered. The first
survey, known as the basic survey, only offers class label, which provides no
transparency into the model’s decision making.

The second survey, known as advanced survey, uses Figure 1 as its output. From
left to right, the first section shows the confidence that the model had in
its classification of each star. The figure shows that it had a confidence of
0.03 for 5 stars. The second portion shows the top features, or words, that
influenced the model’s decision of the label it chose and the ones it did not.
A higher number represents a larger influence in its decision. The third
section repeats the text from the Yelp review, and highlights those specific
words that were mentioned in the second section. A darker highlight
corresponds to higher influence. This figure shows greater transparency in the
model’s decision than Figure 1 does.

The explanation for Figure 1 was created by LIME. In both instances, a
pretrained Fast-text natural language processing model was further trained on
the official Yelp dataset to accomplish the task of predicting the star rating
of a review.

Figure 1: Explanation provided for second survey. (Left) Confidence in each
class prediction, (Middle) Top most influential words in decision making,
(Right) Top influential words, highlighted within entire text

###  2.3 Distribution of survey

Both surveys were separate tasks distributed on Amazon Mechanical Turks. 25
participants were used for each survey, with no additional criteria included.
The sets of participants were not the same set of individuals, and were
anonymous.

##  3 Evaluation of survey results

In order to quantify the 50 respondent answers and understand the relation
between trust and transparency, data was split into multiple groups and
compared between the two surveys using customized metrics.

###  3.1 Splitting the data

Data was split by first which survey was being answered. The first group
contained answers from the basic survey, and the second from the advanced. In
each group, further subgroups were created. In both surveys, the model was
correct in its prediction 6 times, 6 were incorrect by 1 or 2 stars, and 3 by
more than 3 stars. I compared these groups between the surveys in order to
ensure confidence was not the leading factor in decision making, and to
determine exactly how much transparency can influence a users decision’s.
Measuring if there was difference in the sets of data for all three would
ensure that transparency increases trust in all cases, however if there is not
a difference in the less correct sets, it would show that confidence has a
greater influence on trust and transparency cannot overcome the lack of
confidence in the model.

###  3.2 Comparing the data

To compare the data between the groups, a custom metrics was created, the
model influence metric. This was done to show the exactly how the model
changed the users decision. A metric of checking if the user changed their
answer between originally answering and being presented with the model’s
decision would not suffice, because there is the likelihood the respondent
further lowered their answer away from the model’s decision, as they simply
rethought their answer. Therefore, a metric was needed which checked if the
respondent changes their answer to be closer to the model. This would show
that the respondent did use the model’s output in their decision making,
showing trust between the model and the respondent. If the respondent’s answer
was closer to the model’s output by the second question, then a score for the
respondent was incremented by 1. For example, if a respondent believed yelp
review to have a star prediction of 2 stars, but the model predicted 4 and the
respondent then changed their answer to 3, that would increment their score by
1. This was done for each of the subsections based on correctness, and the
results were compared between the basic and advanced sets of data as mentioned
previously. One case which presented an issue was the case of when the
respondent guessed the same answer as the model did, without the model’s
input. If they guessed 3 stars before looking at the model, an the model also
guessed 3 stars, then no trust is being measured if the respondent keeps their
answer. To prevent this from occurring, I omitted those answers from the data
and kept the metric as a percentage value. This ensured that those data points
would not be included, while still maintaining a proper metric that can be
compared over varying sizes of data. The following section will discuss the
results. Equation 1 shows the formula for the metric used.

$\left(\frac{100\%}{25}\right)(\sum\frac{\text{\\#influenced
answers}}{\text{\\#non omitted answers}})$

(1)

##  4 Results

###  4.1 Model influence metric results

After processing all 50 survey responses using metrics and groups described in
the previous section, the results were created and displayed in the Figures
and tables below. Between the groups of correct, slightly incorrect, and
totally incorrect, the metric used showed a statistically significant
difference. A one tailed independent mean t-test was used for finding the p
values listed in the table. In all cases p value is less than 0.05. Table 1
shows the individual means of all 6 groups compared.

###  4.2 Trust evaluation results

Questions taken from the trust in automation questionnaire were placed at the
end of both surveys and analyzed in order to see if there exists any contrast
between the subjective trust measured by the questionnaire and the metric used
in this paper. Both group’s surveys were averaged. The advanced questionnaire
had an average of 4.50 and the basic an average of 4.71 (p-value=0.14, one-
tailed independent mean t-test).The results show no statistically significant
difference between the advanced and basic surveys, which does not align with
the difference shown in the previous section with the model influence metric.

Groups | Basic Survey | Advanced Survey | p-value
---|---|---|---
Correct | 33.27% | 59.54% | .003
Slightly In. | 33.20% | 55.67% | .009
Totally In. | 50.00% | 65.34% | .038
Table 1: Difference between basic and advanced survey in means, p-value
calculated by one-tailed t-test

###  4.3 Discussion

The purpose of this study was to experimentally verify if transparency
increases trust in the case of artificial intelligence aided tasks similar to
the one described. Results from the experiment show statistically significant
difference between the sets of data, and therefore do not reject hypothesis 1.
This shows that with increased transparency, trust increases as well between
the respondent and the model, at least when trust is measured as model’s
influence on the respondent. The data also shows that any additional insight
into the model’s decision making will increase trust, regardless of if the
model is totally incorrect in its results. This is shown by comparing the
results from the ”totally incorrect” sections of the transparent and non
transparent models.

The second section of the experiment pertained to the Trust in Automation
questionnaire, and whether its results align with that from the surveys. The
confidence and dependence questions which were discussed in previous sections
show no statistically significant results between transparent and non
transparent models, which is not reflected by the metrics used for this paper.
This shows that for measuring model influence through change in decision
making after model input, the Trust in Automation questionnaire does not align
with measures used. Subjective trust measured through the Trust in Automation
Questionnaire therefore contrasts with trust measured through the metric used
in this paper in experimental scenarios. This paper comes to its conclusions
for the specific experiment and metric used in the paper. This paper serves as
a starting ground for future research in verifying the hypothesis mentioned
for different experiments and metrics, as well as larger sets of data.

## References

  * [1] Kevin Hoff and Masooda Bashir.  Trust in automation: Integrating empirical evidence on factors that influence trust.  Human Factors The Journal of the Human Factors and Ergonomics Society, 57:407–434, 05 2015.
  * [2] Jiun-Yin Jian, Ann M. Bisantz, and Colin G. Drury.  Foundations for an empirically determined scale of trust in automated systems.  International Journal of Cognitive Ergonomics, 4(1):53–71, 2000\.
  * [3] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  arXiv preprint arXiv:1607.01759, 1, 2016.
  * [4] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”why should I trust you?”: Explaining the predictions of any classifier.  CoRR, abs/1602.04938, 2016.
  * [5] Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André.  ”do you trust me?”: Increasing user-trust by integrating virtual agents in explainable ai interaction design.  Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, IVA ’19:1–3, 07 2019.


Vladimir<EMAIL_ADDRESS>Weidi Xieweidi@robots.ox.ac.uk2,3 Esa
<EMAIL_ADDRESS>Andrew<EMAIL_ADDRESS>Computing Sciences
Tampere University
Tampere, Finland Coop. Medianet Innovation Center
Shanghai Jiao Tong University
Shanghai, China Visual Geometry Group
Department of Engineering Science
University of Oxford
Oxford, UK Audio-visual Synchronisation

# Sparse in Space and Time: Audio-visual Synchronisation with Trainable
Selectors

###### Abstract

The objective of this paper is audio-visual synchronisation of general videos
‘in the wild’. For such videos, the events that may be harnessed for
synchronisation cues may be spatially small and may occur only infrequently
during a many seconds-long video clip, _i.e_ the synchronisation signal is
‘sparse in space and time’. This contrasts with the case of synchronising
videos of talking heads, where audio-visual correspondence is dense in both
time and space.

We make four contributions: (i) in order to handle longer temporal sequences
required for sparse synchronisation signals, we design a multi-modal
transformer model that employs ‘selectors’ to distil the long audio and visual
streams into small sequences that are then used to predict the temporal offset
between streams. (ii) We identify artefacts that can arise from the
compression codecs used for audio and video and can be used by audio-visual
models in training to artificially solve the synchronisation task. (iii) We
curate a dataset with only sparse in time and space synchronisation signals;
and (iv) the effectiveness of the proposed model is shown on both dense and
sparse datasets quantitatively and qualitatively. Project page:
v-iashin.github.io/SparseSync

##  1 Introduction

Audio-visual synchronisation is the task of determining the temporal offset
between the audio (sound) and visual (image) streams in a video. In recent
literature, this task has been explored by exploiting strong correlations
between the audio and visual streams, _e.g_ in human speech [Chung and
Zisserman(2016a), Chung et al.(2019)Chung, Chung, and Kang, Afouras et
al.(2020)Afouras, Owens, Chung, and Zisserman] and playing instruments
[Arandjelovic and Zisserman(2018), Owens and Efros(2018)], to provide a
training signal for deep neural networks. In such scenarios, effective signals
for synchronisation can be discovered between the lip or body movements and
audio at almost every second. Despite the tremendous success achieved by these
methods, for the most part, existing models are still limited to specialised
domains, and not directly applicable to general (non-face, non-music) classes.

Figure 1: Audio-visual synchronisation requires a model to relate changes in
the visual and audio streams. Open-domain videos often have a small visual
indication, _i.e_ sparse in space. Moreover, cues may be intermittent and
scattered, _i.e_ sparse across time, _e.g_ a lion only roars once during a
video clip. This differs from a tight face crop of a speaker where cues are
dense in space and time.

Our goal in this paper is to develop the next-generation audio-visual (AV)
synchroniser. Rather than focusing on a specialised domain, such as human
speech, we explore architectures for AV synchronisation for videos of general
thematic content, e.g. daily videos [Korbar et al.(2018)Korbar, Tran, and
Torresani, Khosravan et al.(2019)Khosravan, Ardeshir, and Puri, Chen et
al.(2020)Chen, Xie, Vedaldi, and Zisserman] and live sports [Ebeneze et
al.(2021)Ebeneze, Wu, Wei, Sethuraman, and Liu]. A solution for this task
would be extremely useful for a number of applications that improve a user’s
viewing experience – in order to avoid or at least automatically detect AV
synchronisation offsets. Applications such as video conferencing, television
broadcasts, and video editing, are currently largely done by ‘off-line’
measurements or heavy manual processing [Shrestha et al.(2010)Shrestha,
Barbieri, Weda, and Sekulovski, Staelens et al.(2012)Staelens, De Meulenaere,
Bleumers, Van Wallendael, De Cock, Geeraert, Vercammen, Van den Broeck,
Vermeulen, Van de Walle, et al., Dassani et al.(2019)Dassani, Bird, and
Cliff].

However, upgrading the existing audio-visual synchronisation systems to
general videos is not straightforward, due to the following challenges: (i),
in general videos, the synchronisation signal is often sparse and
instantaneous in time, (a lion roaring or a tennis volley), rather than dense
in time (a recorded monologue); (ii), objects that emit sounds can vary in
size or appear in the distance making their presence on the frame small or
sparse in space (a ball being hit in tennis), whereas synchronisation of a
talking-head video may rely on visual cues from the localised mouth region,
_i.e_ dense in space; (iii), some sound sources do not have a useful visual
signal for synchronisation,  _e.g_ stationary sounds (a car engine or electric
trimmer), ambient sounds (wind, water, crowds, or traffic), and off-screen
distractors (commentary track or advertisements); (iv), video encoding
algorithms compress unperceived redundancy of a signal, this, however, can
introduce artefacts that may lead to a trivial solution when training for
audio-visual synchronisation; lastly, (v) due to its challenging nature, a
public benchmark to measure progress has not yet been established.

In this paper, (i) we introduce a novel multi-modal transformer architecture,
SparseSelector, that can digest long videos with linear scaling complexity
with respect to the number of input tokens, and predict the temporal offset
between the audio and visual streams. We achieve this by using a set of
learnable queries to select informative signals from the ‘sparse’ video events
across a wide time span. (ii) We show that for specific common audio and
visual coding standards, a model can detect compression artefacts during
training. We present a few simple indicators to determine if a model has
learnt using these artefacts, as well as suggest several ways to mitigate the
problem. Specifically, for the RGB stream, we recommend avoiding the MPEG-4
Part 2 codec, as well as reducing the sampling rate for audio. (iii)
Additionally, to measure the progress of audio-visual synchronisation on
general thematic content, we curate a subset of VGGSound with ‘sparse’ audio-
visual correspondence called VGGSound-Sparse. We validate the effectiveness of
the new model with thorough experiments on the existing lip reading benchmark
(LRS3) and natural videos from VGGSound-Sparse and demonstrate state-of-the-
art performance.

##  2 Related Work

#### Audio-visual synchronisation.

During the pre-deep-learning era, the audio-visual human face synchronisation
models relied on manually crafted features and statistical models [Hershey and
Movellan(1999), Slaney and Covell(2000)]. With the advent of deep learning,
[Chung and Zisserman(2016b)] introduced a two-stream architecture that was
trained in a self-supervised manner using a binary contrastive loss. Later
improvements were brought by multi-way contrastive training [Chung et
al.(2019)Chung, Chung, and Kang], and Dynamic Time Warping [Rabiner and
Juang(1993)] used by [Halperin et al.(2019)Halperin, Ephrat, and Peleg].
Khosravan  _et al_ [Khosravan et al.(2019)Khosravan, Ardeshir, and Puri]
demonstrated the benefits of spatio-temporal attention and Kim _et al_[Kim et
al.(2021)Kim, Heo, Chung, and Lee] employed a cross-modal embedding matrix to
predict the offset for synchronisation. The progress was followed by
[Kadandale et al.(2022)Kadandale, Montesinos, and Haro] who introduced an
architecture called VocaLiST with three transformer decoders [Vaswani et
al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and
Polosukhin]: two that cross-attend individual modalities and a third that
fuses the outputs of the first two. These methods achieve impressive
performance but focus on human speech rather than open-domain videos.

Although audio-visual synchronisation of general classes is a novel task, a
few promising attempts have been made. In particular, Casanovas _et
al_[Casanovas and Cavallaro(2015)] studied a handful of different scenes
captured from a set of cameras. More recently, Chen _et al_[Chen et
al.(2021)Chen, Xie, Afouras, Nagrani, Vedaldi, and Zisserman] adapted the
transformer architecture and used a subset of VGGSound [Chen et al.(2020)Chen,
Xie, Vedaldi, and Zisserman] covering 160 classes. In contrast to prior work,
we focus on more challenging classes that have ‘sparse’ rather than ‘dense’
synchronisation signals.

#### Video coding artefacts.

Since the early work of Doersch _et al_ on self-supervision [Doersch et
al.(2015)Doersch, Gupta, and Efros], it has been known that network training
can find shortcuts. Similarly, shortcuts due to video editing and coding
artefacts have been noted in Wei _et al_[Wei et al.(2018)Wei, Lim, Zisserman,
and Freeman] and Arandjelović _et al_[Arandjelovic and Zisserman(2018)]. In
particular, [Wei et al.(2018)Wei, Lim, Zisserman, and Freeman] tackled the
arrow-of-time in videos and studied artificial cues caused by black regions on
video frames. While [Arandjelovic and Zisserman(2018)] noticed a slight impact
of MPEG-encoding on audio-visual correspondence training and attributed it to
the way negative samples are picked with respect to the start time of a
positive sample. In this work, we study the ways to easily spot that the data
contains artificial signals, as well as provide a few recommendations on how
to prevent leaking such artefacts into data.

##  3 SparseSelector: an Audio-visual Synchronisation Model

In this section, we describe our audio-visual synchronisation model, where the
audio-visual correspondence may only be available at sparse events in the ‘in
the wild’ videos. This requires the model to handle longer video clips so that
there is a high probability that a synchronisation event will occur. To this
end, we propose SparseSelector, a transformer-based architecture that enables
the processing of long videos with linear complexity with respect to the
duration of a video clip. It achieves this by ‘compressing’ the audio and
visual input tokens into two small sets of learnable selectors. These
selectors form an input to a transformer which predicts the temporal offset
between the audio and visual streams.

#### Architecture overview.

The overview of the model is shown in Fig. 2. Given an audio spectrogram
$\mathcal{A}\in\mathbb{R}^{H_{a}\times W_{a}\times 1}$ ($H_{a}$, $W_{a}$ are
frequency and time dimensions) and a stack of RGB frames
$\mathcal{V}\in\mathbb{R}^{T_{v}\times H_{v}\times W_{v}\times 3}$, the audio-
visual synchronisation model outputs the offset $\Delta$ between audio
($\mathcal{A}$) and visual ($\mathcal{V}$) streams:

$\displaystyle\Delta=\Phi_{\text{Sync}}\Big{(}\Phi_{\text{A-Sel}}\big{(}\Phi_{\text{A-Feat}}(\mathcal{A})\big{)},\,\Phi_{\text{V-Sel}}\big{(}\Phi_{\text{V-Feat}}(\mathcal{V})\big{)}\Big{)}.$
(1)

First, audio and visual streams are independently encoded in feature
extraction modules $\Phi_{\text{A/V-Feat}}$. Next, trainable selectors are
passed to Feature Selectors ($\Phi_{\text{A/V-Sel}}$) along with the encoded
features where they ‘summarise’ informative signals from the features that
contain ‘sparse’ information across time and space. Finally, the selectors are
used in Synchronisation Transformer ($\Phi_{\text{Sync}}$) to predict the
temporal offset $\Delta$ between audio and visual streams.

Figure 2: An overview of SparseSelector. The input is a spectrogram of the
audio waveform and RGB frames from the video stream. These are passed through
corresponding feature extractors, and the resulting features are refined with
trainable selectors that ‘pick’ useful cues for synchronisation. As a result,
the synchronisation transformer operates on substantially shorter sequences
than the original input. The visual and audio selector queries are
concatenated with classification (CLS) and separation tokens (MOD) as input to
the transformer. Finally, the CLS token of the transformer output is used to
predict the audio offset using a linear classification head. RGB frames are
zoomed-in for visualisation purposes. The model is trained by off-setting the
audio spectrogram. Dashed lines illustrate train-time behaviour.

#### Feature encoding.

Audio & visual inputs are encoded in spatio-temporal feature extractors:

$\displaystyle a=\Phi_{\text{A-Feat}}(\mathcal{A})\in\mathbb{R}^{h_{a}\times
w_{a}\times d_{a}},\hskip
10.0ptv=\Phi_{\text{V-Feat}}(\mathcal{V})\in\mathbb{R}^{t_{v}\times
h_{v}\times w_{v}\times d_{v}},$ (2)

where $t,\,h,\,w$, and $d$ denote time, height, width and channel dimensions,
respectively. For the audio backbone, we use a variant of ResNet18 [He et
al.(2016)He, Zhang, Ren, and Sun], which we pre-train on VGGSound [Chen et
al.(2020)Chen, Xie, Vedaldi, and Zisserman] for sound classification. As for
the visual backbone, we adopt S3D [Xie et al.(2018)Xie, Sun, Huang, Tu, and
Murphy] pre-trained for action recognition on Kinetics 400 [Kay et
al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola,
Green, Back, Natsev, et al.]. Although the setting allows employing any visual
recognition network, we found that training a synchronisation model with a
frame-wise feature extractor was significantly more difficult.

#### Feature Selectors.

To utilize sparsely occurring synchronisation cues, the model should be able
to handle longer input sequences. Moreover, accurate synchronisation requires
a higher visual frame rate than other video understanding tasks (_e.g_ action
recognition), which further increases the input size. For this reason, drawing
on the idea of trainable queries [Carion et al.(2020)Carion, Massa, Synnaeve,
Usunier, Kirillov, and Zagoruyko, Jaegle et al.(2021)Jaegle, Gimeno, Brock,
Zisserman, Vinyals, and Carreira, Xu et al.(2021)Xu, Xiong, Chen, Li, Xia, Tu,
and Soatto], we propose to use a small number of trainable ‘selectors’ that
learn to attend to the most useful modality features for synchronisation and,
thus, reducing sequence length.

The architecture of the Feature Selector is similar to the transformer decoder
[Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin]. Specifically, we start by flattening audio and visual
features into sequences $a\in\mathbb{R}^{h_{a}w_{a}\times d_{a}}$ and
$v\in\mathbb{R}^{t_{v}h_{v}w_{v}\times d_{v}}$. After adding trainable
positional encoding ($\text{PE}_{*}$) for each dimension, trainable selectors
and modality features are passed to the separate Feature Selectors as follows:

$\displaystyle\hat{q}_{a}=\Phi_{\text{A-Sel}}(a+\text{PE}_{a},q_{a}+\text{PE}_{q_{a}}),\hskip
10.0pt\hat{q}_{v}=\Phi_{\text{V-Sel}}(v+\text{PE}_{v},q_{v}+\text{PE}_{q_{v}}),$
(3)

where $q_{a},\hat{q}_{a}\in\mathbb{R}^{k_{a}\times d}$ and
$q_{v},\hat{q}_{v}\in\mathbb{R}^{k_{v}\times d}$ while $k_{\text{a/v}}$ are
the numbers of selectors.

Note that the selectors provide a ‘short summary’ of the context features
through the cross-attention mechanism while making the memory footprint more
manageable. The reduced memory requirement is a consequence of (a) casting the
complexity from quadratic to linear w.r.t. the input length, and (b) setting
$k_{v}\ll t_{v}\cdot h_{v}\cdot w_{v}$ and $k_{a}\ll h_{a}\cdot w_{a}$. The
number of selectors ($k_{v}$ or $k_{a}$) can be conveniently tweaked according
to the memory budget.

#### Audio-visual synchronisation transformer.

To fuse the audio-visual cues from individual selectors, we adopt the standard
transformer encoder layers to jointly process them, and to predict the offset,
i.e. relative temporal shift between audio and visual streams, as follows:

$\displaystyle\hat{\Delta}=\Phi_{\text{Sync}}([\texttt{CLS};\text{
}q_{v};\text{ }\texttt{MOD};\text{ }q_{a}])$ (4)

Here we concatenate the visual-audio selectors with two learnable special
tokens, namely the classification token $[\texttt{CLS}]$, and the modality
token $[\texttt{MOD}]$ that separates the two modalities. The offset
prediction is obtained by applying a linear prediction head on the first token
of the output sequence (omitted from Eq. (1) and (4) for clarity).

#### Training procedure.

We assume that the majority of videos in the public datasets are synchronised
to a good extent. With this assumption, we can artificially create temporal
offsets between audio and visual streams from a video. We formulate the audio-
visual synchronisation as a classification task onto a set of offsets from a
pre-defined temporal grid space as
$[-2.0,\,-1.8,\,\dots,\,0.0,\,+0.2,\,\dots,\,+2.0]$ sec. The step size is
motivated by the $\pm$0.2 sec human tolerance, where the ITU performed
strictly controlled tests with expert viewers and found that the threshold for
acceptability is $-$0.19sec to $+$0.1sec [Radiocommunication()]. To train the
model, we employ the cross-entropy loss. For our experiments, we randomly trim
a 5-sec segment out of 9 seconds such that both audio and visual streams are
within the 9-second clip to make inputs of the same size and avoid padding
that could hint if the input is off-sync.

##  4 Avoiding Temporal Artefacts

In this section, we describe our discovery of trivial solutions for training
audio-visual synchronisation, that is, the model is able to exploit the video
compression artefacts, to infer the time stamp for the specific video clip.
Additionally, we also detail a suite of techniques that allows us to probe the
artefacts and provide some practical suggestions to avoid them.

###  4.1 Identifying Temporal Artefact Leakage

Codec | Acc@1 | Acc@5
---|---|---
MPEG-4 Part 2 (mpeg4) | 27.2 | 77.1
MPEG-4 Part 10 (H.264) | 2.5 | 11.9
ProRes | 2.7 | 13.4
AAC @ 44100Hz | 86.7 | 100.0
AAC @ 22050Hz | 23.0 | 74.3
AAC @ 16000Hz | 6.3 | 19.3
Lossless @ 22050Hz | 2.9 | 14.6

Table 1: Commonly used coding standards may leak temporal artefacts – it is
easy to test. Left: a simple architecture trained to classify the start of a
trim from a 10s clip to a pre-defined 0.1 sec-step grid (here for RGB). Right:
Accuracy comparison for RGB and audio stream codecs predicting the start of an
RGB or audio trim. Metrics are accuracy at 1 and 5 on 50 classes. Chance
performance is 2 and 10 %. The higher accuracies indicate that an artefact is
being used – see text for discussion.

We present two ways of identifying the temporal artefact leakage. In
particular, training to predict the start time of a temporal crop (discussed
next) and tracking metrics with temporal tolerance (discussed in the
supplementary material).

#### Training to predict a video clip’s time stamp.

A synchronisation model should rely solely on temporal positions of conceptual
cues instead of, what we call, temporal artefacts. To check if data is
polluted with artefacts, we suggest training a model to predict the start time
of a random trim of an available video clip as shown in Tab. 1, left. Of
course, it should not be possible to determine the start time of the trim in
the original clip from the trim itself, and a network trained for this task
should achieve only chance performance. However, for some audio and video
codecs, the performance is far higher indicating artefact leakage.

The start-time classifier is a simple feature extractor (ResNet18). It is
trained on three variations of the MJPEG-AoT dataset [Wei et al.(2018)Wei,
Lim, Zisserman, and Freeman] obtained from the Vimeo streaming service:
original ProRes videos, and ProRes videos transcoded into either MPEG-4 Pt. 2
(aka. mpeg4) or MPEG-4 Pt. 10 (aka. H.264). Note, frames in ProRes are
compressed independently from others. If the visual stream of the video is
encoded using mpeg4, the model trained to predict the start of the trim can do
it significantly beyond a chance performance (Tab. 1, top-right).

Similarly, Advanced Audio Coding (AAC) might also leak temporal cues to the
audio signal (Tab. 1, bottom-right). Since it is challenging to find a large
set of videos with lossless audio compression, we used audio of randomly
generated noise with a specified sampling rate and saved it to a disk
losslessly (PCM) to obtain the performance with lossless compression. To
obtain results on AAC, we transcoded these files to AAC with ffmpeg.

###  4.2 Preventing Temporal Artefact Leakage

#### Avoiding MPEG-4 Part 2 in favour of H.264.

The algorithm that selects key-frames in MPEG-4 Part 2 (mpeg4) is less
flexible than the one of H.264. In particular, ffmpeg, which is commonly used
in practice, by default, encodes key frames every 12 frames. This means that
each of the following 11 frames is merely a residual of the key frame and it
is noticeable on the RGB stream (as we show in the supplementary). Such a
temporal regularity can be picked up by a model and used to solve the task
relying mostly on these artefacts. In contrast, each frame encoded by H.264
can reference up to 16 key-frames, which can be allocated more sparsely and
their presence depends heavily on the scene rather than a rather strict
interval as in MPEG-4 Pt. 2. This benefit is apparent when training a model to
predict the start of a trim (see Tab. 1, top-right). A potential solution
would be to avoid inter-frame codecs (mpeg4 and H.264) in favor of an intra-
frame codec (_e.g_ MJPEG, ProRes). However, this is a strong requirement for
research datasets because it requires avoiding YouTube which stores videos
compressed with inter-frame codecs (H.264 or VP9, according to view count).

#### Reducing audio sampling rate.

There is a substantial difference in the model’s ability to predict the start
of a trim depending on the sampling rate of the audio track (Fig. 1, bottom-
right). While the reason behind the temporal artefacts in AAC is unknown, we
recommend avoiding higher sampling rates. In our experiments, we rely on a
16kHz sampling rate as it provides a reasonable trade-off between audio
quality and the start prediction performance. Ultimately one would want to
have a dataset with lossless audio tracks yet, again, it is a strong
requirement for a dataset as it is commonly used by YouTube.

##  5 Experiments

#### Dense in time dataset.

The dataset is Lip Reading Sentences (LRS3) [Afouras et al.(2018)Afouras,
Chung, and Zisserman] which is obtained from TED talks for many speakers. We
use two variations of the dataset. The first employs strict rectangular face
crop coordinates that are extended to make a square (‘dense in time and
space’). The second variation consists of full-frame videos without cropping
(‘spatially sparse and temporally dense’). The raw videos are obtained from
YouTube with RGB (25fps, H.264) and audio (16kHz, AAC) streams and referred to
as ‘LRS3-H264’ and ‘LRS3-H264 (“No face crop”)’. We utilise the pretrain
subset and split video ids into 8:1:1 parts for train, validation, and test
sets. Only videos longer than 9 sec are used to unify it with the sparse
dataset (discussed next). In total, we use $\sim$58k clips from $\sim$4.8k
videos.

#### Sparse in time dataset.

The dataset uses VGGSound [Chen et al.(2020)Chen, Xie, Vedaldi, and Zisserman]
which consists of 10s clips collected from YouTube for 309 sound classes. A
subset of ‘temporally sparse’ classes is selected using the following
procedure: 5–15 videos are randomly picked from each of the 309 VGGSound
classes, and manually annotated as to whether audio-visual cues are only
sparsely available. After this procedure, 12 classes are selected ($\sim$4 %)
or 6.5k and 0.6k videos in the train and test sets, respectively (for class
names see Fig. 3). Next, the second round of manual verification of a
different subset of 20 videos from each class determines if it is feasible to
align the sound based on the visual content. It is observed that $\sim$70 % of
these video clips are synchronisable. We refer to this dataset as VGGSound-
Sparse.

#### Baseline.

Drawing on architectural details proposed in [Chen et al.(2021)Chen, Xie,
Afouras, Nagrani, Vedaldi, and Zisserman], we design a baseline as a
transformer decoder that uses audio features as queries and visual features as
context (keys and values) to predict the offset. The audio features are pooled
across the spectrogram frequency dimension and trained from scratch. Apart
from that, the feature extractors resemble ours.

#### Offset grid.

We define the synchronisation task as classifying the offset on a 21-class
grid ranging from $-$2 to $+$2 seconds with the 0.2-sec step size, as
explained in Sec. 3. This can be regarded as a more challenging variant of the
sync/off-sync task that prior work solves. We also experiment with a simpler
setting with only 3 offset classes [$-1$, 0, $+$1], that test if a model could
predict if one track either lags, is in sync, or is ahead of the other one.

#### Metrics.

Considering the human off-sync perception tolerance, in our experiments, we
mainly report the Top-1 Accuracy with a $\pm$1 class of temporal tolerance (as
described in Sec. 4.1). Note, that the training loss does not account for this
tolerance. In the supplementary section, we additionally provide performance
on accuracy without tolerance.

###  5.1 Results

| Dense-Dense | Dense-Sparse | Sparse-Sparse
---|---|---|---
| LRS3 (Face crop) | LRS3 (W/o face crop) | VGGSound-Sparse
| 3 cls | 21 cls | 3 cls | 21 cls |  3 cls⋆ | 21 cls
AVST${}_{\text{dec}}$ | 98.4 | 89.8 | 95.8 | 83.1 | 52.2 | 29.3
Ours | 96.4 | 95.6 | 95.5 | 96.9 | 60.3 | 44.3
Table 2: The proposed model handles the increasing complexity of the setting
and dataset better than the baseline while reaching a strong performance
compared to the oracle. ‘Dense-Dense’ refers to the face-cropped speech videos
(LRS3), ‘Dense-Sparse’ for spatially-sparse LRS3 (‘No face crop’), ‘Sparse-
Sparse’ is reported on VGGSound-Sparse which is sparse in time and space,
_e.g_ lion roars once during a clip. The synchronisation performance is
measured in two settings: the 3-class with ($-$1, 0, $+$1) offsets given 5-sec
clips, and 21 classes of offsets from $-2.0$ to $+2.0$ sec with 0.2-sec step
size. The latter setting allows $\pm$1 temporal class tolerance ($\pm$0.2
sec). ⋆: oracle performance is 70 %.  Figure 3:  Performance per data and
offset class on VGGSound-Sparse (test).

#### Dense in time and space.

Tab. 2 shows the comparison between the baseline and proposed architecture. As
the task becomes more difficult (more sparse data or finer offset grid), we
observe a larger gap between our model and the baseline. In particular, the
baseline performs strongly on LRS3 (dense in time and space) in the setting
with just three classes (98.4 %). However, once the task gets more
challenging, _i.e_ when training on finer offset shifts (21 class, 0.2 sec
apart), the baseline performance deteriorates significantly ($\sim$89.8 %). In
contrast, the proposed model performs strongly even in the setting with finer
offsets. This suggests that the proposed model is better suited for more
challenging data tasks.

#### Dense in time and sparse in space.

A similar effect is observed on the dataset that is dense in time and sparse
in space, _i.e_ LRS3 (No face crop). As both architectures drop their
performance slightly on the 3-class setting after switching to a more
difficult dataset, the drop is more significant for the baseline than for our
model. Moreover, the baseline performance drops substantially in the 21-class
setting ($>$6 %), while our model performs strong.

#### Sparse in time and space.

Finally, the experiments on the VGGSound-Sparse reveal an even larger
difference between the baseline method and our final model. For this
experiment, we add additional data augmentation to mitigate overfitting. In
particular, our model significantly outperforms the baseline showing the
benefit of selectors on a more challenging dataset and setting. Ultimately,
our model reaches 60 % in the 3-class setting which is close to the oracle
performance ($\sim$70 %: a human performance on 240 randomly picked videos),
while achieving 44 % in the 21-class setting. We report performance per class
in Fig. 3.

| Sync Model | Pre-trained | Unfrozen |
---|---|---|---|---
| pre-trained | Feature | Feature |
Selectors | on LRS3 | Extractors | Extractors |  Accuracy${}_{\text{21}}$
✗ | ✔ | ✔ | ✔ | 40.1
✔ | ✗ | ✔ | ✔ | 12.1
✔ | ✔ | ✗ | ✔ | 29.6
✔ | ✔ | ✔ | ✗ | 33.5
✔ | ✔ | ✔ | ✔ | 44.3
Table 3: Results of the ablation study. The experiments are conducted on
VGGSound-Sparse with 21 offset classes. Metric is Accuracy with $\pm$1 class
tolerance.

Figure 4: Working with longer sequences quickly becomes infeasible without
selectors.

Figure 5: Visual feature selectors focus on specific parts of the sparse
signal that is useful for synchronisation. Examples are from the hold-out set
of LRS3 (‘No face crop’) and VGGSound-Sparse. Attention is captured from a
selector to a visual or spectrogram feature token from a head within one of
the layers. Attention values are min-max scaled.

###  5.2 Ablation Study

In Tab. 3 we provide results for an ablation study. The results are reported
on the VGGSound-Sparse dataset with 21 offset classes. More results are
provided in the supplementary.

#### Feature selectors.

The architecture with feature selectors outperforms the vanilla transformer
showing the effectiveness of selectors in ‘compressing’ signals from the audio
and visual features. Also, Fig. 3 shows a memory footprint comparison of the
two, omitting the memory consumed by feature extractors that are the same. It
is evident that memory demand grows rapidly with the input duration making it
impossible to work with longer sequences and the transformer that inputs
concatenated audio and visual streams.

#### Pre-training on dense signals.

Pre-training a model on LRS3 (‘No face crop’) is an essential part of the
training procedure on VGGSound-Sparse: 44.3 vs. 12.1 (near chance
performance). For this reason, we also pre-train the baseline architecture
(see Tab. 2).

#### Pre-trained feature extractors.

The initialisation of audio and visual feature extractors with pre-trained
weights has a strong positive effect on model performance. To initialise our
feature extractors, we use weights of S3D pre-trained on Kinetics 400 for
action recognition and ResNet18 pre-trained sound classification on VGGSound.
The initialisation not only improves the final performance (43.3 vs. 33.5 %)
but also significantly speeds up training. We attribute this improvement to
the fact that such initialisation allows the model to ‘skip’ learning of the
generic low-layer features and focus on training for synchronisation.

#### Frozen feature extractors during training.

Allowing the gradients to reach raw data pixels is useful for audio-visual
synchronisation as it makes the model sensitive to the smallest variations in
the signal which is useful for synchronisation. In particular, having feature
extractors to be trainable significantly boosts the performance from 34 to
above 43 %, and the difference is even more pronounced on LRS3 (‘No face
crop‘) – see supplementary material.

#### Attention visualisation.

Fig. 5 shows examples from LRS3-H.264 (‘No Face Crop’) and VGGSound-Sparse.
Specifically, the attention exhibits spatial locality as the selectors learned
to attend to the features extracted from the mouth region as expected from a
model trained on a speech dataset. For a more challenging and diverse dataset,
VGGSound-Sparse, the model highlights important parts of the visual and audio
streams. In particular, the model accounts for the hit of the second badminton
player who is far away in the background or attends to the axe during the
chop, or the roaring mouth of the lion, yet these occur just once per video
clip. Similarly, audio feature selectors point to specific parts of the
spectrogram when the change occurs. More examples are provided in the
supplementary material.

Length | VGGSound-Sparse
---|---
(sec.) | 3 classes | 21 classes
2 | 55.6 |  —
3 | 59.4 | 36.8
4 | 60.8 | 43.0
5 | 60.3 | 44.2
6 | 61.2 | 45.6
7 | 62.9 | 46.5

Table 4: Synchronisation accuracy improves with input length. We report
results on two settings: with 3 offset classes ($-$1, 0, $+$1 sec) and 21
classes ($\pm$2.0 sec grid with 0.2-sec step size). The results are reported
on the test subset and accuracy is used as the metric. The accuracy for the
21-class setting is reported with $\pm$1 class tolerance. We use the same
input lengths for pre-training, fine-tuning, and testing.

#### Input length.

As the sparse synchronisation cues occur only occasionally within a video
clip, processing shorter temporal crops decrease the chance of having
sufficient cues for synchronisation, which, in turn, should decrease the
performance. In Tab. 4, we show how performance varies with respect to the
duration of input video clips. The results on the 3 and 21 offset classes
illustrate the upward trend in model performance as the input duration
extends. Note that the longer the inputs, the less unseen training data the
model processes at each epoch. Specifically, a 10-second clip may be split
into non-overlapping 3-second clips, which is not possible with clips longer
than 5 seconds. Thus, this effect may undermine the current performance.

##  6 Conclusion

In this work, we study ‘in the wild’ videos that often have a synchronisation
signal that is sparse in time. This requires a model to efficiently process
longer input sequences as these synchronisation cues occur only rarely. To
this end, we designed a transformer-based synchronisation model that has
linear complexity with respect to the input length. This was made possible by
using a small set of learnable ‘selectors’ that summarise long audio and
visual features that are employed to solve the synchronisation task. To
evaluate models in this challenging setup, we curate a dataset with only
sparse events and train it on 5-second long clips. Finally, we discovered that
compression artefacts caused by audio and video codecs might pose a threat to
training for synchronisation, yet, as we show, these artefacts are easily
identifiable and could be avoided to a certain extent.

#### Limitations.

First, considering the complicated input-output relationship in the proposed
model, it is challenging to determine which part of the input signal
influences the output. Second, in this work, we considered signals that are
‘dense in time and space’, ‘dense in time but sparse in space’, and ‘sparse in
time and space’. However, there is another interesting setting ‘sparse in time
but dense in space’ yet it is not clear how to design such a dataset without
making it ‘too artificial’. Third, despite showing strong performance on the
proposed dataset, VGGSound-Sparse, there is still room for improvement.

#### Acknowledgements.

Funding for this research was provided by the Academy of Finland projects
327910 and 324346, EPSRC Programme Grant VisualAI EP$/$T028572$/$1, and a
Royal Society Research Professorship. We also acknowledge CSC – IT Center for
Science, Finland, for computational resources.

Figure 6: MPEG-4 Part 2 (mpeg4) leaks regular temporal artefacts every
12${}^{\text{th}}$ frame that are unnoticeable on the RGB stream. The top row
shows a video encoded with H.264 and the pixel-wise first difference between
two consecutive frames. The bottom row shows the same video but now transcoded
(encoded) from H.264 into MPEG-4 Pt. 2 (mpeg4) and its first difference. The
first difference illustration is made sensitive to any change in corresponding
pixel intensity (colored in white).

##  7 Supplementary Material

In this section, we present more details on temporal artefacts (Sec. 7.1),
additional ablation study results (Sec. 7.2), and implementation details (Sec.
7.3).

###  7.1 Further Discussion on Temporal Artefacts

#### Illustrating temporal artefacts.

The temporal artefacts in the audio and visual streams have a periodic nature,
_e.g_ artefacts are equidistant temporally, and can be easily highlighted on
the RGB stream. In Fig. 6, we compare the first-order temporal difference
calculated from a video that was encoded with MPEG-4 Part 10 (H.264 or AVC)
and MPEG-4 Part 2 (aka mpeg4) which was transcoded111Sometimes this process is
incorrectly called ‘re-formatting’ or ‘re-encoding” from H.264. Notice the
white noise (RGB intensity change) on the 12${}^{\text{th}}$,
24${}^{\text{th}}$, 36${}^{\text{th}}$ frames (the period is 12) in the
bottom-right image. In particular, notice that the caption at the bottom of
each frame is fixed across the video yet every 12${}^{\text{th}}$ we see it to
be white, and black otherwise. The 12-frame period originates from the Group
of Picture (GoP) size, _i.e_ how often independently-encoded frames occur, and
is selected by the ffmpeg tool by default.

In contrast, on a video encoded in H.264, these artefacts are also visible but
are not regular (see the top row of Fig. 6) since the H.264 codec follows a
more flexible key-framing algorithm that allows it to reference not 1 as in
mpeg4 but up to 16 different key-frames (I-frames). Moreover, the allocation
of key-frames depends heavily on the scene content, which makes the temporal
spacing of key-frames unique for each video and, thus, less predictable a
priori. Note: this should hold for any video originally encoded with regularly
spaced key-frames, _i.e_ transcoding will not remove temporal artefacts. For
instance, if we crop out and transcode 5 seconds of a 10-second MPEG-4 Part 2
video file into H.264, the artefacts will still be present in the RGB stream.

#### Tracking accuracy metrics with temporal tolerance.

We noticed that, when the model is trained for start-time prediction and there
are temporal artefacts in the data, the model tend to output equidistant
classes in the top-K predictions. For example, if the correct start class is
15 (15${}^{\text{th}}$ frame if a video is encoded at 10 fps), the model would
be ‘confident’ in classes (3, 15, 27, etc) or simply any class where ‘class id
mod 12 = 3’. The 12 results from the GoP size as described in the previous
paragraph. This differs from the expected error behavior where predictions
would be around the true class, _e.g_(13, 15, 14, etc) or 1.3sec $\pm$0.1sec.

Figure 7: Some codecs leak temporal artefacts and it is easy to notice when
training for the temporal crop start-time prediction. Training dynamics for
the start-time prediction experiments. Left: random noise encoded in AAC with
varying sampling rates. Right: ProRes videos encoded as H.264 or mpeg4. Metric
is the top-1 accuracy across 50 classes.

#### Training details.

The artefact leakage can be detected by training a model to predict the start-
time of a temporal segment given a 5-second audio spectrogram or a stack of
RGB frames. In our experiments, we used a 50-way classification on a grid with
the 0.1s (one frame) stride from a 10fps video. The 50 classes is because in
our implementation the start of the temporal crop could only begin during the
first 5 seconds. For simplicity, we used a ResNet18 followed by a spatial
average pooling and a dense layer as a classification head for both audio and
visual inputs. The training dynamic is shown in Fig. 7.

###  7.2 Ablation Study (Additional Results)

| Sync Model | Pre-trained | Unfrozen | Dense-Sparse | Sparse-Sparse
---|---|---|---|---|---
| pre-trained | Feature | Feature | LRS3 (No face crop) | VGGSound-Sparse
Selectors | on LRS3 | Extractors | Extractors |  Acc${}_{\text{21}}$ / Acc${}^{\text{tol.}}_{\text{21}}$ |  Acc${}_{\text{21}}$ / Acc${}^{\text{tol.}}_{\text{21}}$
✗ | ✔ | ✔ | ✔ | 78.3 / 97.2 | 21.2 / 40.1
✔ | ✗ | ✔ | ✔ | — | 5.0 / 12.1
✔ | ✔ | ✗ | ✔ | 76.6 / 94.1 | 16.0 / 29.6
✔ | ✔ | ✔ | ✗ | 38.6 / 69.7 | 14.7 / 33.5
✔ | ✔ | ✔ | ✔ | 80.7 / 96.9 | 26.7 / 44.3
Table 5: Additional results of the ablation study. The metrics are the
accuracy with and without temporal tolerance ($\pm$1 class) on a 21-class
offset grid ranging from $-$2.0 sec to $+$2.0 sec with a step size of 0.2 sec.

|  | LRS3 (No face crop)
---|---|---
Depth | Params. |  Acc${}_{\text{21}}$ / Acc${}^{\text{tol.}}_{\text{21}}$
1 | 31.2M | 61.7 / 89.2
2 | 42.8M | 78.0 / 95.7
3 | 55.3M | 80.7 / 96.9
6 | 89.0M | 81.1 / 97.5

Table 6: The performance scales with the depth of the transformer modules and
saturates by the 3-layer capacity. The results are reported on the setting
with 21 classes with offsets from $-2.0$ to $+2.0$ with step size 0.2 sec.
Metrics are accuracy with and without temporal tolerance ($\pm$1 class).

As mentioned in Sec. 5, we report additional results with metrics without
temporal tolerance as well as the performance on LRS3-H.264 (‘No face crop’)
in Tab. 5. Moreover, we experiment with different numbers of layers in the
transformer modules, provide additional visualisation of attention, and show
the benefit of fine-tuning the model on a more diverse and large-scale
dataset.

#### Results with additional metrics and a dataset.

Similar to the results on the VGGSound-Sparse dataset, we see the importance
of each component in the final model performance when compared on the ‘Dense-
Sparse’ setting, _i.e_ LRS3-H.264 (‘No Face Crop’). Notice, the narrower gap
in performance compared to the models with and without selectors. This
highlights that the use of selectors becomes more important as the complexity
of the task increases (from ‘Dense’ to ‘Sparse’).

Interestingly, the feature extractors pre-training has a much more noticeable
effect on the VGGSound-Sparse compared with the LRS3 (‘No face crop’) dataset.
We attribute it to the simplicity of the talking-face dataset, which imposes
lower requirements on the diversity of audio-visual synchronisation cues. Such
cues are easier and faster to learn because they are quite generic across
videos.

#### Number of transformer layers.

Tab. 6 presents the impact of the transformer’s depth on the model’s
synchronisation performance. We vary the depths of two feature selector
transformers and the synchronisation transformer simultaneously. According to
the results, the performance grows as the capacity of the model increases,
however, the returns are diminishing. Thus, we regard the 3-layer option as
our final model.

Figure 8: Visual feature selectors focus on specific parts of the sparse
signal that is useful for synchronisation. Examples are from the hold-out set
of LRS3 (‘No face crop’). Attention is captured from a selector to a visual or
spectrogram feature token from a head within one of the layers. Attention
values are min-max scaled.

#### Additional attention visualisations.

We provide additional attention visualisation on the LRS3-H.264 (‘No Face
Crop’) dataset in Fig. 8. The illustration shows that the model performs well
regardless of gender, race, partial occlusion, position of a head within a
frame, as well as the different distance to a person, and background.

Pre-trained on | Fine-tuned on |  Acc${}_{\text{21}}$ / Acc${}^{\text{tol.}}_{\text{21}}$
---|---|---
LRS3 (‘No Face Crop’) | VGGSound-Sparse | 26.7 / 44.3
LRS3 (‘No Face Crop’) | VGGSound | 33.5 / 51.2
Table 7: Fine-tuning on a larger general-purpose dataset significantly
improves synchronisation performance. The results are shown on the test-set of
VGGSound-Sparse. Metrics are accuracy with and without temporal tolerance
($\pm$1 class).

#### Fine-tuning on a larger dataset.

We found that fine-tuning on a larger general-purpose dataset, _e.g_ VGGSound
(300+ classes), gives a significant boost to synchronisation performance of
the sparse classes. The results of this experiment are shown in Tab. 7.

###  7.3 Implementation Details

#### Feature extractors.

For visual input, we sample the video with 25fps and resize the frames such
that $\min(H_{v},\,W_{v})=256$ and take a random crop of $224^{2}$. The
5-second stack of cropped frames ($125\times 224\times 224\times 3$) is
encoded with S3D which results in a $16\times 7\times 7\times 1024$ visual
feature map. For the 5-second audio input, we convert a waveform sampled at
16kHz to a spectrogram with the STFT (512 in size with 128 hop length) and
apply a logarithm to the result. The log-spectrogram ($257\times 626$ is
passed to the ResNet18, which outputs an audio feature map of size $9\times
20$.

#### Feature selectors and audio-visual synchronisation transformer.

Before processing the visual feature map in the feature selector, a simple
$1\times 1$ convolution layer is used to map the visual features from $1024$-d
to $512$-d which is the dimension of the transformer blocks across the final
architecture. We use layer norm [Ba et al.(2016)Ba, Kiros, and Hinton] on both
audio and visual features and apply positional encoding after that. Both
feature selectors have the same number of trainable ‘selectors’
($k_{a}=k_{v}=16$) that are initialised with Gaussian noise. As has been
discussed in the ablation study, 3 cross-attention layers are used for each
feature selector and 3 attention layers in the audio-visual synchronisation
transformer. The layers have 8 heads.

#### Data augmentations.

We use two sets of augmentations for (pre-)training on LRS3 and finetuning on
VGGSound-Sparse. For the RGB stream, we use: random spatio-temporal crop222The
range of the temporal crop is limited such that the resulting clip could fit
the audio that was offset as well as the visual clip. and random horizontal
flip (with 50 % probability). For VGGSound-Sparse, we additionally may crop
smaller to $192^{2}$ (with 50 % probability) followed by upscaling to
$224^{2}$ as well as applying colour jittering and converting to a grey scale
(with 20 % probability and only for VGGSound-Sparse).

For the audio, we found that adding short audio jittering around the start of
the offset audio trim (_e.g_ $\sim U[-0.05,\,+0.05]$ seconds) stabilises
training. Additionally, for VGGSound-Sparse, we apply (with 20 % probability)
random reverberation, amplitude jittering, random pitch shift, a low-pass
filter, and add small Gaussian noise. We also employ SpecAugment [Park et
al.(2019)Park, Chan, Zhang, Chiu, Zoph, Cubuk, and Le] except for time
stretching as it could potentially mislead the synchronisation signal.

#### Training details.

The final synchronisation model is trained for 8 days on 8 NVidia V100 GPU
(32Gb) with half-precision on LRS3-H.264 followed by a few hours of finetuning
on VGGSound-Sparse. During finetuning, we use the same hyper-parameters as for
pre-training (except for the data augmentation as mentioned earlier). The
models are trained with a batch size of 10 video clips per GPU with a learning
rate of $5\cdot 10^{-6}$ scaled by the number of GPUs. We warm up training
with a 100-times lower learning rate and linearly increase it during the first
1k iterations. During training, Adam optimizer is used with
$\beta_{1,2}=[0.99,0.999]$ and the norm of the gradients is max-clipped to 1.
The code base relies on PyTorch (including torchvision and torchaudio) [Paszke
et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin,
Gimelshein, Antiga, et al.] and ffmpeg (version 4.3.2). Before training, we
fix the set of offsets for every video in the validation and test sets such
that we could evaluate using the same data during experimentation.

## References

  * [Afouras et al.(2018)Afouras, Chung, and Zisserman] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.  LRS3-TED: a large-scale dataset for visual speech recognition.  _arXiv preprint arXiv:1809.00496_ , 2018.
  * [Afouras et al.(2020)Afouras, Owens, Chung, and Zisserman] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * [Arandjelovic and Zisserman(2018)] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In _Proceedings of the European Conference on Computer Vision_ , pages 435–451, 2018.
  * [Ba et al.(2016)Ba, Kiros, and Hinton] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * [Casanovas and Cavallaro(2015)] Anna Llagostera Casanovas and Andrea Cavallaro.  Audio-visual events for multi-camera synchronization.  _Multimedia Tools and Applications_ , 2015.
  * [Chen et al.(2020)Chen, Xie, Vedaldi, and Zisserman] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  VGG-Sound: A large-scale audio-visual dataset.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [Chen et al.(2021)Chen, Xie, Afouras, Nagrani, Vedaldi, and Zisserman] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman.  Audio-visual synchronisation in the wild.  In _Proceedings of the British Machine Vision Conference_ , 2021.
  * [Chung and Zisserman(2016a)] Joon Son Chung and Andrew Zisserman.  Out of time: automated lip sync in the wild.  In _Workshop on Multi-view Lip-reading, ACCV_ , pages 251–263, 2016a.
  * [Chung and Zisserman(2016b)] Joon Son Chung and Andrew Zisserman.  Lip reading in the wild.  In _Proceedings of the Asian Conference on Computer Vision_ , pages 87–103, 2016b.
  * [Chung et al.(2019)Chung, Chung, and Kang] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang.  Perfect match: Improved cross-modal embeddings for audio-visual synchronisation.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , 2019.
  * [Dassani et al.(2019)Dassani, Bird, and Cliff] Vansh Dassani, Jon Bird, and Dave Cliff.  Automated composition of picture-synched music soundtracks for movies.  In _Proceedings of the European Conference on Computer Vision_ , 2019\.
  * [Doersch et al.(2015)Doersch, Gupta, and Efros] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In _Proceedings of the International Conference on Computer Vision_ , pages 1422–1430, 2015.
  * [Ebeneze et al.(2021)Ebeneze, Wu, Wei, Sethuraman, and Liu] Joshua P. Ebeneze, Yongjun Wu, Hai Wei, Sriram Sethuraman, and Zongyi Liu.  Detection of audio-video synchronization errors via event detection.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , 2021.
  * [Halperin et al.(2019)Halperin, Ephrat, and Peleg] Tavi Halperin, Ariel Ephrat, and Shmuel Peleg.  Dynamic temporal alignment of speech to lips.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , 2019.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * [Hershey and Movellan(1999)] John Hershey and Javier Movellan.  Audio vision: Using audio-visual synchrony to locate sounds.  _Advances in Neural Information Processing Systems_ , 1999.
  * [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Zisserman, Vinyals, and Carreira] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.  Perceiver: General perception with iterative attention.  _arXiv preprint arXiv:2103.03206_ , 2021.
  * [Kadandale et al.(2022)Kadandale, Montesinos, and Haro] Venkatesh S Kadandale, Juan F Montesinos, and Gloria Haro.  VocaLiST: An audio-visual synchronisation model for lips and voices.  _arXiv preprint arXiv:2204.02090_ , 2022.
  * [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola, Green, Back, Natsev, et al.] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ , 2017.
  * [Khosravan et al.(2019)Khosravan, Ardeshir, and Puri] Naji Khosravan, Shervin Ardeshir, and Rohit Puri.  On attention modules for audio-visual synchronization.  In _Workshop on Sight and Sound, CVPR_ , 2019.
  * [Kim et al.(2021)Kim, Heo, Chung, and Lee] You Jin Kim, Hee Soo Heo, Soo-Whan Chung, and Bong-Jin Lee.  End-to-end lip synchronisation based on pattern classification.  In _SLT Workshop_ , 2021.
  * [Korbar et al.(2018)Korbar, Tran, and Torresani] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Co-training of audio and video representations from self-supervised temporal synchronization.  In _Advances in Neural Information Processing Systems_ , 2018.
  * [Owens and Efros(2018)] Andrew Owens and Alexei A. Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In _Proceedings of the European Conference on Computer Vision_ , 2018\.
  * [Park et al.(2019)Park, Chan, Zhang, Chiu, Zoph, Cubuk, and Le] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  SpecAugment: A simple data augmentation method for automatic speech recognition.  _arXiv preprint arXiv:1904.08779_ , 2019.
  * [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, et al.] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * [Rabiner and Juang(1993)] Lawrence Rabiner and Biing-Hwang Juang.  _Fundamentals of speech recognition_.  Prentice-Hall, Inc., 1993.
  * [Radiocommunication()] ITU Radiocommunication.  Relative timing of sound and vision for broadcasting.
  * [Shrestha et al.(2010)Shrestha, Barbieri, Weda, and Sekulovski] Prarthana Shrestha, Mauro Barbieri, Hans Weda, and Dragan Sekulovski.  Synchronization of multiple camera videos using audio-visual features.  _IEEE Transactions on Multimedia_ , 2010.  10.1109/TMM.2009.2036285.
  * [Slaney and Covell(2000)] Malcolm Slaney and Michele Covell.  Facesync: A linear operator for measuring synchronization of video facial images and audio tracks.  _Advances in Neural Information Processing Systems_ , 2000.
  * [Staelens et al.(2012)Staelens, De Meulenaere, Bleumers, Van Wallendael, De Cock, Geeraert, Vercammen, Van den Broeck, Vermeulen, Van de Walle, et al.] Nicolas Staelens, Jonas De Meulenaere, Lizzy Bleumers, Glenn Van Wallendael, Jan De Cock, Koen Geeraert, Nick Vercammen, Wendy Van den Broeck, Brecht Vermeulen, Rik Van de Walle, et al.  Assessing the importance of audio/video synchronization for simultaneous translation of video sequences.  _Multimedia systems_ , 2012.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * [Wei et al.(2018)Wei, Lim, Zisserman, and Freeman] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman.  Learning and using the arrow of time.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8052–8060, 2018.
  * [Xie et al.(2018)Xie, Sun, Huang, Tu, and Murphy] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _Proceedings of the European Conference on Computer Vision_ , pages 305–321, 2018.
  * [Xu et al.(2021)Xu, Xiong, Chen, Li, Xia, Tu, and Soatto] Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto.  Long short-term transformer for online action detection.  In _Advances in Neural Information Processing Systems_ , 2021.