# AVA-AVD: Audio-visual Speaker Diarization in the Wild

Eric Zhongcong Xu1 Zeyang Song1 Chao Feng1 Mang Ye2 Mike Zheng Shou1

1National University of Singapore 2Wuhan University
Corresponding author

###### Abstract

Audio-visual speaker diarization aims at detecting “who spoken when“ using
both auditory and visual signals. Existing audio-visual diarization datasets
are mainly focused on indoor environments like meeting rooms or news studios,
which are quite different from in-the-wild videos in many scenarios such as
movies, documentaries, and audience sitcoms. To create a testbed that can
effectively compare diarization methods on videos in the wild, we annotate the
speaker diarization labels on the AVA movie dataset and create a new benchmark
called AVA-AVD. This benchmark is challenging due to the diverse scenes,
complicated acoustic conditions, and completely off-screen speakers. Yet, how
to deal with off-screen and on-screen speakers together still remains a
critical challenge. To overcome it, we propose a novel Audio-Visual Relation
Network (AVR-Net) which introduces an effective modality mask to capture
discriminative information based on visibility. Experiments have shown that
our method not only can outperform state-of-the-art methods but also is more
robust as varying the ratio of off-screen speakers. Ablation studies
demonstrate the advantages of the proposed AVR-Net and especially the modality
mask on diarization. Our data and code will be made publicly available at
https://github.com/zcxu-eric/AVA-AVD.

##  1 Introduction

Diarization is an appealing topic in both speech technology and computer
vision. A diarization system can localize the start time and end time of a
person’s speech and identify the speaker’s identity. It can benefit many real-
world applications such as news video summarization and human-robot
interaction.

Currently, audio-visual diarization datasets [7, 9, 22] are mainly captured in
indoor environments such as meeting rooms and news studios, which are less
challenging due to:

Limited diversity. In indoor environments, traffic noise, background music,
and scene changes are less frequently observed. Also, most of them are not
multilingual. The diversity of these datasets is limited, thus systems
developed on these datasets are hard to generalize well for in-the-wild
scenarios (Fig. 1), such as movies, documentaries, and audience sitcoms.

Lack of completely off-screen speakers. Another special feature for in-the-
wild videos is the completely off-screen speakers, such as the speaker on the
phone or radio, and voiceover. These speakers never appear in the field of
view and audio-visual correspondence is missing. Thus, completely off-screen
speakers deserve further investigation.

Figure 1: Example scenes in audio-visual diarization datasets. Existing
datasets (AMI [7] and AVDIAR [22]) only have silent indoor environments, while
our AVA-AVD has more complex scenarios with daily activities. In particular,
AVA-AVD has many compeletely off screen speakers such as speech on the phone
and voiceover.

While early work in speaker diarization solely depends on the audio stream
[54, 64, 71], recent works [10, 9, 22] start to attempt leveraging both audio
and visual cues. These methods either fuse the similarity scores of two
independent modalities or incorporate the synchronization between utterance
and lip motion. However, they have several issues when dealing with videos in
the wild:

Ignore cross-modal biometrics. The consistency between the face and voice [43,
44, 20] is a useful cue for the speaker diarization. For example, a female
speaker may have a higher voice frequency; a heavier person may have a wide
range of sound intensities [20, 3]. Humans can by nature pay attention to such
cross-modal link and then separate and group the speakers in a long video.
However, the commonly used late fusion strategy cannot fully explore the
relationships between utterances and faces.

Miss off-screen speakers. Several works adopt speaker enrollment using audio-
visual correspondence. The speakers who never appear in the video frame will
not be successfully enrolled. This failure makes these methods less robust to
the off-screen speakers. The recent work [10] runs an audio-visual
synchronization model offline and enroll speakers in advance based on the
profiles provided by the users, but this paradigm would fail to enroll the
invisible speakers.

The roadblock to remedy the above issues is the lack of a proper testbed i.e.
an in-the-wild audio-visual diarization benchmark. To this end, we first
introduce a challenging audio-visual diarization dataset, AVA-Audio-visual
Diarization (AVA-AVD). The video sources of this dataset are the AVA-Active
Speaker [50]. We select 117 unique videos which have high resolution and no
dub. As shown in Fig. 1, AVA-AVD has diverse complicated scenarios with a wide
spectrum of daily activities.

In addition, we propose an Audio-visual Relation Network (AVR-Net). Depending
on the face visibility of two candidate speakers, we design 4 different
subtasks and introduce a task-aware mask for each subtask. Specifically, our
subtasks are A vs. A, A vs. A-V, A-V vs. A, and A-V vs. A-V, where A denotes
audio, and A-V denotes audio-visual. Moreover, instead of learning
discriminative embedding vectors and measuring distances linearly [44, 43, 53,
66, 49], our model utilizes Siamese [38] architecture and the distance metric
is also optimized during training.

In summary, our main contributions are as follows:

  * •

We introduce a new challenging audio-visual diarization dataset AVA-AVD, and
experimentally show that SOTA methods perform poorly on such in-the-wild
videos.

  * •

Besides the test set, we also provide a good amount of valuable training data,
which can decrease diarization error rate by 17.1%.

  * •

We propose Audio-visual Relation Network (AVR-Net) which can handle missing
modalities using learnable task-aware masks. Comparing to SOTA methods, our
model improves from 22.02 to 20.57 with an absolute gain of 6.6%, and its
average diarization error rate with missing modality is 2.8% lower.

##  2 Related Work

Audio-only Speaker Diarization. Speaker diarization using only audio streams
has been studied extensively in the recent years. Most of the diarization
systems [51, 68, 41, 64] employ multi-stage framework, including voice
activity detection [35], embedding extraction [37, 15, 56, 63], and clustering
[14, 46, 18]. Each component of the system will affect the final diarization
performance; handling overlapping speeches is challenging for these methods.
Another promising direction is end-to-end diarization [42, 27, 31]. It has
exhibited several advantages, such as simpler pipeline and the ability to cope
with speech overlap. But end-to-end methods can only deal with a limited
number of speakers [42, 27] and they rely heavily on the synthetic data for
training [31]. Currently, the performance of end-to-end systems still cannot
outperform their multi-stage counterparts.

Audio-visual Speaker Diarization. Facial attributes and lip motion are highly
related to speech [69]. Therefore, several methods that leverage audio and
visual cues for diarization are motivated by the synergy between utterances
and lip movements. These methods adopt techniques such as mutual information
[26, 47], canonical correlation analysis [28], and deep learning [12, 13, 10].
In recent works, audio-visual correspondence is also used for associating
talking faces and voice tracks [60, 9, 67]. Based on the paired audio-visual
inputs, different modalities are fused by linear combination [9, 10], temporal
alignment [19, 36], or bayesian method [22].

In contrast, our model takes both audio and visual features as input and can
learn to compute the similarity of two speakers. Instead of using linear
metrics such as cosine similarity or Euclidean distance, the measurement
function in our model is also optimized during training.

Speaker Verification with Faces and Voices. Off-the-shelf verification methods
including speech verification [63, 8] and face recognition [17, 48] are often
applied to extract discriminative embeddings. These models are pretrained on
large-scale datasets [11, 45, 5, 23] with a large number of identities. Thus,
they can generalize well on the unseen videos. Based on the pretrained models,
many interesting downstream tasks have been explored, such as deep clustering
[33, 30], face clustering [61, 34], person search [67], person clustering [4],
as well as speaker diarization [41, 10]. The above verification models are all
uni-modal. However, cross-modal links between one person’s face and voice can
also reveal the speaker’s identity and has been extensively studied in [43,
44, 20, 66, 65]. To this end, our model also leverages this cross-modal
correlation and dynamically learns both uni-modal and cross-modal similarities
in a unified framework.

Dataset | access | scenario | #videos | duration | #IDs | #speakers | #os speakers | speech% | language
---|---|---|---|---|---|---|---|---|---
AMI [7] Corpus | A✓V✓ | meetings | 684 | 100h | 189 | 3/4.0/5 | 0/0/0 | 50.5/83.2/96.4 | En
AVDIAR [22] | A✓V✓ | chat | 27 | 21m | 11 | 1/2.2/4 | 0/0/0 | 23.9/82.6/96.2 | En,Fr
VoxConverse [9] | A✓V✗ | debate,discussion | 448 | 63h50m | - | 1/5.6/21 | - | 10.7/91.3/100.0 | En
AVA-AVD (Ours) | A✓V✓ | diverse daily activities | 351 | 29h15m | 1500 | 2/7.7/24 | 0/2.4/20 | 12.0/46.0/84.3 | Multi
Table 1: Comparison with existing audio-visual speaker diarization datasets.
os speakers denote completely off-screen speakers. Numbers separated by $/$
are minimum, average, and maximum. AVA-AVD is multilingual while others have
only 1 or 2 languages.

Modeling Cross-modal Relationships. Apart from feature extraction, deep neural
networks can also model the relationship between two samples. Learning a
feature comparison function achieves great success in a range of interesting
tasks, such as patch-based matching [24, 70], few-shot classification [58, 62,
38], and object relation reasoning [52, 29, 73]. Especially, several previous
works have explored modeling audio-visual relationships for cross-modal
verification [44], audio-visual speaker recognition [59], and unsupervised
audio-visual correspondence learning [2, 39, 12]. Unlike any of the above, our
work tackles audio-visual speaker diarization by predicting similarity scores
rather than using fixed linear metrics [55].

##  3 AVA-AVD – An In-the-wild Audio-visual Diarization Benchmark

In this section, we describe the video source (Sec. 3.1), our annotation
process (Sec. 3.2), and the statistics (Sec. 3.3) of AVA Audio-visual
Diarization (AVA-AVD) dataset. Moreover, recruitment of participants will be
discussed in Sec. 3.4.

###  3.1 Video Source

Following the recent works [1, 4], instead of collecting or recording new
videos, we build AVA-AVD dataset upon a publicly available dataset, i.e. AVA-
Active Speaker[50].

Motivations. There are three main reasons: (1) the diverse acoustic and visual
scenarios in [50] is challenging for audio-visual diarization task; (2) active
speaker detection is a prior step for diarization, by adding extra
annotations, we can enrich this dataset and set up a complete benchmark; (3)
with the ground truth face labels and active speaker labels, we can stay
focused on the components which are more related to diarization task itself
(similar to the protocol in [54]).

Original annotations. AVA-Active Speaker has 144 videos in total and this
collection features movies with different races, ample dialogues, and diverse
languages and genres. It also has labels for faces as well as active speakers
which can aid the association between utterances and faces. We select 117
high-quality videos without any dub because: (1) audio-visual correspondence
is destroyed in dubbed videos; (2) it could be ambiguous if the same person
dubs for more than one characters. In each video, a 15-minute clip is
annotated by AVA-Active Speaker. Noted that the time complexity to verify
speaker ID is $O(N^{2})$, we need to control the number of speakers within the
video to reduce the time cost. Therefore, we further divide a 15-minute clip
into three 5-minute clips and add diarization labels respectively.

###  3.2 Annotation Process

Each video clip is annotated through the following steps: Generating face and
voice tracks. Annotating faces frame by frame and labeling speech segments are
time-consuming. Since AVA-Active Speaker provides annotated face crop tracks,
we keep face tracks that contain audible speeches. Noted, one single face
track will be divided into several short tracks if the speech is not
continuous. Further, rough speech segment proposals are automatically
generated using the timestamp of speaking face tracks.

Labeling face tracks. Given all kept face tracks, we need to identify their
speaker ID. Human annotators will watch the video and listen to the concurrent
utterances to assign video-level identity labels for the faces. Details of our
annotation tool can be found in Sec. A.3.

Adjusting speech segments. Some speakers are temporally invisible in the video
frame. Thus, human annotators need to refine the onset and offset of the
speech segment proposals and get accurate segment labels. Also, the
neighboring short segments produced by the same speaker will be merged.

Labeling off-screen speakers. Another challenging feature of in-the-wild
videos is that there may exist completely off-screen speakers, e.g. speaker
talking on the phone, broadcasting on the radio, or voiceover. In this stage,
annotators watch and listen to the video clips thoroughly and annotate the IDs
for all the utterances generated by off-screen speakers. Relying only on the
voice similarities, annotators will set the ID of each utterance to either an
existing speaker or a new speaker.

Manual verification. At last, the annotations will be manually verified by
different auditors. The average time cost for completing the above steps is 8
times of the video duration.

Figure 2: Overview of our audio-visual diarization system. We use a sliding
window approach to split active audio segments into short 2s segments using a
stride of 0.5s. The faces in red boxes are invisible. Our model (AVR-Net)
directly predicts similarity score for two candidate speakers.

(a)

(b)

(c)

(d)

Figure 3: Visualization of AVA-AVD statistics. (a) Histogram showing the
number of clips vs. the number of speakers in each clip; (b) Histogram showing
the number of clips vs. the number of completely off-screen speakers in each
clip; (c) Histogram showing the duration of speech segments; (d) Histogram
showing the duration of speech segments from off-screen speakers.

###  3.3 Statistics

Comparisons with existing datasets. Table 1 summarizes statistics of existing
audio-visual diarization datasets. Compared with the existing datasets which
only have indoor activities such as meetings, free chat, and debates, AVA-AVD
has a wide spectrum of human daily activities. It provides around 1500 unique
identities with voice tracks and/or faces. We compute the number of identities
by mapping the face tracks into global labels based on the annotations
released by [1], the number of visible speakers is 1044, and the number of
off-screen speakers is approximated using speech length. Fig. 3(a) and 3(c)
illustrate the number of speakers and the long-tail utterance length
distribution. Especially, the statistics for the off-screen speakers can be
found in Fig. 3(b) and 3(d). Also, differing from AMI and VoxConverse, our
AVA-AVD also has multiple languages.

Train/val/test split. We split AVA-AVD dataset into training, validation, and
testing subsets. Current audio-visual diarization datasets [9, 4] do not hold
out train set because: (1) the number of identities is limited; (2)
diarization dataset only provides video-level identity labels. However,
considering that AVA-AVD has a large number of identities, it could
potentially benefit training audio-visual speaker verification models and end-
to-end diarization models. Thus, we still partition train set, the necessity
of training data will be further discussed in Sec. 5.4. Since each 15-minute
video is divided into three 5-minute clips, we will not split the clips from
the same movie into different subsets to keep the disjoint property across
subsets. Therefore, our split can reflect models’ generalization ability in
the unseen scenarios. In addition, we also keep the test split consistent with
the validation set in AVA-Active Speaker. The main advantage is that an active
speaker detection model pretrained on AVA can be applied directly without any
speaker leakage issue.

Finally, we hold out 243 clips for training, 54 clips for validation, and 54
clips for testing.

###  3.4 Participants

The video source of AVA-AVD is an existing public video dataset. We do not
record any new videos and thus do not recruit any participants for data
collection. Thus, we could waive the participant-related requirements from
Institutional Review Board (IRB) protocols.

In total, we recruit 10 annotators and 2 auditors. The only requirement for
these human annotators is that they must be able to identify and localize a
person leveraging both visual and auditory cues.

##  4 Method

Our audio-visual diarization system is shown in Fig. 2. We first define the
diarization problem (Sec. 4.1); then we present the audio-visual relation
network and introduce how we learn audio-visual similarity with missing faces
(Sec. 4.2); more details of training, inference, and implementation can be
found in Appendix A.1.

###  4.1 Problem Definition

Prior steps. Given a video with multiple speakers (Fig. 2), our goal is to
localize the audible utterances and label them with video level speaker
identity leveraging audio and visual information. Following a commonly adopted
multi-stage paradigm [9, 10, 64], we can either run face detection and
tracking [72], voice activity detection[40], and active speaker detection [60]
in advance or directly use the ground truth results [41]. Then, we get a set
of speech segments $A_{n}^{[s_{n},e_{n}]}$, where $n\in\\{1,2...N\\}$, $s_{n}$
and $e_{n}$ are onset and offset. We also have the talking faces $I_{t}^{k}$
at each time instant $t$, where $t\in\\{1,2...T\\}$, $k\in\\{1,2...K_{t}\\}$,
and $K_{t}$ is the maximum number of speakers at $t$. For each speech segment,
we ensemble an utterance $A_{n}^{[s_{n},e_{n}]}$ and one image $I_{i}$ sampled
from track $\\{I_{t}^{k}$, $t\in[s_{n},e_{n}]\\}$ into an audio-visual pair.

Diarization. Our goal is to train a network $F_{\phi}$, which computes the
similarities between two audio-visual pairs
$s_{i,j}=F_{\phi}((A_{i}^{[s_{i},e_{i}]},I_{i}),(A_{j}^{[s_{j},e_{j}]},I_{j}))$.
Then we get the similarity graph represented by adjacent matrix $S^{N\times
N}$. Based on the graph $S$, we perform agglomerative hierarchical clustering
[14] following [9, 10, 41] and finally get the diarization results. In
previous works, similarities are measured by cosine similarity or Euclidean
distance based on the embedding vectors extracted by off-the-shelf models,
while we integrate these two separated steps into one model (Sec. 4.2).

###  4.2 Audio-visual Relation Network

Backbone. Pretrained on large-scale datasets, speaker verification, and face
recognition models have strong generalization ability and are directly applied
in various downstream tasks [33, 30, 61, 34, 4, 67, 41, 10]. We follow these
works and utilize the backbones of off-the-shelf models [8, 17] to encode
voice and face features respectively. The architectures of these models are
similar to ResNet [25]. Since our audio-visual relation network needs to learn
both uni-modal and cross-modal relationships, we use the feature maps of the
last ResNet layer. This design will be studied in Sec. 5.6.

Relation module. As shown in Fig. 4, given two audio-visual pairs
$(A_{i}^{[s_{i},e_{i}]},I_{i}),(A_{j}^{[s_{j},e_{j}]},I_{j})$, the audio and
visual backbones encode them into intermediate feature pairs. The audio
dimension is $C_{A}\times H_{A}\times W_{A}$ and the face dimension is
$C_{I}\times H_{I}\times W_{I}$. We first apply adaptive pooling on audio
features and then concatenate two modalities along the channel axis.
Therefore, each audio-visual pair has a dimensionality of $(C_{I}+C_{A})\times
H_{I}\times W_{I}$. Then we can further concatenate these two pairs and feed
them into the relation module. However, in the audio-visual diarization
dataset, it is common to encounter off-screen speakers due to low resolution,
sharp motion, light conditions, head pose, or voiceover. Depending on the face
visibility, we distinguish the comparison cases. Specifically, here we have
four cases: A vs. A, A vs. A-V, A-V vs. A, and A-V vs. A-V. To handle
different cases within a single network, we design learnable task-aware masks
$M_{i}=\mathbb{R}_{i}^{D}$, $i\in\\{0,1,2,3\\}$, where $D=(C_{I}+C_{A})*2$.
The learnable input masks are inspired by [6, 57]. Prior to the relation
module, we compute interactions between audio-visual features and the task-
aware mask by multiplying them in the channel dimension. With the task-aware
masks, our model can dynamically choose the input features. Our relation
module consists of two stacked ResNet-style layers. Finally, a simple MLP
layer maps the output into one dimension and then predicts the similarity
score using Sigmoid.

Learning objective. AVR-Net is trained by:

$\phi\leftarrow\underset{\phi}{\operatorname{argmin}}\sum_{i=1}^{N}\sum_{j=1}^{N}\left(s_{i,j}-\mathbf{1}\left(y_{i}==y_{j}\right)\right)^{2},$
(1)

where $N$ is the batch size, $y_{i}$ and $y_{j}$ denote the identity labels.
The goal of our AVR-Net is regressing 1 if two pairs are from the sample
person and 0 vice versa. Similar to [58], we adopt mean square error (MSE)
loss because the similarity score is continuous and can be considered a
regression problem.

Figure 4: Our Audio-visual Relation Network ingests two audio-visual pairs. We
use pretrained backbones to extract voice and face features. The intermediate
features are concatenated and multiplied with a learnable task-aware mask,
which is selected according to the face visibility. Relation module predicts
the similarity score of two candidate speakers directly.

##  5 Experiments

In this section, we first run the state-of-the-art systems to study the
challenges of AVA-AVD (Sec. 5.3). Then, we investigate the effectiveness of
AVA-AVD training data (Sec. 5.4). Third, we compare AVR-Net with baseline
methods (Sec. 5.5) and finally present the ablation study (Sec. 5.6).

###  5.1 Datasets

VoxCeleb [45] contains over 100,000 utterances for 1,251 celebrities extracted
from Youtube videos. We use their official train set which has 1211 speakers,
and their faces are extracted by [44] at 1 fps. In the pretraining stage, the
VoxCeleb test set is used for evaluating the equal error rate (EER) for
speaker verification.

VoxCeleb2 [11] contains over 1 million utterances for 6,112 celebrities
extracted also from videos on Youtube with cropped and associated face tracks.
In our experiments, we only use the development set which contains 5994 unique
identities.

###  5.2 Evaluation Metric

The diarization performance is evaluated by Diarization error rate (DER). It
contains three components: Missing Detection (MS), False Alarm (FA), and
Speaker Error (SPKE). All of these components are lower the better. We use the
evaluation tools released by [32] and adopt the Fair protocol defined in [41],
which considers overlap speech and includes an acceptance margin of 0.25s.

###  5.3 Evaluate the Challenges of AVA-AVD

To evaluate the challenging characteristics of our AVA-AVD dataset, we first
run the state-of-the-art audio-visual (WST [10]) and audio-only (VBx [41])
diarization systems. These systems do not require any training data, we
directly finetune the threshold on AVA-AVD validation set.

| Test set | VAD | MS | FA | SPKE | DER
---|---|---|---|---|---|---
WST[10] | AMI-ES | GT | 5.6 | 0.0 | 4.8 | 10.4
W | 10.5 | 6.6 | 4.0 | 21.1
AMI-IS | GT | 6.5 | 0.0 | 7.1 | 13.6
W | 11.2 | 4.0 | 6.2 | 21.3
AVA-AVD | GT | 3.08 | 0.0 | 38.95 | 42.04
E | 11.63 | 40.63 | 36.15 | 88.41
VBx[41] | AMI | GT | 9.55 | 0.0 | 2.98 | 12.53
E | 30.28 | 1.05 | 7.32 | 38.65
AVA-AVD | GT | 2.92 | 0.0 | 18.45 | 21.37
E | 8.80 | 75.03 | 20.54 | 104.37
Table 2: Performance of previous diarization systems. DER is lower the better.
WST method is reproduced by ourselves. GT means oracle VAD derived from
diarization labels. W means WebRTC[35] VAD algorithm. E means the simple
energy-based VAD method introduced in [40]. We report results with energy-
based VAD because it yields slightly better results.

Results of SOTA audio-visual diarization method. The WST system has good
performance on AMI: 10.4 on AMI-ES subset, and 13.6 on AMI-IS subset. It
adopts an enrollment pipeline, they use the ground truth speaker profiles and
a synchronization model [12] to enroll the audio segments of each speaker.
However, this enrollment process would fail for the completely off-screen
speakers due to the lack of audio-visual correspondence. Therefore, WST only
achieves a DER of 42.04 on AVA-AVD. We can see that DER is much higher because
of the severe speaker error which mainly comes from the enrollment failures.

Results of SOTA audio-only diarization method. Another method (VBx) is an
audio-only diarization system. We use the ResNet101 model released by [41].
Different from WST, it employs clustering instead of enrollment. Completely
off-screen speakers will not cause any system-level failure for VBx, thus it
outperforms WST by 49.17% on AVA-AVD. However, compared with AMI (SPKE of
2.98, DER of 12.53), VBx has a much higher SPKE (18.45) and DER (21.37) on
AVA-AVD. The reason for the high speaker error is that AVA-AVD has more
speakers and complicated scenarios.

Challenging acoustic environments. To evaluate the acoustic conditions in our
AVA-AVD, we also study the effects of voice activity detection (VAD). For all
of the experiments, we report the results using energy-based VAD and oracle
VAD respectively. In general, the false alarm of both WST (40.63) and VBx
(75.03) on AVA-AVD is quite high because AVA-AVD has a large amount of
background music, traffic noise, laughter, etc. WST is slightly better due to
the enrollment mechanism, but its false alarm is significantly higher than AMI
(6.6 and 4.0) where videos are only recorded in silent meeting rooms. Thus,
different from other datasets with only indoor scenarios, simple energy-based
VAD systems cannot work well on AVA-AVD. Other VAD methods such as combining
VAD with ASR [40] deserve further investigation but this is out of the scope
of this work. In the following experiments, to study the performance of AVR-
Net, we only report results using oracle VAD and ground truth active speaker
labels.

###  5.4 Necessity of AVA-AVD Train Set

Diarization is considered as a downstream task for speaker verification,
existing audio-visual datasets [7, 22] have less diversity and thus no data is
reserved for training. However, since our AVA-AVD dataset has much more
identities, challenging auditory environments, and multiple languages, we also
split a train set and investigate its effectiveness.

Figure 5: Diarization results of AVR-Net trained on different datasets. The
circle size represents the scale of the training data. Stars represent the
results of AVR-Net combined with a pretrained face recognition model [17]
using late-fusion.

Training data mixture. VoxCeleb and VoxCeleb2 development set do not have
speaker overlap, the combination is by far the largest speaker verification
dataset, thus they are usually mixed up [51, 37] for training. To evaluate the
AVA-AVD train set, we train AVR-Net on different dataset combinations and
their diarization results on AVA-AVD are shown in Fig. 5. Although the scale
of AVA-AVD is much smaller, the model trained only on AVA-AVD achieves a
comparable result (27.35) to the model trained only on VoxCeleb2 (25.70). With
the increasing amount of data, the performance can be further improved. DER
drops slightly by 3.5% when the VoxCeleb2 and VoxCeleb are mixed. Yet, adding
AVA-AVD yields a much larger performance gain, the DER is further decreased
from 24.81 to 20.57 by a margin of 17.1%. Moreover, the model trained on AVA-
AVD can get faster convergence, the iterations of training on AVA-AVD is 26.2%
less than on VoxCeleb mixture.

Fused with pretrained model. Although the mixture of VoxCeleb series datasets
is the largest, the scale of face data in this combination is not comparable
with large-scale face recognition datasets [5, 23]. Therefore, we also use a
pretrained face recognition model [17] and fuse the predicted similarity
scores with AVR-Net to evaluate the performance gain with the existence of the
powerful face model. The results are marked with stars in Fig. 5. For the
models trained only on AVA-AVD or VoxCeleb series datasets, this late fusion
strategy increases their results. But the improvement of the models trained on
three datasets is negligible. However, although late-fused with a pretrained
model, AVR-Net trained together with AVA-AVD can still outperform others. The
main reason is the domain gap, AVA-AVD has complex scenarios while VoxCeleb
series are all extracted from interviews on Youtube. Also, AVA-AVD has
multiple languages. By including the AVA-AVD train set, we can boost the
diarization performance by 19.09%.

| SPKE | DER
---|---|---
Single | 22.17 | 25.09
Siamese | 17.65 | 20.57
(a) Model architecture.

| SPKE | DER
---|---|---
Raw | 25.45 | 28.37
Embed | 18.08 | 21.00
Feat | 17.65 | 20.57
(b) Input features.

| SPKE | DER
---|---|---
Concat | 19.37 | 22.29
Alter Concat | 19.69 | 21.98
Mask | 17.65 | 20.57
Alter Mask | 17.87 | 20.79
(c) Alternative training.

| SPKE | DER
---|---|---
None | 18.37 | 21.29
Hand-crafted Mask | 18.44 | 21.36
Partially Learnable Mask | 18.54 | 21.46
Learnable Mask | 17.65 | 20.57
(d) Learnable task-aware masks.

Table 3: AVR-Net ablation experiments. We report diarization performance of
different alternatives. Default settings are marked with gray.

###  5.5 Comparisons with State-of-the-art

####  5.5.1 Baselines

In this section, we compare AVR-Net with strong baseline models on AVA-AVD
benchmark:

Speech+Face is a multi-stage diarization method using the off-the-shelf face
recognition model [17] and speaker verification model [8] which is pretrained
by ourselves on the same datasets (VoxCeleb, VoxCeleb2, and AVA-AVD);

DIMNet is a disjoint mapping network [66], it learns to map the speech and
face into a shared embedding space, where embeddings of the same person are
close and otherwise far apart. It has been proved effective in both speaker
verification and cross-modal verification [53];

GMU is a recent SOTA audio-visual speaker verification model [49], which
learns to fuse the audio and visual embeddings using gated multimodal unit.

####  5.5.2 Comparisons with State-of-the-art

Table 4 shows that our AVR-Net outperforms all of its strong counterparts. The
pretrained baseline, Speech+Face, achieves a DER of 24.18. Our AVR-Net yields
the best result 20.57 because it learns to compute the similarity of two
audio-visual pairs, while the pretrained model only employs linear similarity
metric and late-fusion strategy, which are suboptimal. Also, the cross-modal
biometrics are ignored by this pretrained baseline. DIMNet maps audio and
visual inputs to the same embedding space by sharing a classifier, but it is
an extension of the pretrained baseline and does not explore the cross-modal
links between face and voice from different speakers neither. It is even
slightly worse than Speech+Face because the face train set is much smaller.
The result of GMU is 1.76 higher than AVR-Net. Although GMU can learn cross-
modal attention to fuse the audio and visual embeddings dynamically, it is a
single-stream model which is harder to be optimized on diarization dataset.

| SPKE | Avg SPKE | DER | Avg DER
---|---|---|---|---
Speech[8]+Face[17] | 21.48 | 22.98 | 24.40 | 25.90
DIMNet[66] | 19.77 | 21.06 | 22.69 | 23.98
GMU[49] | 19.10 | 20.10 | 22.02 | 23.02
Ours | 17.65 | 19.45 | 20.57 | 22.37

Table 4: Comparisons with state-of-the-art models. SPKE and DER are computed
with an missing rate of 0. Avg SPKE and Avg DER are average results across
different missing rates. Figure 6: Diarization results under different missing
rates. We increase the missing rate from 0 to 1.0 with a step of 0.1.

Robustness against increasing missing rates. AVA-AVD has off-screen speakers,
the original missing faces in AVA-AVD account for 50%. To investigate the
models’ robustness against missing faces, we further synthesize additional
missing faces with a certain probability (missing rate) and evaluate the
diarization performance. All of the parameters are fixed during evaluation.
Fig. 6 shows that our AVR-Net has the best performance when missing rate is
smaller than 0.8. However, when the missing rate increases further, GMU is
better than our model. The main reason is that during training, the missing
probability is set to 50%, it is suboptimal for higher missing rates. The
average values are summarized in Table 4, although our model cannot outperform
GMU under high missing rates, the average SPKE 19.45 and average DER 22.37 are
still the best.

###  5.6 Ablation Study

Learning objective. Training AVR-Net is not straightforward because AVA-AVD
only has video-level labels. Two solutions are discussed in Appendix A.1.1.
Table 3(a) compares the Siamese network [38] with single-stream architecture.
It shows that single-stream network has a much higher DER than AVR-Net. AVR-
Net can regress the similarity of two speakers, this objective is not as
strict as the classification supervision used in the single-stream network.
Our architecture is well-suited for the large-scale diarization dataset with
only video-level identity labels.

Input features. Recent face clustering [61], multi-modal clustering [30, 33],
and verification [49] methods learn the threshold or unified representation
based on the embedding vectors extracted by pretrained models. Hence, we also
study the influence of inputs for our AVR-Net. We have three input options:
(1) raw face and audio stream; (2) embeddings extracted by pretrained models;
(3) feature maps encoded by pretrained backbones (ours default setting).

Table 3(b) shows that the AVR-Net using raw input has a much higher DER of
28.37. Although the embedding vectors are discriminative enough and can
achieve 21.00, our AVR-Net architecture yields a better result because AVR-Net
encodes informative features which are more discriminative and helpful for
computing cross-modal links and comparing speakers.

Figure 7: Task masks learned by AVR-Net. Red represents larger weights while
blue represents smaller ones. Our task-ware mask can dynamically attend to
different channels depending on the visibility.

Task-aware masks. The task-ware mask in AVR-Net is multiplied by the input
features. Table 3(d) studies other alternatives, i.e. concatenating masks with
the input features. Moreover, the task-aware masks are passed into the
relation module, they are part of the input signals. Therefore, we also study
the alternative training manners. Results in Table. 3(c) demonstrate that: (1)
multiplied task-aware mask is more effective than concatenation; (2)
alternative training cannot boost the performance. Our AVR-Net updates task-
ware masks simultaneously during training, it is simple yet effective.

In addition to learnable masks, binary task-aware masks can be manually
assigned to the features depending on the visibility. Table 3(d) summarizes
the comparisons between hand-crafted masks and learnable ones. Using no task-
aware masks, hand-crafted masks, and partially learnable masks have similar
performance. While learnable masks are optimized during training and can learn
to encode the most informative or discriminative features, thus it is
consistently better. To further interpret the learnable masks, weights of
masks are visualized in Fig. 7. It reveals that: (1) when all of the faces are
invisible, the network will only rely on audio inputs, different channels of
audio inputs are equally important; (2) if only one face is visible, the
network can utilize the cross-modal links to predict the similarities, both
face and audio inputs are activated; (3) when both of the speakers have
visible faces, the network can easily predict similarity based on sparse face
features, and the speech features are negligible.

##  6 Discussion and Conclusion

In this work, we first introduce a new challenging audio-visual diarization
dataset AVA-AVD. We experimentally show that both the train and test sets in
AVA-AVD are useful. Also, we propose an audio-visual relation network, which
can learn to compare two audio-visual pairs and handle missing modalities in a
unified model. Experiments demonstrate that our method outperforms all of the
strong baselines and is more robust.

Limitations. The limitation of this work is that we still follow the multi-
stage diarization framework. The future work is to develop an end-to-end
audio-visual diarization model.

Negative societal impact. We have two potential negative impacts: (1) AVA-AVD
dataset contains speaker identity information; (2) AVR-Net could be used to
develop surveillance algorithms or identify a person, which may cause privacy
issues. Our mitigation strategies are: (1) check annotations manually and
remove personally identifiable information under IRB guidelines before dataset
release; (2) confine the purposes in the model and annotation licenses.

##  7 Acknowledgment

This project is supported by the National Research Foundation, Singapore under
its NRFF award NRF-NRFF13-2021-0008. Thanks to Tongxin Wang, Yunyi Zhu, and
Kang He for help with dataset annotation.

## References

  * [1] Juan Leon Alcazar, Fabian Caba, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbelaez, and Bernard Ghanem.  Apes: Audiovisual person search in untrimmed video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1720–1729, 2021.
  * [2] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 609–617, 2017.
  * [3] Ben Barsties, Rudi Verfaillie, Nelson Roy, and Youri Maryn.  Do body mass index and fat volume influence vocal quality, phonatory range, and aerodynamics in females?  In CoDAS, volume 25, pages 310–318. SciELO Brasil, 2013.
  * [4] Andrew Brown, Vicky Kalogeiton, and Andrew Zisserman.  Face, body, voice: Video person-clustering with multiple modalities.  ICCV Workshop, 2021.
  * [5] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision (ECCV), pages 213–229. Springer, 2020.
  * [7] Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al.  The ami meeting corpus: A pre-announcement.  In International workshop on machine learning for multimodal interaction, pages 28–39. Springer, 2005.
  * [8] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han.  In defence of metric learning for speaker recognition.  In INTERSPEECH, 2020.
  * [9] Joon Son Chung, Jaesung Huh, Arsha Nagrani, Triantafyllos Afouras, and Andrew Zisserman.  Spot the conversation: speaker diarisation in the wild.  In INTERSPEECH, 2020.
  * [10] Joon Son Chung, Bong-Jin Lee, and Icksang Han.  Who said that?: Audio-visual speaker diarisation of real-world meetings.  In INTERSPEECH, 2019.
  * [11] J. S. Chung, A. Nagrani, and A. Zisserman.  Voxceleb2: Deep speaker recognition.  In INTERSPEECH, 2018.
  * [12] J. S. Chung and A. Zisserman.  Out of time: automated lip sync in the wild.  In Workshop on Multi-view Lip-reading, ACCV, 2016.
  * [13] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang.  Perfect match: Improved cross-modal embeddings for audio-visual synchronisation.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965–3969. IEEE, 2019.
  * [14] William HE Day and Herbert Edelsbrunner.  Efficient algorithms for agglomerative hierarchical clustering methods.  Journal of classification, 1(1):7–24, 1984.
  * [15] Najim Dehak, Patrick J Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet.  Front-end factor analysis for speaker verification.  IEEE Transactions on Audio, Speech, and Language Processing, 19(4):788–798, 2010.
  * [16] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [17] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4690–4699, 2019.
  * [18] Mireia Diez, Lukáš Burget, Federico Landini, Shuai Wang, and Honza Černockỳ.  Optimizing bayesian hmm based x-vector clustering for the second dihard speech diarization challenge.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6519–6523. IEEE, 2020.
  * [19] Elie El Khoury, Christine Sénac, and Philippe Joly.  Audiovisual diarization of people in video content.  Multimedia tools and applications, 68(3):747–775, 2014.
  * [20] Ruohan Gao and Kristen Grauman.  Visualvoice: Audio-visual speech separation with cross-modal consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15495–15505, 2021.
  * [21] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, and Hongsheng Li.  Self-paced contrastive learning with hybrid memory for domain adaptive object re-id.  In Advances in Neural Information Processing Systems, 2020.
  * [22] Israel D Gebru, Sileye Ba, Xiaofei Li, and Radu Horaud.  Audio-visual speaker diarization based on spatiotemporal bayesian fusion.  IEEE transactions on pattern analysis and machine intelligence, 40(5):1086–1099, 2017.
  * [23] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In European conference on computer vision (ECCV), pages 87–102. Springer, 2016.
  * [24] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg.  Matchnet: Unifying feature and metric learning for patch-based matching.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3279–3286, 2015.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016.
  * [26] John Hershey and Javier Movellan.  Audio vision: Using audio-visual synchrony to locate sounds.  Advances in neural information processing systems, 12:813–819, 1999\.
  * [27] Shota Horiguchi, Yusuke Fujita, Shinji Watanabe, Yawen Xue, and Kenji Nagamatsu.  End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors.  In Proc. Interspeech 2020, pages 269–273, 2020.
  * [28] Harold Hotelling.  Relations between two sets of variates.  In Breakthroughs in statistics, pages 162–190. Springer, 1992.
  * [29] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3588–3597, 2018.
  * [30] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama.  Learning discrete representations via information maximizing self-augmented training.  In International conference on machine learning, pages 1558–1567. PMLR, 2017.
  * [31] Zili Huang, Shinji Watanabe, Yusuke Fujita, Paola García, Yiwen Shao, Daniel Povey, and Sanjeev Khudanpur.  Speaker diarization with region proposal network.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6514–6518. IEEE, 2020.
  * [32] Dan Istrate, Corinne Fredouille, Sylvain Meignier, Laurent Besacier, and Jean François Bonastre.  Nist rt’05s evaluation: pre-processing techniques and speaker diarization on multiple microphone meetings.  In International Workshop on Machine Learning for Multimodal Interaction, pages 428–439. Springer, 2005.
  * [33] Yangbangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang.  Dm2c: Deep mixed-modal clustering.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 5888–5892, 2019.
  * [34] SouYoung Jin, Hang Su, Chris Stauffer, and Erik Learned-Miller.  End-to-end face detection and cast grouping in movies using erdos-renyi clustering.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5276–5285, 2017.
  * [35] Alan B Johnston and Daniel C Burnett.  WebRTC: APIs and RTCWEB protocols of the HTML5 real-time web.  Digital Codex LLC, 2012.
  * [36] Ioannis Kapsouras, Anastasios Tefas, Nikos Nikolaidis, Geoffroy Peeters, Laurent Benaroya, and Ioannis Pitas.  Multimodal speaker clustering in full length movies.  Multimedia Tools and Applications, 76(2):2223–2242, 2017.
  * [37] Patrick Kenny.  Bayesian speaker verification with heavy-tailed priors.  In Odyssey, volume 14, 2010.
  * [38] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al.  Siamese neural networks for one-shot image recognition.  In ICML deep learning workshop, volume 2. Lille, 2015.
  * [39] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7774–7785, 2018.
  * [40] Federico Landini, Ondřej Glembek, Pavel Matějka, Johan Rohdin, Lukáš Burget, Mireia Diez, and Anna Silnova.  Analysis of the but diarization system for voxconverse challenge.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5819–5823. IEEE, 2021.
  * [41] Federico Landini, Ján Profant, Mireia Diez, and Lukáš Burget.  Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: theory, implementation and analysis on standard tasks.  Computer Speech & Language, 71:101254, 2022.
  * [42] Ivan Medennikov, Maxim Korenevsky, Tatiana Prisyach, Yuri Khokhlov, Mariya Korenevskaya, Ivan Sorokin, Tatiana Timofeeva, Anton Mitrofanov, Andrei Andrusenko, Ivan Podluzhny, et al.  Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario.  In INTERSPEECH, 2020.
  * [43] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Learnable pins: Cross-modal embeddings for person identity.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 71–88, 2018.
  * [44] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Seeing voices and hearing faces: Cross-modal biometric matching.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 8427–8436, 2018.
  * [45] A. Nagrani, J. S. Chung, and A. Zisserman.  Voxceleb: a large-scale speaker identification dataset.  In INTERSPEECH, 2017.
  * [46] Andrew Y Ng, Michael I Jordan, and Yair Weiss.  On spectral clustering: Analysis and an algorithm.  In Advances in neural information processing systems, pages 849–856, 2002.
  * [47] Athanasios Noulas, Gwenn Englebienne, and Ben JA Krose.  Multimodal speaker diarization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(1):79–93, 2011.
  * [48] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman.  Deep face recognition.  2015\.
  * [49] Yanmin Qian, Zhengyang Chen, and Shuai Wang.  Audio-visual deep neural network for robust person verification.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1079–1092, 2021.
  * [50] Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, et al.  Ava active speaker: An audio-visual dataset for active speaker detection.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4492–4496. IEEE, 2020.
  * [51] Neville Ryant, Kenneth Church, Christopher Cieri, Alejandrina Cristia, Jun Du, Sriram Ganapathy, and Mark Liberman.  The second dihard diarization challenge: Dataset, task, and baselines.  In INTERSPEECH, 2019.
  * [52] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap.  A simple neural network module for relational reasoning.  Advances in Neural Information Processing Systems, 30, 2017.
  * [53] Leda Sarı, Kritika Singh, Jiatong Zhou, Lorenzo Torresani, Nayan Singhal, and Yatharth Saraf.  A multi-view approach to audio-visual speaker verification.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6194–6198. IEEE, 2021.
  * [54] Gregory Sell, David Snyder, Alan McCree, Daniel Garcia-Romero, Jesús Villalba, Matthew Maciejewski, Vimal Manohar, Najim Dehak, Daniel Povey, Shinji Watanabe, et al.  Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge.  In INTERSPEECH, pages 2808–2812, 2018.
  * [55] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4080–4090, 2017.
  * [56] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur.  X-vectors: Robust dnn embeddings for speaker recognition.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329–5333. IEEE, 2018.
  * [57] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14454–14463, 2021.
  * [58] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1199–1208, 2018.
  * [59] Ruijie Tao, Rohan Kumar Das, and Haizhou Li.  Audio-visual speaker recognition with a cross-modal discriminative network.  In INTERSPEECH, 2020.
  * [60] Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li.  Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 3927–3935, 2021.
  * [61] Makarand Tapaswi, Marc T Law, and Sanja Fidler.  Video face clustering with unknown number of clusters.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5027–5036, 2019.
  * [62] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29:3630–3638, 2016.
  * [63] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno.  Generalized end-to-end loss for speaker verification.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879–4883. IEEE, 2018.
  * [64] Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopz Moreno.  Speaker diarization with lstm.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5239–5243. IEEE, 2018.
  * [65] Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, and Qingming Huang.  Seeking the shape of sound: An adaptive framework for learning voice-face association.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16347–16356, 2021.
  * [66] Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh.  Disjoint mapping network for cross-modal matching of voices and faces.  In International Conference on Learning Representations, 2018.
  * [67] Jiangyue Xia, Anyi Rao, Qingqiu Huang, Linning Xu, Jiangtao Wen, and Dahua Lin.  Online multi-modal person search in videos.  In European Conference on Computer Vision (ECCV), pages 174–190. Springer, 2020.
  * [68] Xiong Xiao, Naoyuki Kanda, Zhuo Chen, Tianyan Zhou, Takuya Yoshioka, Sanyuan Chen, Yong Zhao, Gang Liu, Yu Wu, Jian Wu, et al.  Microsoft speaker diarization system for the voxceleb speaker recognition challenge 2020.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5824–5828. IEEE, 2021.
  * [69] Hani Yehia, Philip Rubin, and Eric Vatikiotis-Bateson.  Quantitative association of vocal-tract and facial behavior.  Speech Communication, 26(1-2):23–43, 1998.
  * [70] Sergey Zagoruyko and Nikos Komodakis.  Learning to compare image patches via convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 4353–4361, 2015.
  * [71] Aonan Zhang, Quan Wang, Zhenyao Zhu, John Paisley, and Chong Wang.  Fully supervised speaker diarization.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6301–6305. IEEE, 2019.
  * [72] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li.  S3fd: Single shot scale-invariant face detector.  In Proceedings of the IEEE international conference on computer vision (ICCV), pages 192–201, 2017.
  * [73] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.  Temporal relational reasoning in videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 803–818, 2018.

##  Appendix A Appendix

###  A.1 Details of Training and Inference

####  A.1.1 Training

Diarization datasets only provide annotations in Rich Transcription Time
Marked (rttm) files. Several end-to-end diarization systems [42, 27, 31] are
trained on synthesized datasets due to the limited dataset scales, while
multi-stage works [51, 68, 41, 64] never train verification models on the
diarization dataset. The only parameter they need to learn is the threshold
for clustering.

In contrast, we explore training the audio-visual relation network on both
speaker verification datasets [11, 45] and our AVA-AVD diarization dataset.
One of the challenges on diarization datasets is that they usually only
provide video-level speaker labels rather than global identities. There are
two solutions to utilize the local identities and at the same time mitigate
the drawbacks: (1) treat the video level identity labels as global ones: It
has been studied in un-supervised re-identification works [21] that splitting
the same instance into different instances will not degrade performance too
much; (2) sample positive data within one video and sample negative data
across videos: This sampling strategy can avoid ambiguous labels while largely
keep the diversity of negative samples [1]. Experimentally, we find the second
solution more effective in training our AVR-Net (Sec. 5.6).

In addition, previous methods finetune a threshold on the validation set and
fix this parameter during inference. In this work, we propose to automatically
search the parameters during training and select the best model according to
validation performance rather than evaluation metrics of pretraining tasks.

####  A.1.2 Inference

During inference, we use a common sliding window approach to process the input
sequence segment by segment for videos of arbitrary length. We randomly select
one image from the face tracks because, unlike averaging embedding vectors
[61], averaging feature maps is harmful to our AVR-Net and thus does not bring
any performance gain in practice. Moreover, cannot-link and must-link
constraints are widely used by face clustering algorithms [61, 34, 4].
However, this technique does not work in our system and is ignored
consequently.

####  A.1.3 Implementation Details

Our audio-visual relation network is implemented in PyTorch. For all of the
experiments, the input faces are resized to $112\times 112$, we follow a
standard preprocessing step [66, 17] and run face alignment using RetinaFace
[16] in advance. During training, to synthesize the missing cases, we replace
the visible face with zero under a missing probability of 0.5. The input
speech segments are 2s and sub-sampled to 16 kHz. The spectrogram is extracted
every 10ms with a window of 25ms on the fly. We use pretrained face (ResNet50)
and audio (ResNet34) backbones. Similar to [20, 61], we sample one face
uniformly from face tracks. The task-aware masks have a dimension of 1536. Our
network is optimized using Adam with a learning rate of 5$\times$10-4. We
train the model for 80000 iterations on a single RTX 3090 GPU. We only
finetune diarization parameters on AVA-AVD validation set and fix all of the
parameters during testing.

###  A.2 Additional Experiments

####  A.2.1 Ablations on missing probability

prob | SPKE | DER
---|---|---
0.0 | 18.02 | 20.94
0.1 | 19.23 | 22.15
0.3 | 19.12 | 22.04
0.5 | 17.65 | 20.57
0.7 | 21.01 | 23.93
1.0 | 50.48 | 53.40
Table 5: Additional AVR-Net ablation experiments on missing face probability.
Default settings are marked with gray.

During training, to synthesize the cases where faces are invisible, we apply a
missing face augmentation. Each speaker’s face has a certain probability to be
removed. Table 5 studies the missing probability. The missing probability of 0
achieves comparable results with 0.5. In this case, the missing face
distribution is close to the test set. However, since there is no synthesized
missing faces, the model cannot learn the relations for A vs. A, A vs. AV, and
AV vs. A effectively on VoxCeleb and VoxCeleb2. Another extreme case is 100%
missing probability, where no faces are passed into the AVR-Net, thus the
performance deteriorates a lot and DER is only 53.40. Among all of the values,
0.5 has the best result since it achieves a balance between visible and
invisible speakers.

####  A.2.2 Qualitative Results

The qualitative results are shown in Fig. 8. Although there are background
music and frequent cutaway in these clips, the results are robust and there
are only short speaker errors. Please refer to the videos attached for more
details.

(a) Results for video oD_wxyTHJ2I from 21 min to 23 min.

(b) Results for video zR725veL-DI from 23 min to 25 min.

Figure 8: Qualitative results of AVR-Net. Segments shorter than 1s are
removed. Images are the key frames sampled from video clips. Pink boxes denote
active speakers. Please refer to the attached videos for complete results.

###  A.3 Annotation Tool

Our annotation tool is developed based on an open-source project
VIA111https://gitlab.com/vgg/via. Its GUI is visualized in Fig. 9.

Figure 9: Annotation tool and process. Our annotation tool is built on VIA.
Details of the annotation steps can be found in Sec. 3.2.