# Reinforcement Learning Agents for Ubisoft’s Roller Champions

Nancy Iskander
Riot Games
<EMAIL_ADDRESS>
Aurelien Simoni
Ubisoft
<EMAIL_ADDRESS>
Eloi Alonso
Ubisoft
<EMAIL_ADDRESS>
Maxim Peter
Ubisoft
<EMAIL_ADDRESS>
Work done while at Ubisoft.

###### Abstract

In recent years, Reinforcement Learning (RL) has seen increasing popularity in
research and popular culture. However, skepticism still surrounds the
practicality of RL in modern video game development. In this paper, we
demonstrate by example that RL can be a great tool for Artificial Intelligence
(AI) design in modern, non-trivial video games. We present our RL system for
Ubisoft’s Roller Champions, a 3v3 Competitive Multiplayer Sports Game played
on an oval-shaped skating arena. Our system is designed to keep up with agile,
fast-paced development, taking 1–4 days to train a new model following
gameplay changes. The AIs are adapted for various game modes, including a 2v2
mode, a Training with Bots mode, in addition to the Classic game mode where
they replace players who have disconnected. We observe that the AIs develop
sophisticated co-ordinated strategies, and can aid in balancing the game as an
added bonus. Please see the accompanying video at https://vimeo.com/466780171
111Password: rollerRWRL2020 for examples.

##  1 Introduction and Related Work

Reinforcement Learning (RL) is an active topic of research that has received a
lot of attention in the literature [1] [2] [3] [4]. However, the role of RL in
AI design is still unclear in the video games industry, and is highly doubted
by some [5]. By presenting our RL system for Roller Champions, we demonstrate
that RL can be a powerful and practical AI design tool for modern, non-trivial
video games. We tackle the challenges of agile development, real-time
inference, competitive and collaborative gameplay, and multi-purpose AI where
winrate is secondary to player experience.

Baker et al. [3] observe emergent complexity in their Hide And Seek
environment, and propose that multi-agent co-adaptation is capable of
producing complex and intelligent behaviour. We observe analogous kinds of
multi-agent co-ordinated co-dependent behaviours in Roller Champions (section
4.4).

The role of RL in balancing games is an open area of research. Recently, RL
has been used to assess game balance in chess variants [4]. While aiding in
game balance was not one of our key goals, we found that the AIs readily
adapted to changes in game mechanics, and inadvertently revealed exploits
(section 4.2).

##  2 Roller Champions

At the time of writing, Roller Champions is set to be released by Ubisoft in
early 2021. The Roller Champions official website [6] provides additional
details.

###  2.1 Gameplay

Roller Champions: Rules of the Arena [7] describes the high-level gameplay.

###  2.2 Challenging application for RL

Roller Champions presents a number of characteristics that make it an
interesting case-study as an application of RL in modern video game
development, namely:

##### Collaborative.

Roller Champions is a collaborative game, where winning hinges upon the
players’ ability to pass the ball to each other, goal tend, and peel for the
ball bearer. Most importantly, they need to coordinate and assume self-
assigned roles depending on the situation.

##### Competitive.

The competitive gameplay further increases the complexity of Roller Champions.
For instance, one mistake can cost the whole team all of its completed laps
and checkpoints. Players need to weigh the advantage of completing additional
laps for extra points against the risk of losing their progress due to the
enemy team getting ahold of the ball.

##### Reward Sparsity.

In Roller Champions, players must pass a series of ordered checkpoints, while
holding the ball, in order to activate the goal for their team, at which point
only that team is allowed to score. The long sequence of events needed to
score makes for sparse rewards.

##### Multi-purpose AI.

Our goal was to develop AI that is not just good at winning, but that would
also be fun to play with and against. For `Training With Bots` mode, we wanted
an all-around well-performing competitive AI that would still be fun. But in
order to be able to use the AI to replace players who disconnect, we wanted
another model that would be more fun than it is competitive.

##### Live Game.

Roller Champions is set to be a free-to-play, live game. This means that the
training environment has to be efficient enough to keep up with continuous
gameplay and balance changes.

##### Real-time Inference.

Since the AI is to be deployed in real games alongside human players,
inference needs to be fast enough to maintain the target framerate.

##  3 Training Environment

Roller Champions is built in Unity, and the Unity ML-Agents Toolkit [8] is
used for training. Proximal Policy Optimization [2] is used as the training
algorithm. The policy network was kept relatively small for training and
inference speed, containing 3 layers of 512 neurons (excluding input and
output layers). We use a decision interval of 15 (i.e. observations are
collected and an action is decided every 15 `FixedUpdates`).

Self-play [9] is used to obtain AIs that can play against a wide-range of
strategies and skill levels. In the absence of penalties (see section 3.3), we
found self-play useful in preventing collaboration between the two enemy
teams.

###  3.1 Observations

Observations are derived directly from the game state. We experimented with
two approaches:

##### Raycasts

Rays are cast from each agent’s position at different angles to detect nearby
objects and their types. Relevant game state observations are added.

##### Game Entities

Observations of relevant game entities are always included. These entities
include: ball, enemies, allies, goal, checkpoints, and laps. For each entity,
relative position, speed, and clear line-of-sight flag are observed. For
players, flags indicating whether they are hurt, in air, or performing certain
actions are added. The agent also observes its own position, state, and speed.

Even though the `Raycasts` approach gave satisfactory results, we eventually
chose the `Game Entities` approach. Collecting game entity observations is
much faster due to requiring a small number of raycasts (needed for line-of-
sight flags only). The performance advantage is useful during both inference
and training.

In addition, the total number of observations can be kept low –at 60 and 78
total observations in `2v2` and `3v3` modes respectively– without omitting any
relevant information. A smaller observation size makes training easier and
reduces the number of training steps required to learn a satisfactory policy.

Observations of time and score are intentionally omitted. This makes it
possible to reuse models in multiple game modes with different end conditions.
The system is easily adapted to `2v2` mode by simply omitting two player
observations.

###  3.2 Action Space

The action space is discrete and contains two branches: `Agent Action` and
`Agent Direction`.

##### Agent Action

Near one-to-one correspondence with actions available to human players:
`Accelerate, Pump, Brake, Throw, Pass, Call For Pass, Dash, Dodge, Jump`.
`Throw` distance is fixed for AIs. The decision interval is chosen to be
slightly longer than certain player actions in order to make it possible for
the AIs to perform combos:`Dive`: `Dash`$\,\to\,$`Dash`, `Dodge Dive`:
`Dodge`$\,\to\,$`Dash`, `Uppercut`: `Dash`$\,\to\,$`Jump`.

##### Agent Direction

Controls agent’s rotation _only_ , except with `Throw`, where it is used as
throw direction. Initially, standard 2D cartesian coordinates were used. This
allowed the agents to navigate the arena well enough, but the amount of time
it was taking them to learn the game was impractical. Therefore, we made the
switch to directions that took the shape of the arena into account, namely:
`Clockwise`, `Anti-Clockwise`, `Up Inner Wall`, and `Up Outer Wall`. These
were further augmented with the following high-level directions: `Ally1`,
`Ally2`, `Enemy1`, `Enemy2`, `Enemy3`, `Ball`, and `Goal`. Compared to
cartesian co-ordinates, this action space significantly reduced the time
required for agents to learn the basics of the game.

##### Action Masking

Actions that cannot be performed are masked. `Agent Actions` can be masked due
to cooldown, ball state, game state, etc. while `Agent Directions` are masked
for targets that the observing agent does not have a clear line-of-sight to.
Action masking was imperative to overcoming superstition [10] and for
successful training. Before implementing action masking, training was
unstable, eventually leading to a policy where the AIs did not perform any
actions if penalties were used.

###  3.3 Rewards

Rewards are only given for progress towards winning a match, namely:
Progressing a checkpoint (0.05 scaled by checkpoint index) and scoring to win
the match (2 per point scored).

##### Teamwork

Teamwork is encouraged through the whole team receiving the same reward or
penalty.

##### Penalties

Penalties are given to team A whenever team B receives a reward. The magnitude
of the penalty has a huge effect on the resulting behaviours, and is
controlled using a penalty multiplier parameter.

Since agents can receive rewards and penalties due to actions of other players
that do not directly affect them, they are susceptible to superstition [10].
Batch size is increased to prevent superstition.

### Tailoring rewards for custom behaviours

In a nutshell, tailoring rewards to obtain behaviours that are not directly
related to win-rate did not work. There were two custom behaviours that we
tried to encourage through rewards: Co-operation through the `Pass` action,
and wall skating. In both cases, the rewards were exploited; the AIs quickly
unlearned scoring, opting to maximize the more immediate rewards instead (see
figure 1).

However, it is worthwhile to mention that the desired behaviours (i.e. wall
skating and co-ordinated passes) automatically and consistently emerge using
our current system (section 4.4).

Figure 1: Assigning rewards for wall skating interferes with training. The
reward was scaled by the agent’s position on the wall, with a maximum value of
0.003 for the highest position, and was only given to the ball bearer, while
grounded. Training is technically successful and agents are able to maximize
their rewards, but they stop playing the game, opting to maximize the more
immediate wall skate reward instead.

###  3.4 Adapting the game for training

The game was adapted to run in server-only mode, without graphics or
networking. Time scaling is used to speed up the simulation (accuracy of the
simulation is not compromised due to game logic being implemented in Unity’s
`FixedUpdate`). We typically run 3–15 simultaneous environments. Rather than
using an entire match as an episode, shorter one-goal episodes are used.
Tensorflow’s GPU acceleration is used to speed up training.

With these adaptations, episodes initially conclude in about 30 seconds,
reduced to about 15 seconds after the AIs learn how to score.

##  4 Results and Discussion

###  4.1 Performance

#### Training Performance

It takes roughly 10 hours per 10 million steps of training. The total number
of training steps depends on the desired result; the AIs can improve their
strategy (compared to previous policies) for up to 100 million steps, before
hitting a plateau. At 12 million steps, the AIs have a high degree of tendency
to clump together in one spot of the arena. At 20 million, they start to
become more goal-oriented, utilizing the entire arena and completing laps
faster and more frequently. At 30 million, co-ordinated co-dependent emergent
behaviours are observed (section 4.4). At 60 million+ steps, a distinct
strategy emerges, with ELO continuing to rise at a slower rate.

In practice, the model obtained at 30–40 million steps can be preferable to
that obtained at 60 million steps due to the latter model being lower in
entropy and thus more predictable. Overall, it only takes 1–4 days to produce
a new model following gameplay or balance changes. This makes the system very
practical for fast-paced, agile development, which is suitable for a live
game.

#### Inference Performance

Framerate is throttled to 30 frames per second (FPS) in Roller Champions. This
framerate is easily achieved with all players replaced by AIs, and can easily
go up to 60 FPS without throttling. Inference efficiency can be attributed to
the usage of a small policy network and CPU-efficient observations derived
directly from the game state (see section 3.1).

###  4.2 Assessing Game Balance

Observing how the AIs evolve over the training period can be helpful in
assessing game balance. For instance, figure 2 shows that even though the
`Dodge` as well as combos such as `Uppercut` and `Dive` are well utilized by
the agents, the `Dodge Dive` combo is almost never used after the 30 million
step mark, indicating that this action is underpowered compared to other
actions.

In earlier models, all players of the same team tended to skate together
throughout the match, as shown in figure 3(a). We speculated this to be due to
the `Draft` mechanic –whereby players skating in the wake of other players
gain bonus speed– being overpowered. This mechanic has since been nerfed. In
recent models, the agents have a low tendency to skate together, opting for
more strategic positioning instead (see section 4.4).

An earlier version of the `Dodge` action caused a sharp change in direction
(as shown in figure 3(b)), which was difficult to accomplish using normal
navigation controls. The `Dodge` action was exploited by the agents to
abruptly switch from clockwise to anti-clockwise skating and vice versa.
Currently, `Dodge` causes the player to side-step while continuing in the same
direction.

Figure 2: Player Action graphs demonstrate how the AIs evolve over the
training period, and are helpful in assessing game balance. This graph
indicates that the Dodge Dive combo is underpowered compared to the other two
combos (Uppercut and Dive) and coordinated actions such as Passing. We
speculate that the Tumble penalty caused by these combos is perhaps more
penalizing when the player is holding the ball (as is the case for Dodge
Dive).

(a) The Draft mechanic gave players bonus speed for skating in the wake of
other players. This was exploited by the agents who always skated together.

(b) The Dodge action used to result in a sharp change in direction (as
indicated by the orange trail). This was often used at checkpoint 1 (at the
goal line) to switch from clockwise to anti-clockwise skating in order to
progress checkpoints.

Figure 3: The Draft mechanic and the Dodge action were exploited in earlier
models before nerfing.

###  4.3 Multi-purpose AIs

The AIs are agnostic to game mode due to generic observations (section 3.1).
They are used in `Classic` game mode to replace human players who disconnect,
and in `Training with Bots`, which is a practice mode for one human player.
`3v3` and `2v2` versions exist for both game modes.

Lower difficulty models are preferred for `Classic` game mode, while any
difficulty level can be used in `Training with Bots`. The penalty multiplier
(section 3.3) can be used to obtain different difficulty levels. Figure 4
shows how the penalty multiplier affects the number of points scored.

Figure 4: The penalty multiplier significantly affects the resulting
behaviours. Zero-sum rewards result in AIs that are cautious and defense-
oriented– it is clear from observing the model that multi-point goals are
never attempted. Half Penalty results in AIs that are balanced; frequently
attempting multi-point goals while also developing sophisticated defense
strategies. No Penalty results in AIs that learn to score multi-point goals
very quickly, but it takes them longer to develop good defense strategies.

###  4.4 Emergent Behaviours

The AIs consistently develop intelligent, co-ordinated, and co-dependent
techniques and strategies. Some of the techniques developed by the agents were
inline with the intent of the game designers, while others were exploits or
simply off-brand uses of game mechanics. This section highlights some of the
most commonly observed behaviours. Please refer to https://vimeo.com/466780171
1 for a video demonstrating some of these behaviours, along with longer goal
sequences.

#### Self-assigned Roles

Agents are able to strategically position themselves in offensive or defensive
positions depending on the game state.

A commonly observed strategy is for `Player A` to progress `checkpoints 1` and
`2`, then pass the ball to `Player B` who is positioned at `checkpoint 3`.
`Player A` then goes back to the goal (at `checkpoint 1`) to either `Call For
Pass` and complete more laps, or to peel for `Player B` who proceeds to
progress the final checkpoint before either passing to `Player A` or scoring.
The defending players typically position themselves either at the goal or at
`checkpoint 3` to intercept a pass or to tackle the receiving player. Figure 5
illustrates these positions.

(a) Orange agent is strategically positioned at the goal, performing Call for
Pass as their teammate (indicated by orange ball marker) passes checkpoint 3
with the ball.

(b) Blue agent goal tending by performing high jumps near the goal, while the
goal is open for the Orange team.

Figure 5: Strategic positioning depending on game state

#### Navigation

Agents use almost all available player actions and combos for navigation.
Figure 6 shows the agents using different player actions to reach the ball
depending on its relative position and speed.

Figure 7 shows agents skating on the walls of the arena to avoid the enemy
team. Earlier models exploited game mechanics for navigation, as discussed in
section 4.2.

(a) Blue agent using Jump to reach a ball in air (indicated by orange ball
marker).

(b) Using Brake to catch a ball rolling towards the agent

Figure 6: Agents can utilize multiple player actions to navigate to the ball
depending on its relative position and speed

(a) Orange agent skating on outer arena wall

(b) Orange agent skating on inner arena wall

Figure 7: Agents skate on the walls of the arena to avoid the enemy team

#### Other

Figure 8 demonstrates agents effectively using action combos to quickly
traverse the arena and tackle enemies. Combos are made possible given the
action space and observations described in sections 3.1 and 3.2.

Even though the `Pass` mechanic automatically selects a target, agents
sometimes favour the `Throw` action. For instance, when enemy players are in
the way between the ball bearer and their target ally, the ball bearer passes
the ball by bouncing it off the wall rather than passing it directly to their
ally.

(a) Blue agent performing a combo involving Uppercut and Dash while jumping
off the wall to target an enemy

(b) Orange agent performing a combo to target a far away enemy

Figure 8: Agents are able to effectively use combos to traverse the arena and
tackle enemies

## Conclusion

This work demonstrates that RL can be a powerful and practical AI design tool
for competitive, cooperative, modern games. Our RL system for Roller Champions
accomplished the following goals:

  * •

The system is highly efficient memory and cpu-wise due to the use of a small
policy, simple observations derived directly from the game state, and high-
level directions in the action space. This makes it suitable for real-time
inference.

  * •

The turnover for training is 1–4 days, making it suitable for a live game or a
game that’s still in development.

  * •

By keeping the rewards simple, the AIs readily adapt to changes in game
mechanics, inadvertently identifying over-powered or under-powered mechanics.

  * •

The AIs consistently develop sophisticated co-operative strategies despite the
system’s relative simplicity and efficiency.

In conclusion, our system is capable of producing sophisticated game AIs with
complex strategies and co-dependent co-operative behaviours, is practical for
fast-paced agile development, and suitable for an online multiplayer game.

## Acknowledgments and Disclosure of Funding

The authors would like to thank Ubisoft and the Roller Champions team for
enthusiastically embracing this project.

## References

  * [1] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, Nov 2019. [Online]. Available: https://doi.org/10.1038/s41586-019-1724-z
  * [2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [3] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch, “Emergent tool use from multi-agent autocurricula,” 2019.
  * [4] N. Tomašev, U. Paquet, D. Hassabis, and V. Kramnik, “Assessing game balance with alphazero: Exploring alternative rule sets in chess,” 2020.
  * [5] A. Irpan, “Deep reinforcement learning doesn’t work yet,” https://www.alexirpan.com/2018/02/14/rl-hard.html, 2018.
  * [6] Ubisoft (US), “Roller champions,” 2019. [Online]. Available: https://roller-champions.ubisoft.com/
  * [7] Ubisoft [NA]. (2019, Jun 18) Roller champions: Rules of the arena. [YouTube video]. Accessed Oct. 5, 2020. [Online]. Available: https://www.youtube.com/watch?v=i3EMHh3QisU
  * [8] Unity, “Unity machine learning tools and resources.” [Online]. Available: https://unity3d.com/machine-learning
  * [9] T. Bansal, J. Pachocki, S. Sidor, I. Sutskever, and I. Mordatch, “Emergent complexity via multi-agent competition,” 2018.
  * [10] P. Bontrager, A. Khalifa, D. Anderson, M. Stephenson, C. Salge, and J. Togelius, “Superstition in the network: Deep reinforcement learning plays deceptive games,” 2019.


# Learning Branched Fusion and Orthogonal Projection for Face-Voice
Association

Muhammad Saad Saeed1∗, Shah Nawaz2∗†, Muhammad Haris Khan3, Sajid Javed4,
Muhammad Haroon Yousaf1, Alessio Del Bue2,5
1Swarm Robotics Lab (SRL)-NCRA, University of Engineering and Technology
Taxila, 2Pattern Analysis & Computer Vision (PAVIS) - Istituto Italiano di
Tecnologia (IIT), 3Mohamed Bin Zayed University of Artificial Intelligence,
4Khalifa University of Science and Technology, 5Visual Geometry & Modelling
(VGM) - Istituto Italiano di Tecnologia (IIT)

###### Abstract

††$\ast$ Equal contribution††$\dagger$ Current Affiliation: Deutsches
Elektronen-Synchrotron

Recent years have seen an increased interest in establishing association
between faces and voices of celebrities leveraging audio-visual information
from YouTube. Prior works adopt metric learning methods to learn an embedding
space that is amenable for associated matching and verification tasks. Albeit
showing some progress, such formulations are, however, restrictive due to
dependency on distance-dependent margin parameter, poor run-time training
complexity, and reliance on carefully crafted negative mining procedures. In
this work, we hypothesize that an enriched representation coupled with an
effective yet efficient supervision is important towards realizing a
discriminative joint embedding space for face-voice association tasks. To this
end, we propose a light-weight, plug-and-play mechanism that exploits the
complementary cues in both modalities to form enriched fused embeddings and
clusters them based on their identity labels via orthogonality constraints. We
coin our proposed mechanism as fusion and orthogonal projection (FOP) and
instantiate in a two-stream network. The overall resulting framework is
evaluated on VoxCeleb1 and MAV-Celeb datasets with a multitude of tasks,
including cross-modal verification and matching. Results reveal that our
method performs favourably against the current state-of-the-art methods and
our proposed formulation of supervision is more effective and efficient than
the ones employed by the contemporary methods. In addition, we leverage cross-
modal verification and matching tasks to analyze the impact of multiple
languages on face-voice association. Code is available:
https://github.com/msaadsaeed/FOP

###### Index Terms:

Multimodal, Face-voice association, Cross-modal verification and matching

##  I Introduction

It is a well-studied and understood fact that humans can associate voices and
faces of people because the neuro-cognitive pathways for voices and faces
share same structure [1, 2]. Recently, Nagrani et al. [3, 4, 5] introduced the
face-voice association task to vision community with the creation of a large-
scale audio-visual dataset, comprising faces and voices of $1,251$
celebrities. Since then, the face-voice association task has gained increased
research interest [6, 7, 4, 3, 8, 9, 10, 11]. In addition, we observe the
creation of new audio-visual datasets for studying novel tasks [7, 12, 13,
14]. For example, fundamental face-voice association tasks are studied to
analyze the impact of multiple languages, see Fig. 1.

Most existing works [7, 3, 4, 12] tackle face-voice association as a cross-
modal biometric task. The two prominent challenges in developing an effective
method for this task are learning of a common yet discriminative embedding
space, where instances from two modalities are sufficiently aligned and
instances of semantically similar identities are nearby. Often separate
networks for face and voice modalities are leveraged to obtain the respective
feature embeddings and contrastive or triplet loss formulations are employed
to construct this embedding space.

Figure 1: Face-voice association is established in cross-modal verification
and matching tasks. The goal of cross-modal verification task is to verify if
a clip of a voice and face image belong to the same identity. In a cross-modal
matching task, Voice-Face(V-F): given a clip of a voice and two or more face
images, select the face image that belongs to the voice. (Face-Voice) F-V:
given an image of a face, determine the respective voice. In addition, we
analyze the impact of multiple of languages on cross-modal verification and
matching tasks.

Although showing some effectiveness in this task, such loss formulations,
however, are restrictive in following ways. First, they require tuning of a
margin hyperparameter, which is hard as the distances between instances can
alter significantly while training. Secondly, the run-time training complexity
for contrastive and triplet losses are $\mathcal{O}(n^{2})$ and
$\mathcal{O}(n^{3})$, respectively, where $n$ is the number of available
instances for a modality. Finally, to mitigate the high run-time training
complexity challenge, different variants of carefully crafted negative mining
strategies are used, which are both time-consuming and performance sensitive.

A few methods e.g., [8] have attempted to replace the contrastive/triplet loss
formulations by utilizing auxiliary identity centroids [15]. The training
process alternates between the following two steps: 1) clustering embeddings
around their identity centroids and pushing embeddings away from all other
identity centroids, and 2) updating these centroids using the mini-batch
instances. Such centroid based losses are used with traditional classification
loss (i.e. softmax cross-entropy (CE)). However, their co-existence is
unintuitive and ineffective because the former promotes margins in Euclidean
space whereas latter implicitly achieves separability in the angular domain.

In this work, we hypothesize that an enriched unified feature representation,
encompassing complementary cues from both modalities, alongside an effective
yet efficient supervision formulation is crucial towards realizing a
discriminative joint embedding space for improved face-voice association. To
this end, we propose a light-weight, plug-and-play mechanism that exploits the
best in both modalities through fusion and semantically aligns fused
embeddings with their identity labels via orthogonality constraints. We
instantiate our proposed mechanism in the two-stream pipeline, which provides
face and voice embeddings, and the resulting overall framework is an effective
and efficient approach for face-voice matching and verification tasks.

We summarize our key contributions as follows. 1) We propose to harness the
complementary features from both modalities in forming enriched feature
embeddings, that are consistent with semantics of identity, thereby allowing
improved identity recognition. 2) We propose to impose orthogonality
constraints on the fused embeddings. They are not only coherent with the
angular characteristic of the commonly employed classification loss but are
very efficient as they operate directly on mini-batches. 3) Experimental
results on large-scale VoxCeleb$1$ [5] show the effectiveness of our method on
both face-voice verification and matching tasks. Further, we note that our
method performs favourably against the existing state-of-the-art methods. 4)
We perform a thorough ablation study to analyze the impact of different
components and empirically show that the proposed supervision formulation for
face-voice retrieval is more effective and efficient than the ones employed by
the contemporary works.

A preliminary version of this work was published in [16]. In addition, the
current manuscript makes following new contributions. We study the
effectiveness of our approach in the challenging setting of establishing face-
voice association across multiple languages spoken by the same set of persons.
The experiments are performed on Multilingual Audio-Visual (MAV-Celeb) dataset
[12]. Our results indicate that, when training on language ‘A’ and testing on
unheard language ‘B’, the performance in face-voice association tasks
deteriorates notably, and this performance is significantly poor when the same
model (i.e. trained on language ‘A’) is tested on heard language ‘A’. However,
the performance in the former setting (unheard language ‘B’) is still better
than random, which is of significance considering the challenging nature and
the configuration of the proposed evaluation protocol. Our analysis reveal
that this performance drop is due to notable mismatch between the feature
distributions of the two languages, typically known as the domain shift
problem. Finally, to our knowledge, we provide a first rigorous comparison of
loss functions employed in existing face-voice association methods by
utilizing the same underlying branch architecture, input features, and
evaluation protocols on VoxCeleb$1$ dataset (see Fig. 3).

##  II Related Work

We begin by discussing face-voice association from a cognitive neuroscience
and psychological perspective, and then detail overview of related work on
face-voice (F-V) association.

Cognitive Psychological Perspective. Studies have shown that humans are
capable of associating voices of unknown identities to their faces. In [17,
18], authors performed a studies to show that humans memorize and recall
voices of persons which they have seen (learned) previously. In a similar
study [1, 2], authors reveal the presence of equivalent amount of information
about an identity from audio and visual streams. Hence, we can say that
multimodality can improve the perception capability of humans [19]. In [19],
authors show that human perceptual accuracy improves when visual modality
supplements the auditory modality in a noisy environment. We note that these
studies have been conducted previously under the cognitive psychological
perspective in detail and have recently inspired the vision community [3, 4].

Face-voice Association. The work of Nagrani et al. [3] leveraged audio and
visual information to establish an association between faces and voices in a
cross-modal biometric matching task. Similarly, some recent metric learning
work [7, 4, 9, 12, 20, 21] introduced joint embeddings to establish
correlation between face and voice of an identity. These methods extract audio
and face embeddings and then minimize the distance between embeddings of same
identities while maximize the distance among embeddings from different ones.
Zheng et al. [22] proposes Adversarial-Metric Learning model combing metric
and adversarial learning. The aim of adversarial learning is to generate
modality independent representation while the metric learning learns a metric
for audio-visual cross-modal matching and retrieval tasks. Wen et al. [10]
presented a disjoint mapping network to learn a shared representation for
audio and visual information by mapping them individually to common covariates
(gender, nationality, identity) without needing to construct pairs or triplets
at the input. Similarly, Nawaz et al. [8] extracted audio and visual
information with a single stream network to learn a shared deep latent
representation, leveraging identity centroids to eliminate the need of pairs
or triplets [4, 3]. Both Wen et al. [10] and Nawaz et al. [8] show that
effective face-voice representations can be learned without pairs or triplets
formation. More recently, Ning et al. [23] proposed to disentangle the
alignable latent identity factors and nonalignable the modality dependent
factors for cross-modal biometric matching.

In contrast to existing, our method differs in following respects. First, it
proposes to construct enriched embeddings via exploiting complementary cues
from the embeddings of both modalities through a attention-based fusion.
Second, it clusters the embeddings of same identity and separates embeddings
of different identities via orthogonality constraints. The instantiation of
both proposals in a two-stream pipeline results in an effective and efficient
face-voice association framework.

Figure 2: Architecture of our method. Fundamentally, it is a two-stream
pipeline which generates face and voice embeddings. We believe that an
enriched embedding coupled with an effective yet efficient supervision is key
in learning a discriminative joint embedding space for improved face-voice
association. To this end, we propose a light-weight, plug-and-play mechanism,
dubbed as fusion and orthogonal projection (FOP) (shown in dotted red box). It
is capable of exploiting complementary cues in both modalities and clustering
the enriched fused embeddings based on their identity labels via imposing
orthogonality constraints with the traditional classification loss (i.e.
softmax cross-entropy).

##  III Overall Framework

Our aim is to learn a discriminative joint face-voice embedding that is
amenable to a multitude of tasks such as face-voice/voice-face matching,
verification and retrieval. To actualize this, we develop a new framework for
cross-modal face-voice association (See Fig. 2) that is fundamentally a two-
stream pipeline (sec. III-A) and features a light-weight module that exploits
complementary cues from both face and voice embeddings and facilitates
discriminative identity mapping via orthogonality constraints (sec. III-B).

###  III-A Preliminaries

Problem Settings. Without the loss of generality, we consider cross-modal
retrieval of bimodal data, i.e., for face and voice. Given that we have N
instances of face-voice pairs,
$\mathcal{D}=\\{(x_{i}^{f},x_{i}^{v})\\}_{i=1}^{N}$, where $x_{i}^{f}$ and
$x_{i}^{v}$ are the face and voice examples of the $i_{th}$ instance,
respectively. Each pair of an instance $(x_{i}^{f},x_{i}^{v})$ has an
associated label $y_{i}\in\\{0,1\\}$, where $y_{i}=1$ if $x_{i}^{f}$ and
$x_{i}^{v}$ belong to the same identity and $y_{i}=0$ if $x_{i}^{f}$ and
$x_{i}^{v}$ belong to a different identity. Both face and voice embeddings
typically lie in different representation spaces owing to their different
superficial statistics and are mostly unaligned semantically, rendering them
incomparable for cross-modal tasks. Cross-modal learning aims at projecting
both into a common yet discriminative representation space, where they are
sufficiently aligned and instances from the same identity are nearby while
from a different are far apart.

Two-stream pipeline. We employ a two-stream pipeline [4] to obtain the
respective feature embeddings of both face and voice inputs. The first stream
corresponds to a pre-trained convolutional neural network (CNN) on image
modality. We take the penultimate layer’s output, denoted as $\mathbf{b}_{i}$,
of this CNN as the feature embeddings for an input face image. Likewise, the
second stream is a pre-trained audio encoding network that outputs a feature
embedding, denoted as $\mathbf{e}_{i}$, for an input audio signal (typically a
short-term spectrogram). Existing approaches handling face-voice retrieval [4,
8], mostly resort to triplet and contrastive objectives with carefully crafted
negative mining strategies, which significantly increases computational time
and are performance-sensitive, to learn a discrminative embedding space. To
this end, we introduce a light-weight mechanism that exploits complementary
cues from both modality embeddings to form enriched fused embeddings and
imposes orthogonal constraints on them for learning discriminative joint face-
voice embeddings.

###  III-B Learning Discriminative Joint Embedding

In this section, we first describe extracting complementary cues, via
multimodal fusion, from both face and voice embeddings obtained through their
respective pre-trained networks. We then discuss clustering fused embeddings
belonging to the same identity and pushing away the ones with different
identity via orthogonality constraints.

Prior to multimodality fusion, we project the face embeddings
$\mathbf{b}_{i}\in\mathbb{R}^{F}$ to a new d-dimensional embedding space
$\mathbf{u}_{i}\in\mathbb{R}^{d}$ with a fully-connected layer. Similarly, we
project the voice embedding $\mathbf{e}_{i}\in\mathbb{R}^{V}$ to a similar
d-dimensional embedding space $\mathbf{v}_{i}\in\mathbb{R}^{d}$ with another
fully-connected layer. We then L2 normalize both $\mathbf{u}_{i}$ and
$\mathbf{v}_{i}$ which can now be fused to get $\mathbf{l}_{i}$, using the
procedure described next.

Multimodal fusion. We propose to extract complementary features from both
modalities, some of which could be related to age, gender and nationality, to
form an enriched unified feature representation which is crucial towards
learning a discriminative joint embedding space. Inspired by [24, 25], we
employ an attention mechanism to first compute the attention scores (affinity)
between the embeddings of two modalities and then fuse these individual
modality embeddings after recalibrating them with the attention scores (see
Fig. 2). We compute attention scores $\mathbf{k}$ between $\mathbf{u}_{i}$ and
$\mathbf{v}_{i}$ as:

$\mathbf{k}=\sigma(F_{att}([\mathbf{u}_{i},\mathbf{v}_{i}]),$ (1)

where $\sigma$ is a sigmoid operator, and $F_{att}$ are the attention layers.
Finally, we fuse $\mathbf{u}_{i}$ and $\mathbf{v}_{i}$ after modulating them
with the attention scores $\mathbf{k}$ to obtain the fused embeddings
$\mathbf{l}_{i}$ as:

$\mathbf{l}_{i}=\mathbf{k}\odot tanh(\mathbf{u}_{i})+(1-\mathbf{k})\odot
tanh(\mathbf{v}_{i}),$ (2)

where $\odot$ is element-wise multiplication.

Supervision via orthogonality constraints. We want the fused embeddings to
encapsulate the semantics of the identity. In other words, these embeddings
should be able to predict the identity labels with good accuracy. This is
possible if the instances belonging to the same identity are placed nearby
whereas the ones with different identity labels are far away. A popular choice
to achieve this is softmax cross entropy (CE) loss, which also allows stable
and efficient training. Specifically, we use an identity linear classifier
with weights denoted as
$\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{C}]\in\mathbb{R}^{d\times
C}$ to compute the logits corresponding to $\mathbf{l}_{i}$. Where $d$ is the
dimensionality of embeddings and $C$ is the number of identities. Now,
identity classification loss with fused embeddings is computed as:

$\mathcal{L}_{CE}=-log\frac{exp(\mathbf{l}_{i}^{T}\mathbf{w}_{y_{i}})}{\sum_{j=1}^{C}exp(\mathbf{l}_{i}^{T}\mathbf{w}_{j})}$
(3)

Since softmax CE loss does not enforce margins between pair of identities, it
is prone to constructing differently-sized class regions which affects
identity separability [26, 27]. Some works attempt to include margin between
classes in the Euclidean space [15, 28], which is not well synergized with the
CE loss as it achieves separation in the angular domain. Therefore, we propose
to impose orthogonality constraints on the fused embeddings to explicitly
minimize intra-identity separation while maximizing inter-identity
separability [29]. These constraints complement better with the innate angular
characteristic of CE loss. Further, since they directly operate on mini-
batches, they show greater training efficiency compared to the complex
negative mining procedures required in contrastive and triplet loss
formulations [4, 30] (sec. IV). Formally, the constraints enforce fused
embeddings of different identities to be orthogonal and the fused embeddings
with same identity to be similar:

$\mathcal{L}_{OC}=1-\sum_{i,j\in
B,y_{i}=y_{j}}\langle\mathbf{l}_{i},\mathbf{l}_{j}\rangle+\displaystyle\left\lvert\sum_{i,j\in
B,y_{i}\neq y_{k}}\langle\mathbf{l}_{i},\mathbf{l}_{k}\rangle\right\rvert,$
(4)

where $\langle.,.\rangle$ is the cosine similarity operator, and $B$
represents the mini-batch size. The first term in Eq. 4 ensures intra-identity
compactness, while the second term enforces inter-identity separation. Note,
the cosine similarity involves the normalization of fused embeddings, thereby
projecting them to a unit hyper-sphere:

$\langle\mathbf{l}_{i},\mathbf{l}_{j}\rangle=\frac{\mathbf{l}_{i}.\mathbf{l}_{j}}{\lVert\mathbf{l}_{i}\rVert_{2}.\lVert\mathbf{l}_{j}\rVert_{2}}.$
(5)

Overall Training Objective. To train the proposed framework, we minimize the
joint loss formulation, comprising of $\mathcal{L}_{CE}$ and
$\mathcal{L}_{OC}$ as:

$\mathcal{L}=\mathcal{L}_{CE}+\alpha\mathcal{L}_{OC},$ (6)

where $\alpha$ balances the contribution of two terms in $\mathcal{L}$. We
empirically set $\alpha$ to 1.0 based on validation set performance. It is
important to mention that CE loss operates in logit space and orthogonal
constraints are imposed in the embedding space, however, both of them
synergizes well with each other owing to their common angular domain
characteristic.

Figure 3: Multimodal frameworks based on two-stream network to embed face and
voice modalities in the latent space. We compare our loss formulation with
various others typically adopted in existing face-voice association methods,
including Contrastive [4], Triplet [3], Center [8] and Git [28], under the
same feature embeddings, network architecture, and evaluation protocol with an
aim to provide a fair comparison.

###  III-C Existing Loss Formulations in F-V Methods

In this section, we formally overview several existing loss formulations
typically employed in existing face-voice association methods with an aim to
provide a fair comparison with ours, see Fig. 3.

Center Loss. It simultaneously learns class centers from features in a mini-
batch and penalizes the distance between each class center and corresponding
features [31]. Recently, Nawaz et al. [8] introduces a single stream
architecture to extract audio-visual information to bridge the gap between
them, leveraging center loss. The loss is formulated as follow:

$\mathcal{L}_{C}=\frac{1}{2}\sum_{i=1}^{b}\left\|\mathbf{I}_{i}-\mathbf{c}_{y_{i}}\right\|_{2}^{2}$
(7)

The $\mathbf{c}_{y_{i}}$ represents the $\mathbf{y_{i}}$th class center of
features. Wen et al. [15] observed that the center loss is very small which
may degraded to zeros, thus, it is jointly trained with CE loss as follow:

$\mathcal{L}=\mathcal{L}_{CE}+\alpha_{c}\mathcal{L}_{C}$ (8)

A scalar value $\alpha_{c}$ is used for balancing center loss and CE loss.

Git Loss. It improves center loss by maximizing the distance between features
belonging to different classes (push) while keeping features of the same class
compact (pull) [28]. The loss is formulated as follow:

$\displaystyle\mathcal{L}_{G}$ $\displaystyle=\sum_{i,j=1,i\neq
j}^{b}\frac{1}{1+\left\|\mathbf{I}_{i}-\mathbf{c}_{y_{j}}\right\|^{2}_{2}}$
(9)

Git loss is trained jointly trained with center loss and CE loss as follow:

$\mathcal{L}=\mathcal{L}_{CE}+\alpha_{c}\mathcal{L}_{C}+\alpha_{g}\mathcal{L}_{G}$
(10)

Scalar values $\alpha_{c}$ and $\alpha_{g}$ are used for balancing center
loss, git loss and CE loss.

Contrastive Loss [4]. It construct positive and negative pairs to learn joint
embedding. A set $\rho$ is formulated comprising of training pairs
$\\{(\mathbf{u}_{i},\mathbf{v}_{i})\\}$ using d-dimensional face
($\mathbf{u}_{i}$) and voice ($\mathbf{v}_{i}$) embeddings. Each pair has an
associated label $y_{i}\in\\{0,1\\}$, where $y_{i}=1$ if $\mathbf{u}_{i}$ and
$\mathbf{v}_{i}$ belong to the same class and $y_{i}=0$ if $\mathbf{u}_{i}$
and $\mathbf{v}_{i}$ belong to different class.

Triplet Loss [3]. It encourages that dissimilar pairs be distant from similar
pairs by a certain margin value by formulating a triplet. A triplet consists
of an anchor, positive and negative sample to formulate a triplet. Given
$d-$dimensional face embedding $\textbf{u}_{i}$ and voice embeddings
$\textbf{v}_{i}$, we take a voice anchor $\textbf{v}_{i}^{a}$ and choose a
positive face $\textbf{u}_{i}^{p}$ and negative face $\textbf{u}_{i}^{n}$
constructing a triplet {$\textbf{v}_{i}^{a}$, $\textbf{u}_{i}^{p}$,
$\textbf{u}_{i}^{n}$} such that:

$\||\mathbf{v}_{i}^{a}-\mathbf{u}_{i}^{p}||_{2}^{2}<||\mathbf{v}_{i}^{a}-\mathbf{u}_{i}^{n}||_{2}^{2}~{}~{}\forall(v_{i}^{a},u_{i}^{p},u_{i}^{n})\in\rho$
(11)

where $\rho$ is the set of all triplets. In this work, we have used negative
hard mining to generate triplets [30]. The loss function can now be defined
as:

$\mathcal{L}_{triplet}=\sum_{i=1}^{N}[||\mathbf{v}_{i}^{a}-\mathbf{u}_{i}^{p}||_{2}^{2}-||\mathbf{v}_{i}^{a}-\mathbf{v}_{i}^{n}||_{2}^{2}+m]$
(12)

where $m$ is margin factor which helps keeping a distance between anchor and
positive and negative samples. The loss function minimizes the distance
between the anchor voice and positive face i.e. push
$||\textbf{v}_{i}^{a}-\textbf{u}_{i}^{p}||$ and penalize
$||\textbf{v}_{i}^{a}-\textbf{u}_{i}^{n}||$ to be greater than
$||\textbf{v}_{i}^{a}-\textbf{u}_{i}^{p}||+m$. Intuitively, this helps in
producing closely packed clusters of samples from same class while
simultaneously pushing apart clusters of other classes.

##  IV Experiments

We perform various cross-modal experiments on verification and matching tasks
to evaluate the effectiveness of our method and provide ablation study with
analysis to show the impact of different components in our method. In
addition, we analyze the impact of multiple languages on face-voice
association.

###  IV-A Experimental Setup

Training Details and Datasets. We train our method on Quadro P$5000$ GPU for
$50$ epochs using a batch-size of $128$ using Adam optimizer with
exponentially decaying learning rate (initialised to $10^{-5}$). We extract
face and voice embeddings from VGGFace [32] and Utterance Level Aggregation
[33]. Note that, we only backprop. through FOP module while the weights of
face and voice subnetworks remains unchanged.

Nagrani et. al [5] introduced a large-scale dataset of audio-visual human
speech videos extracted ‘in the wild’ from YouTube. Learnable Pins [4]
introduced two train/test splits out of this dataset to perform various cross-
modal verification and matching tasks. The first split consists of disjoint
videos from the same set of speakers while the second split contains disjoint
identities. We train the model using these two training sets, allowing us to
evaluate on both test sets, the first one for seen-heard identities, and the
second for unseen-unheard identities. Note that we followed the same train,
validation and test split configurations as used in [4] for fair comparisons.

Recently Nawaz et al. [12] curated MAV-Celeb dataset to study the impact of
language on face-voice association task; it consists of video and audio
recordings of celebrities speaking more than one language. The dataset
contains two splits English–Urdu (EU) and English–Hindi (EH) to analyze
performance measure across multiple languages, leveraging unseen-unheard and
completely unheard evaluation protocol. In this work, we only used EU split in
our experiments.

Experimental Setup. We perform our experiments on following two tasks: The
first task is to perform cross-modal verification where the goal is to verify
if an audio segment and a face image belong to the same identity. Two inputs
are considered i.e. face and voice and verification between the two depends
upon a threshold on the similarity value. The threshold can be adjusted in
accordance to wrong rejections of true match and/or wrong acceptance of false
match. We report results on standard verification metrics i.e. ROC curve (AUC)
and equal error rate (EER). The second task consists of cross-modal matching
where the goal is to match the input modality to the varying gallery size
$n_{c}$ which consists of the other modality. In our experiments, we increase
$n_{c}$ to analyze its impact on the performance. For example, in $1:2$
matching task, we are given a modality at input, e.g. face, and the gallery
consists of two inputs from other modality, e.g. audio. One of them contains a
true match while the other serves as an imposter. We employ matching metric
i.e. accuracy to report performance. We perform this task in five settings
where in each setting the $n_{c}$ is increased as $2,4,6,8,10$.

###  IV-B Results

TABLE I: Cross-modal verification results for our (joint) loss and other
losses under two configurations and two error metrics.

Method | EER | AUC | EER | AUC
---|---|---|---|---
| Seen-Heard | Unseen-Unheard
CE Loss | 21.8 | 86.6 | 26.8 | 81.7
Center Loss [15, 8] | 19.8 | 88.6 | 29.7 | 77.5
Git Loss [28] | 19.6 | 88.9 | 29.5 | 77.8
Contrastive Loss [4] | 23.4 | 84.7 | 29.1 | 79.5
Triplet Loss [30] | 20.7 | 88.0 | 27.1 | 81.4
Ours | 19.3 | 89.3 | 24.9 | 83.5

TABLE II: Theoretical/empirical training complexity of our (joint) loss and
others. $n$: # training instances in a modality, and $B$ is mini-batch size.

Method | Empirical | Theoretical
---|---|---
| Time (s) | Worst Case
CE Loss | .02 | $\mathcal{O}(n)$
Center Loss [15, 8] | 6.8 | $\mathcal{O}(n+\frac{n^{2}}{B})$
Git Loss [28] | 6.2 | $\mathcal{O}(n+\frac{n^{2}}{B})$
Contrastive Loss [4] | 568.2 | $\mathcal{O}(n^{2})$
Triplet Loss [30] | 619.7 | $\mathcal{O}(n^{3})$
Ours | 0.7 | $\mathcal{O}(n)$

Comparison with other F-V losses. We compare our proposed FOP against various
losses typically employed in F-V association methods. The first is center loss
[15], and is adapted by [8] in a single stream network to learn F-V
association for cross-modal verification and matching tasks. The second is Git
loss [28], which in addition to center loss also maximizes the intra-identity
distances of embeddings with other centroids and has shown effectiveness for
verification and matching tasks. In implementation, we simply replaced our
orthogonal constraints loss formulation ($\mathcal{L}_{OC}$) with center loss
and then Git loss while keeping all other settings unaltered. The third one is
Contrastive Loss that is adapted by [4] to associate face and voice of a
person in a cross-modal verification task. The last is Triplet Loss, that was
leveraged by [3] in their face-voice association framework. In implementation,
we remove our FOP module in the overall architecture and plug first
contrastive and then triplet loss with hard negative mining strategy [30]
while keeping the rest of settings fixed.

Table II shows the comparison of our supervision formulation with center and
Git losses, and our FOP mechanism with the contrastive and triplet losses on
cross-modal verification task with seen-heard and unseen-unheard
configurations. We observe that our proposal performs better than other loss
formulations across all configurations and both error metrics. We can
attribute this improved performance across full spectrum to two possible
reasons. First, our orthogonality constraints don’t depend on some margin
hyperparameter which is certainly difficult to tune with large number of
instances and identities (as in VoxCeleb$1$). Second, as opposed to centroid
based losses e.g., center or Git, these orthogonality constraints complement
well with the innate angular characteristic of softmax CE loss. We compare
both the theoretical and empirical run-time training complexity of our (joint)
loss formulation and various others commonly employed in existing F-V methods
(Table II). The empirical run-time is the time to complete one epoch. Our loss
formulation is superior than all others listed in terms of both theoretical
and empirical training efficiency. To further validate the effectiveness of
our proposed FOP mechanism, we examine the effect of Gender (G), Nationality
(N), Age (A) and its combination (GNA) separately, which influence both face
and voice verification (Table III). FOP achieves consistently better
performance on G, N, A and the combination (GNA) in both seen-heard and
unseen-unheard configurations than other loss formulations.

TABLE III: Cross-modal biometrics results under varying demographics for seen-heard and unseen-unheard configurations. Demographic | Random | G | N | A | GNA | Random | G | N | A | GNA
---|---|---|---|---|---|---|---|---|---|---
| Seen-Heard | Unseen-Unheard
CE | 86.6 | 78.0 | 85.0 | 86.3 | 77.3 | 81.7 | 65.9 | 53.6 | 76.0 | 52.8
Center | 88.6 | 79.2 | 87.0 | 88.2 | 78.1 | 77.5 | 62.4 | 51.7 | 72.5 | 54.2
Git | 88.9 | 79.7 | 87.4 | 88.6 | 78.5 | 77.9 | 62.6 | 51.8 | 72.8 | 54.2
Contrastive | 84.7 | 69.7 | 83.7 | 84.5 | 69.2 | 79.5 | 61.0 | 53.5 | 74.7 | 51.8
Triplet | 88.0 | 76.3 | 86.7 | 87.6 | 75.6 | 81.7 | 65.5 | 53.4 | 76.3 | 52.2
Ours | 89.3 | 76.7 | 87.9 | 88.6 | 76.6 | 83.5 | 68.8 | 54.9 | 78.1 | 54.2

Furthermore, we compare our FOP against aforementioned loss functions on a
cross-modal matching task, $1:n_{c}$ with $n_{c}=2,4,6,8,10$ in Fig. 4. In
this setting, the input is voice while the matching gallery consists of faces.
For example, consider the case where the input is voice and the task is $1:2$
matching, we find that out the entry in gallery which best matches the input.
We see that our proposed FOP outperforms the counterpart loss formulations for
all values of $n_{c}$. Note, this is a relatively challenging task as the
common trend for all competitors is that upon increasing $n_{c}$ the matching
performance deteriorates. In addition to the numerical results, we present
qualitative results in Fig. 5.

Figure 4: Cross-modal matching results with varying gallery size for our
proposed FOP and different loss formulations employed by the existing face-
voice association methods. Figure 5: Qualitative evaluation of V-F matching
task: We evaluate the performance of our method by examining the failure cases
of CE Loss. A voice query is shown on the left, and a gallery of 10 template
images are shown on the right. There are three successful predictions by our
proposed method (highlighted in green). On the last row the wrong prediction
is highlighted in red whereas ground truth is highlighted in green. It is
clear that the embedding produced with FOP is discriminative in contrast to
the embedding produced by the traditional CE loss. Best viewed zoomed in and
in colour.

Comparison with state-of-the-art. Table IV compares our method against
existing state-of-the-art works (DIMNet [10], Learnable Pins [4], MAV-Celeb
[12], Single Stream Network [8], Multi-view Approach [34], Adversarial-Metric
Learning [22], Disentangled Representation Learning [23], Voice-face
Discriminative Network [20]). Our method shows the best performance on unseen-
unheard protocol, however, it achieves the second best performance on seen-
heard protocol. Adversarial-Metric Learning [8] records top performance on
seen-heard configuration. We further show a comparison of our method with
existing state-of-the-art methods on cross-modal matching (Fig. 6). In
particular, we perform the $1:n_{c}$ matching tasks, where $n_{c}=2,4,6,8,10$
and report the results. Our method outperforms [4] while achieves competitive
performance against DIMNet [10]. Note that, in addition to identity, DIMNet
leverages two additional source of supervisions from gender and nationality,
which are not always readily available.

TABLE IV:  Cross-modal verification results on seen-heard and unseen-unheard configurations of our method and existing state-of-the-art methods. Methods | Dataset | EER | AUC | EER | AUC
---|---|---|---|---|---
|  | Seen-Heard | Unseen-Unheard
Voice-face Discriminative Network [20] | VoxCeleb2 | - | - | 22.5 | 85.4
DIMNet [10] | VoxCeleb1 | - | - | 24.9 | -
Learnable Pins [4] | VoxCeleb1 | 21.4 | 87.0 | 29.6 | 78.5
MAV-Celeb [12] | VoxCeleb1 | - | - | 29.0 | 78.9
Single Stream Network [8] | VoxCeleb1 | 17.2 | 91.1 | 29.5 | 78.8
Multi-view Approach [34] | VoxCeleb1 | - | - | 28.0 | -
Adversarial-Metric Learning [22] | VoxCeleb1 | - | 92.3 | - | 80.6
Disentangled Representation Learning [23] | VoxCeleb1 | - | - | 25.0 | 84.6
Ours | VoxCeleb1 | 19.3 | 89.3 | 24.9 | 83.5
Figure 6: Cross-modal matching results of our method and existing state-of-
the-art methods with varying gallery size.

Ablation study and analysis. Table V reports the results when varying the
hyperparameter $\alpha$ that is used to balance the contribution of softmax CE
and orthogonal constraints based loss in our joint formulation (Eq. 6). Our
method performance is mostly robust to the choice of $\alpha$. There is only a
0.8% drop in EER when varying $0.5\leq\alpha\leq 2.0$ around the best reported
value of 1.0. However, the performance starts to deteriorate beyond 0.5 and
above 2.0, where the contribution of softmax CE, which synergizes with
orthogonality constraints, is almost negligible. We also experiment with
replacing gated multimodal fusion with a much simpler linear fusion (Table
VI). Linear fusion naively sums element-wise the embeddings from two
modalities. Results reveal the gated fusion significantly works better than
the linear fusion. This highlights the importance of first finding the
correlating features between the two modality embeddings, modulating them
accordingly, and finally fusing them.

TABLE V: Coss-modal Verification results on unseen-unheard configurations to
illustrate the effect of varying $\alpha$ value.
$\mathcal{L}_{CE}+\alpha\mathcal{L}_{OC}$
---
$\alpha$ | 0.0 | 0.1 | 0.5 | 1.0 | 2.0 | 5.0
EER | 26.8 | 26.1 | 25.8 | 24.9 | 25.9 | 26.0
AUC | 81.7 | 82.4 | 82.8 | 83.5 | 82.7 | 82.6
TABLE VI: Coss-modal Verification results on unseen-unheard configurations
with linear and gated fusion strategies.

Fusion Strategy | EER | AUC
---|---|---
Linear Fusion | 25.6 | 82.7
Gated Fusion | 24.9 | 83.5

Decomposing Orthogonal Constraints (OC). Taking a step further, we break down
OC into its sub-components as defined in Eq. 4. These components are employed
to quantify the intra-class compactness and inter-class separation in any
given embedding space. Therefore, it will be interesting to compare the
contribution of OC towards inter-class maximisation and intra-class
minimization of an embedding space in contrast to the traditional CE based
training scenario. In Fig. 7, the effect of OC on learning face-voice
association is visualized. It is evident that OC has a strong impact on
overall feature orthogonality in fused embedding space. Moreover, it helps to
increase the similarity index between features of similar classes as compared
to only CE while simultaneously enforcing higher intra-class dissimilarity.

Figure 7: (a) Feature Orthogonality (↓) (b) Similarity of same class features
(↑) (c) Similarity of different class features (↓). FOP simultaneously
achieves higher inter-class similarity and intra-class dissimilarity in
comparison with the standard CE.

Impact of multiple languages on face-voice association. MAV-Celeb dataset
provides language annotation of celebrities to analyze the impact of multiple
languages on cross-modal verification and matching tasks. For example, a
celebrity named ‘Imran Khan’ has audio information in Urdu and English
languages. Thus, we can a train a model on one language and tested on an
unheard one. Table VII shows the performance of face-voice association on
cross-modal verification task across multiple languages using unseen-unheard
and completely unheard configuration. The performance is dropped when our
face-voice association model is trained on language ‘A’ (English) and tested
on a completely unheard ‘B’ (Urdu) language. Similarly, Fig. 8 and 9 show the
performance of face-voice association on cross-modal matching task with
varying gallery size ($n_{c}$). Interestingly, the performance is better than
random verification and matching, which is not trivial considering the
challenging configuration unseen-unheard and completely unheard evaluation
protocol. In order to find the reason of performance drop, we use feature
distributions of three classes in test sets belonging to Urdu and English
languages, see Fig. 10. We observed that the performance degradation is due to
different feature distributions of the two languages. In addition, both
languages share common characteristics.

TABLE VII: Cross-modal verification between face and voice across multiple
language on various test configurations of MAV-Celeb dataset. The down
arrow($\downarrow$) represents percentage decrease in performance. (EER: lower
is better)

|  | EU
---|---|---
Method | Configuration | Eng. test | Urdu test
|  | (EER) | (EER)
MAV-Celeb [12] | Eng. train | 45.1 | 48.3$\downarrow$7.1
Urdu train | 47.0$\downarrow$6.3 | 44.3
Ours | Eng. train | 31.0 | 36.6$\downarrow$17.4
Urdu train | 44.4$\downarrow$41.7 | 31.2

Figure 8: Cross-modal matching results of our method on heard language (Urdu)
and completely unheard language (English). Figure 9: Cross-modal matching
results of our method on heard language (English) and completely unheard
language (Urdu).

Figure 10: Features distribution of English and Urdu test sets to show domain
gap. First row represents feature distribution of English and Urdu test sets
with a model trained on English language while the second row represents
distribution of English and Urdu test sets with a model trained on Urdu
language.

##  V Conclusion

We proposed a light-weight module, namely FOP, for F-V association task. It
attempts to harness the best in both face and voice modalities through
attention-based fusion and clusters the fused embeddings based on their
identity-labels via orthogonality constraints. We integrated this module in a
two-stream pipeline, used for extracting face and voice embeddings, and the
resulting overall framework is evaluated on a large-scale VoxCeleb$1$ dataset
for F-V matching and verification tasks. Our method performs favourably
against the existing state-of-the-art methods and proposed FOP outperforms
competitors both in accuracy and efficiency. Moreover, we study the
performance of our method in the challenging setting of F-V association across
multiple languages. Results indicate the opportunity of developing more
effective F-V methods that are robust to varying feature distributions in
cross-modal verification and matching tasks.

## References

  * [1] Miyuki Kamachi, Harold Hill, Karen Lander, and Eric Vatikiotis-Bateson,  “Putting the face to the voice’: Matching identity across modality,”  Current Biology, vol. 13, no. 19, pp. 1709–1714, 2003.
  * [2] Pascal Belin, Shirley Fecteau, and Catherine Bedard,  “Thinking the voice: neural correlates of voice perception,”  Trends in cognitive sciences, vol. 8, no. 3, pp. 129–135, 2004\.
  * [3] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman,  “Seeing voices and hearing faces: Cross-modal biometric matching,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8427–8436.
  * [4] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman,  “Learnable pins: Cross-modal embeddings for person identity,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 71–88.
  * [5] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  arXiv preprint arXiv:1706.08612, 2017.
  * [6] Shota Horiguchi, Naoyuki Kanda, and Kenji Nagamatsu,  “Face-voice matching using cross-modal embeddings,”  in Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 1011–1019.
  * [7] Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojciech Matusik,  “On learning associations of faces and voices,”  in Asian Conference on Computer Vision. Springer, 2018, pp. 276–292.
  * [8] Shah Nawaz, Muhammad Kamran Janjua, Ignazio Gallo, Arif Mahmood, and Alessandro Calefati,  “Deep latent space learning for cross-modal mapping of audio and visual signals,”  in 2019 Digital Image Computing: Techniques and Applications (DICTA). IEEE, 2019, pp. 1–7.
  * [9] Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, and Qingming Huang,  “Seeking the shape of sound: An adaptive framework for learning voice-face association,”  arXiv preprint arXiv:2103.07293, 2021.
  * [10] Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh,  “Disjoint mapping network for cross-modal matching of voices and faces,”  in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
  * [11] Boqing Zhu, Kele Xu, Changjian Wang, Zheng Qin, Tao Sun, Huaimin Wang, and Yuxing Peng,  “Unsupervised voice-face representation learning by cross-modal prototype contrast,”  arXiv preprint arXiv:2204.14057, 2022.
  * [12] Shah Nawaz, Muhammad Saad Saeed, Pietro Morerio, Arif Mahmood, Ignazio Gallo, Muhammad Haroon Yousaf, and Alessio Del Bue,  “Cross-modal speaker verification and recognition: A multilingual perspective,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1682–1691.
  * [13] Cho-Ying Wu, Chin-Cheng Hsu, and Ulrich Neumann,  “Cross-modal perceptionist: Can face geometry be gleaned from voices?,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10452–10461.
  * [14] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy,  “CelebV-HQ: A large-scale video facial attributes dataset,”  in ECCV, 2022.
  * [15] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao,  “A discriminative feature learning approach for deep face recognition,”  in European conference on computer vision. Springer, 2016, pp. 499–515.
  * [16] Muhammad Saad Saeed, Muhammad Haris Khan, Shah Nawaz, Muhammad Haroon Yousaf, and Alessio Del Bue,  “Fusion and orthogonal projection for improved face-voice association,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7057–7061.
  * [17] Hunter A McAllister, Robert HI Dale, Norman J Bregman, Allyssa McCabe, and C Randy Cotton,  “When eyewitnesses are also earwitnesses: Effects on visual and voice identifications,”  Basic and Applied Social Psychology, vol. 14, no. 2, pp. 161–170, 1993.
  * [18] Stefan R Schweinberger, David Robertson, and Jürgen M Kaufmann,  “Hearing facial identities,”  Quarterly Journal of Experimental Psychology, vol. 60, no. 10, pp. 1446–1456, 2007.
  * [19] Kevin G Munhall and Eric Vatikiotis-Bateson,  “The moving face during speech communication,”  Hearing by eye II: Advances in the psychology of speechreading and auditory-visual speech, pp. 123–139, 1998.
  * [20] Ruijie Tao, Rohan Kumar Das, and Haizhou Li,  “Audio-Visual Speaker Recognition with a Cross-Modal Discriminative Network,”  in Proc. Interspeech 2020, 2020, pp. 2242–2246.
  * [21] Guangyu Chen, Deyuan Zhang, Tao Liu, and Xiaoyong Du,  “Self-lifting: A novel framework for unsupervised voice-face association learning,”  in Proceedings of the 2022 International Conference on Multimedia Retrieval, 2022, pp. 527–535.
  * [22] Aihua Zheng, Menglan Hu, Bo Jiang, Yan Huang, Yan Yan, and Bin Luo,  “Adversarial-metric learning for audio-visual cross-modal matching,”  IEEE Transactions on Multimedia, vol. 24, pp. 338–351, 2021.
  * [23] Hailong Ning, Xiangtao Zheng, Xiaoqiang Lu, and Yuan Yuan,  “Disentangled representation learning for cross-modal biometric matching,”  IEEE Transactions on Multimedia, vol. 24, pp. 1763–1774, 2021.
  * [24] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González,  “Gated multimodal units for information fusion,”  arXiv preprint arXiv:1702.01992, 2017.
  * [25] Zhengyang Chen, Shuai Wang, and Yanmin Qian,  “Multi-modality matters: A performance leap on voxceleb,”  Proc. Interspeech 2020, pp. 2252–2256, 2020.
  * [26] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou,  “Arcface: Additive angular margin loss for deep face recognition,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4690–4699.
  * [27] Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao,  “Gaussian affinity for max-margin class imbalanced learning,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6469–6479.
  * [28] Alessandro Calefati, Muhammad Kamran Janjua, Shah Nawaz, and Ignazio Gallo,  “Git loss for deep face recognition,”  in Proceedings of the British Machine Vision Conference (BMVC), 2018.
  * [29] Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan,  “Orthogonal projection loss,”  arXiv preprint arXiv:2103.14021, 2021.
  * [30] Florian Schroff, Dmitry Kalenichenko, and James Philbin,  “Facenet: A unified embedding for face recognition and clustering,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815–823.
  * [31] Liwei Wang, Yin Li, and Svetlana Lazebnik,  “Learning deep structure-preserving image-text embeddings,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5005–5013.
  * [32] O Parkhi, A Vedaldi, and A Zisserman,  “Deep face recognition,”  2019, pp. 1–12, British Machine Vision Association.
  * [33] Weidi Xie, Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Utterance-level aggregation for speaker recognition in the wild,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5791–5795.
  * [34] Leda Sarı, Kritika Singh, Jiatong Zhou, Lorenzo Torresani, Nayan Singhal, and Yatharth Saraf,  “A multi-view approach to audio-visual speaker verification,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6194–6198.