#  Attention for Robot Touch: Tactile Saliency
Prediction for Robust Sim-to-Real Tactile Control

Yijiong Lin, Mauro Comi, Alex Church, Dandan Zhang, Nathan F. Lepora
YL was supported by a UoB-CSC-joint scholarship. NL was supported by a
Leadership Award from the Leverhulme Trust on ‘A biomimetic forebrain for
robot touch’ (RL-2016-39). (Corresponding author: Yijiong Lin)All authors are
with the Department of Engineering Mathematics and Bristol Robotics
Laboratory, University of Bristol, Bristol BS8 1UB, U.K. (email: {yijiong.lin,
<EMAIL_ADDRESS>
###### Abstract

High-resolution tactile sensing can provide accurate information about local
contact in contact-rich robotic tasks. However, the deployment of such tasks
in unstructured environments remains under-investigated. To improve the
robustness of tactile robot control in unstructured environments, we propose
and study a new concept: tactile saliency for robot touch, inspired by the
human touch attention mechanism from neuroscience and the visual saliency
prediction problem from computer vision. In analogy to visual saliency, this
concept involves identifying key information in tactile images captured by a
tactile sensor. While visual saliency datasets are commonly annotated by
humans, manually labelling tactile images is challenging due to their
counterintuitive patterns. To address this challenge, we propose a novel
approach comprised of three interrelated networks: 1) a Contact Depth Network
(ConDepNet), which generates a contact depth map to localize deformation in a
real tactile image that contains target and noise features; 2) a Tactile
Saliency Network (TacSalNet), which predicts a tactile saliency map to
describe the target areas for an input contact depth map; 3) and a Tactile
Noise Generator (TacNGen), which generates noise features to train the
TacSalNet. Experimental results in contact pose estimation and edge-following
in the presence of distractors showcase the accurate prediction of target
features from real tactile images. Overall, our tactile saliency prediction
approach gives robust sim-to-real tactile control in environments with unknown
distractors. Project page: https://sites.google.com/view/tactile-saliency/.

##  I INTRODUCTION

High-resolution tactile sensing is seeing greater use in robot manipulation as
a complement to vision, due to its ability to reveal fine-grained details in
local contact [1]. Despite its potential, the research community has
predominantly focused on tasks with idealized experimental conditions,
disregarding the impact of noise and distractors [2, 3, 4]. This has resulted
in a limited understanding of how to achieve robust tactile control in
unstructured environments, where unexpected stimuli can impair the accuracy of
controllers or policies relying on tactile sensing (as shown in Fig. 1b),
making it difficult to achieve precise control in tactile-oriented tasks such
as contour following and tactile exploration [5, 6, 7, 8]. Therefore, it is
crucial to develop a methodology that can effectively distinguish between
target and noise features in tactile feedback, enabling robust tactile control
in unstructured environments.

To better describe this problem, we propose a new concept for robot touch:
tactile saliency. Analogous to visual saliency in computer vision, we define
tactile saliency to describe the critical regions of interest for a robot in a
tactile image obtained by a tactile sensor. For example in Fig. 1b, a tactile
saliency map can indicate a target feature of edge from a real tactile image
during a contour-following task in an unstructured environment. However,
collecting and labelling tactile saliency data presents unique challenges
compared to visual saliency data. Unlike the latter, which is typically
collected through human labelling using eye trackers, mouse clicks, etc. [9],
it is challenging for humans to distinguish between target and noise features
in a raw tactile image. Additionally, tactile sensors are often soft and
delicate, making it impractical to collect a large amount of data through
direct physical contact with various stimuli while avoiding damaging the
sensors, particularly if the goal is to learn the joint distribution of target
and noise features.

Figure 1: Visual saliency vs tactile saliency: (a) an example of visual
saliency map (right) and its source visual image (mid) from SALICON dataset
[10]; (b) an example of tactile saliency map (right) and its source tactile
image (mid) obtained by a TacTip (left) contacting a target edge (white
cylinder) and a distractor (pink toy) in an edge-following task in a cluttered
scene.

To address these challenges, here we propose a novel approach for tactile
saliency prediction (Fig. 2) comprised of three interrelated networks: 1) a
Contact Depth Network (ConDepNet), which generates a contact depth map to
localize deformation for an input real tactile image that contains target and
noise features in a simplified format; 2) a Tactile Saliency Network
(TacSalNet), which predicts a tactile saliency map to describe the target
areas for an input contact depth map; 3) and a Tactile Noise Generator
(TacNGen), which generates noise features to train the TacSalNet, making it
more generalizable to unseen contact depth maps.

The proposed approach presents several advantages. Firstly, it eliminates the
need for human labelling, leading to a more practical and efficient data
collection process. Secondly, with the high fidelity of generated tactile
noise, it enables accurate prediction of target features from real tactile
images in the presence of unseen noise, enhancing the robustness of tactile
control in unstructured environments. Thirdly, the TacSalNet can be seen as a
simple plug-in module for various types of tactile-oriented control methods,
such as pose-based tactile servoing with PID control [11] or image-based deep
reinforcement learning (deep-RL) [2, 4]. Moreover, this approach has the
potential to apply to other types of optical tactile sensors, making it a
general solution for predicting tactile saliency in a manner applicable to a
wide range of uses.

The main contributions of this research are as follows:
1) We initiate the concept of tactile saliency for robot touch, and
investigate this problem for robust sim-to-real tactile control in
unstructured environments.
2) We propose a novel approach to predict tactile saliency from real tactile
images without relying on human labelling.
3) We achieve precise contact pose prediction and robust sim-to-real tactile
control for an edge-following task in the presence of distractors, by
successfully learning a tactile saliency prediction network from simulation
and applying it to the real world.

Figure 2: Overview of the proposed 3-stage approach for tactile saliency
prediction: (a) ConDepNet training for generating a contact depth map to
localize deformation in a real tactile image using a dataset of paired real
and simulated tactile images. (b) TacSalNet training to predict a tactile
saliency map to capture a desired target feature for an input contact depth
map with noise. (c) TacNGen training for generating noise features to train
the TacSalNet.

##  II RELATED WORK

Visual saliency has been researched extensively in computer vision [12, 13,
14, 15]. This concept refers to the regions or objects in an image that
attracts more visual attention or are more noticeable than other regions [9].
In recent years, researchers have applied this concept to touch-relevant tasks
as well. Xu et al. [16] and Ni et al. [17] explored the use of visual saliency
in 2D images on mobile devices to better understand and predict users’ touch
behaviours. Lau et al. [18] introduced tactile mesh saliency to measure the
salient parts of 3D object mesh models that are more likely to be contacted by
humans. Similarly, Jiao et al. [19] proposed tactile sketch saliency, which
uses supervised learning to jointly estimate the tactile saliency, depth map,
and semantic category of a sketch. Additionally, Jain et al. [20] utilized 3D
point clouds as visual input to determine an object saliency map for touch,
which guides the robot to collect more informative tactile data. Cao et al.
[21] proposed the Spatio-Temporal Attention Model for tactile texture
recognition.

Nevertheless, these research studies addressed different problems than the one
we focus on in this study. Instead, our aim is to identify the most salient
contact areas in tactile images obtained from high-resolution optical tactile
sensors. This tactile saliency can assist tactile robots in achieving complex
tasks in unstructured environments. For example, a robot may need to explore
an object’s contour [5, 6, 7, 8] in clutter using touch. In such scenarios,
the identification of tactile saliency is critical since the robot may lack
full knowledge of the object and its surrounding stimuli.

##  III Methodology

###  III-A Tactile Saliency Prediction

Given a real grey-scale tactile image $\text{I}=\\{\text{I}_{ij}\in[0,1]\mid
i\\!\\!\in\\!\\!\\{1,...,w\\},j\\!\\!\in\\!\\!\\{1,...,h\\}\\}$ of size
$w\times h$ and a target feature type F, we define a tactile saliency map
${\text{S}}^{\text{F}}$ = $\left\\{\text{s}^{\text{F}}_{ij}\in[0,1]\mid
i\in\\{1,...,w\\},j\in\\{1,...,h\\}\right\\}$ as a matrix of probabilities
where $\text{s}^{\text{F}}_{ij}$ is the probability of pixel $\text{I}_{ij}$
being part of the feature F (e.g. an edge or surface) in the tactile image I.
We define a mapping
$\psi_{\text{F}}(.)\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{w\times
h}\rightarrow\mathbb{R}^{w\times h}$ as a function that maps a real tactile
image I to a tactile saliency map ${\text{S}}^{\text{F}}$,

$\displaystyle\psi_{\text{F}}(\text{I})\mathrel{\mathop{\ordinarycolon}}={\text{S}}^{\text{F}}.$
(1)

Our aim is to learn a tactile saliency prediction model to approximate
$\psi_{\text{F}}$, which can be used to predict a tactile saliency map
${\text{S}}^{\text{F}}$ representing the probabilities of target feature areas
in image I. However, unlike visual saliency datasets, it is challenging for
humans to construct a tactile saliency dataset by accurately labelling real
tactile images with tactile saliency maps. This is due to the inherent nature
of marker-based tactile images, which can present counterintuitive patterns
that are difficult for humans to identify and separate the target features
from the noise features. Thus, it is impractical to learn a tactile saliency
model $\psi_{\text{F}}$ for F that directly predicts ${\text{S}}^{\text{F}}$
from I.

###  III-B Contact Depth Prediction

While it is challenging to learn $\psi_{\text{F}}$ to predict
$\text{S}^{\text{F}}$ directly from I, we can alternatively predict
$\text{S}^{\text{F}}$ from a simplified tactile image that only represents
contact areas, giving a more straightforward and interpretable representation.
Here, we define a simplified tactile image of I as a contact depth map C =
$\left\\{\text{c}_{ij}\in[0,1]\mid
i\in\\{1,\ldots,w\\},j\in\\{1,\ldots,h\\}\right\\}$ where $\text{c}_{ij}$
describes the contact depth level of the tactile skin in pixel
$\text{I}_{ij}$. We define a mapping
$\phi\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{w\times
h}\rightarrow\mathbb{R}^{w\times h}$ as a function that maps a real tactile
image I to a contact depth map C,

$\displaystyle\phi(\text{I})\mathrel{\mathop{\ordinarycolon}}={\text{C}}.$ (2)

Our aim is to learn a contact depth prediction model $G_{\text{C}}$ (referred
to as ConDepNet) to approximate $\phi$ to predict a contact depth map C
representing the contact areas in tactile image I. Following a general image-
to-image translation approach in [22], we use pix2pix GAN to learn
$G_{\text{C}}$ with an objective in an adversarial training manner,

$\displaystyle G_{\text{C}}=G^{\ast}=\text{arg\ }\underset{G}{\text{min}\
}\underset{D}{\text{max}\ }\mathcal{L}_{\text{C}}(G,D)\text{,}$ (3)

where $G$ and $D$ are the generator and the discriminator respectively in a
GAN setting, with a standard definition

$\displaystyle\mathcal{L}_{\text{C}}(G,D)$
$\displaystyle=\mathcal{L}_{\text{cGAN}}(G,D)+\alpha\mathcal{L}_{L_{1}}(G)\text{,}$
(4) $\displaystyle\mathcal{L}_{\text{cGAN}}(G,D)$
$\displaystyle=\mathbb{E}_{\text{I}\in\Gamma,\text{C}\in\Phi}[\text{log}D(\text{I},\text{C})]$
$\displaystyle\hskip
50.00008pt+\mathbb{E}_{\text{I}\in\Gamma}[\text{log}(1-D(\text{I},G(\text{I}))]\text{,}$
(5)

where $\Gamma=\left\\{\text{I}_{k}\mid k\in\\{1,\ldots,n\\}\right\\}$ are $n$
real tactile images and $\Phi=\left\\{\text{C}_{k}\mid
k\in\\{1,\ldots,n\\}\right\\}$ are $n$ contact depth maps, $L_{1}$ represents
the Manhattan norm. Note that, unlike [22], we do not consider a random vector
as input to $G$, since we need a deterministic contact depth image for each
real tactile image.

To collect a dataset
$\mathcal{D}_{\Gamma,\Phi}=\left\\{(\text{I},\text{C})\mid(\text{I}\in\Gamma,\text{C}\in\Phi\right\\}$
with auto-labelling, we leverage Tactile Gym [2], a simulator for tactile
robotics based on rigid-body physics. We use this to generate simulated
tactile images rendered as the contact depth map captured by a simulated
tactile sensor when contacting stimuli. These contact depth maps serve as a
simplified representation of real tactile images, which can be used to label
the corresponding contact areas in real tactile images obtained with the same
relative contact poses (see Fig. 2a). Thus, a set of real tactile images
$\Gamma$ labelled with corresponding simulation-generated contact depth maps
$\Phi^{\Gamma}$ is accessible and enables us to learn a ConDepNet without the
need for human labelling.

###  III-C Predicting Tactile Saliency from Contact Depth

To achieve our primary aim of mapping saliency, we define a mapping
$\delta_{\text{F}}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{w\times
h}\rightarrow\mathbb{R}^{w\times h}$ as a function that maps a contact depth
map C to a tactile saliency map $\text{S}^{\text{F}}$ for a given target
feature F:

$\displaystyle\delta_{\text{F}}(\text{C})\mathrel{\mathop{\ordinarycolon}}=\text{S}^{\text{F}}.$
(6)

In other words, we can solve Eq. 1 with a composite function of Eq. 2 and Eq.
6, because a contact depth map C preserves the contact information in a real
tactile image I,

$\displaystyle\delta_{\text{F}}(\phi(\text{I}))=\delta_{\text{F}}(\text{C})=\text{S}^{\text{F}}=\psi_{\text{F}}(\text{I}).$
(7)

Thus our aim reduces to learning a tactile saliency prediction model
$G_{\text{S}^{\text{F}}}$ (referred to as TacSalNet) to approximate
$\delta_{\text{F}}$ for a target feature F that predicts a saliency map
$\text{S}^{\text{F}}$ representing the target areas in C. Similar to
ConDepNet, we also apply pix2pix GAN to learn $G_{\text{S}^{\text{F}}}$, with
an objective

$\displaystyle G_{\text{S}^{\text{F}}}=G^{\ast}=\text{arg\
}\underset{G}{\text{min}\ }\underset{D}{\text{max}\
}\mathcal{L}_{\text{S}^{\text{F}}}(G,D)\text{,}$ (8)

where $\mathcal{L}_{\text{S}^{\text{F}}}(G,D)$ is defined like
$\mathcal{L}_{\text{C}}(G,D)$ in Eqs (4,5) with $\text{S}^{\text{F}}$
replacing C, drawn from $\Psi^{\text{F}}=\left\\{\text{S}^{\text{F}}\right\\}$
a set of tactile saliency maps for a target feature F.

A dataset $\mathcal{D}$ consisting of contact depth maps $\Phi$ and their
corresponding tactile saliency maps $\Psi^{\text{F}}$ is generated using
Tactile Gym as follows (see Fig. 2b):
1) We first collect a set of contact depth maps
$\Phi^{\text{F}}=\left\\{\text{C}^{\text{F}}_{k}\mid
k\in\\{1,..,n\\}\right\\}$ for the target feature type F, and a set of noise
contact depth maps $\Phi^{\text{N}}=\left\\{\text{C}^{\text{N}}_{k}\mid
k\in\\{1,..,m\\}\right\\}$ for random noise features, where $n$ and $m$ denote
the sizes of $\Phi^{\text{F}}$ and $\Phi^{\text{N}}$ respectively.
2) Then, we generate a set of contact depth maps containing both target and
noise features:
$\Phi=\left\\{\text{C}_{kl}=\text{C}_{k}^{\text{F}}+\text{C}_{l}^{\text{N}}\mid\text{C}_{k}^{\text{F}}\in\Phi^{\text{F}},\text{C}_{l}^{\text{N}}\in\Phi^{\text{N}}\right\\}$,
where we overlay $\text{C}_{k}^{\text{F}}$ with $\text{C}_{l}^{\text{N}}$ by
the pixel-by-pixel sum of pixel values. We label each $\text{C}_{kl}\in\Phi$
with its corresponding $\text{C}_{k}^{\text{F}}\in\Phi^{\text{F}}$ to generate
a dataset
$\mathcal{D}_{\Phi,\text{F}}=\left\\{(\text{C},\text{C}^{\text{F}})\mid\text{C}\in\Phi,\text{C}^{\text{F}}\in\Phi^{\text{F}}\right\\}$.
3) Finally, we consider that the saliency map $\text{S}^{\text{F}}$ is a min-
max normalized form of $\text{C}^{\text{F}}$, therefore we can obtain
$\mathcal{D}_{\Psi,\text{F}}=\left\\{(\text{C},\text{S}^{\text{F}})\mid\text{S}^{\text{F}}={\rm
Norm}(\text{C}^{\text{F}}),\text{C}\in\Phi,\text{C}^{\text{F}}\in\Phi^{\text{F}}\right\\}$
by linearly scaling each $\text{C}^{\text{F}}$ in
$\mathcal{D}_{\Phi,\text{F}}$ to the range of $[0,1]$. Note that while here we
only demonstrate a simple and straightforward rescaling method (i.e. Min-Max
Normalization) to prove our concept; there are alternative methods like
’standardization’ for converting a $\text{C}^{\text{F}}$ to
$\text{S}^{\text{F}}$.

###  III-D Tactile Noise Generator

We could use a simulated tactile sensor to interact with noise stimuli in
Tactile Gym to collect a set of noise contact depth maps $\Phi^{\text{N}}$ for
learning a TacSalNet. However, this is inefficient as various stimuli CAD
models are required. Also, the noise patterns are unlike those we see in
contact depth maps generated from a ConDepNet in real-world experiments.
Therefore, we propose a generative model $\tau$ to generate random noise
features, which we call Tactile Noise Generator (TacNGen), as shown in Fig.
2c. Specifically, We use a conventional structure of Variational AutoEncoder
(VAE) to learn an encoder parameterised by $\theta$:
$f_{\theta}(\text{C})=(\mu_{\theta,\text{C}},\Sigma_{\theta,\text{C}})$ where
$\mu_{\theta,\text{C}}\in\mathbb{R}^{K}$ and
$\Sigma_{\theta,\text{C}}=\text{diag}(\sigma_{1}^{2},..,\sigma_{K}^{2})$, and
a decoder parameterised by $\lambda$: $f_{\lambda}(z)=\hat{\text{C}}$ to
reconstruct a contact depth map $\hat{\text{C}}$ from a $K$-dim latent vector
$z\sim\mathcal{N}(\mu_{\theta,\text{C}},\Sigma_{\theta,\text{C}})$. Thus, we
construct a loss function $\mathcal{L}_{\text{TN}}$:

$\displaystyle\mathcal{L}_{\text{TN}}=\mathbb{E}_{\text{C}\in\Phi^{\text{N}}}[\alpha\text{KL}[p_{\theta}(z\mid\text{C})\parallel
q(z)]-\mathbb{E}_{p_{\theta}(z\mid\text{C})}\text{log}p_{\lambda}(\text{C}\mid
z)]\text{.}$ (9)

The regularization term $\text{KL}[p_{\theta}(z\mid\text{C})\parallel q(z)]$
is the Kullback–Leibler divergence between
$p_{\theta}(z\mid\text{C})=\mathcal{N}(\mu_{\theta,\text{C}},\Sigma_{\theta,\text{C}})$
and $q(z)=\mathcal{N}(0,\mathbb{I})$ where $\mathbb{I}$ is an identity matrix.
The second term
$\mathbb{E}_{p_{\theta}(z\mid\text{C})}\text{log}p_{\lambda}(\text{C}\mid z)$
is the reconstruction log-probability; we control the trade-off between these
two terms with a hyperparameter $\alpha$. After minimizing
$\mathcal{L}_{\text{TN}}$ with dataset $\Phi^{\text{N}}$ by simultaneously
optimizing with respect to $\theta$ and $\lambda$, we can then obtain a
TacNGen $\tau$ to generate a random noise contact depth map
$\hat{\text{C}^{\text{N}}}$ with

$\displaystyle\tau(z)=f_{\lambda^{\ast}}(z)=\hat{\text{C}^{\text{N}}}\text{ ,
}$ (10)

where $z\sim\mathcal{N}(\mu,\Sigma)$ and
$\lambda^{\ast}=\text{argmin}_{\theta,\lambda}\mathcal{L}_{\text{TN}}$.

##  IV Experiments and Results

###  IV-A Preliminary

####  IV-A1 Hardware

For all experiments in this paper, we use a desktop robot (Dobot MG400), which
is a relatively low-cost, high-accuracy robot well-suited for tactile robot
control tasks [8, 23]. We attach a low-cost, high-resolution optical tactile
sensor called the TacTip [1] as an end-effector. This tactile sensor features
an internal array of markers on biomimetic protruding pins inside the soft
tactile skin to capture marker-based movement that amplifies the skin
deformation induced by physical contact against external stimuli.

####  IV-A2 Dataset and Networks Training

To streamline our analysis, in this paper we limit our focus to an edge
feature as the target feature. To this end, we first collect 7000 tactile
images of edge-feature data by conducting labelled random contacts between the
TacTip and a square stimulus with flat edges (Fig. 4a, top image). These
contacts are made within ranges of position $y\in[-6,6]$ mm, depth $z\in[3,6]$
mm, and rotation $R_{z}\in[-180^{\circ},180^{\circ}]$ both in the real world
and in simulation. Then we develop a TacNGen in simulation for creating
various noisy contact depth maps. Specifically, we train a VAE with conv
layers to reconstruct contact depth maps of simulated cone-shaped distractors
that vary in radii and angles (see Fig. 2c). Further VAE training
hyperparameters can be found in the supplementary files. In practice, we found
that generating the noise tactile images from TacNGen and then applying
conventional image augmentation to them in every epoch of training the
TacSalNet resulted in similar performance compared to regenerating the noise
tactile images every epoch. With these datasets and the proposed methods (Secs
III-B,III-C), we can learn a ConDepNet and a TacSalNet using pix2pixGAN [22]
with hyperparameters specified in [2]. It is important to note that the real
tactile images are used only for the ConDepNet, while the TacSalNet and the
TacNGen are trained solely in simulation.

###  IV-B TacSalNets with Different Tactile Noise Generation

First, we conduct an ablation study to compare the performance of our tactile
saliency network (TacSalNet) when the noise is generated using our proposed
noise generation model (TacNGen) versus a 2-dim multivariate Gaussian
distribution. For conciseness, we will refer to the TaSalNet trained with
noise generated from TacNGen as TacSalNet-1, and the one trained with Gaussian
noise as TacSalNet-2. The evaluation process, depicted in Fig. 3a, uses the
same pose ranges as those used during data collection (see Sec. IV-A2). We
evaluate the performance of both TacSalNets with four commonly-used metrics in
visual saliency research: AUC-Judd, similarity measure (SIM), Pearson’s linear
coefficient (CC), and normalized scanpath saliency (NSS), and the testing
results are reported in Table I. The results demonstrate that the TacSalNet-1
outperforms the TacSalNet-2 in three metrics and one the same, indicating the
effectiveness of our TacNGen in improving the performance of tactile saliency
prediction compared to domain-agnostic stochastic noise generation.
Additionally, we found that even though they are only trained with straight-
edge features, the TacSalNet-1 can generalize well to unseen corner-edge
features while the TacSalNet-2 tends to preserve the noise features (see the
last 3 rows in Fig. 3b). Thus, we only consider the TacSalNet-1 (TacNGen-
based) for the evaluation of our whole framework in the contact pose
prediction task and the edge-following task.

Figure 3: (a) Illustration of the evaluation process for TacSalNets trained with TacNGen-generated noise (TacSalNet-1) and Gaussian distribution noise (TacSalNet-2). Blue arrows indicate the pipeline for predicted saliency map generations, while green arrows show the pipeline for target saliency map generation. (b) Tactile saliency maps predicted from TacNGen-based (left) and Gaussian-based (right) TacSalNets respectively using the same input tactile images (overlay) in each row. Table I: Real-world validation with similarity metrics for TacSalNet-1 and TacSalNet-2 over 1000 samples. Bold numbers denote the best results. | AUC-J $\uparrow$ | SIM $\uparrow$ | CC $\uparrow$ | NSS $\uparrow$
---|---|---|---|---
TacSalNet-1 | 0.995 | 0.984 | 0.957 | 4.629
TacSalNet-2 | 0.995 | 0.972 | 0.936 | 4.288

###  IV-C Tactile Saliency Prediction for Contact Pose Estimation

Figure 4: Real-world evaluation of tactile saliency prediction based on the
tactile PoseNet prediction accuracy on the target feature (edge) distracted by
various everyday objects: (a) the distraction setup with everyday objects of
various shapes; (b) real tactile images of the local contact paired with
related contact depth map and tactile saliency map; (c) tactile PoseNet
prediction accuracy on 2D edge pose without and (d) with tactile saliency
prediction.

In our second experiment, we evaluate the performance of tactile saliency
prediction in improving contact pose prediction accuracy in the presence of
distractors. To achieve this, we apply the TacSalNet-1 to a tactile
PoseNet111Here we use a sim-to-real version of the PoseNet:
https://github.com/ac-93/tactile_gym_servo_control., which is a Convolutional
Neural Network that predicts pose from tactile images [7], and investigate its
impact on predicting the target edge 2D pose (position $y$ and orientation
$R_{z}$) from a tactile image when the TacTip statically contacts the target
edge and distractors (see Fig. 4a). In particular, we consider five everyday
objects as distractors varied in shapes, including a bolt, a comb with soft
tines, a wooden clip, a toy figure, and a spoon (the results of the last two
objects are shown in the supplementary materials). Each distractor is fixed
next to the target edge, with distances ranging between $[7,14]$ mm. The
contact poses are sampled from position $x\in[-10,10]$ mm, $y\in[-3,3]$ mm,
$z\in[3,6]$ mm, and rotation $R_{z}\in[-45^{\circ},45^{\circ}]$.

The results show that the presence of the distractors significantly impairs
the standard PoseNet performance, resulting in mean absolute errors (MAEs) of
$1.84$-$2.41$ mm and $14.0^{\circ}$-$30.0^{\circ}$ for predicting position $y$
and orientation $Rz$ respectively (Fig. 4c). However, when the TacSalNet-1 is
applied, the PoseNet can maintain its accuracy with position $y$ MAE of
$0.26$-$0.40$ mm and orientation $R_{z}$ MAE of $4.44^{\circ}$-$5.06^{\circ}$
(Fig. 4d, note that we calibrated the unavoidable prediction drift caused by
the distractors). Importantly, tactile saliency prediction does not degrade
the PoseNet performance, as evident from the marginal difference in MAEs
between the results with and without TacSalNet-1 is minor (the first row in
Fig. 4).

To demonstrate the generalizability of our tactile saliency prediction method
on various real-world noise features, we randomly selected real tactile images
with paired contact depth maps and tactile saliency maps induced by static
contact with different distractors (Fig. 4b). In the last pair of tactile
images of Fig. 4b, we see the TacSalNet-1 accurately predicts the target edge
shape even in the presence of distracting contacts that were unseen during
training.

We observe that the performance of the PoseNet with tactile saliency
prediction drops in all cases when the TacTip position $y$ is around $-2.5$
mm. We attribute this to the close proximity of the TacTip to the distractors,
and accordingly the TacSalNet-1 tends to instead predict the distractor as
part of the target feature, causing a decrease in PoseNet accuracy.

###  IV-D Tactile Saliency Prediction in the Edge Following Task

Figure 5: The diagram of two types of sim-to-real tactile control methods augmented with tactile saliency prediction for the edge-following task with unexpected distractors. The orange arrows denote the pipeline of tactile image processing, the blue and purple arrows denote the pipelines of the pose-based PID control and image-based deep RL policy respectively. The network in green is trained with real-world and simulated data, while the ones in red are obtained solely in simulation and applied to the real world without fine-tuning.  Table II: MAEs of the trajectories from the ground truth for the edge-following task with distractors using pose-based PID control and Image-based deep RL policy without and with TacSalNet-1 (short for TSN-1 in this table). Objects | Pose-based PID | Image-based deep RL
---|---|---
w/o TSN-1 | w/ TSN-1 | w/o TSN-1 | w/ TSN-1
Square | Fail | 0.77mm | Fail | 1.04mm
Foil | Fail | 0.53mm | Fail | 0.94mm
Flower | Fail | 0.76mm | Fail | 1.03mm
Volute | Fail | 0.91mm | Fail | 1.21mm
Figure 6: Evaluation of the proposed tactile-saliency-based control framework
in the real-world 2D edge-following task: (a) the target objects are
surrounded with distractors; (b) the TacSalNet-1 can accurately predict the
target-feature shapes that are unseen during training; edge-following
performance for both (c) pose-based PID control and (d) image-based deep-RL
control: the green contour shows the ground truth for each target object;
errors of the traced contour from the ground truth are colour-coded (side
colour bar); the purple crosses indicate the distractors that the TacTip
remained stuck within 50 steps; the blue arrow denotes the starting pose; and
the red arrow denotes the orientation of the TacTip with the pose-based PID
controller (the image-based deep-RL policy only outputs x and y actions).

In our third experiment, we investigate the performance of the tactile
saliency prediction in improving the robustness of different tactile control
methods during an edge-following task that involves distractors. Specifically,
the robot is tasked to control its tactile sensor to follow along the edges of
four target objects chosen to have distinct edge features (a square, volute,
flower and foil). These target objects are surrounded by at least four
distractors fixed next to their edges with distances between $[7,12]$ mm (Fig.
6a). In this task, we focus on the effect of the distractor as the edge
changes in curvature, considering just one type (the bolt) given the
effectiveness of the TacSalNet-1 for all distractors considered in Sec. IV-C.

We consider two distinct state-of-the-art tactile control methods: 1) tactile
pose-based PID control [11], and 2) tactile image-based deep reinforcement
learning [2]. Note that both the pose-based PID controller and the image-based
deep-RL policy are learned solely in simulation with the contact depth map as
input, and apply to the real world without fine-tuning. To evaluate whether
our proposed method can improve the robustness of these control methods in the
presence of distractors, we augment them with TacSalNet-1 (pipeline in Fig.
5).

The experimental results (Fig. 6) indicate that the use of either tactile
control methods alone in the presence of distractors is insufficient to
successfully complete the task, as the robot becomes distracted by bolts and
fails away from the edge of the target objects. Thus neither control method
alone is robust to unexpected distractions. However, when the TacSalNet-1 is
used to predict tactile saliency to augment the tactile input to both control
methods, the robot successfully achieves the task by completing a circuit of
the object.

The accuracies of the pose-based PID controller and the image-based deep-RL
policy with tactile saliency prediction from 80 real-world tests (10 for each
object with each method combination) are 0.53-0.91 mm and 0.94-1.21 mm
respectively (Table II), compared to an overall distance travelled of 300-520
mm. Overall, the tactile saliency prediction significantly improves the
robustness of sim-to-real tactile control.

Moreover, we observed that the sim-to-real tactile control methods with
saliency prediction performed consistently well on all objects, despite
variations in their edge features, such as sharp and right-angle corners,
concave and convex edges with noise features (Fig. 6a). This finding indicates
the generalizability of our saliency prediction framework, which is a key
advantage for tactile robotic systems that operate in diverse and unstructured
environments. A broader range of distracting objects was also considered with
this edge-following task, and the method found to be similarly robust to
distractors; the videos for these experiments are available in the
supplementary materials.

##  V DISCUSSION AND FUTURE WORK

In this work, we introduce the concept of tactile saliency inspired by visual
saliency, to describe the target features from real tactile images captured by
optical tactile sensors, for improving the accuracy and robustness of tactile
control in the presence of distractors. To develop a generalizable tactile
saliency prediction network, we propose a 3-stage approach. First, we employ a
ConDepNet to generate contact depth maps, which can extract the global contact
information that contains target and noise features in real tactile images.
Second, we introduce a TacSalNet to predict a tactile saliency map that
accurately describes the target areas within an input contact depth map.
Finally, we develop a TacNGen, which generates various noise features to train
the TacSalNet, thereby improving its generalizability to unseen contact depth
maps. Notably, we only train the ConDepNet using real-world data, while the
TacSalNet and TacNGen are trained solely in simulation.

We validated our approach in the real world by demonstrating the efficacy of
tactile saliency prediction alongside previous works in: 1) improving tactile
pose prediction accuracy [7] in the presence of different distractors (Sec.
IV-C); 2) enhancing the robustness of two different sim-to-real tactile
control methods, namely pose-based PID control [11] and image-based deep
reinforcement learning [2], during an edge-following task that involves
distractors (Sec. IV-D).

Our proposed approach offers several key advantages. Firstly, it eliminates
the need for labour-intensive human labelling, resulting in a more efficient
and practical data collection process. Secondly, with the high fidelity of
generated tactile noise, it enables accurate prediction of target features
from contact depth maps in the presence of unseen noise, enhancing the
accuracy of the contact pose prediction and the robustness of tactile control
in unstructured environments. Thirdly, the TacSalNet can be seen as a simple
plug-in module that can extend readily for various types of sim-to-real
tactile control methods, such as the ones we considered here [11, 2].

Our approach was tested with a marker-based optical tactile sensor (the
TacTip) and relies on a depth-based tactile simulator (Tactile Gym). Previous
work [8] has demonstrated that different tactile sensors (DIGIT [24] and
DigiTac [23]) can be easily incorporated into tactile gym for sim-to-real
tactile policy learning. It will be interesting in future work to explore how
our approach applies to other types of tactile sensors.

Overall, we expect the tactile saliency approach developed here can be applied
more widely to various other exploration and manipulation tasks involving
optical tactile sensors that are adversely affected by unexpected tactile
distractors.

## References

  * [1] N. F. Lepora, “Soft biomimetic optical tactile sensing with the tactip: A review,” _IEEE Sensors Journal_ , vol. 21, no. 19, pp. 2131–2143, 2021.
  * [2] A. Church, J. Lloyd, R. Hadsell, and N. F. Lepora, “Tactile sim-to-real policy transfer via real-to-sim image translation,” in _Proceedings of the 5th Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, A. Faust, D. Hsu, and G. Neumann, Eds., vol. 164. PMLR, 08–11 Nov 2022.
  * [3] L. Pecyna, S. Dong, and S. Luo, “Visual-tactile multimodality for following deformable linear objects using reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 3987–3994.
  * [4] Y. Lin, A. Church, M. Yang _et al._ , “Bi-touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 9, pp. 5472–5479, 2023.
  * [5] I. Huang _et al._ , “Soft tactile contour following for robot-assisted wiping and bathing,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 7797–7802.
  * [6] Z. Kappassov, J. A. C. Ramon, and V. Perdereau, “Tactile-based task definition through edge contact formation setpoints for object exploration and manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5007–5014, 2022.
  * [7] N. F. Lepora and J. Lloyd, “Optimal deep learning for robot touch: Training accurate pose models of 3d surfaces and edges,” _IEEE Robotics & Automation Magazine_, vol. 27, no. 2, pp. 66–77, 2020.
  * [8] Y. Lin, J. Lloyd, A. Church, and N. Lepora, “Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 10 754–10 761, August 2022.
  * [9] R. Cong _et al._ , “Review of visual saliency detection with comprehensive information,” _IEEE Transactions on circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 2941–2959, 2018.
  * [10] M. Jiang _et al._ , “Salicon: Saliency in context,” in _IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1072–1080.
  * [11] N. F. Lepora and J. Lloyd, “Pose-based tactile servoing: Controlled soft touch using deep learning,” _IEEE Robotics & Automation Magazine_, vol. 28, no. 4, pp. 43–55, 2021.
  * [12] A. Borji, “Saliency prediction in the deep learning era: Successes and limitations,” _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * [13] S. Jia and N. D. Bruce, “Eml-net: An expandable multi-layer network for saliency prediction,” _Image & Vision Computing_, p. 1037, 2020.
  * [14] Y. A. Mejjati _et al._ , “Look here! a parametric learning based approach to redirect visual attention,” in _European Conference on Computer Vision_. Springer, 2020, pp. 343–361.
  * [15] K. Aberman, J. He _et al._ , “Deep saliency prior for reducing visual distraction,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 9819–9828.
  * [16] M. Xu, B. Ni, J. Dong, Z. Huang, M. Wang, and S. Yan, “Touch saliency,” in _Proceedings of the 20th ACM international conference on Multimedia_ , 2012, pp. 1041–1044.
  * [17] B. Ni _et al._ , “Touch saliency: Characteristics and prediction,” _IEEE Transactions on Multimedia_ , vol. 16, no. 6, pp. 1779–1791, 2014.
  * [18] M. Lau, K. Dev _et al._ , “Tactile mesh saliency,” _ACM Transactions on Graphics (TOG)_ , vol. 35, no. 4, pp. 1–11, 2016.
  * [19] J. Jiao _et al._ , “Tactile sketch saliency,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 3072–3080.
  * [20] S. Jain _et al._ , “Visual 3d perception for interactive robotic tactile data acquisition,” in _2021 IEEE International Conference on Automation Science and Engineering (CASE)_ , Aug. 2021.
  * [21] G. Cao, Y. Zhou, D. Bollegala, and S. Luo, “Spatio-temporal attention model for tactile texture recognition,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020, pp. 9896–9902.
  * [22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1125–1134.
  * [23] N. F. Lepora, Y. Lin, B. Money-Coomes, and J. Lloyd, “Digitac: A digit-tactip hybrid tactile sensor for comparing low-cost high-resolution robot touch,” _IEEE Robotics and Automation Letters_ , 2022.
  * [24] M. Lambeta, P.-W. Chou, _et al._ , “Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 3838–3845, 2020.


# ClusterFace: Joint Clustering and Classification for Set-Based Face
Recognition

Samadhi Wickrama Arachchilage and Ebroul Izquierdo  Multimedia and Vision
Group
School of Electronic Engineering and Computer Science
Queen Mary University of London, United Kingdom
Email: {s.wickramaarachchilage<EMAIL_ADDRESS>
###### Abstract

Deep learning technology has enabled successful modeling of complex facial
features when high quality images are available. Nonetheless, accurate
modeling and recognition of human faces in real world scenarios ‘on the wild’
or under adverse conditions remains an open problem. When unconstrained faces
are mapped into deep features, variations such as illumination, pose,
occlusion, etc., can create inconsistencies in the resultant feature space.
Hence, deriving conclusions based on direct associations could lead to
degraded performance. This rises the requirement for a basic feature space
analysis prior to face recognition. This paper devises a joint clustering and
classification scheme which learns deep face associations in an easy-to-hard
way. Our method is based on hierarchical clustering where the early iterations
tend to preserve high reliability. The rationale of our method is that a
reliable clustering result can provide insights on the distribution of the
feature space, that can guide the classification that follows. Experimental
evaluations on three tasks, face verification, face identification and rank-
order search, demonstrates better or competitive performance compared to the
state-of-the-art, on all three experiments.

##  I Introduction

††© 20XX IEEE. Personal use of this material is permitted. Permission from
IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional
purposes, creating new collective works, for resale or redistribution to
servers or lists, or reuse of any copyrighted component of this work in other
works.

Face recognition made tremendous advancements over the last decade with a
plethora of systems based on deep convolutional neural networks (DCNN).
Starting from DeepFace in 2014, face recognition systems consistently reported
near-human or even advanced performance on classic datasets like LFW [1].
Followed by the performance saturation on LFW, subsequent benchmarks like
IJB-A [2] aimed to address the rather challenging problem of unconstrained
face recognition. Consequently, diverse architectural enhancements and
learning based solutions addressing issues related to low-quality images,
varying pose, illumination changes, emotional expressions, etc., have been
proposed and studied over the recent years.

Generic advances of face recognition include increasing dataset size,
employing more sophisticated deep networks and using ensembles of multiple
models etc. Facebook and Google used large in-house datasets of millions in
scale to train face recognition models [3, 4]. The success of these systems,
resulted in large and openly accessible face datasets like VGGFace2 [5].
Parallel progress was reported in research landscapes such as deeper and
sophisticated network architectures and advanced loss functions. These leading
contributions act as a strong foundation for novel systems by providing richer
feature representations for faces.

Learning based solutions aim to increase the representative power of deep
features assuming powerful feature space will be more robust to variations in
unconstrained data. However, face recognition under adverse conditions entails
specific complications that cannot be easily remedied by generic solutions.
These faces are naturally less informative due to blur, low-resolution, poor
illumination or insufficient identity cues due to extreme poses. Additionally,
these faces could be confusing and even contradictory. For example, the weak
samples of different individuals could be of more visual similarity than
between a weak sample and a stable sample of the same individual. When such
face samples are mapped into deep features, the inconsistencies are carried
onto the feature space and hence plain associations based on first-perception
could be misleading. While these complications cannot be eliminated entirely,
their impact can be minimized by feature distribution learning and
neighbourhood analysis. A simple and creative solution can be composed via an
easy-to-hard self learning of deep feature distribution where one begins by
associating the most confident neighbors and incrementally progress towards
confusing samples.

This paper presents a joint clustering and classification scheme code-named
‘ClusterFace’, where clustering is introduced as an intermediary leaning step.
During this stage, we gather the closest neighbours into confident clusters.
Next we use the cluster base result to classify the less close neighbours. We
use this incrementally learnt information, along with local smoothness
assumption to formulate a set of constraints which guide the final recognition
task. In doing so, we formulate associations in a gradual and incremental
fashion rather than conventional single-prediction based recognition.

Figure 1: The benefit of hierarchical clustering based associations.
${x1,..x6}$ are faces of individual x and ${y1,...,y6}$ are of individual y.
$c\\_x$ denotes cluster x and $c\\_y$ denotes cluster y. Dashed lines show
direct associations, solid lines show ClusterFace associations. Top: Direct
associations incorrectly conclude that x1 and y1 (faces of different
individuals under similar illumination conditions) are more similar than y1
and y6 (faces of same individual under different settings). Bottom:
ClusterFace associations begin by merging the closest faces. Gradual and
incremental associations correctly group the samples of same individual in to
a single cluster.

Figure 1 provides a visual demonstration of ClusterFace association
formulation. It exemplifies how misleading information can be restored by
incremental learning. This cluster based association is applicable on tasks
such as face verification, identification and retrieval. In face verification,
ClusterFace assists in identifying faces of the same individual under
different environments by chaining through similarities. In face
identification and retrieval tasks where a search term probe is compared
against a gallery of enrolled faces, we map the probe and the gallery in to
the same space and develop ClusterFace learning. This mapping is particularly
helpful in cross-media recognition where where there is a gap between probe
and gallery samples. In such circumstances, ClusterFace identifies and
exploits instances where probe and gallery has clear relationships.

The main contributions of the paper are as follows. We present a clustering
based deep feature distribution learning scheme which aims to minimize the
impact of confusing samples. The solution is composed as a generic-set based
recognition scheme which is invariant to the image order, and hence is
applicable on both videos and images. On top of the proposed cluster based
learning, we formulate face verification, recognition and rank-order search as
constrained classification problems. Experimental evaluation on challenging
benchmarks show that the proposed approach exploits the deep features in an
optimal way.

The rest of the paper is organized as follows. Section 2 discusses the related
literature on unconstrained face recognition. Section 3 presents the
ClusterFace workflow followed by the performance evaluation in section 4.
Section 5 presents the conclusion.

##  II Related Work

The approach we proposed in this paper aims to address unconstrained face
recognition by building on advanced deep networks and self formulated
constraints. Therefore, we briefly review related works from three aspects,
including deep network architectures, unconstrained face recognition and self-
formulated constraints.

###  II-A DCNN architectures

Starting from LeNet in 1989 [6], DCNNs have evolved into sophisticated
networks particularly fuelled by classification challenges like The ImageNet
Large Scale Visual Recognition Challenge (ILSVRC) [7]. These sophisticated
networks were exploited in many computer vision tasks including face
recognition. Some successful face recognition applications that used image
classification networks are: DeepId3 [8] which was influenced by VGGNet [9]
and GoogLeNet [10], Google’s Facenet [11] which used GoogleNet [10]
architecture and VGGFace [12] that exploited concepts from VGGNet [9].
Inspired, we use Inception ResNet V1 network [13] which has shown to be
effective in image recognition with a 4.3% top-5 error rate in ILSVRC 2012.

A deep network is generally underpinned by an optimization loss function which
plays an important role in adding discriminative power to the generated
features. Over the years, loss functions have evolved complementing the
network architectures. These loss functions can be categorized as
classification based approaches, i.e., softmax loss and it’s variants [14,
15], and metric learning approaches, i.e., contrastive loss [16] and triplet
loss [11]. This paper uses a combination of two approaches (i.e., softmax loss
for pre-training and triplet loss for further fine-tuning) to achieve high
representational power.

###  II-B Unconstrained recognition

Face recognition under unconstrained settings, requires additional guidance on
top of deep features. Popular guidance measures include, using multiple
classifiers which guide each other, adaptive self-learning, guidance through
visual or spatio-temporal cues, or strong sample based predictions.

Adaptive facial models employ single or multiple classifiers like support
vector machines, nearest neighbour classifiers, etc., that are updated based
on face tracking results [17, 18]. These approaches are applicable only on
videos where, the additional spatio-temporal information present in videos can
be used to generate face tracks. Co-training is the process of training
multiple classifiers where the classifiers guide each other based on confident
predictions [19, 20]. While co-training has reported considerable performance,
their effectiveness mainly depends on the quality and quantity of the labelled
data that initiate the classifiers. Bhatt et al. [19] used an ensemble of
support vector machines trained on labelled gallery and some labelled probe
samples. Such semi-supervised approaches require some labelled test data. Co-
training and adaptive facial model generation have similarities to ClusterFace
in that both approaches progress through operational data in an easy-to-hard
way. However, unlike the other two approaches, ClusterFace initiates by
clustering the feature space into unknown number of clusters, and hence
requires no additional information or labelled data.

Another approach of minimizing the effect of adverse samples is explicit
subset selection, where high quality samples are selected through quality
assessment [21, 22, 23]. While this approach has been proven to be effective
through empirical studies, the challenge in this is finding a solid definition
for ‘face quality’ [24]. Yang et al. proposed a neural aggregation network
which is trained to automatically advocate high-quality face images while
repelling low-quality ones [25]. ClusterFace is a much simpler alternative
where instead of identifying strong samples, we identify strong relationships.
The quality of a relationship is defined in terms of the semantic similarity
of corresponding deep features.

###  II-C Self formulated constraints

Self formulated constraints have been used in video based tracking and
clustering. These applications use two pairwise affinity constraints, Cannot-
Link (CL) and Must-Link (ML) [26, 27, 28]. These constraints are formed
exploiting spatio-temporal affinities of multi-face videos where faces within
a single shot (a shot is a set of adjacent frames which covers a single motion
pattern) are considered must-link and faces in a single frame are considered
as cannot link. While these constraints certainly does provide guidance for
face tracking and clustering, these are not applicable on tasks such as face
recognition on singleton videos (because there are no overlapping faces to
form cannot link constraints), face image or image-set based recognition.
Unlike the conventional CL and ML constraints, the ClusterFace constraints can
be formulated without spatio-temporal relationships. To the best of our
knowledge, this is the first face recognition based study which uses self-
formulated constraints on un-ordered face image-sets.

##  III ClusterFace Workflow

The problem of face recognition requires association learning among faces. We
device a multi-step process which first learns the strong predictions and
second builds a set of neighbourhood predictions on top of the strong
predictions. These predictions are then used as constraints that guide the
remaining predictions. As seen is figure 2, the process begins with deep
feature space generation and is followed by salient clustering, constraint
formulation and final recognition.

Given a face detection $d$, the deep network maps the complex high-dimensional
image information into a n-dimensional proprietary feature vector
$\phi(d)\epsilon R^{n}$. The generated feature vectors can be interpreted as
points in a fixed-dimensional space where the distance between two points is
analogous to the level of similarity between the two corresponding faces. The
distance measure used in this paper is the cosine similarity between two face
detections $d_{a},d_{b}$, which is calculated as follows.

$SIM(d_{a},d_{b})=\frac{\phi(d_{a}).{\phi(d_{b})}}{||\phi(d_{a})||||\phi(d_{b})||}$
(1)

Conventional face verification process employs a threshold $\beta$ such that
if the distance between two faces is less than $\beta$, they are of the same
individual. Building on this convention, we introduce a margin of uncertainty
$\gamma$ such that if the distance between two faces is less than
$\beta-\gamma$, they are concluded as of the same individual with higher
degree of certainty (i.e strong positive predictions).

###  III-A Salient Clustering

The clustering is based on bottom-up Hierarchical Agglomeration (HAC) which
combines the closest cluster in each iteration. Hence, the process exploits
two inherent advantages of hierarchical clustering: preserving higher
reliability in early iterations and not requiring a pre-specified number of
clusters. By setting $\beta-\gamma$ as the termination height in HAC, this
clustering step identifies the most stable associations within the feature
space. The clustering algorithm is formulated for generic set-based face
recognition tasks where each input face image set $f_{i}$ is represented by a
single point in the deep feature space $\phi(f_{i})$. The clustering is
detailed in algorithm 1 where the distance between two clusters is measured as
the cosine similarity between the two cluster centroids as denoted in the
following equation.

$dist(c_{x},c_{y})=\frac{(\frac{\sum_{i=1,..N_{x}}{\phi(x_{i})}}{N_{x}})\text{
.
}(\frac{\sum_{i=1,..N_{y}}{\phi(y_{i})}}{N_{y}})}{||\frac{\sum_{i=1,..N_{x}}{\phi(x_{i})}}{N_{x}}||\text{
}||\frac{\sum_{i=1,..N_{y}}{\phi(y_{i})}}{N_{y}}||},$ (2)

where the two clusters $c_{x}$ and $c_{y}$ are of size $N_{x}$ and $N_{y}$.

Algorithm 1 Salient Clustering For Face image sets

Input: F:
$[\\{f_{1},\phi(f_{1})\\},\\{f_{2},\phi(f_{2})\\},...,\\{f_{n},\phi(f_{3})]$}
Output: F where confident positives are merged
Initialize: $dist\\_arr=[]$
$termination\\_dist=\beta-\gamma$

1: $pairs\leftarrow$ Generate the set of all face-set pairs.

2: for each pair $[f_{1},f_{2}]$ in $pairs$ do

3: $dist\\_arr\leftarrow SIM(f_{1},f_{2})$

4: Find $[f\\_min_{1},f\\_min_{2}]$ corresponding to $min(dist\\_arr)$

5: while $min(dist\\_arr)<termination\\_dist$ do

6: $merge\\_cluster\leftarrow Merge(f\\_min_{1},f\\_min_{2})$

7: Delete $dist\\_arr$ elements with $f\\_min_{1}$ or $f\\_min_{2}$

8: Delete $f\\_min_{1},f\\_min_{2}$ from $F$

9: for each face $f$ in $F$ do

10: $dist\\_arr\leftarrow dist(f,merge\\_cluster)$

11: $F\leftarrow merge\\_cluster$

$MA_{i,j}$[{x,$\phi(x)$}]$NA_{i,j}$ Constrained Classification (1) Face
Verification (2) Face Identification (3) Rank-Order Retrieval Input Faces Deep
Feature Generation Constraint Learning DCNN [$\phi(x)$] Salient Clusters

Figure 2:  The dataflow of the ClusterFace face recognition system. The input
faces are the test faces including both gallery and probe images. The pre-
trained DCNN model generates feature vectors and the resulting feature space
is used to generate a set of salient clusters. The cluster label based
constrained matrixes guide the classification that follows.

###  III-B Constraint Formulation

We formulate two constraints Must Associate (MA) and Neighbourhood Associate
(NA) based on the clustering result, local smoothness assumption and an
additional classification step. Local smoothness assumption is imposed in two
levels. The label-level smoothness means that if two observations $x_{i}$ and
$x_{j}$ are similar, then their labels $y_{i}$ and $y_{j}$ should be similar;
the constraint-level smoothness tells that given a must-associate between
$x_{1}$ and $x_{2}$, if $x_{3}$ is close to $x_{2}$, then it is assumed that
there is also a must-associate between $x_{1}$ and $x_{3}$ [27].

####  III-B1 Must Associate Constraint

MA constraint states that faces within the same salient cluster are of the
same individual. The MA matrix is calculated as follows.

$MA_{i,j}=\begin{cases}c_{x},&\text{if}\ i,j\epsilon C_{x}\\\
NILL,&\text{otherwise}\end{cases}$ (3)

where $c_{x}$ is the cluster label of cluster $C_{x}$.

The cluster labels are assigned based on the label-level local smoothness
assumption. In particular, all elements within a single salient cluster are
identified by a single identity label which is the highest occurring identity.
Hence the label of a cluster $C_{x}$ is given by $MODE\\{x_{i}\\}$ where
$x_{i}$ denotes the identity label of cluster element $i$.

####  III-B2 Neighbourhood Associate Constraint

NA is based on the constraint-level local smoothness assumption and k-nearest
neighbour search. Given a face $i$, its nearest neighbours
$\\{n_{1},n_{2},n_{3},...,n_{k}\\}$ are selected such that
$SIM(i,n_{1})<SIM(i,n_{2}),...,<SIM(i,n_{k})$ and subjected to the regularity
constraint $SIM(i,n_{x})<\beta$. The NA matrix is calculated as follows.

$NA_{i,j}=\begin{cases}c_{x},&\text{if}\ i\epsilon C_{x}\text{ AND
}N\\{j,C_{x}\\}=TRUE\\\ NILL,&\text{otherwise}\end{cases}$ (4)

where $c_{x}$ is the cluster label of cluster $C_{x}$ and $N\\{j,C_{x}\\}$ is
the neighbourhood constraint between a face $j$ and a cluster $C_{x}$ computed
as follows.

$\displaystyle N\\{j,C_{x}\\}=len[(n_{1},n_{2},...,n_{k})\epsilon C_{x}]$ (5)
$\displaystyle<len[(n_{1},n_{2},...,n_{k})\epsilon C_{y}]$
$\displaystyle\text{ for any cluster }C_{y}\text{ where }y!=x$

where $n_{1},n_{2},...,n_{k}$ are the k-nearest neighbours of $j$ and
$[(n_{1},n_{2},...,n_{k})\epsilon C_{x}]$ is the sub-set of the k-neighbours
that are also elements of cluster $C_{x}$.

###  III-C Classification

This section details the three main constrained classification tasks, face
identification, verification and rank-order search.

####  III-C1 Face verification

Face verification is the task of determining if two faces belong to the same
identity or not. The constrained verification of two faces i and j, can be
stated as follows.

$V_{i,j}=\begin{cases}True,&\text{if}\ MA_{i,j}\\\ True,&\text{if}\
NA_{i,j}\text{ and }d(m,j)<\beta-\gamma\\\ False,&\text{otherwise}\end{cases}$
(6)

####  III-C2 Face identification

Given a probe and a gallery, face identification is the task of matching the
probe to its mate in gallery. Given a face $i$, face identification problem is
formulated as finding the optimal closest neighbour by a rank order search for
rank-1.

####  III-C3 Rank-Ordered Results

Given a probe and a gallery, rand-ordered identification measures what
percentage of probe searches return the probe’s gallery mate within the top k
rank-ordered results. This is formulated as a constrained ranked search as
denoted in algorithm 2 which ranks the input nearest neighbours ($NN$) of a
probe $p$, in the order of similarity closest first.

Algorithm 2 Constrained Rank-Order Search

Input: Probe p
$MA=\\{MA_{i,j}\\}$ i,j={1,2,…,n}
$SA=\\{NA_{i,j}\\}$ i,j={1,2,…,n}
$NN=\\{n_{1},n_{2},...,n_{k}\\}$
Output:  Ranked Neighbours  $RN$
Initialize:  $RN=[]$

1: for n in NN do

2: if $MA_{p,n}$ then

3: $RN\leftarrow n$

4: DELETE $n$ from $NN$

5: $RN\leftarrow x$ $\forall$ $MA_{p,x}$ WHERE $x$ $!\epsilon$ $RN$

6: Repeat 1-4 for $\\{NA_{i,j}\\}$

7: $RN\leftarrow n$ $\forall$ $n$ $\epsilon$ $NN$

##  IV Databases and Evaluations

The proposed scheme was evaluated in face recognition and verification
experiments on two publicly available datasets: including the YouTube and
IJB-A database. This section describes the details of the experiments and
results.

###  IV-A Implementation Details

We use an implementation of the MTCNN architecture described in [29] for face
detection. As mentioned before, to leverage the recent advent of DCNN
architectures, we use Inception ResNet V1 network discussed in [13]. The
network is trained with VGGFace2 dataset [5] based on softmax loss, and fine-
tuned on CASIA-WebFace dataset (overlapping subjects with the evaluation
benchmarks removed) [30] based on triplet loss.

We formulate both video face recognition and set based recognition as a set
based recognition problem and perform feature aggregation to assign a single
representation to each set of faces. For a more reliable representation, we
heavily weigh the stable face appearances using the normalised detection
scores, as follows.

$\phi(S)=\sum_{k}w_{s_{i}}\phi(s_{i}),\forall s_{i}\epsilon S$ (7)

where $\phi(S)$ is the set level representation of face image set
$S=\\{s_{1},s_{2},...s_{n}\\}$ of n faces. $w_{s_{i}}$ is the weight
corresponding to the face $s_{i}$. For image based face recognition, each
image is considered as an image-set with cardinality 1.

###  IV-B Face Recognition Benchmarks

####  IV-B1 YouTube Celebrities Face Recognition Dataset

The YouTube Celebrities (YTC) video dataset consists of 1,910 video sequences
of 47 celebrities from YouTube. There are large variations of pose,
illumination, and expression on face videos in this dataset. Moreover, the
quality of face videos is very poor because most videos are of high
compression rate. The experiment setting is the same as [31, 32]. Five fold
cross validation was carried out with three video sequences per subject for
training and six for testing in each fold.

####  IV-B2 IARPA Janus Benchmark A

The IJB-A Dataset (IJB-A) [2] contains 5712 images and 2085 videos of 500
subjects. The average numbers of images and videos per subject are 11.4 images
and 4.2 videos. The images are manually aligned as opposed to the general
practise of using a commodity face detector. The manual alignment process
preserves challenging variations such as pose, occlusion, illumination and
etc., that are generally filtered out with automated detection. The dataset is
a collection of media in the wild which contains both images and videos.

TABLE I: Performance evaluation for verification on IJB-A benchmark. The true accept rates (TAR) vs. false positive rates (FAR). System | FAR=0.001 | FAR=0.01 | FAR=0.1
---|---|---|---
Triplet Similarity [33] | 59.0 | 79.0 | 94.5
Multi-pose (WACV16) [34] | - | 78.7 | 91.1
Triplet Emb (BTAS16) [35] | 81.3 | 91 | 96.4
FastSearch (TPAMI17) [36] | 51.0 | 72.9 | 89.3
Joint Bayesian (WACV16) [37] | - | 83.8 | 96.7
PAM (CVPR16) [38] | 65.2 | 82.6 | -
NAN (CVPR16) [25] | 88.1 | 94.1 | 97.8
Template (FG17) [39] | 83.6 | 93.9 | 97.9
DR GAN (CVPR17) [40] | 53.9 | 77.4 | -
Contrastive (ECCV18) [41] | 63.91 | 84.01 | 95.31
Direct Associations | 84.95 | 93.76 | 98.13
ClusterFace Associations | 86.60 | 94.23 | 98.30
TABLE II: Performance evaluation closed-set face recognition on IJB-A benchmark. The percentage identification accuracies in rank-N retrievals System | Rank-1 | Rank-5 | Rank-10
---|---|---|---
Triplet Similarity [33] | 88 | 95 |
Deep Multi-pose (WACV 16) [34] | 84.6 | 92.7 | 94.7
Triplet Emb (BTAS 16) [35] | 93.2 | - | 97.7
FastSearch (TPAMI 17) [36] | 82.2 | 93.1 | -
Joint Bayesian (WACV 16) [37] | 90.3 | 96.5 | 97.7
PAM (CVPR 16) [38] | 84 | 92.5 | 94.6
NAN (CVPR 16) [25] | 95.8 | 98.0 | 98.6
Template (FG 17) [39] | 92.8 | 97.7 | 98.6
DR GAN (CVPR 17) [40] | 85.5 | 94.7 | -
Direct Associations | 94.23 | 97.05 | 97.71
ClusterFace Associations | 94.28 | 97.05 | 97.72
TABLE III: Classification Rates (%) on the YouTube Celebrities Dataset. System | Classification Accuracy (%)
---|---
SANP (TPAMI 12) [31] | 65.60
MMDML (CVPR 15) [32] | 78.5
DRM-PWV (TPAMI 15) [42] | 72.55
Fast FR (ICCVW 17) [43] | 72.1
GJRNP (IVC 17) [44] | 81.3
Direct Associations | 90.71
ClusterFace Associations | 91.06

###  IV-C Comparison With the Baseline

We compare the performance levels on ClusterFace associations to the
conventional direct associations. This comparison provides a fair evaluation
of the CluterFace impact, since both approaches are based on the same deep
features. Tables I and II, III reports the evaluation results on IJB-A and YTC
databases. It is clear that the proposed approach achieves improvements over
the baseline on all three experiments. In particular, two main observations
depicted in the results are (1): the deep features provide a strong foundation
for recognition, (2) the Clusterface constraints have guided in correcting a
set of confusing predictions.

###  IV-D Comparison With the State-Of-The-Art

Furthermore, our proposed method is compared with a some of the important
state-of-the-art face recognition systems. Despite all our experiments being
carried out on a a simple and plain network without any data augmentation
techniques or using multiple ensembles and under single crop experiments, the
proposed approach achieves better or highly competitive results on all
occasions.

###  IV-E Computational complexity

The additional clustering step results in an added computational complexity.
Since the computational complexity of hierarchical clustering is generally
polynomial, the clustering process could be expensive. In particular, given
$n$ number of discrete face images, the complexity of hierarchical clustering
is generally $O(n^{3})$. To avoid such complexity, the presented clustering
algorithm replaces each image set with its centroid. Moreover, after each
merge, the merged cluster is represented by a single point in the feature
space and hence the number of comparisons in the next iteration is reduced.
Therefore the computational complexity of the presented algorithm is
$O(s^{2}log(s))$, where s is the number of sets and $s<<n$.

##  V Conclusion

We have presented a novel face recognition algorithm for matching faces across
variations such as pose, illumination and resolution. Associations are learned
in an easy-to-hard way where the most confident predictions guide the
confusing predictions. The guiding constraints are formulated based on
hierarchical clustering. The main advantage of the proposed approach is that
there it does not require any additional informational guidance and is solely
based on facial features. Hence our approach is easily applicable on a wide
range of face recognition based tasks face verification, identification and
retrieval. The usefulness of our algorithms is justified with experiments
conducted on the IJB-A database and YTC database in which very good
recognition performance is obtained under wide range of pose and illumination
and resolution conditions.

## Acknowledgment

The research activities leading to this publication has been partly funded by
the European Union Horizon 2020 Research and Innovation program under grant
agreement No. 786629 (MAGNETO RIA project).

## References

  * [1] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments,” in _Workshop on Faces in ’Real-Life’ Images: Detection, Alignment, and Recognition_. Marseille, France: Erik Learned-Miller and Andras Ferencz and Frédéric Jurie, Oct. 2008. [Online]. Available: https://hal.inria.fr/inria-00321923
  * [2] B. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. E. Allen, P. Grother, A. Mah, M. Burge, and A. K. Jain, “Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1931–1939.
  * [3] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Web-scale training for face identification,” _CoRR_ , vol. abs/1406.5266, 2014. [Online]. Available: http://arxiv.org/abs/1406.5266
  * [4] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , Jun. 2014, pp. 1701–1708.
  * [5] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _International Conference on Automatic Face and Gesture Recognition_ , 2018.
  * [6] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
  * [7] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision_ , vol. 115, no. 3, pp. 211–252, Dec. 2015. [Online]. Available: https://doi.org/10.1007/s11263-015-0816-y
  * [8] Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very deep neural networks,” _CoRR_ , vol. abs/1502.00873, 2015. [Online]. Available: http://arxiv.org/abs/1502.00873
  * [9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , Sept. 2015.
  * [10] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2015, pp. 1–9.
  * [11] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2015, pp. 815–823.
  * [12] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in _Proceedings of the British Machine Vision Conference (BMVC)_. BMVA Press, Sept. 2015, pp. 41.1–41.12. [Online]. Available: https://dx.doi.org/10.5244/C.29.41
  * [13] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017.
  * [14] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in _Computer Vision – ECCV 2016_ , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 499–515.
  * [15] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017, pp. 6738–6746.
  * [16] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2, Jun. 2006, pp. 1735–1742.
  * [17] K. Lu, Z. Ding, J. Zhao, and Y. Wu, “A novel semi-supervised face recognition for video,” in _2010 International Conference on Intelligent Control and Information Processing_ , Aug 2010, pp. 313–316.
  * [18] M. D. la Torre, E. Granger, P. V. Radtke, R. Sabourin, and D. O. Gorodnichy, “Partially-supervised learning from facial trajectories for face recognition in video surveillance,” _Information Fusion_ , vol. 24, pp. 31 – 53, 2015\. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1566253514000700
  * [19] H. S. Bhatt, R. Singh, M. Vatsa, and N. K. Ratha, “Improving cross-resolution face matching using ensemble-based co-transfer learning,” _IEEE Transactions on Image Processing_ , vol. 23, no. 12, pp. 5654–5669, Dec 2014.
  * [20] N. El Gayar, S. A. Shaban, and S. Hamdy, “Face recognition with semi-supervised learning and multiple classifiers,” in _Proceedings of the 5th WSEAS International Conference on Computational Intelligence, Man-Machine Systems and Cybernetics_ , ser. CIMMACS’06. Stevens Point, Wisconsin, USA: World Scientific and Engineering Academy and Society (WSEAS), 2006, p. 296–301.
  * [21] X. Gao, S. Z. Li, R. Liu, and P. Zhang, “Standardization of face image sample quality,” in _Advances in Biometrics_ , S.-W. Lee and S. Z. Li, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 242–251.
  * [22] K. Nasrollahi and T. B. Moeslund, “Face quality assessment system in video sequences,” in _Biometrics and Identity Management_ , B. Schouten, N. C. Juul, A. Drygajlo, and M. Tistarelli, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 10–18.
  * [23] M. Subasic, S. Loncaric, T. Petkovic, H. Bogunovic, and V. Krivec, “Face image validation system,” in _ISPA 2005. Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, 2005._ , Sep. 2005, pp. 30–33.
  * [24] Y. Wong, S. Chen, S. Mau, C. Sanderson, and B. C. Lovell, “Patch-based probabilistic image quality assessment for face selection and improved video-based face recognition,” in _CVPR 2011 WORKSHOPS_ , June 2011, pp. 74–81.
  * [25] J. Yang, P. Ren, D. Chen, F. Wen, H. Li, and G. Hua, “Neural aggregation network for video face recognition,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5216–5225, 2016.
  * [26] B. Wu, S. Lyu, B. Hu, and Q. Ji, “Simultaneous clustering and tracklet linking for multi-face tracking in videos,” in _2013 IEEE International Conference on Computer Vision (ICCV)_ , Dec. 2013, pp. 2856–2863.
  * [27] B. Wu, Y. Zhang, B. Hu, and Q. Ji, “Constrained clustering and its application to face clustering in videos,” in _2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2013, pp. 3507–3514.
  * [28] S. Zhang, J.-B. Huang, J. Lim, Y. Gong, J. Wang, N. Ahuja, and M.-H. Yang, “Tracking persons-of-interest via unsupervised representation adaptation,” _International Journal of Computer Vision_ , Sep. 2019. [Online]. Available: https://doi.org/10.1007/s11263-019-01212-1
  * [29] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, Oct. 2016.
  * [30] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” _CoRR_ , vol. abs/1411.7923, 2014. [Online]. Available: http://arxiv.org/abs/1411.7923
  * [31] Y. Hu, A. S. Mian, and R. Owens, “Face recognition using sparse approximated nearest points between image sets,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 34, no. 10, pp. 1992–2004, Oct 2012.
  * [32] J. Lu, G. Wang, W. Deng, P. Moulin, and J. Zhou, “Multi-manifold deep metric learning for image set classification,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015, pp. 1137–1145.
  * [33] S. Sankaranarayanan, A. Alavi, and R. Chellappa, “Triplet similarity embedding for face verification,” _CoRR_ , vol. abs/1602.03418, 2016. [Online]. Available: http://arxiv.org/abs/1602.03418
  * [34] W. AbdAlmageed, Y. Wu, S. Rawls, S. Harel, T. Hassner, I. Masi, J. Choi, J. Lekust, J. Kim, P. Natarajan, R. Nevatia, and G. Medioni, “Face recognition using deep multi-pose representations,” in _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , March 2016, pp. 1–9.
  * [35] S. Sankaranarayanan, A. Alavi, C. D. Castillo, and R. Chellappa, “Triplet probabilistic embedding for face verification and clustering,” in _2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS)_ , Sep. 2016, pp. 1–8.
  * [36] D. Wang, C. Otto, and A. K. Jain, “Face search at scale,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1122–1136, June 2017.
  * [37] J. Chen, V. M. Patel, and R. Chellappa, “Unconstrained face verification using deep cnn features,” in _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , March 2016, pp. 1–9.
  * [38] I. Masi, S. Rawls, G. Medioni, and P. Natarajan, “Pose-aware face recognition in the wild,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016, pp. 4838–4846.
  * [39] N. Crosswhite, J. Byrne, C. Stauffer, O. Parkhi, Q. Cao, and A. Zisserman, “Template adaptation for face verification and identification,” in _2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)_, May 2017, pp. 1–8.
  * [40] L. Tran, X. Yin, and X. Liu, “Disentangled representation learning gan for pose-invariant face recognition,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017, pp. 1283–1292.
  * [41] C. Han, S. Shan, M. Kan, S. Wu, and X. Chen, “Face recognition with contrastive convolution,” in _Computer Vision – ECCV 2018_ , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 120–135.
  * [42] M. Hayat, M. Bennamoun, and S. An, “Deep reconstruction models for image set classification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 4, pp. 713–727, April 2015.
  * [43] H. Cevikalp and H. S. Yavuz, “Fast and accurate face recognition with image sets,” in _2017 IEEE International Conference on Computer Vision Workshops (ICCVW)_ , Oct 2017, pp. 1564–1572.
  * [44] M. Yang, X. Wang, W. Liu, and L. Shen, “Joint regularized nearest points for image set based face recognition,” _Image and Vision Computing_ , vol. 58, pp. 47 – 60, 2017. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S026288561630124X