# Nowhere to Hide:
Cross-modal Identity Leakage between Biometrics and Devices

Chris Xiaoxuan Lu  University of Liverpool<EMAIL_ADDRESS>,  Yang
Li  New York University<EMAIL_ADDRESS>,  Yuanbo Xiangli  The Chinese
University of Hong Kong<EMAIL_ADDRESS>and  Zhengxiong Li  University at
Buffalo, SUNY<EMAIL_ADDRESS>
(2020)

###### Abstract.

Along with the benefits of Internet of Things (IoT) come potential privacy
risks, since billions of the connected devices are granted permission to track
information about their users and communicate it to other parties over the
Internet. Of particular interest to the adversary is the user identity which
constantly plays an important role in launching attacks. While the exposure of
a certain type of physical biometrics or device identity is extensively
studied, the compound effect of leakage from both sides remains unknown in
multi-modal sensing environments. In this work, we explore the feasibility of
the compound identity leakage across cyber-physical spaces and unveil that co-
located smart device IDs (e.g., smartphone MAC addresses) and physical
biometrics (e.g., facial/vocal samples) are side channels to each other. It is
demonstrated that our method is robust to various observation noise in the
wild and an attacker can comprehensively profile victims in multi-dimension
with nearly zero analysis effort. Two real-world experiments on different
biometrics and device IDs show that the presented approach can compromise more
than $70\%$ of device IDs and harvests multiple biometric clusters with $\sim
94\%$ purity at the same time.

Identity Leakage; Cross-modality Association; Internet of Things

††journalyear: 2020††conference: Proceedings of The Web Conference 2020; April
20–24, 2020; Taipei, Taiwan††booktitle: Proceedings of The Web Conference 2020
(WWW ’20), April 20–24, 2020, Taipei, Taiwan††doi:
10.1145/3366423.3380108††isbn: 978-1-4503-7023-3/20/04††ccs: Security and
privacy Pseudonymity, anonymity and untraceability††ccs: Human-centered
computing Ubiquitous and mobile computing

##  1\. Introduction

Internet of Things (IoT) devices are gaining popularity: it is expected that
$20$ billion IoT devices will be globally deployed by $2020$ (Meulen, 2017).
Along with the benefits of them come potential privacy risks. By connecting
rich sensors (e.g., cameras, microphones, geophones) to the Internet, many IoT
devices can potentially _suggest or expose_ extensive information about their
users. Moreover, due to the mature manufacturing technologies, IoT devices can
be made very small and easily hidden in commonplace (team, 2018). According to
the Identity Theft Research Center, $\sim 4.5$ million sensitive records
containing personally identifiable information were exposed in 2017 alone,
representing a $126\%$ increase from the previous year (Burkhalter, 2019).
Users who have their identity information compromised may suffer from threats
ranging from persecution by governments (Marthews and Tucker, 2017), to
targeted frauds/advertising (Christin et al., 2010), to spoofing attacks
(Hadid et al., 2015).

Given the severe consequences of identity theft, prior efforts have been made
to investigate leak channels and effective countermeasures. These efforts can
be broadly categorized into two types, where one focuses on revealing the
identity of user via biometrics (e.g., face, voice, gait), while the other
type concerns the link between user identity and devices identities (IDs),
such as (MAC addresses of smartphones, web/browser cookies of laptops etc.).
For biometrics, Ilia et al. proposed to rethink the access control of face
photos to prevent users being recognized by unwanted individuals on social
media (Ilia et al., 2015). Meanwhile, privacy breach via device IDs also draws
substantial attention. For example, Yen et al. showed how several digital
identities, such as IP, cookies and usernames, can be combined to track users’
devices reliably on the web (Yen et al., 2012). Cunche et al. demonstrated
that the MAC addresses of Wi-Fi enabled devices can be abused to track device
owners’ locations (Cunche, 2014).

The above works unveil possible compromisations brought by either biometrics
or device IDs. However, few studies discuss the risk of information leakage
from a compound channel. Intuitively, given a hidden camera contributing
user’s facial biometrics, the device IDs (e.g., phone MAC addresses (Cunche,
2014)) captured by a _co-located_ WiFi sniffer can be utilized as _side
information_ to complete its eavesdropping view. Similarly, a co-located
hidden camera is also a side channel to the WiFi sniffer and can be
maliciously used to augment its knowledge base. Though the tactic to combine
side channels is straightforward, associating one’s biometrics and devices is
non-trivial under real-world constraints. The first challenge is the spatial-
temporal mismatch between biometric and device ID observations. While both of
the observations conceptually describe the ‘identity’ of a user, they are not
necessarily sensed at the same time. For instance, sniffing the MAC address of
a victim’s smartphone does not imply that the victim is speaking at the exact
same instant and vice versa. In addition, device IDs and biometrics are
usually sensed by different sensors with different spatial coverage. This
difference can further exacerbate the mismatch issue. For example, as radio
wave can penetrate obstacles, a WiFi sniffer is able to sniff devices behind a
shelf which occludes the cameras for face capture. Such spatial mismatch is
inevitable when the attacker seeks to eavesdrop, where the field covered by
_hidden/spy_ cameras or microphones becomes more constrained (JLaservideo,
2018). To further complicate the problem, the spatial-temporal mismatch cannot
be easily addressed by naive elimination, because eavesdropped data often
contains unexpected disturbances from the crowd which cause a substantial
amount of observations that are unimportant to the attacker. Concretely, in
our experiment, $1,147$ facial images from $35$ people outside the target
(victim) group were observed (see Sec. 3.1). Many of these non-targets were
temporal visitors to the eavesdropping space. Additionally, we also found
$109,768$ _distinct_ MAC addresses in our $238,833,555$ sniffed packets, with
only $27$ MAC addresses belonged to targets’ device IDs. As a result, a naive
elimination method is insufficient to handle these unexpected disturbances,
while a more sophisticated method is required to associate biometrics and
device IDs.

In this paper, we first validate the feasibility of identity breach across
biometrics and device IDs. We then investigate a cross-modal association
method that addresses the aformentioned challenges an attacker might face. Our
method draws the intuition that: even if sensed at different instants, long-
term eavesdropped biometrics and device IDs still form a shared context across
modalities. Given the shared context, an adversary is able to assign a set of
MAC addresses to biometric clusters based on their attendance consistency
throughout eavesdropped sessions. To deal with real-world complexity, we
present a method that allows an attacker to robustly associate cross-modal
identities under observation mismatches and substantial noises. Our
contributions are both conceptual and empirical:

  * •

We observe a new privacy issue of cross-modal identity leakage and formalize
an unprecedented threat in multi-modal sensing environments. Our work unveils
a compound identity leak from the combined side channels between human
biometrics and device identities.

  * •

We present a novel approach that robustly associates physical biometrics with
device IDs under substantial sensing noises and observation disturbances. If
maliciously used, an attacker can automatically profile the victims’
identities in multiple dimensions at the same time.

  * •

We conduct extensive evaluation on two real-world data collection of different
biometrics, showing that our presented attack is a real threat. In certain
cases, an attacker is able to correctly de-anonymize over $70\%$ device IDs
and harvest multiple biometric clusters of $\sim 94\%$ purity.

Our prototype and code are available at
https://github.com/zjzsliyang/CrossLeak.

##  2\. Background

###  2.1. WiFi MAC Address and Sniffing

WiFi MAC Addresses WiFi communication is on the $802.11$ radio between end
devices and access points. Similar to the IP address in network, every
$802.11$ radio has a $48$-bit MAC address in link layer which is globally
unique as the identifier of that device. The uniqueness of MAC addresses
across devices is guaranteed by the Institute of Electrical and Electronics
Engineers (IEEE) and the organizations such as smartphone manufacturer and
retailer.

WiFi Sniffing. The information exchange on a network following IEEE $802$
standards is through packets. A packet is a chunk of data consists of
‘header’, ‘body’ and ‘trailer’. Particularly, ‘header’ is where the sender
address and destination address are stored, which provides the most crucial
information for attackers who are interested in identification. The
communication between two devices on WiFi is by convention via packets but not
necessarily in an one-on-one form. Instead, the sender broadcasts the packet
on the airwaves where every device within range can ‘listen’ to it. The packet
is then checked by any receiving node, and compared with their own MAC
addresses. As a result, besides the desired device whose MAC address is
specified in the header, ‘header’ information is also exposed to the
surrounding ones which causes passive sniffing.

Figure 1. Attack Scenario. Through multi-modal eavesdropping (e.g., via a
hidden camera and/or microphone integrated with a WiFi sniffer), an attacker
stealthily collects biometric and device data in multiple sessions. The
attacker can then leverage our presented approach to automatically _identify_
targets’ (e.g., Alex and Bob in the figure) device IDs and _harvest_ their
biometric clusters. Subjects other than Alex and Bob are _out-of-set_ subjects
(i.e. non-targets).

###  2.2. Threat Model

The adversary model considered in this work is an attacker somehow
eavesdropped the device IDs and biometrics of victims for a period of time.
The eavesdropping tool can be a combination of a WiFi/Bluetooth/ZigBee sniffer
and hidden microphone/camera, or an integrated solution customized from
existing devices (e.g., WiFi spy microphone (JLaservideo, 2018)). These tools
are cheap and commercially available, allowing an easy setup for attackers.
Deploying such an eavesdropping tool is feasible, with threats coming from
both insiders and outsiders.

Insider Threat. A threat from insiders has the potential to compromise a group
of other co-workers’ identities (see Fig. 1). Owning to the privileged access
granted to an insider, deploying the eavesdropping tool is easy to disguise.
The attacker can place it as an office decoration without arousing suspicion.
Moreover, since such a setup does not generate packets, it will not be
detected even if the wireless traffics in the target workplace is regularly
inspected by a security guard. In fact, recent studies (team, 2018) found that
disguised eavesdropping on a particular type of biometrics or device IDs
already exists in domestic and commercial environments. Under this assumption,
the adversary can stealthily gather device identities (e.g., MAC addresses)
and physical biometrics (e.g., faces/voices). When sufficient information is
collected, the attacker can use our presented approach for bilateral ID
association. As visually or acoustically identifying a co-worker is easy for
the adversary, such association builds a bridge for the attacker to figure out
who owns which device and enables further attacks (e.g., online or physical
tracking (Korolova and Sharma, 2018; Castelluccia, 2012; Cunche, 2014)).

Outsider Threat. Even if long-term sensor deployment is inaccessible to
outsiders, an adversary can still eavesdrop enough information by carrying
disguised _mobile devices_ to public spaces. Here we provide a concrete
example with a WiFi sniffer and camera. As WiFi sniffing is available on many
laptops, we imagine that an attacker uses his laptop (probably equipped with a
back camera) as the eavesdropping system. The attacker then brings his laptop
and sits in the corner of a public place (e.g., a cafe or vehicles with WiFi
services). Using this setup, the adversary is able to continuously and
covertly capture high-profile or normal customers’ facial images and sniff
WiFi packets of surrounding user smartphones for days. Eventually, the
adversary can associate individuals’ facial images to personal device IDs with
our presented approach. The multi-dimensional IDs of the victims can make a
profit by selling them to third-party stakeholders, or can be used for
launching personalized advertising/fraud (Kshetri and Voas, 2019) and
biometric spoofing (Rodrigues et al., 2010) subsequently.

##  3\. Dataset and Feasibility

To understand how one can associate device IDs to biometrics, we designed an
eavesdropping prototype to collect multi-modal data in indoor environments. We
then study the feasibility of associating these two types of identities on the
collected datasets that provides insights for the presented approach.

(a) Prototype

(b) Testbed

Figure 2. Data collection. (2(a)): assembled eavesdropping prototype; (2(b)):
floor plan of the eavesdropping testbed; data were collected in its three
shared spaces - public office, common and meeting rooms.

###  3.1. Data Collection Methodology

Eavesdropping Prototype. As shown in Fig. 2(a), the assembled eavesdropping
prototype is built on a Raspberry Pi (RPi) 3 (rpi, 2018). To collect
facial/vocal samples and WiFi MAC addresses, the sensing modules consist of
(1) a Matrix-IO board (mat, 2019) to record audios in the far field, (2) an
RPi 8MP Camera Board (rpi, 2016) to capture videos and (3) an Ralink RT5572
WiFi Dongle (ral, 2016) to sniff MAC addresses. To reduce storage overheads,
the system records video/audio input to a circular buffer, while
simultaneously analyze the input to detect motion or voice activity on the
RPi. When motion or voice activity is detected, the buffer is saved to a file.
On the side of sniffing, RPi runs TShark in the background and continuously
captures packets transmitted on the airwaves. The RPi then extracts the source
MAC address from each packet and saves it along with the time of it being
captured and its corresponding received signal strength in a file. In order to
eavesdrop as many MAC addresses as possible, channel hopping is launched along
with sniffing such that live packets on all channels of different WiFi bands
can be captured. It is worth mentioning that as an alternative for integrated
eavesdropping, the information can also be collected from separate devices as
long as they are co-located.

Testbed and Ethics. As shown in Fig. 2(b), our testbed is a certain floor in a
commercial building that has $32$ regular residents. A subset of those
residents volunteered to participate as _victims_ in our experiment. We
deployed three eavesdropping tools in a _public office_ , a _common room_ and
a _meeting room_ , with areas of $60m^{2}$, $46m^{2}$ and $25m^{2}$
respectively. Our system unobtrusively collected _victims_ ’ data from their
daily activities, which is referred as _sessions_ (e.g., meetings, chats etc.)
in our context111The study has received ethical approval _R50950_.. Sessions
being accidentally recorded without full consents were deleted permanently.

Face-domain Dataset. This dataset consists of $123$ sessions and there are
$22$ victims and $35$ non-victims (e.g., temporary visitors) captured by the
camera. After face detection and image pre-procssing, the video clips give
$27,482$ facial samples. Over $2\times 10^{8}$ packets are sniffed during
eavesdropping, where approximately $110,000$ of them are distinct. Please
refer to Tab.1 for more detail.

Voice-domain Dataset. This dataset contains $49$ sessions contributed by $21$
victims and $9$ non-victims through different conversation scenarios (e.g.,
meetings). The collected audio recordings are segmented into $3,555$
utterances. In total, there are around $2\times 10^{6}$ packets sniffed during
data collection coming from about $3,500$ distinct MAC addresses. Please refer
to Tab.1 for more detail.

| Face Dataset | Voice Dataset
---|---|---
Sessions | 123 | 49
Victims (Volunteers) | 22 | 21
Non-Victims | 35 | 9
Biometric Samples | 27,482 | 3,555
Sniffed Packets | 238,833,555 | 1,985,862
Distinct Device IDs | 109,755 | 3,478
| Average Victims / Session
---
5.90 | 2.14
| Average Duration / Session
---
2 hr | 15 min
Table 1. Key Features of Our Dataset.

###  3.2. Feasibility Exploration

We are now in a position to explore the feasibility by inspecting the
eavesdropped data. For the ease of illustration, we use the term _session_
hereafter to broadly refer to settings in which victims interact with entities
in an environment for a given time interval, e.g., a physical visit or a
meeting in the meeting room from 10am to 11am. Therefore, a dataset comprises
a set of sessions, each of which is uniquely determined by the time-slot, the
location and the involved subjects.

(a) Face Cluster $\rightarrow$ Device ID: Left - User #3; Right - User#19

(b) Device ID $\rightarrow$ Face Cluster: Left - User #5; Right - User#9

Figure 3. Attendance map of $14$ sessions: dark squares indicate absence and
bright squares indicate presence. From the compatibility of attendance in
(3(a)), one can identify which device belongs to a target face cluster and
vice versa in (3(b)).

####  3.2.1. How Biometrics Aid the De-Anonymization of MAC addresses?

Take facial biometric as an example, we firstly perform clustering on the
eavesdropped facial images according to the similarity between their deep
features, similar to (Lu et al., 2019a; Xiangli et al., 2019). Note that
because of the intrinsic uncertainty, the derived clusters are impure, mixed
with samples of other victims or non-targets (to be further discussed in Sec.
6.2). Fig. 3(a) shows the attendance of $2$ randomly selected victims derived
from their facial biometric clusters observed in $14$ random sessions. Below
the map of face clusters, we show the top-$4$ device IDs with the most similar
attendance patterns. One can notice that, the session attendance map of the
‘true’ smartphone has a very similar pattern with the victim’s face cluster.
As a result, if the face cluster is visually identifiable by the attacker
(e.g., an insider threat), such similarity can be maliciously used to
deanonymize the MAC address of a target device.

####  3.2.2. How MAC Addresses Aid the Harvest of Biometrics?

Equally, an attacker can use the eavesdropped MAC addresses to improve the
efficacy of biometrics harvesting. It is inevitable that eavesdropped
biometrics in real-world include erroneous samples coming from non-victims or
non-targets (e.g., short term visitors to a compromised office as demonstrated
in Fig. 1). These samples can not only contaminate the victims’ biometric
clusters, but result in irrelevant clusters that mislead the attacker. Even
worse, if the eavesdropping is conducted in a public space (e.g., cafe in a
commercial building) by an outsider threat, the amount of irrelevant clusters
can be non-negligible. For example, in our face-domain dataset, there are
$1,147$ facial images belongs to $35$ non-victims. Despite these factors, as
shown in Fig. 3, the face clusters’ attendance maps of non-targets are much
different from the target devices (i.e., cluster $\\#5$ and $\\#9$ in Fig.
3(b)). The attacker can therefore focus on the successfully associated
cluster-to-MAC address pairs and ignore the unassociated ones of non-targets.

####  3.2.3. Uniqueness Analysis

Knowing that there exists session attendance similarity between one’s physical
biometrics and his/her personal device, a natural question to ask is, does
every user have a _discriminative attendance pattern_ such that the target can
be isolated? To answer this, we investigate the possibility of distinguishing
a group of victims by selecting only a small subset of sessions in their
attendance history. Two different sampling strategies are used to obtain the
subset: (1) randomly selecting $G$ attended sessions of the victim (Rand-G)
and (2) randomly selecting $G$ _continuous_ sessions attended by the victim
(Cont-G). Under these strategies, we calculate the percentage of users that
can be distinguished from others. Fig. 4 shows that on the face-domain
dataset, $18$ out of the $22$ victims can be distinguished from the randomly
selected $25$ sessions (Rand-$25$ strategy); when the number of sessions
reaches $50$, all victims can be perfectly distinguished. Similar behaviour
can be observed on face-domain dataset when adopting Cont-G strategy. As for
the voice-domain dataset which has $49$ sessions, $19$ out of the $21$ victims
can be identified via Rand-$20$ and Cont-$20$ strategies; when all sessions
are used, all victims are perfectly separated.

_Takeaway:_ The above results indicate that the attendance pattern of a victim
can be significantly different from others. Such uniqueness probably is due to
the distinct living/working habits of individuals. Based on the above
analysis, we conclude that cross-modal identity breach is feasible and is a
real threat.

(a) Face-domain Dataset

(b) Voice-domain Dataset

Figure 4. Quantified Feasibility: percentage of distinguishable users based on
different session sampling strategies.

##  4\. Overview and Formulation

###  4.1. Attack Overview

We now present an attack vector that leverages the above intuition to
associate victims’ device IDs and physical biometrics. Our approach features
its robustness to eavesdropping disturbances, requiring minimal prior
knowledge from an adversary to launch the attack. As shown in Fig. 5, our
framework consists of three modules that operates in a pipelined manner: (1)
Multi-modal eavesdropping (2) Device filtering and (3) Cross-modal ID
association.

The _multi-modal eavesdropping_ module described in Sec. 3.1 is used to gather
data. An attacker then feeds the sniffed device IDs to the _device filtering_
module so that a majority of devices unimportant to the attacker can be
removed. Given the filtered device IDs and biometric collection, the attacker
can associate them by tasking the _cross-modal association_ module. In what
follows, we will first formulate the generic association problem after
eavesdropping and then introduce the filtering and association modules in Sec.
5 and Sec. 6 respectively.

###  4.2. Problem Formulation

Definition. Without loss of generality, suppose the adversary aims to
compromise the identities of $P$ victims in a target environment. The
adversary can either be an insider that attacks co-workers or an outsider who
collects data of people in public spaces. We assume the scenario where the
attacker managed to eavesdrop identities in $G$ sessions,
$\mathcal{S}=\\{s_{j}|j=1,2,\ldots,G\\}$. The term _session_ is introduced in
Sec. 3.2 that broadly refers to settings in which victims interact with
entities in an environment for a certain amount of time. The set of
eavesdropped device IDs are denoted as
$\mathcal{Y}=\\{y_{j}|j=1,2,\ldots,H\\}$. A set of biometric samples are also
captured, denoted as $\mathcal{X}=\\{x_{j}|j=1,2,\ldots,W\\}$.

Modeling Real-world Disturbances. In real-world scenarios, it is possible that
eavesdropping will experience unexpected disturbance from the out-of-set (OOS)
observations, including device IDs and biometrics outside the target group.
For instance, a subject visiting the vulnerable workspace in a short term
without connecting to WiFi will only have their facial images captured.
Similarly, since WiFi signals are able to penetrate walls, the sniffed MAC
addresses are likely to include devices from a long distance, while the owner
has never appeared in the camera. Without imposing strong assumptions, we
consider the case in the wild where OOS observations occur in both
eavesdropping $\mathcal{X}$ and $\mathcal{Y}$. Aside from the OOS
disturbances, a large portion of the sniffed MAC addresses are dummies or
noises, owning to the MAC address randomization mechanism (Martin et al.,
2017) and nuisance packets generated by other WiFi devices, e.g., access
points, workstations etc. Our problem setup considers that these observation
noises are included in $\mathcal{Y}$.

Attack Goal. _Given the very noisy eavesdropped data of $\mathcal{X}$ and
$\mathcal{Y}$, discover the device IDs and biometric feature clusters
belonging to $P$ target victims, and then correctly associate these cross-
modal identities_. For readability, we describe our presented approach in the
context using MAC addresses as the device IDs with facial/vocal samples as
biometrics.

Figure 5. Attack Overview. Our attack framework consists of three modules that
operate in a pipelined manner: (1) Multi-modal eavesdropping (2) Device
filtering and (3) Cross-modal ID association.

##  5\. Device Filtering

As the sniffed MAC addresses (i.e., device IDs) contain instances outside the
target environment, one needs to filter the eavesdropped device IDs in
$\mathcal{Y}$ and derive a smaller and cleaner set of MAC addresses
$\mathcal{L}=\\{l_{j}|j=1,2,\ldots,M\\}$. Our proposed filter draws
characteristics from (1) MAC address format and (2) wireless signal strength.
It should be noted that while we describe the following filtering approach
with the case of WiFi MAC addresses, the similar concept can generalize to
device IDs attained by other means, e.g., Bluetooth sniffing (Albazrqaoe et
al., 2016; Loveless, 2017).

###  5.1. MAC Address Format Filter

The goal of this sub-module is to remove device IDs that do not comply with
end devices’ (e.g., smartphones) formats by inspecting the information hidden
in MAC addresses. Particularly, we focus on two dominant types of disturbance:
(1) Randomized MAC addresses and (2) MAC addresses of WiFi access points.

####  5.1.1. MAC Randomization

To prevent WiFi tracking, both Android and Apple iOS operating systems allow
devices in a _disassociated state_ to use random, locally assigned MAC
addresses in active WiFi scanning. These addresses are dummy bytes pointing to
no real devices. The MAC addresses remain random until the device is
associated with an access point (AP). However, abiding by the 802.11 protocol
of IEEE, one very specific bit of the MAC address is the seventh bit of its
first byte: the Locally Administered (LA) bit. If a LA bit set to $1$, the MAC
address is implied as randomized/changed by the Administrator of the device
(WifiNigel, 2018). Therefore, the first step of our filter utilizes this LA
bit and inspects each sniffed packet. Only those MAC addresses identified as
non-randomization are passed to next-step analysis. As we can see in Tab. 2,
randomized MAC addresses account for about $65\%$ and $20\%$ in the distinct
sniffed device IDs on our face and voice data collection.

####  5.1.2. Vendor Information

Aside from the information about randomization, one can also find the vendor
information from a MAC address. To guarantee the uniqueness of MAC addresses
across devices, the IEEE assigns blocks of addresses to organizations in
exchange for a fee. An Organizationally Unique Identifier (OUI), also the
first three bytes of a non-randomized MAC address, is such an information
snippet that may be purchased and registered with the IEEE (OUI, 2017). By
looking up the public database of OUI, we first find the manufacturers of all
MAC addresses. As the attacker targets the mobile devices of victims, the
filter removes those addresses whose manufacturers are corresponding to WiFi
access points and switches (e.g., TP-Link, Cisco, 3Com, Juniper, LinkSys,
D-Link, NetGear). Tab. 2 lists that $91$ and $96$ MAC addresses in our face
and voice data collection are such kind of MAC addresses.

Filter Module | Face Dataset | Voice Dataset
---|---|---
Number | Percentage | Number | Percentage
Random | 71,720 | 65% | 693 | 20%
Vendor | 91 | 0.08% | 96 | 3%
RSS | 108,639 | 99% | 3,378 | 97%
Table 2. Statistics of MAC addresses removed by different filter modules.
NOTE: one nuisance MAC address is possibly detected by multiple modules,
leading to over $100\%$ total percentage in the table.

###  5.2. Received Signal Strength (RSS) Filter

The second sub-module of our filter aims to filter out distant devices outside
the effective eavesdropping range. Here, we turn to the WiFi Received Signal
Strength (RSS) information in sniffed packets. Following the signal
propagation laws of wireless radios, RSS reflects _how far_ the sniffed
devices are from the WiFi sniffer. This spatial information is pivotal to
determining the collocation with biometric samples. Although WiFi sniffing can
capture packets in distance ($\sim 40$m), biometric sensing often works only
in a short range (e.g., $7$m for microphones). In our context, MAC addresses
of interest to attackers are those can be associated with co-located biometric
observations; our filter thus discards those MAC addresses with low RSS
values, e.g., less than $-45$ dBm on voice-domain dataset and less than $-55$
on face-domain dataset. This helps filter out $108,639$ and $3,378$
unimportant MAC addresses in our two datasets (see Tab. 2). The threshold is
derived empirically by following the geo-fence concept introduced in (Lu et
al., 2016). In Sec. 7.3.1, we will discuss the impacts of different RSS
thresholds on the overall performance.

_Summary:_ Applying all above filtering sub-modules substantially narrows down
the association range of device IDs. In our case, we found that the MAC
addresses of interest (i.e., $\mathcal{L}$) is reduced from $109,768$ to $174$
on face datasets and from $3,478$ to $54$ on the voice dataset. This reduction
makes the follow-up cross-modal association tractable and efficient.

##  6\. Cross-modal ID Association

Given the filtered MAC addresses $\mathcal{L}$ and biometric samples
$\mathcal{X}$, the next step is to discover the MAC addresses of victims and
associate them across two identity spaces. Our association approach consists
of three sequential processes: (1) biometric feature extraction (2) linkage
tree construction and (3) node selection. At a high level, the algorithms
works by first extracting deep features from biometric samples $\mathcal{X}$
and constructing a linkage tree based on their feature similarity. Each node
in the tree is a biometric cluster. Then by leveraging the advocated concept
of session attendance pattern (see Sec. 3.2), victims’ MAC addresses can be
discovered in $\mathcal{L}$ and assigned to the corresponding biometric
clusters. Algo. 1 illustrates the association steps.

Algorithm 1 Cross-modality association.

Input:  pre-trained feature model $f_{\theta}$, biometric samples
$\mathcal{X}$, device IDs $\mathcal{Y}$and estimated number of victims $K$.
Output:  decision variable $A=(a_{i,j})_{N*M}$ that victims’ device ID $j$
with corresponding biometric cluster node $i$ are selected.

1:$\mathcal{L}\leftarrow\texttt{filtering\\_device}(\mathcal{Y})$
$\triangleright$ Sec. 5

2:$\mathbf{Z}\leftarrow f_{\theta}(\mathcal{X})$ $\triangleright$ Feature
Extraction in Sec. 6.1

3:$T\leftarrow\texttt{linkage\\_tree\\_construction}(\mathbf{Z})$
$\triangleright$ Sec. 6.3

4:$r_{l}\leftarrow\texttt{device\\_context\\_vector}(\mathcal{L})$

5:$r_{t}\leftarrow\texttt{treenode\\_context\\_vector}(T)$

6:$A\leftarrow\texttt{node\\_selection}(T,r_{t},r_{l},K)$ $\triangleright$
Sec. 6.4

###  6.1. Biometric Feature Extraction

The key to merging biometric samples lies in how to measure their similarity.
For biometric samples such as facial images or vocal segments, accurately
quantifying their similarity is challenging owning to multiple factors. For
instance, face images can differ a lot due to lighting conditions (Li et al.,
2007); voice quality may vary across HiFi microphones to low-cost microphones.
Thanks to the recent advances in DNNs, it is proven that they are able to
handle variable observation conditions through feature learning from massive
amounts of training samples. To utilize this advantage, biometric samples are
first encoded into a feature embedding $\mathbf{Z}$ via a deep neural network
$f_{\theta}$ designed for biometric recognition (e.g., facenet (Schroff et
al., 2015) for face recognition and x-vector (Snyder et al., 2018) for speaker
recognition). Such $f_{\theta}$ can be pre-trained on public datasets and
learns an effective feature representation. The similarity between biometric
samples are based on the extracted features with the pre-trained model.

###  6.2. A Naive Method and Its Limitation.

Given the similarity between biometric samples and filtered device IDs, a
naive approach originated from the feasibility analysis (see Sec. 3.2) is to
leverage the diverse participatory information in multiple sessions and adopt
a two-step association procedure: a) in the Clustering Step, biometric
observations $\mathcal{X}$ are firstly grouped into clusters across all
sessions, each of which potentially contains the biometric samples of a single
victim; and then b) in the Data Association Step, the clusters are assigned
with device IDs based on the similarity between their attendance patterns
across eavesdropped sessions. Although this approach is simple and easy to
implement, it is not robust against OOS observations. For example, a session
may contain faces or voices of temporal visitors while their device ID is not
captured. Due to the disturbance incurred by OSS subjects, the number of
clusters is intractable as a priori for the attacker. A misleading clustering
result, however, could further degrade the quality of data association.

###  6.3. Linkage Tree Construction

To address the limitation of the naive method, we use a linkage tree to merge
similar biometric samples in a hierarchy manner. Each node in the tree is a
biometric cluster candidate, potentially belonging to a single victim.

Tree Structure. Based on the extracted features $\mathbf{Z}$, our presented
algorithm compiles biometric samples into a linkage tree $T$. The leaf nodes
$T_{leaf}$ are samples, while a branch node represents the cluster of all its
descendant leaf nodes. Essentially $T$ represents the hierarchical clustering
of all biometric observations in different sessions. Selecting a combination
of nodes from the tree gives a specific clustering plan. Each node $t_{i}$ in
$T$ is associated with a _linkage score_ $q_{link_{i}}$, describing the
average feature similarity or compatibility between the data within the
cluster it represents.

Context Vector. A basic linkage tree can be developed as the above. However,
due to the presence of domain differences, the features learned via a pre-
trained DNN can deviate and mislead the node merging process. For example, the
learned feature representations on public datasets of front faces are not
effective for the side profile of a face. When ineffective representation
features are used, two face images of different subjects can be falsely merged
into the same node which may cause unrecoverable knock-on effects on the
ensuing association step. In order to combat this problem, we augment the
linkage tree by introducing context vectors to augment the knowledge of each
node. A context vector of the MAC address is essentially a binary vector
describing the session attendance. The vector’s length is equal to the total
number of eavesdropped sessions. Concretely, let
$\mathbf{r}_{t_{i}}=(r_{t_{i}}^{1},r_{t_{i}}^{2},\ldots,r_{t_{i}}^{G})$ be the
context vector of a tree node $t_{i}$, where $G$ is the total number of the
eavesdropped sessions. $r_{t_{i}}^{j}$ is set to $1$ if node $t_{i}$ contains
biometric samples eavesdropped from session $s_{j}$. Similarly, a MAC address
$l_{k}$ is also linked with a context vector $\mathbf{r}_{l_{j}}$, and
$\mathbf{r}_{l_{j}}^{j}$ is set to 1 only if $l_{j}$ is detected in session
$s_{j}$. Intuitively, for a node $t_{i}$ and a MAC address $l_{j}$, if
$\mathbf{r}_{t_{i}}$ and $\mathbf{r}_{l_{j}}$ are similar enough, it is very
likely that the grouped samples under node $t_{i}$ are actually the biometric
data of the subject who owns that MAC address $l_{j}$, as their presence
patterns in sessions agree with each other the most.

Node Scores and Dice Coefficient. Given the context vectors, we can
additionally introduce a data association score to each of its node $t_{i}$,
which represents the fitness of assigning an identity label to $t_{i}$ given
the filtered MAC addresses $\mathcal{L}$. For a biometric node $t_{i}$, we
define its data association scores with respect to $M$ filtered MAC addresses
as a vector
$\mathbf{q}_{assoc_{i}}=(q_{assoc_{i}}^{1},q_{assoc_{i}}^{2},\ldots,q_{assoc_{i}}^{M})$,
where the $j$-th score $q_{assoc_{i}}^{j}$ is the Dice coefficient (Duarte et
al., 1999) between the node context vector $\mathbf{r}_{t_{i}}$ and the device
context vector $\mathbf{r}_{l_{j}}$. Intuitively, when matching with device
IDs, the absence information is not as informative as the presence
information, especially for _users present rarely_. Dice coefficient is known
to be suitable in handling such cases which favors the presence information
(i.e., ‘1’) when comparing two binary vectors. We will further explain this in
Sec. 7.3.3. Together with the linkage score, the final score assigned to the
cluster node $t_{i}$ to a MAC address $l_{j}$ is a composite score function:

(1) $\small q_{i}^{j}=(1-\omega)*q_{link_{i}}+\omega*q_{assoc_{i}}^{j}$

where the parameter $\omega$ governs how much the adversary trusts the learned
biometric features and to what extent the adversary wants them to impact the
result of cross-modal association.

###  6.4. Node Selection

Optimization Program. Once node scores are all assigned, the attack of cross-
modal ID association is equivalent to selecting the top $K$ nodes from the
tree with a maximum sum. With the previously introduced terms and notations,
we formulate this node selection problem as follows:

(2)
$\small\max_{\mathbf{A}}\quad\sum^{N}\limits_{i=1}\sum^{M}\limits_{j=1}q_{i}^{j}*a_{i,j}\\\
$

subject to $\sum^{M}\limits_{j=1}a_{i,j}\leq 1,\forall i\in{1,\ldots,N}$ (3),
$\sum^{N}\limits_{i=1}a_{i,j}\leq 1,\forall j\in{1,\ldots,M}$ (4),
$\sum^{N}\limits_{i=1}\sum^{M}\limits_{j=1}a_{i,j}=K$ (5),
$\sum\limits_{i\in\Pi_{k}}\sum^{M}\limits_{j=1}a_{i,j}\leq 1,\forall k\in
T_{leaf}$ (6), $a_{i,j}\in\\{0,1\\},\forall i\in\\{1,\ldots,N\\},\forall
j\in\\{1,\ldots,M\\}$ (7).

where $\mathbf{A}=(a_{i,j})_{N\times M}$ is the decision variable and
$q_{i}^{j}$ is the composite score determined by Eq. 1. $T_{leaf}$ represents
the set of all leaf nodes in the linkage tree. The objective function aims to
maximize the total scores when selecting $K$ nodes in the linkage tree $T$
with size of $N$. Intuitively, the selected $K$ nodes are the optimal clusters
out of these $N$ biometric observations. The inequalities in Eq. 2 simply mean
that a node can be assigned to at most one device ID. Similarly, the
constraints in Eq. 2 are used to ensure each device ID is associated with a
single node. Eq. 2 decides how many nodes the program should select, i.e., the
number of victims. As a clustering tree, a node cannot be selected with its
ancestors or descendants at the same time since they contain duplicate data.
In order to compile this tree structure in optimization, the constraint Eq. 2
is enforced to guarantee that on any path leading to a leaf node, at most one
node is assigned to a device ID. Finally, the constraint Eq. 2 is there to
make sure that decision variable $a_{i,j}$ can take on the integer value $0$
and $1$ only. The above optimization formulation is essentially an integer
linear programming (ILP) problem and can be solved with off-the-shelf tools
(Karmarkar, 1984).

Choice of K. To run the above optimization program, a key step is to estimate
a rough number of subjects $K$ in the target environment. Such estimation is
made based on the adversary’s observation as either an insider or an outsider.
However, due to the disturbance from the unknown number of OOS subjects, $K$
is often inaccurate in practice but treated as an approximation of the true
number of victims. Despite this, because the above association approach adopts
a node selection strategy, it is able to _pick out_ and associate as many
correct nodes as possible with either under- or over-estimated $K$, as
illustrated in Fig. 6.

Figure 6. An illustrative example with $3$ victims and $1$ OOS subject. Our
presented method inherits the concept of ‘early stop’ and tolerates the noisy
biometric samples from OOS subjects before their contamination in merging.

##  7\. Evaluation

In this section, we conduct experiments on both real-world (refer to Sec. 3.1)
and simulation datasets to quantitatively verify the performance of the
presented attack.

###  7.1. Baseline and Performance Measure

####  7.1.1. Baseline Methods

Two variants of the proposed association approach are considered: (1) Ours
(Euc.) uses an Euclidean distance metric rather than the proposed Dice
coefficient to calculate a node score $\mathbf{q}_{assoc_{i}}$ (see Sec. 6.3);
(2) Naive is the the intuitive strategy introduced in Sec. 6.2 that performs
clustering and association step sequentially in contrast to our simultaneous
approach.

####  7.1.2. Performance Measure - Association Accuracy

From the perspective of _device ID compromise_ , to represent how well one can
associate physical biometrics with device IDs, we calculate the portion of
correctly associated pairs of biometric clusters and device IDs among all
victims, denoted as the association accuracy. The true identity of a device ID
is retrieved from a pre-registered table provided by our volunteered
‘victims’, which maps device MAC addresses to their owners. On the other side,
the identity of an individual biometric sample is manually labeled via
inspecting. Considering that biometric clusters might be noisy, the true label
of a biometric cluster is determined by majority voting. For the ease of
readability, we refer to association accuracy as _accuracy_ hereafter.

####  7.1.3. Performance Measure - Cluster Purity

From the perspective of a _biometric compromise_ , we introduce an additional
metric purity that concerns sample-level accuracy. Given a pair of correctly
associated biometric cluster and MAC address, cluster purity describes the
ratio of samples that actually belong to the corresponding victim. This metric
stems from the real-world application where pure biometric clusters are more
valuable to an attacker for quick identity determination (e.g., majority
voting) and launching subsequent attacks, such as biometric spoofing or
impersonation attacks. For the ease of readability, we refer cluster purity as
_purity_ hereafter.

(a) Face-domain.

(b) Voice-domain.

Figure 7. Overall Attack Performance.

###  7.2. Attack Performance

We quantify the cross-modal ID leakage on RealWorld datasets. Recall in
Sec.6.4, an adversary needs to first make an estimation on the number of
potential victims $K$ in the target environment. Such $K$ is generally
inaccurate due to OOS disturbances yet deeply interwoven in almost all real-
world attacks. Obviously, the attack algorithm needs to be robust against
different levels of noises. We therefore report _accuracy_ and _purity_ under
different choices of $K$ to provide a comprehensive view on attack
performance.

Setup. As a reference, we examine a set of $K$s ranging from 50% to 150% of
the true number of victims $P$ on the RealWorld datasets in both face- and
voice-domain. In this experiment, the regularization parameter $\omega$ is set
to $0.5$ (refer to Eq. 1) and the RSS threshold is set to -55 dBm and -45 dBm
for face- and voice-domain respectively. All eavesdropped sessions of our
real-world datasets (see Sec. 3.1) are used in this experiment.

Overall Performance. Fig.7 demonstrates the performance of the presented
mechanism and two baselines as $K$ varies. In all cases, a clear winning
margin of our method against baselines is observed, with a relatively wide
region of $K$ achieving comparable performance. Our approach achieves 72.7%
and 71.4% _accuracy_ , and 93.9% and 76.2% _purity_ on face and voice domain
datasets respectively.

Dataset Comparison. It can be seen that the advantage is much more prominent
on face-domain. This is due to the fact that faces captured by cameras in the
wild are much noisier than utterances captured by recorders as described in
Sec.3. Consequently, the quality of face clusters is inferior to that of voice
clusters.

Comparison with Ours (Euc.). Observing our method (Ours) and its Euclidean
variant Ours (Euc.), it can be observed that Dice coefficient yields a
significantly higher accuracy than Euclidean distance, especially on face-
domain dataset. This is because, compared to Dice, Euclidean distance evenly
punishes unmatched and matched instances when searching for similar context
vectors of biometric clusters and MAC addresses. However, in our scenario,
Dice is more desirable as the matched pairs should be of higher importance
over unmatched ones to combat attendance sparsity and observation noise (see
Sec. 6.3). Consequently, Euclidean distance usually leads to purer clusters if
correctly associated, but it is also very error-prone when pairing clusters to
MAC addresses. Inspecting purity, our method with Dice coefficient is able
achieve comparable results with Ours (Euc.) when $K$ is reasonably set.

Comparison with Naive. Compared to the Naive approach, the advantage of using
simultaneous clustering and association (Ours) is also obvious, especially for
purity. This can be attributed to the nature of clustering. Commonly, a
clustering algorithm requires one to pre-define the number of
clusters222Clustering algorithms without the requirement of number of clusters
often require other hyper-parameter specification, e.g., the neighborhood size
for DBSCAN (Schubert et al., 2017)., then separates the dataset accordingly.
However, under the scenario of attack, such behavior makes association rely on
the guessed number of clusters (i.e., the $K$). However, noises will be
scattered inside each cluster and this effect is particularly significant in
face-domain. Despite this, our method alleviates the impact of uncertain $K$
and gives best accuracy since it only takes effect after identity association.
The purity of the Naive approach is compromised due to the same reason.

###  7.3. Impact Factors

We now continue our evaluation on the effects of different impact factors,
including (1) WiFi RSS Threshold, (2) Collection Span, (3) regularization
parameter $\omega$ and (4) Number of OOS subjects. Notably, for the ease of
scaling the number of OOS subjects, the fourth experiment is conducted on two
large-scale simulation datasets. Hereafter, unless otherwise stated, all
experiments in (1-3) are conducted on 123 sessions in face-domain and 49
sessions in voice-domain. $K$ is set to $1.25$ times of the true number of
victims $P$ (see Sec. 6.4) and the regularization parameter $\omega$ is equal
to $0.5$ (refer to Sec. 6.3). The RSS threshold is set to -$55$ dBm and -$45$
dBm for face- and voice-domain respectively.

####  7.3.1. Impact of WiFi RSS Threshold

(a) Face-domain.

(b) Voice-domain.

Figure 8. Impact of WiFi RSS threshold on RealWorld datasets.

As described in Sec.5.2, a critical factor of filtering nuisance MAC addresses
is the spatial distance. Leveraging the RSS information contained in sniffed
packets, an adversary can determine whether a device is presented in the
target environment by comparing with a pre-defined ‘fence’. Normally, devices
closer to the sniffer have stronger RSS. However, without accurate
calibration, an adversary can only empirically guess a threshold value based
on this fact which results in ambiguity.

As we can see in Fig.8, the influence of varying RSS threshold is global on
all three methods. It shows that for face-domain, a rigorous threshold harms
association result with a decrease ranging from 22.2% to 56.2% in accuracy. A
drastic drop happens in purity when the threshold goes beyond -50 dBm where
Ours decreases to 30% and Ours (Euc.) fails at -40 dBm. In voice-domain,
performance degrades when the threshold is relaxed for all three approaches,
where accuracy shows 26.7%, 53.3% and 50.0% decrements in Ours, Ours (Euc.)
and Naive respectively. Similar degradation happens to purity. On the other
side, the presented approach maintains its advantage in both accuracy and
purity. The discrepancy between face- and voice-domain is mainly due to the
difference of locations where faces were captured by the hidden camera
deployed in a relative large public office ($60m^{2}$) while dialogues were
mostly recorded in a small meeting room ($25m^{2}$).

####  7.3.2. Impact of Collection Span

Figure 9. Impact of collection span on face-domain.

Figure 10. Impact of omega on RealWorld datasets.

Another key parameter to consider when executing the attack is the number of
eavesdropped sessions, referred to as collection span. Recall that in
Sec.3.2.3, we discussed the prerequisite for victims to be distinguishable.
Naturally, more sessions lead to more diverse attendance patterns, indicating
that the probability of having inseparable victims is lower. However, it also
increases the amount of OSS observations in both biometric samples and device
IDs. In this experiment, we study the influence of collection span by
gradually decreasing the number of eavesdropped sessions used in association,
mimicking the variation from long-term to short-term monitoring. This
experiment is not practical on voice-domain dataset since the number of
eavesdropped sessions is limited ($49$ sessions) and further decrements in
collection span leads nowhere.

Fig.9 shows that on the face-domain dataset, the presented approach (Ours) can
achieve reasonable accuracy and purity without using all sessions. This
observation coincides with our feasibility analysis in Sec. 3.2.3 that
attendance uniqueness is widely available even among co-workers. Note that
Dice similarity (Ours) achieves similar purity level with Euclidean distance
(Ours (Euc.)) while maintains its advantage in accuracy. Ours (Euc.) manages
to maintain its high purity but the accuracy is corrupted due to its non-
discriminative constraint on matched and unmatched attendance. Last but not
least, there is a drastic drop in Naive approach’s purity when the number of
sessions increases from $80$ to $100$, caused by the increment amount of noise
added to clusters.

####  7.3.3. Impact of $\omega$

Recall that in Sec.6.3, we introduced a regularization parameter $\omega$ to
balance the importance of biometric feature similarity and session attendance
similarity. In this experiment, we examine the impact of varying this
parameter on the presented approach (Ours). Due to the intrinsic difference
between biometric features, $\omega$ takes effect at different scales (i.e.,
$\times 20^{-1}$ for voice domain against face domain). As shown in Fig.10,
our method is robust to this parameter while a slight lift of $3.8\%$ in
purity can be observed in face-domain when we put more trust on the context
vector. Such behavior proves that, even though the features extracted via the
pre-trained DNNs are already trustworthy, there is still a gap caused by
domain difference. In our approach, context vector can bootstrap association
by providing more insights from the eavesdropped sessions. As a comparison,
since recorded vocal segments are much less noisy in voice-domain, biometric
clusters become more trustworthy.

####  7.3.4. Impact of Number of OOS Subjects

(a) Face-domain.

(b) Voice-domain.

Figure 11. Impact of OOS subjects on Simulation datasets.

In this experiment, we examine the robustness of the demonstrated approach
where the number of OOS subjects increases, corresponding to different levels
of disturbance in biometric observations. To this end, we composed a
Simulation dataset for each scenario to flexibly adjust this parameter. The
Simulation dataset was synthesized from publicly available datasets, namely
VGGFace2 (Cao et al., 2018) for face-domain and VoxCeleb2 (Chung et al., 2018)
for voice-domain. Both simulation datasets contain $100$ sessions with $50$
victims and the number of OOS subjects varies from $20$ to $80$ with a step
size of $20$. By randomly allocating biometric samples and pseudo device IDs,
the datasets contain $26,901$ facial images and $10,100$ voice segments.

As we can see in Fig.11, Ours still outperforms baselines and yields strong
robustness against different levels of OOS disturbance. Respectively, in face-
and voice-domain, the standard deviations of accuracy are $0.02$ and $0.01$,
while the standard deviations of purity are less than $0.005$ and $0.02$. This
result implies that an attacker can leverage our approach to robustly
compromise identities in public area, such as public shops with many OOS
subjects.

##  8\. Related Work

Linkage Attack. The linkage attack is a class of attacks that breaks
k-anonymity and reidentifies users (Sweeney, 2002). In such an attack, the
adversary collects auxiliary information about a specific individual from
multiple data sources and then combine that data to form a whole picture about
their target, which is often an individual’s personally identifiable
information. Linkage attacks have a long history and can be dated back to the
famous de-anonymization of a Massachusetts hospital discharge database by
joining it with a public voter database (Sweeney, 1997). Recent studies
demonstrated that linkage attacks are able to infer privacy across different
scenarios, including movie ratings forums (Narayanan and Shmatikov, 2008;
Frankowski et al., 2006), crowdsourced sensor/mobility data (Lane et al.,
2012; Jia et al., 2017; Xu et al., 2017) and social networks (Zheleva and
Getoor, 2009; Mislove et al., 2010). Clearly, our work draws a similar flavor
with the linkage attack on utilizing entity uniqueness. But in contrast, our
work takes one step forward and formulate the attack problem in the context of
IoT-rich environments. To our best knowledge, our work is the first one that
systemically studies the vulnerable linkage between biometrics and device IDs.

Side Channel Attacks. The side channel attack is proven to be widely powerful
in practice. By measuring side channel information, the attacker is able to
recover other information that is often sensitive (Joy Persial et al., 2011).
There mainly two types of side channel attacks. First, it is the side channel
attack based on a single field. Such observed side channel attacks in
literature include timing attacks (Dhem et al., 1998), power analysis attacks
(Amiel et al., 2007), electromagnetic analysis attacks (Réal et al., 2009),
acoustic attacks (Backes et al., 2010) and traffic analysis attacks (Kadloor
et al., 2010). The second type is known as a combined side channels attack,
which could improve the accuracy of existing attacks or cope with more
sophisticated attack scenarios. Some research recently started to explore the
attacks from combined side channels (Spreitzer et al., 2017; Lipp et al.,
2016; Lin et al., 2014; Fan and Gierlichs, 2018). However, these work only
focus on a single domain, which has limitations in employing the information
from both sides. In contrast, our system is the first to explore the privacy
issue of identity leakage between physical (e.g., user biometrics) and digital
domains (e.g., smart devices ID).

Cross-modality Association. Technically, our presented cross-modal association
method is related to data association methods. Given a track of sensor
readings, data association aims to figure out inter-frame correspondences
between them. Data association is widely used in radar systems, when tracking
blips on a radar screen (Blackman, 1986), as well as in object monitoring of
surveillance systems (Hu et al., 2004). When it comes to the cross-modal
association, research attention is limited and all dedicated to location
tracking of humans (Teng et al., 2014; Alahi et al., 2015; Papaioannou et al.,
2015). These methods heavily rely on the hypothesis that both sensor
modalities are observing evolving state spaces matched precisely in the
temporal domain. However, this hypothesis is invalid in our case, since
eavesdropping a MAC address does not imply that someone will be speaking at
that exact instant. On the other side, association methods are also proposed
for autonomous human identification (Lu et al., 2019a; Lu et al., 2017; Lu et
al., 2019b). However, owing to the purpose of learning, these methods require
a _known mapping_ between users and their MAC addresses which are _unknowns_
to de-anonymize in our context.

##  9\. Discussion

###  9.1. Potential Mitigation Techniques

Since the presented attack is cross-modal, we discuss possible mitigation from
both sides of device IDs and biometrics.

####  9.1.1. Mitigation from the side of Device ID

In this work, the eavesdropped device IDs (MAC addresses) are obtained from
the WiFi sniffing module. Perhaps the most intuitive mitigation against is MAC
addresses randomization, a perturbation approach that regularly changes a
device’s MAC address in _probe request frames_ and hence adds more noise to
the sniffed data. However, as pointed out in (Martin et al., 2017; Vanhoef et
al., 2016), random MAC addresses, on its own, does not guarantee ID privacy.
As randomization is not imposed on the sniffed _association frames_ and
_transmission frames_ , an attacker is still able to capture the true/global
MAC addresses of devices. Our analysis on random MAC addresses filter in Sec.
5 also validates this observation. Applying randomization to association and
transmission frames is difficult as their response mechanisms are managed
within the 802.11 chipset, instead of the operating system, implying the only
way is to develop a firmware patch that has to be distributed by manufactures.
Moreover, even without WiFi sniffing, an attacker may also attain device ID by
sniffing on connectivity without using any randomization, e.g., ZigBee (Ltd,
2018). Injecting noise to the context vectors by turning off connectivity
every once in a while seems a plausible mitigation as well. However,
considering the session duration in Tab. 1, an effective noise injection would
either cause network interruptions if configured by device manufacturer or
require constant user effort when configured manually.

####  9.1.2. Mitigation from the side of Biometrics

Mitigation solutions on the side of biometrics are boiled down to the long-
standing issues of spy cameras and microphones. With the maturity of
manufacturing technology and advent in energy harvesting, these hidden devices
nowadays can be made smaller and batteryless, making long-term eavesdropping
widely available. A promising way is to use e-devices recognition methods such
as (Li et al., 2018; Connolly, 2008; Hertenstein and Jagannathan, 2011).
However, these detection solutions usually require scanning at the entrance
before making sure of the space clear, which is heavily dependent on user’s
cooperation and awareness of hidden devices.

###  9.2. Attack Limitation

The association between device IDs and physical biometrics is largely based on
the discriminative patterns of individuals’ session attendance. It is expected
that our presented cross-modal association would become ineffective when these
patterns become ambiguous and target victims will ‘Hide in a Crowd’. Such
pattern ambiguity can be attributed to over-few eavesdropping sessions, or
tricky attendance scenarios where two or more users always appear together
during eavesdropping.

##  10\. Conclusion

In this work, we describe a new type of privacy leakage under multi-modal
attack. We systemically validated that co-located physical biometrics and
device IDs are side channels to each other, which can significantly facilitate
the malicious inference of identity thefts. A cross-modal association approach
is presented by which the adversary can automatically pair victims with their
digital devices and harvest their biometric clusters at the same time, even
under complicated sensing disturbances in the real world. With this strategy,
the adversary can comprehensively profile victims in a target environment and
launch more personal attacks. Experimental results show that our approach is
feasible in two real-world scenarios, where face images and voice segments are
captured and associated with device MAC addresses. We discussed the limitation
of the demonstrated attack vector as well as potential countermeasures against
it. By raising awareness, we invite the community and IoT manufacturers to
look into this new privacy threat.

## References

  * (1)
  * ral (2016) 2016\.  Ralink RT5572.  https://wikidevi.com/wiki/Ralink_RT5572.
  * rpi (2016) 2016\.  Raspberry Pi Camera Module.  https://www.raspberrypi.org/products/camera-module-v2/.
  * OUI (2017) 2017\.  Guidelines for Use Organizationally Unique Identifier (OUI) and Company ID (CID).  https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/tutorials/eui.pdf.
  * rpi (2018) 2018\.  Raspberry Pi 3.  https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/.
  * mat (2019) 2019\.  MATRIX Ecosystem.  https://matrix-io.github.io/matrix-documentation/.
  * Alahi et al. (2015) Alexandre Alahi, Albert Haque, and Li Fei-Fei. 2015.  RGB-W: When vision meets wireless. In _2015 IEEE International Conference on Computer Vision (ICCV)_.
  * Albazrqaoe et al. (2016) Wahhab Albazrqaoe, Jun Huang, and Guoliang Xing. 2016\.  Practical bluetooth traffic sniffing: Systems and privacy implications. In _Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services_. ACM, 333–345.
  * Amiel et al. (2007) Frederic Amiel, Karine Villegas, Benoit Feix, and Louis Marcel. 2007.  Passive and active combined attacks: Combining fault attacks and side channel analysis. In _Fault Diagnosis and Tolerance in Cryptography, 2007. FDTC 2007. Workshop on_. IEEE, 92–102.
  * Backes et al. (2010) Michael Backes, Markus Dürmuth, Sebastian Gerling, Manfred Pinkal, and Caroline Sporleder. 2010\.  Acoustic Side-Channel Attacks on Printers.. In _USENIX Security symposium_. 307–322.
  * Blackman (1986) Samuel S Blackman. 1986\.  Multiple-target tracking with radar applications.  _Dedham, MA, Artech House, Inc., 1986, 463 p._ (1986).
  * Burkhalter (2019) Max Burkhalter. 2019\.  Recent data leak highlights the importance of IoT back-end security.  https://www.perle.com/articles/recent-data-leak-highlights-the-importance-of-iot-back-end-security-40185881.shtml.
  * Cao et al. (2018) Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. 2018.  VGGFace2: A dataset for recognising faces across pose and age. In _International Conference on Automatic Face and Gesture Recognition_.
  * Castelluccia (2012) Claude Castelluccia. 2012\.  Behavioural tracking on the internet: a technical perspective.  In _European Data Protection: In Good Health?_ Springer, 21–33.
  * Christin et al. (2010) Nicolas Christin, Sally S Yanagihara, and Keisuke Kamataki. 2010\.  Dissecting one click frauds. In _Proceedings of the 17th ACM conference on Computer and communications security_. ACM, 15–26.
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Connolly (2008) Christine Connolly. 2008\.  X-ray systems for security and industrial inspection.  _Sensor Review_ 28, 3 (2008), 194–198.
  * Cunche (2014) Mathieu Cunche. 2014\.  I know your MAC Address: Targeted tracking of individual using Wi-Fi.  _Journal of Computer Virology and Hacking Techniques_ 10, 4 (2014), 219–227.
  * Dhem et al. (1998) Jean-Francois Dhem, Francois Koeune, Philippe-Alexandre Leroux, Patrick Mestré, Jean-Jacques Quisquater, and Jean-Louis Willems. 1998\.  A practical implementation of the timing attack. In _International Conference on Smart Card Research and Advanced Applications_. Springer, 167–182.
  * Duarte et al. (1999) Jair Moura Duarte, João Bosco dos Santos, and Leonardo Cunha Melo. 1999.  Comparison of similarity coefficients based on RAPD markers in the common bean.  _Genetics and Molecular Biology_ 22, 3 (1999), 427–432.
  * Fan and Gierlichs (2018) Junfeng Fan and Benedikt Gierlichs. 2018.  _Constructive Side-Channel Analysis and Secure Design_.  Springer.
  * Frankowski et al. (2006) Dan Frankowski, Dan Cosley, Shilad Sen, Loren Terveen, and John Riedl. 2006.  You are what you say: privacy risks of public mentions. In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 565–572.
  * Hadid et al. (2015) Abdenour Hadid, Nicholas Evans, Sebastien Marcel, and Julian Fierrez. 2015.  Biometrics systems under spoofing attack: an evaluation methodology and lessons learned.  _IEEE Signal Processing Magazine_ 32, 5 (2015), 20–30.
  * Hertenstein and Jagannathan (2011) Jake Hertenstein and S Jagannathan. 2011.  Detection of unintended electromagnetic emissions from super-regenerative receivers. In _Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XVI_ , Vol. 8017. International Society for Optics and Photonics, 80170F.
  * Hu et al. (2004) Weiming Hu, Tieniu Tan, Liang Wang, and Steve Maybank. 2004\.  A survey on visual surveillance of object motion and behaviors.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 34, 3 (2004), 334–352.
  * Ilia et al. (2015) Panagiotis Ilia, Iasonas Polakis, Elias Athanasopoulos, Federico Maggi, and Sotiris Ioannidis. 2015\.  Face/off: Preventing privacy leakage from photos in social networks. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_. ACM, 781–792.
  * Jia et al. (2017) Ruoxi Jia, Fisayo Caleb Sangogboye, Tianzhen Hong, Costas Spanos, and Mikkel Baun Kjærgaard. 2017\.  Pad: Protecting anonymity in publishing building related datasets. In _Proceedings of the 4th ACM International Conference on Systems for Energy-Efficient Built Environments_. ACM, 4.
  * JLaservideo (2018) JLaservideo. 2018\.  DIY WIFI Spy Microphone Inside a Football.  https://www.youtube.com/watch?v=a8avue5cPFE.
  * Joy Persial et al. (2011) G Joy Persial, M Prabhu, and R Shanmugalakshmi. 2011.  Side channel attack-survey.  _Int J Adva Sci Res Rev_ 1, 4 (2011), 54–57.
  * Kadloor et al. (2010) Sachin Kadloor, Xun Gong, Negar Kiyavash, Tolga Tezcan, and Nikita Borisov. 2010.  Low-cost side channel remote traffic analysis attack in packet networks. In _Communications (ICC), 2010 IEEE International Conference on_. IEEE, 1–5.
  * Karmarkar (1984) Narendra Karmarkar. 1984\.  A new polynomial-time algorithm for linear programming. In _Proceedings of the sixteenth annual ACM symposium on Theory of computing_. ACM, 302–311.
  * Korolova and Sharma (2018) Aleksandra Korolova and Vinod Sharma. 2018.  Cross-App Tracking via Nearby Bluetooth Low Energy Devices. In _Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy_. ACM, 43–52.
  * Kshetri and Voas (2019) Nir Kshetri and Jeffrey Voas. 2019.  Online Advertising Fraud.  _Computer_ 52, 1 (2019), 58–61.
  * Lane et al. (2012) Nicholas D Lane, Junyuan Xie, Thomas Moscibroda, and Feng Zhao. 2012.  On the feasibility of user de-anonymization from shared mobile sensor data. In _Proceedings of the Third International Workshop on Sensing Applications on Mobile Phones_. ACM, 3\.
  * Li et al. (2007) Stan Z Li, RuFeng Chu, ShengCai Liao, and Lun Zhang. 2007\.  Illumination invariant face recognition using near-infrared images.  _IEEE Transactions on pattern analysis and machine intelligence_ 29, 4 (2007), 627–639.
  * Li et al. (2018) Zhengxiong Li, Zhuolin Yang, Chen Song, Changzhi Li, Zhengyu Peng, and Wenyao Xu. 2018\.  E-Eye: Hidden Electronics Recognition Through mmWave Nonlinear Effects. In _Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems_ _(SenSys ’18)_. ACM, New York, NY, USA, 68–81.  https://doi.org/10.1145/3274783.3274833
  * Lin et al. (2014) Chia-Chi Lin, Hongyang Li, Xiao-yong Zhou, and XiaoFeng Wang. 2014.  Screenmilker: How to Milk Your Android Screen for Secrets.. In _NDSS_.
  * Lipp et al. (2016) Moritz Lipp, Daniel Gruss, Raphael Spreitzer, Clémentine Maurice, and Stefan Mangard. 2016\.  Armageddon: Cache attacks on mobile devices. In _25th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 16)_. 549–564.
  * Loveless (2017) Mark Loveless. 2017\.  BLUETOOTH HACKING TOOLS COMPARISON.  https://duo.com/decipher/bluetooth-hacking-tools-comparison.
  * Ltd (2018) Z-Smart Systems UK Ltd. 2018\.  Wireshark ZigBee Sniffer.  https://www.cd-jackson.com/downloads/ZigBeeWiresharkSniffer.pdf.
  * Lu et al. (2019a) Chris Xiaoxuan Lu, Xuan Kan, Bowen Du, Changhao Chen, Hongkai Wen, Andrew Markham, Niki Trigoni, and John Stankovic. 2019a.  Autonomous Learning for Face Recognition in the Wild via Ambient Wireless Cues. In _The World Wide Web Conference_. ACM, 1175–1186.
  * Lu et al. (2017) Chris Xiaoxuan Lu, Hongkai Wen, Sen Wang, Andrew Markham, and Niki Trigoni. 2017.  SCAN: Learning speaker identity from noisy sensor data. In _Proceedings of the 16th ACM/IEEE International Conference on Information Processing in Sensor Networks_. ACM, 67–78.
  * Lu et al. (2019b) Chris Xiaoxuan Lu, Yuanbo Xiangli, Peijun Zhao, Changhao Chen, Niki Trigoni, and Andrew Markham. 2019b.  Autonomous Learning of Speaker Identity and WiFi Geofence From Noisy Sensor Data.  _IEEE Internet of Things Journal_ 6, 5 (2019), 8284–8295.
  * Lu et al. (2016) Xiaoxuan Lu, Hongkai Wen, Han Zou, Hao Jiang, Lihua Xie, and Niki Trigoni. 2016\.  Robust occupancy inference with commodity WiFi. In _International Conference on Wireless and Mobile Computing, Networking and Communications_.
  * Marthews and Tucker (2017) Alex Marthews and Catherine E Tucker. 2017.  Government surveillance and internet search behavior.  _Available at SSRN 2412564_ (2017).
  * Martin et al. (2017) Jeremy Martin, Travis Mayberry, Collin Donahue, Lucas Foppe, Lamont Brown, Chadwick Riggins, Erik C Rye, and Dane Brown. 2017\.  A study of MAC address randomization in mobile devices and when it fails.  _Proceedings on Privacy Enhancing Technologies_ 2017, 4 (2017), 365–383.
  * Meulen (2017) Rob van der Meulen. 2017\.  Gartner says 8.4 billion connected “Things” will be in use in 2017 up 31 percent from 2016.  _Gartner. Letzte Aktualisierung_ 7 (2017), 2017.
  * Mislove et al. (2010) Alan Mislove, Bimal Viswanath, Krishna P Gummadi, and Peter Druschel. 2010.  You are who you know: inferring user profiles in online social networks. In _Proceedings of the third ACM international conference on Web search and data mining_. ACM, 251–260.
  * Narayanan and Shmatikov (2008) Arvind Narayanan and Vitaly Shmatikov. 2008.  Robust de-anonymization of large sparse datasets. In _Security and Privacy, 2008. SP 2008. IEEE Symposium on_. IEEE, 111–125.
  * Papaioannou et al. (2015) Savvas Papaioannou, Hongkai Wen, Zhuoling Xiao, Andrew Markham, and Niki Trigoni. 2015\.  Accurate Positioning via Cross-Modality Training. In _ACM SenSys_.
  * Réal et al. (2009) Denis Réal, Frédéric Valette, and Mhamed Drissi. 2009.  Enhancing correlation electromagnetic attack using planar near-field cartography. In _Proceedings of the Conference on Design, Automation and Test in Europe_. European Design and Automation Association, 628–633.
  * Rodrigues et al. (2010) Ricardo N Rodrigues, Niranjan Kamat, and Venu Govindaraju. 2010\.  Evaluation of biometric spoofing in a multimodal system. In _2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS)_. IEEE, 1–5.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015\.  Facenet: A unified embedding for face recognition and clustering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 815–823.
  * Schubert et al. (2017) Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. 2017\.  DBSCAN revisited, revisited: why and how you should (still) use DBSCAN.  _ACM Transactions on Database Systems (TODS)_ 42, 3 (2017), 19\.
  * Snyder et al. (2018) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018\.  X-vectors: Robust dnn embeddings for speaker recognition. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5329–5333.
  * Spreitzer et al. (2017) Raphael Spreitzer, Veelasha Moonsamy, Thomas Korak, and Stefan Mangard. 2017.  Systematic classification of side-channel attacks: a case study for mobile devices.  _IEEE Communications Surveys & Tutorials_ 20, 1 (2017), 465–488.
  * Sweeney (1997) Latanya Sweeney. 1997\.  Weaving technology and policy together to maintain confidentiality.  _The Journal of Law, Medicine & Ethics_ 25, 2-3 (1997), 98–110.
  * Sweeney (2002) Latanya Sweeney. 2002\.  k-anonymity: A model for protecting privacy.  _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ 10, 05 (2002), 557–570.
  * team (2018) FindLaw’s team. 2018\.  Are Hidden Cameras at Work Legal?  https://employment.findlaw.com/workplace-privacy/are-hidden-cameras-at-work-legal-.html.
  * Teng et al. (2014) Jin Teng, Boying Zhang, Junda Zhu, Xinfeng Li, Dong Xuan, and Yuan F Zheng. 2014\.  Ev-loc: integrating electronic and visual signals for accurate localization.  _IEEE/ACM Transactions on Networking_ (2014).
  * Vanhoef et al. (2016) Mathy Vanhoef, Célestin Matte, Mathieu Cunche, Leonardo S Cardoso, and Frank Piessens. 2016.  Why MAC address randomization is not enough: An analysis of Wi-Fi network discovery mechanisms. In _Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security_. ACM, 413–424.
  * WifiNigel (2018) WifiNigel. 2018\.  Randomized MAC addresses in 802.11 Probe Frames.  http://wifinigel.blogspot.com/2018/04/to-address-perceived-privacy-issues.html.
  * Xiangli et al. (2019) Yuanbo Xiangli, Chris Xiaoxuan Lu, Peijun Zhao, Changhao Chen, and Andrew Markham. 2019\.  iSCAN: automatic speaker adaptation via iterative cross-modality association. In _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_.
  * Xu et al. (2017) Fengli Xu, Zhen Tu, Yong Li, Pengyu Zhang, Xiaoming Fu, and Depeng Jin. 2017\.  Trajectory recovery from ash: User privacy is not preserved in aggregated mobility data. In _Proceedings of the 26th International Conference on World Wide Web_. 1241–1250.
  * Yen et al. (2012) Ting-Fang Yen, Yinglian Xie, Fang Yu, Roger Peng Yu, and Martin Abadi. 2012.  Host Fingerprinting and Tracking on the Web: Privacy and Security Implications.. In _NDSS_ , Vol. 62. 66.
  * Zheleva and Getoor (2009) Elena Zheleva and Lise Getoor. 2009.  To join or not to join: the illusion of privacy in social networks with mixed public and private user profiles. In _Proceedings of the 18th international conference on World wide web_. ACM, 531–540.