††thanks: These two authors contributed equally††thanks: These two authors
contributed equally

# Identify Hidden Spreaders of Pandemic over Contact Tracing Networks

Shuhong Huang  Institute of Neuroscience, Technical University of Munich,
Munich 80802, Germany  Jiachen Sun  MIT Center for Collective Intelligence,
Cambridge 02142, MA, USA  Ling Feng  Institute of High Performance Computing,
A*STAR, 138632 Singapore  Department of Physics, National University of
Singapore, Singapore 117551  Jiarong Xie  School of Data and Computer Science,
Sun Yat-sen University, Guangzhou 510006, China  Dashun Wang  Kellogg School
of Management, Northwestern University, Evanston, IL, USA  Yanqing Hu
<EMAIL_ADDRESS>School of Data and Computer Science, Sun Yat-sen
University, Guangzhou 510006, China

###### Abstract

The COVID-19 infection cases have surged globally, causing devastations to
both the society and economy. A key factor contributing to the sustained
spreading is the presence of a large number of asymptomatic or hidden
spreaders, who mix among the susceptible population without being detected or
quarantined. Here we propose an effective non-pharmacological intervention
method of detecting the asymptomatic spreaders in contact-tracing networks,
and validated it on the empirical COVID-19 spreading network in Singapore. We
find that using pure physical spreading equations, the hidden spreaders of
COVID-19 can be identified with remarkable accuracy. Specifically, based on
the unique characteristics of COVID-19 spreading dynamics, we propose a
computational framework capturing the transition probabilities among different
infectious states in a network, and extend it to an efficient algorithm to
identify asymptotic individuals. Our simulation results indicate that a
screening method using our prediction outperforms machine learning algorithms,
e.g. graph neural networks, that are designed as baselines in this work, as
well as random screening of infection’s closest contacts widely used by China
in its early outbreak. Furthermore, our method provides high precision even
with incomplete information of the contract-tracing networks. Our work can be
of critical importance to the non-pharmacological interventions of COVID-19,
especially with increasing adoptions of contact tracing measures using various
new technologies. Beyond COVID-19, our framework can be useful for other
epidemic diseases that also feature asymptomatic spreading.

As the COVID-19 pandemic continues to spread at rapid rates hui2020continuing
; Megan2020pandemic ; world2020director , and the development of effective
pharmacological treatments is still uncertain according to WHO, non-
pharmacological interventions like isolation of the infectious through
quarantines hellewell2020feasibility ; maier2020effective  are the most
effective and possibly the only means of containing the continued outbreaks,
as it effectively reduces the person to person transmissions chan2020familial
. Yet, unlike other infectious diseases like SARS and Ebola, COVID-19 is
unique in that a large portion of its infected population is mild or
asymptotic gao2020systematic . Even some of the asymptotic infections do not
exhibit any clinic symptoms until self-recovery kimball2020asymptomatic ;
long2020clinical . Without being detected and subsequently quarantined, the
asymptomatic population (i.e. hidden spreaders) sustains the ongoing spreading
of the disease to the susceptible population unknowingly liu2020locally ;
rothe2020transmission . This poses a major challenge in the effective
mitigation of the pandemic spreading. Furthermore, empirical studies have
shown that such asymptomatic infections accounts for a large proportion of the
population quilty2020effectiveness ; byambasuren2020estimating ;
nishiura2020estimation ; day2020covid ; yu2020covid ; mizumoto2020estimating ;
Heneghan2020asymptomatic , as much as up to 80% Heneghan2020asymptomatic .
Currently, estimation of the asymptomatic cases is done through exhaustive
screening of close contacts of the known infected cases in the contact tracing
networks mizumoto2020estimating . This untargeted method requires large amount
of resources and is time consuming, that in turn leads to ineffective or
delayed interventions to quarantine the asymptomatic cases. Hence, a targeted
screening in the contact tracing network is pertinent, such that asymptomatic
individuals can be estimated with high precision for intervention and
spreading mitigation.

Here we incorporate the empirical characteristics of the COVID-19 spreading
dynamics into a Markovian process, i.e. vectors that represent the different
infection stages and their associated transition probabilities. By embedding
the transition process into a contact tracing network that includes the known
infected nodes (individuals), we develop a method that predicts the infectious
states of the rest of the network with high precision. By combining such
predictions with the network structure, we then derive the spreading power of
every node taking into account of both its infectious state and its specific
location in the network, such that screening of the asymptomatic can be
prioritised accordingly. The effectiveness of our method is validated by
empirical data from two COVID-19 transmission networks in Singapore. Moreover,
in the simulated COVID-19 transmission experiment of contact-tracing network,
we find that a screening scheme designed by the proposed computational
framework outperforms several machine-learning baselines designed in this work
and the random screening of infection neighbors. The latter was widely used in
early COVID-19 outbreaks in China. Furthermore, even in the realistic
situation of incomplete information on the contact tracing network, with
missing links or sub networks consisting of only contacts of the infected
cases, our method retains high accuracy. Thus our method is highly effective
in asymptomatic case estimation and can be implemented to any contact-tracing
networks either constructed manually againstcovid19.com  or through
technological means kondylakis2020covid  such as Bluetooth drew2020rapid ;
ferretti2020quantifying , GPS chang2020mobility  and digital check-in check-
out technologies (e.g. health QR codes mozur2020coronavirus  widely used in
China).

##  I results

Given the spreading of the COVID-19 occurring over the contact network, the
challenge is to identify asymptomatic nodes with the information of infected
symptomatic individuals (nodes) that have been identified from a certain time
$T$. We approach this by estimating the probability of each node being in the
infected state as illustrated in Fig. 1. Specifically, we first construct the
transition dynamical equations among different infection stages and states
based on the empirically observation of COVID-19 disease progression. The set
of transition equations is then combined with the contact network topology and
data on the observed infection history to deduce the state of each node in the
network.

As observed in many clinical studies, the hidden spreaders of COVID-19 fall
into two different categories. One is the presymptomatic infections who are
asymptomatic and infected, but will later develop clinical symptoms (e.g.
fever, cough, dyspnea, etc.); The other type corresponds to the asymptomatic
patients who carry the virus but have never exhibited any symptoms until
recovery. As a result, an individual can have a total of 5 different states in
the process of COVID-19 spreading (see Fig. 1a), namely, Susceptible ($S$),
Presymptomatic ($P$), Asymptomatic ($A$), Symptomatic Infectious ($I$) and
Recovered ($R$). Since the infectious duration in the states of $P$, $I$ and
$A$ follows a specific probability distribution, here we further break down
the $P,I$ and $A$ states into finer states representing the progression in
each of the 3 states, i.e., the number of days passed since the beginning of
the states. For better clarity, we denote $t$ as the number of days of the
COVID-19 evolution on the entire network and $d$ as the number of days in a
particular infected state for a particular individual. Since an individual $i$
can be at any stage in the process, we can use $\mathbf{Z}_{i}(t)$ to
represent the state probabilities at time $t$:

$\displaystyle\mathbf{Z}_{i}(t)=$
$\displaystyle(S_{i}(t),P^{1}_{i}(t),...,P^{d}_{i}(t),...,I^{1}_{i}(t),...,$
(1) $\displaystyle
I^{d}_{i}(t),...,A^{1}_{i}(t),...,A^{d}_{i}(t),...,R_{i}(t))$

where $S_{i}(t)$ and $R_{i}(t)$ is the probability that the individual $i$ is
susceptible and recovered at day $t$, respectively.
$P^{d}_{i}(t),I^{d}_{i}(t)$ and $A^{d}_{i}(t)$ are the probabilities that $i$
is in the state of $P$, $I$ and $A$ for $d$ days at the time of $t$. Since all
of asymptomatic, presymptomatic and symptomatic states are infected states,
their total probability corresponds to that of a node is infectious, and we
use $C_{i}(t)$ to represent it:

$C_{i}(t)=P_{i}(t)+I_{i}(t)+A_{i}(t)$ (2)

where $P_{i}(t)=\sum_{d=1}^{\infty}P^{d}_{i}(t)$,
$I_{i}(t)=\sum_{d=1}^{\infty}I^{d}_{i}(t)$,
$A_{i}(t)=\sum_{d=1}^{\infty}A^{d}_{i}(t)$. Throughout this work we use
$C_{i}(t)$ as a key indicator to infer whether an individual is infected .

From here, we can extract the probability transition dynamics among the 5
different states as follows. First, for a node who is in the susceptible state
$S$ at $t$, its next state at $t+1$ will be jointly determined by the state of
its neighbors in the network at $t$. Specifically, the probability of a node
$i$ in $S$ state remains in $S$ on day $t+1$ (i.e., not infected by any of its
infected neighbor on the next day) is:

$S_{i}(t+1)=S_{i}(t)\cdot\prod_{j\in\partial i}(1-\mathcal{F}(t,j,\beta))$ (3)

where $\partial i$ represents the set of neighbors (contacts) of $i$ in the
network, $\mathcal{F}(t,j,\beta)$ represents the probability that $i$ is
infected by $j$. This can only happen if $j$ is in the infected state on day
$t$ (probability $C_{j}(t)$), and happens to transmit it to $i$ (probability
$\beta$). Then we have:

$\mathcal{F}(t,j,\beta)=C_{j}(t)\cdot\beta$ (4)

Here $\beta$ can be estimated from the empirically observed disease
reproduction number $R_{0}$ for COVID-19 and the average number of neighbors
in the contact tracing network $\langle k\rangle$. Specificaly,
$\beta=\frac{R_{0}}{\lambda\langle k\rangle}$, where $\lambda$ is the average
time a susceptible person carries the virus, which can be expressed as
$\lambda=p\cdot\mu_{A}+(1-p)\cdot(\exp(\mu_{P}+\frac{\sigma_{P}^{2}}{2})+\mu_{I})$,
where $p$ is the proportion of asymptomatic infected cases,
$\mu_{A},\exp(\mu_{P}+\frac{\sigma_{P}^{2}}{2}),\mu_{I}$ are the average time
of the virus carried by infected individuals in $A$, $P$ and $I$ states
li2020early ; zhou2020clinical  respectively.

Next, for an individual under $S$ state at time $t$, the probability of
becoming presymptomatic state $P$ at $t+1$ is:

$P_{i}^{1}(t+1)=S_{i}(t)\cdot(1-p)\cdot(1-\prod_{j\in\partial
i}(1-\mathcal{F}(t,j,\beta))$ (5)

Accordingly, we can calculate the probability that the state of $i$ become $A$
at $t+1$ as:

$A_{i}^{1}(t+1)=S_{i}(t)\cdot p\cdot(1-\prod_{j\in\partial
i}(1-\mathcal{F}(t,j,\beta))$ (6)

In the third case where a node $i$ is in the infected state (i.e. $E$, $I$ or
$A$, $d\geq 1$) on day $t$, the transition probabilities that they will stay
in the same state on day $t+1$ are:

$\displaystyle
P^{d+1}_{i}(t+1)=P^{d}_{i}(t)\cdot(\frac{1-F_{P}(d)}{1-F_{P}(d-1)})$ (7)
$\displaystyle
I^{d+1}_{i}(t+1)=I^{d}_{i}(t)\cdot(\frac{1-F_{I}(d)}{1-F_{I}(d-1)})$
$\displaystyle
A^{d+1}_{i}(t+1)=A^{d}_{i}(t)\cdot(\frac{1-F_{A}(d)}{1-F_{A}(d-1)})$

where $F_{P}(d)=\int_{-\infty}^{d}{f_{P}(t)dt}$,
$F_{I}(d)=\int_{-\infty}^{d}f_{I}(t)dt$, $F_{A}(d)$
$=\int_{-\infty}^{d}f_{A}(t)dt$ are the cumulative distribution functions of
duration length $d$ for $P,I,A$ states, respectively. For mathematical
convince, we simply set $F_{P}(0)=F_{I}(0)=F_{A}(0)=0$. The fourth case is
that individual in the presymptomatic state $P$ turns into the symptomatic
infectious state $I$ at the next day, and can be described with the following
transition probability:

$I^{1}_{i}=\sum_{d=1}^{\infty}P^{d}_{i}\cdot\frac{F_{P}(d)-F_{P}(d-1)}{1-F_{P}(d-1)}$
(8)

In the fifth case, an individual in the state $I$ or the state $A$ has a
certain probability of being recovered i.e, turning into the $R$ state on the
next day. From the above equation, we obtain the probability that the
individual $i$ is in the state of $R$ at the time $t+1$ is:

$\displaystyle R_{i}(t+1)=$ $\displaystyle
R_{i}(t)+\sum_{d=1}^{\infty}A_{i}^{d}\cdot\frac{F_{A}(d)-F_{A}(d-1)}{1-F_{A}(d-1)}$
(9) $\displaystyle+$
$\displaystyle\sum_{d=1}^{\infty}I_{i}^{d}\cdot\frac{F_{I}(d)-F_{I}(d-1)}{1-F_{I}(d-1)}$

To validate our mathematical framework, we test it on a real contact-tracing
network in the Infectious Stay Away exhibitionisella2011s  (ISA network, see
Sec. SI 1 for data detailed description) with 410 individuals and average
degree $\langle k\rangle$ of 13 (more experiments on another social network
are illustrated in Sec. SI 5). We simulate the spreading with the empirically
observed parameters on COVID-19 spreading mechanisms li2020early  (see Methods
for the simulation details). From repeated simulations, we then obtain the
probability of every possible state of a node, and compare this baseline with
the theoretical results from Eqns. 3-9. Here we set the dimension of $Z$ to 77
according to the empirical temporal distributions of the infected states
li2020early ; zhou2020clinical  (see Method for detail). From Fig. 2a and b,
we can see that our theoretical result on the temporal evolutions of the
disease in the whole network is well validated by the simulations. These show
that our transition probability framework is accurate in producing the real
spreading dynamics.

Table 1: COVID-19’s clinical parameters and infectious characteristics used in this work. Parameter |  Meaning |  value |  Origin
---|---|---|---
$R_{0}$ |  basic reproduction number |  3.50 |  Average from 10 researches tang2020estimation ; riou2020pattern ; zhao2020preliminary ; imaireport ; cao2020estimating ; wu2020nowcasting ; shen2020modelling ; liu2020transmission ; read2020novel ; majumder2020early
$p$ |  fraction of asymptomatic infections |  15% |  minimal value from 5 researchesnishiura2020estimation ; byambasuren2020estimating ; nishiura2020estimation ; yu2020covid ; mizumoto2020estimating
$f_{P}(d)$ |  Distribution of during length of Presymptomatic state |  Logarithmic normal distribution with $\mu_{P}=0.62$ and $\sigma_{P}=0.64$ |  fitted value from clinical data of li2020early
$f_{I}(d)$ |  Distribution of during length of Symptomatic state |  Normal distribution with $\mu_{I}=8.8$ and $\sigma_{I}=3.88$ |  fitted value from clinical data of li2020early
$f_{A}(d)$ |  Distribution of during length of Asymptomatic state |  Normal distribution with $\mu_{A}=20.0$ and $\sigma_{A}=5.0$ |  $\mu_{A}$ is estimated from clinical data of zhou2020clinical

Now we extend the proposed transition probability equations to identify nodes
with high risk of being asymptomatic, assuming the infection history on
symptomatic nodes is already known. The underlying principle is to update
every node’s state by incorporating the information of known infection into
Eqns.(2-9) in the subsequent days, and then deduce the infection probability
$C_{i}(T)$ for each node $i$ in the network (see the details in the Method).
The nodes with higher $C_{i}(T)$ are identified as having high risk of being
infected at day $T$. We test the effectiveness by applying it on two sets of
real COVID-19 spreading data on the contact-tracing network in Singapore
againstcovid19.com  (see Fig. 2c and d). The details of network is provided in
Method and in Sec. SI 1. We find that the ranking our $C_{i}(T)$ values are
highly correlated with the date of infection $t$ of nodes (Fig. 2e and f),
meaning nodes with higher infection probabilities indeed have higher risk of
being infected in the real COVID-19 spreading data.

The Singapore empirical datasets have the constraint of merely including the
symptomatic individuals’ identities in the network. Therefore, to further
evaluate our method, we simulate a realistic COVID-19 spreading process on the
ISA network for $T$ days to obtain the detailed infection history of every
node in the network, such that the exact infection history on the asymptomatic
nodes can be obtained. Assuming only the symptomatic nodes with state $I$ are
observed, we use our above method to identify those infected individuals among
the rest of the nodes. Specifically, we select the nodes with the highest
$C_{i}(T)$ values as the mostly likely infected nodes. To our best knowledge,
there is few prior works for estimating asymptomatic nodes in the network.
Therefore, we also design several screening baselines based on the popular
graph neural networks methods including Node2Vec grover2016node2vec , graph
convolutional network kipf2016semi  and graph attention networks
velivckovic2017graph  to further compare our results. (Detailed methodologies
for those methods in Sec. SI 3).

The simulations results show that our transition probabilistic method (i.e.
static screening) significantly outperforms the other methods in terms of the
accuracy and recall on the local network where one can only observe the
nearest neighbors of the known nodes in states $I$ (see SI Figure 1). Such
advantage is still evident when we consider the alternative scenario that one
can observe the full network structure drew2020rapid ; ferretti2020quantifying
(see SI Figure 3), and the intermediate scenario when only nearest and second
nearest neighbors are known in the network (see SI Figure 2). In a more
realistic setting, the screening of the contact tracing network happens
continuously in time. Here one can update the set of known infected nodes
after every screening, and subsequently update the infected risk for the rest
of the network from time to time. Therefore, we develop a dynamic screening
method by updating the evaluation of $C_{i}(t)$ every time a new infected node
is found through selective screening of the network (see the details in the
Method Section). This dynamic screening method outperforms (see Fig. 3a-d)
other screening methods and even our previous static screening method (see
Fig. 3d inset), implying that such dynamic screening method is highly
effective in identifying infected nodes by screening less people.

Very often, the contact tracing network collected through either manual survey
or digital tracking is at best incomplete, such that it is important to have a
screening method that is still robust when there is missing information on the
network structure. To test such robustness of our method, we randomly remove
up to 80% of the edges in the ISA network, and test the accuracy of the method
based on the remaining network (see the results on another network in SI
Figures 14-19). We find that the dynamic screening method on the various
scenarios can still reliably identify the infected nodes in terms of accuracy
and recall rate, as shown in Fig. 3e-h (see the robustness result on the
static screening method in SI Figures 14-17).

Lastly, we study the effectiveness of our method in containing the overall
spread of COVID-19. In the widespread of COVID-19, limited resource on
screening constrains the number of individuals the government can screen in a
given day. Hence, targeted screening and mitigation can have significant
impact on ‘flattening the curve’ of daily infected cases. To study such
effect, we again simulate the COVID-19 spreading on the ISA network
isella2011s , and start screening/testing from day 10 using our method (i.e.
‘neighbor containment’). Each day 2% (4% in SI Figure 20) of the whole network
are tested for the disease, and the positive ones are immediately quarantined,
corresponding a transition to the state $R$ (see details of the containment
strategy in Sec. SI 4). As shown in Fig. 4a-e, our method is highly effective
in suppressing the daily infection cases and total infection cases,
outperforming both the baseline strategy of only quarantining the infected
ones (labeled as ‘infection containment’) and the strategy randomly screening
$2\%N$ among the neighbors of the known infections (labelled as ‘neighbor
containment’), where $N$ is the network size. In addition, we find that even
with up to 80% missing links, our method is still robust enough to effectively
suppress the spreading, close to that of knowing the full network structure.
It shows that our method is expected to be highly effective in containing
COVID-19 spread in practice.

##  II Conclusion

In this paper, based on the transmission rule of COVID-19 and the underlying
physical spreading equations, we for the first time studied the estimation of
asymptomatic infections in the contact-tracing network, which is a current
major concern in the prevention and containment of COVID-19 worldwide. We
provided a complete computational framework of inferring latent infection on
contact network. Based on this, we proposed a feasible method for optimal
detection of latent infection in combination with nodal transmission ability
in the network. We show that the COVID-19 transmission can be broken in a
timely and efficient manner by the proposed method, which outperforms the
direct contact screening, a typical method widely used in China. In addition,
our simulation on a real contact network demonstrated that, this method is
robust even with incomplete network information, demonstrating its
effectiveness in practical scenarios. We believe that the theory and the
corresponding methods in identifying COVID-19 hidden spreaders are of great
practical significance. In principle, it provides policymakers and front-line
workers in COVID-19 with important and effective guidance and tools that could
be deployed swiftly to fight COVID-19, and save billions of people around the
world who are still suffering as the epidemic continues to spread throughout
the world.

## Method

Singapore COVID-19 datasets. The data was collected by the Singapore
government againstcovid19.com , and contains comprehensive records on the
dates of showing symptoms and confirming the disease, as well as their contact
networks. We pick the infected nodes from the first two different time points,
and set $T$ to January 26, 2020 and February 19, 2020 for Singapore A and
Singapore B, respectively. Then based on the known infection history of the
nodes, our transition probability method estimates the infection probabilities
$C_{i}(T)$ of every other node in the networks.

The dimension of the state vector $\mathbf{Z}_{i}(t)$. The dimension of the
state vector $\mathbf{Z}_{i}(t)$ corresponds to the total number of sub states
possible during the various disease progression paths, i.e., the number of
days that an individual can be in each of the 3 different infected states.
From the empirical temporal distributions of the infected states li2020early ;
zhou2020clinical  (Tab. 1), we use 3 standard deviations
wheeler1992understanding  as cut off on the max number of days in states
$P,I,A$, which are $20,20,35$ days, yielding a dimension of 77 for $Z$. ($S$
and $R$ states have no sub states).

COVID-19 spreading simulation. At the starting time $T=0$, we select the 3
nodes with the largest degree in the network as initial infected nodes, whose
infected states are determined as either Asymptomatic or Presymptomatic
according to the parameter $p$ of Tab. 1. Then we apply the empirically
observed parameters on COVID-19 spreading mechanisms li2020early  including
reproductive number $R_{0}=3.50$ and asymptotic infection ratio $p=15\%$ on
our equations to simulate the spreading. The set of values are listed in
Table. 1(see Sec. SI 1 for the detail description of parameters and Sec. SI 3
for the discussion of the parameter sensitivity). Each simulation corresponds
to one realization of the actual spreading based on the realistic dynamics,
and the actual states of each node at every time step can be captured. More
details of the simulation of COVID-19 are provided in Sec. SI 2.

Identifying infection probability $C_{i}(T)$. The goal is to identify nodes
with high risk of being asymptomatic with infection history on known
symptomatic nodes, and we extend our transition probability equations to study
this problem. At a certain time $T$, given the set $I$ of infected
individuals, the first day of infection $s_{j}$ and the day of recovery
$r_{j}$ for each individual $j\in I$, we aim to develop a method from Eqns.
2-10 to deduce the infection probability $C_{i}(T)$ for each node $i$ in the
network. Note that the day of recovery can also be the day of death or
quarantine. The initial condition at $t=0$ is that every node in the network
is in susceptible state, i.e. ${\bf Z}_{i}(0)=\\{1,0,\cdots,0\\}$. The day of
first infection in the network is set to 1, i.e. $\min_{j\in I}s_{j}=1$, and
we update every node’s state in the subsequent days depending on whether their
infection history is known at time $T$. For the known nodes $j\in I$, we
artificially assign their infection states according to the known information,
meaning that $j$ is assigned state $S$ when $t<s_{j}$, state $R$ when
$t>r_{j}$, and infectious state when $s_{j}<t<r_{j}$. For the other nodes, we
evaluate their state vector $Z_{i}(t)$ at every time step $t$ according to the
transition probabilities in Eqns. 2-9, until the final day $T$, such that
their probabilities $C_{i}(T)$ of being infected can be evaluated from
$Z_{i}(T)$.

Dynamic screening method. Every time we screen only node $k$ that is of
highest risk according to the algorithm; if node $k$ is COVID-19 positive, it
is added to the known infected nodes set $I$, and its neighbors are added to
the unknown set, and we repeat the transition probability calculations
according to Eqns. 2-10 from time $0<t\leq T$; if $k$ is negative, its
probability state vector is set to be ${\bf Z}_{k}(t)=\\{1,0,\cdots,0\\}$ in
the calculation of Eqns. 2-10. Next, the revised estimations of infection
probabilities for each unknown node from Eqns. 2-10 tells us which node is the
most risky and to be tested.

## References

  * [1] David S Hui, Esam I Azhar, Tariq A Madani, Francine Ntoumi, Richard Kock, Osman Dar, Giuseppe Ippolito, Timothy D Mchugh, Ziad A Memish, Christian Drosten, et al.  The continuing 2019-ncov epidemic threat of novel coronaviruses to global health–the latest 2019 novel coronavirus outbreak in wuhan, china.  International Journal of Infectious Diseases, 91:264–266, 2020\.
  * [2] Scudellari Megan.  How the pandemic might play out in 2021 and beyond.  Nature, 2020.
  * [3] World Health Organization et al.  Who director-general’s opening remarks at the media briefing on covid-19-11 march 2020, 2020.
  * [4] Joel Hellewell, Sam Abbott, Amy Gimma, Nikos I Bosse, Christopher I Jarvis, Timothy W Russell, James D Munday, Adam J Kucharski, W John Edmunds, Fiona Sun, et al.  Feasibility of controlling covid-19 outbreaks by isolation of cases and contacts.  The Lancet Global Health, 2020.
  * [5] Benjamin F Maier and Dirk Brockmann.  Effective containment explains subexponential growth in recent confirmed covid-19 cases in china.  Science, 368(6492):742–746, 2020.
  * [6] Jasper Fuk-Woo Chan, Shuofeng Yuan, Kin-Hang Kok, Kelvin Kai-Wang To, Hin Chu, Jin Yang, Fanfan Xing, Jieling Liu, Cyril Chik-Yan Yip, Rosana Wing-Shan Poon, et al.  A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-to-person transmission: a study of a family cluster.  The Lancet, 395(10223):514–523, 2020.
  * [7] Zhiru Gao, Yinghui Xu, Chao Sun, Xu Wang, Ye Guo, Shi Qiu, and Kewei Ma.  A systematic review of asymptomatic infections with covid-19.  Journal of Microbiology, Immunology and Infection, 2020.
  * [8] Anne Kimball, Kelly M Hatfield, Melissa Arons, Allison James, Joanne Taylor, Kevin Spicer, Ana C Bardossy, Lisa P Oakley, Sukarma Tanwar, Zeshan Chisty, et al.  Asymptomatic and presymptomatic sars-cov-2 infections in residents of a long-term care skilled nursing facility—king county, washington, march 2020\.  Morbidity and Mortality Weekly Report, 69(13):377, 2020.
  * [9] Quan-Xin Long, Xiao-Jun Tang, Qiu-Lin Shi, Qin Li, Hai-Jun Deng, Jun Yuan, Jie-Li Hu, Wei Xu, Yong Zhang, Fa-Jin Lv, et al.  Clinical and immunological assessment of asymptomatic sars-cov-2 infections.  Nature medicine, pages 1–5, 2020.
  * [10] Ying-Chu Liu, Ching-Hui Liao, Chin-Fu Chang, Chu-Chung Chou, and Yan-Ren Lin.  A locally transmitted case of sars-cov-2 infection in taiwan.  New England Journal of Medicine, 382(11):1070–1072, 2020.
  * [11] Camilla Rothe, Mirjam Schunk, Peter Sothmann, Gisela Bretzel, Guenter Froeschl, Claudia Wallrauch, Thorbjörn Zimmer, Verena Thiel, Christian Janke, Wolfgang Guggemos, et al.  Transmission of 2019-ncov infection from an asymptomatic contact in germany.  New England Journal of Medicine, 382(10):970–971, 2020.
  * [12] Billy J Quilty, Sam Clifford, Stefan Flasche, Rosalind M Eggo, et al.  Effectiveness of airport screening at detecting travellers infected with novel coronavirus (2019-ncov).  Eurosurveillance, 25(5):2000080, 2020.
  * [13] Oyungerel Byambasuren, Magnolia Cardona, Katy Bell, Justin Clark, Mary-Louise McLaws, and Paul Glasziou.  Estimating the extent of true asymptomatic covid-19 and its potential for community transmission: systematic review and meta-analysis.  Available at SSRN 3586675, 2020.
  * [14] Hiroshi Nishiura, Tetsuro Kobayashi, Takeshi Miyama, Ayako Suzuki, Sung-mok Jung, Katsuma Hayashi, Ryo Kinoshita, Yichi Yang, Baoyin Yuan, Andrei R Akhmetzhanov, et al.  Estimation of the asymptomatic ratio of novel coronavirus infections (covid-19).  International journal of infectious diseases, 94:154, 2020.
  * [15] Michael Day.  Covid-19: identifying and isolating asymptomatic people helped eliminate virus in italian village.  BMJ: British Medical Journal (Online), 368, 2020.
  * [16] Yang Yu, Yu-Ren Liu, Fan-Ming Luo, Wei-Wei Tu, De-Chuan Zhan, Guo Yu, and Zhi-Hua Zhou.  Covid-19 asymptomatic infection estimation.  medRxiv, 2020.
  * [17] Kenji Mizumoto, Katsushi Kagaya, Alexander Zarebski, and Gerardo Chowell.  Estimating the asymptomatic proportion of coronavirus disease 2019 (covid-19) cases on board the diamond princess cruise ship, yokohama, japan, 2020\.  Eurosurveillance, 25(10):2000180, 2020.
  * [18] Heneghan C, Brassey J, and Jefferson T.  Covid-19: What proportion are asymptomatic?  Centre for Evidence-Based Medicine, 2020.
  * [19] Dashboard of the covid-19 virus outbreak in singapore.  https://www.againstcovid19.com/singapore/dashboard Accessed April 4, 2020.
  * [20] Haridimos Kondylakis, Dimitrios G Katehakis, Angelina Kouroubali, Fokion Logothetidis, Andreas Triantafyllidis, Ilias Kalamaras, Konstantinos Votis, and Dimitrios Tzovaras.  Covid-19 mobile apps: A systematic review of the literature.  Journal of medical Internet research, 2020.
  * [21] David A Drew, Long H Nguyen, Claire J Steves, Cristina Menni, Maxim Freydin, Thomas Varsavsky, Carole H Sudre, M Jorge Cardoso, Sebastien Ourselin, Jonathan Wolf, et al.  Rapid implementation of mobile technology for real-time epidemiology of covid-19.  Science, 2020.
  * [22] Luca Ferretti, Chris Wymant, Michelle Kendall, Lele Zhao, Anel Nurtay, Lucie Abeler-Dörner, Michael Parker, David Bonsall, and Christophe Fraser.  Quantifying sars-cov-2 transmission suggests epidemic control with digital contact tracing.  Science, 368(6491), 2020.
  * [23] Serina Chang, Emma Pierson, Pang Wei Koh, Jaline Gerardin, Beth Redbird, David Grusky, and Jure Leskovec.  Mobility network models of covid-19 explain inequities and inform reopening.  Nature, pages 1–8, 2020.
  * [24] Paul Mozur, Raymond Zhong, and Aaron Krolik.  In coronavirus fight, china gives citizens a color code, with red flags.  New York Times, 1, 2020.
  * [25] Qun Li, Xuhua Guan, Peng Wu, Xiaoye Wang, Lei Zhou, Yeqing Tong, Ruiqi Ren, Kathy SM Leung, Eric HY Lau, Jessica Y Wong, et al.  Early transmission dynamics in wuhan, china, of novel coronavirus–infected pneumonia.  New England Journal of Medicine, 2020.
  * [26] Fei Zhou, Ting Yu, Ronghui Du, Guohui Fan, Ying Liu, Zhibo Liu, Jie Xiang, Yeming Wang, Bin Song, Xiaoying Gu, et al.  Clinical course and risk factors for mortality of adult inpatients with covid-19 in wuhan, china: a retrospective cohort study.  The lancet, 2020.
  * [27] Lorenzo Isella, Juliette Stehlé, Alain Barrat, Ciro Cattuto, Jean-François Pinton, and Wouter Van den Broeck.  What’s in a crowd? analysis of face-to-face behavioral networks.  Journal of theoretical biology, 271(1):166–180, 2011.
  * [28] Biao Tang, Xia Wang, Qian Li, Nicola Luigi Bragazzi, Sanyi Tang, Yanni Xiao, and Jianhong Wu.  Estimation of the transmission risk of the 2019-ncov and its implication for public health interventions.  Journal of clinical medicine, 9(2):462, 2020.
  * [29] Julien Riou and Christian L Althaus.  Pattern of early human-to-human transmission of wuhan 2019 novel coronavirus (2019-ncov), december 2019 to january 2020.  Eurosurveillance, 25(4):2000058, 2020.
  * [30] Shi Zhao, Qianyin Lin, Jinjun Ran, Salihu S Musa, Guangpu Yang, Weiming Wang, Yijun Lou, Daozhou Gao, Lin Yang, Daihai He, et al.  Preliminary estimation of the basic reproduction number of novel coronavirus (2019-ncov) in china, from 2019 to 2020: A data-driven analysis in the early phase of the outbreak.  International journal of infectious diseases, 92:214–217, 2020\.
  * [31] N Imai, I Dorigatti, A Cori, C Donnelly, C Riley, and NM Ferguson.  Report 2: Estimating the potential total number of novel coronavirus cases in wuhan city, china. 22 january 2020-imperial college london. who collaborating centre for infectious disease modelling. mrc centre for global infectious disease analysis, j-idea, imperial college london, uk.
  * [32] Zhidong Cao, Qingpeng Zhang, Xin Lu, Dirk Pfeiffer, Zhongwei Jia, Hongbing Song, and Daniel Dajun Zeng.  Estimating the effective reproduction number of the 2019-ncov in china.  MedRxiv, 2020.
  * [33] Joseph T Wu, Kathy Leung, and Gabriel M Leung.  Nowcasting and forecasting the potential domestic and international spread of the 2019-ncov outbreak originating in wuhan, china: a modelling study.  The Lancet, 395(10225):689–697, 2020.
  * [34] Mingwang Shen, Zhihang Peng, Yanni Xiao, and Lei Zhang.  Modelling the epidemic trend of the 2019 novel coronavirus outbreak in china.  BioRxiv, 2020.
  * [35] Tao Liu, Jianxiong Hu, Min Kang, Lifeng Lin, Haojie Zhong, Jianpeng Xiao, Guanhao He, Tie Song, Qiong Huang, Zuhua Rong, et al.  Transmission dynamics of 2019 novel coronavirus (2019-ncov).  2020\.
  * [36] Jonathan M Read, Jessica RE Bridgen, Derek AT Cummings, Antonia Ho, and Chris P Jewell.  Novel coronavirus 2019-ncov: early estimation of epidemiological parameters and epidemic predictions.  MedRxiv, 2020.
  * [37] Maimuna Majumder and Kenneth D Mandl.  Early transmissibility assessment of a novel coronavirus in wuhan, china.  China (January 23, 2020), 2020.
  * [38] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * [39] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In 5th International Conference on Learning Representations (ICLR), 2016.
  * [40] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In 6th International Conference on Learning Representations (ICLR), 2017.
  * [41] Donald J Wheeler and David S Chambers.  Understanding statistical process control.  uspc, 1992.
  * [42] Thomas J DiCiccio and Bradley Efron.  Bootstrap confidence intervals.  Statistical science, pages 189–212, 1996.

Figure 1: Identifying Asymptomatic and Presymptomatic COVID-19 Infections on
Contact-tracing Networks. (a) The COVID-19 state transition of an individual.
A susceptible will become either asymptomatic or presymptomatic after being
infected. An asymptomatic patient will further turns to the symptomatic state
after an incubation period, while an presymptomatic patient will never exhibit
any symptoms until the recover. (b) Illustration of asymptotic/presymptomatic
node identification problem. In a contact-tracing network, only a small
fraction of the nodes’ states are known (marked with color), while the hidden
asymptomatic/presymptomatic individuals within the population (marked with
grey) are potential spreaders. Our purpose is to find
asymptomatic/presymptomatic infected individuals in the population using the
contact-tracing network and the information of known confirmed cases. (c)
Diagram of the proposed method. The state transition of an unknown node $k$ is
modeled as a Markov process, i.e., a vector $\mathbf{Z}_{k}$ where the
elements represent the probabilities of different infection stages in (a). The
specific value of the vector $\mathbf{Z}_{k}(t+1)$ at $t+1$ is determined by
the infection status of known nodes at $t$ and the structure of the contact
network. (d) After iterations over the whole network, each unknown node will
be assigned with an infection indicator according to the eventual values of
its state vector $\mathbf{Z}_{k}$, which represents the risk of being
infected. Figure 2: Empirical and Simulated Validation of the Proposed Model.
(a) The number of people in each of the 5 states in the simulation process of
COVID-19 spreading on the ISA network. At $T=0$, we select three nodes with
the maximum degree in the ISA network as the initial infected spreaders. Dash
lines represent the theory values calculated by Eqns.(6-10). Dots represent
the average value of 1000 simulations. (b) The theoretical probability vs.
numerical frequency of each individual being in various states on $T=10$ days.
Each dot corresponds to a certain state of a node in the ISA network while the
errorbar is the 95% confidence interval obtained by the bootstrapping method
[42]. (c) The topological structure of a real COVID-19 spreading network in
Singapore (Singapore A), where dots are patients and curves are contacts
between patients (see Sec. SI 1 for the description of the network). The
points on the timeline indicate the date of the patient’s presence. (d) The
topological structure of another network of Singapore (Singapore B). (e)
Relationship between the individual symptomatic time and the estimated
infection probability in Singapore A. The network has a total of $T=30$, from
January 23, 2020 to February 21, 2020. Here we utilize the information of
infections from the first two different time points as known set to infer the
rest nodes’ states in the network by Eqns. (1-9). Using the obtained state
vector of each unknown node, we rank them according to the infection
probability and compare with it’s real symptomatic time. Since all patients
are symptomatic in the dataset, the rank is based on $C_{i}(t)-A_{i}(t)$. The
line denotes the linear fitted result and the shaded area denotes the 95%
confidence interval. (f) Similar to (e), the value of the rank of infection
probability vs. symptomatic time in Singapore B. The network has a total of
$T=40$, from February 11, 2020 to March 21, 2020. We use infections who got
infected the first two different time points for training.

Figure 3: Screening Performance Assessment. (a-d) Performance of the dynamic
screening method on the ISA network compared with other machine-learning
baselines (see details in Sec. SI 3). (a) The accuracy vs. rank of infection
(divided by the network size). The accuracy is defined as the proportion of
non-Susceptible individuals in the ranking list. Since all nodes we screen at
$T$ do not have symptoms, here we use the value of $C_{i}(t)-I_{i}(t)$ to rank
these nodes. (b) Similar to (a), the relative accuracy of the machine-
learning-based algorithms, which is the ratio between the accuracy of our
proposed algorithm and other algorithms. (c) The relationship between the rank
of infection and the recall rate (i.e., the proportion of successfully
identified non-Susceptible individuals to those in the whole network. (d)
Similar to (c), the relative recall rate of the machine-learning-based
algorithms. (inset) Recall rate of the static algorithms on the 1-step
neighbor subnetwork and on the whole network (see SI Figures 14-17), compared
with the dynamic algorithm. (e-h) Performance of the dynamic screening method
with incomplete network information. We randomly remove a fraction of links in
the ISA network (see the result of another social network in Sec. SI 5) and
then employ the proposed screening schemes on the remaining network. (e) The
relationship between the accuracy and the ranking value of the infection
probability with different proportions of the removed edges. Here we normalize
the ranking value to range [0,1] by dividing it with the total number of
individuals who have been screened. (f) Accuracy of the dynamic screening
method vs. the proportion of the removed edges by measuring the infection rank
with different proportions. For example, top 20% rank means the 20% nodes with
the highest infection possibility which is equal to 0.2 of norlized rank of
infection in (e). (g) The dependencies of recall rate of the dynamic screening
method on the infection rank. (h) Recall vs. the remaining edges. Figure 4:
Containment Effectiveness of the Proposed Scheme on Simulated COVID-19
Spreading. From $T=10$ of COVID-19 spreading simulation on the ISA network, we
conduct different approaches separately to contain the pandemic, including the
proposed method (i.e. Dynamic Containment), the Infection Containment and the
Neighbor Containment (see Sec. SI 4 for details). For the Neighbor Containment
and the proposed method, we select a total of $2\%N$ of individuals to screen
and then quarantine those tested positive at each time step. (a) Visualization
of infected population without conducting any contain scheme and (b-d) the 3
different control schemes). (e) The number of daily new infections in
different control schemes. The value of each curve is divided by the highest
point at that time to normalize to [0,1]. (f) The cumulative number of
infections corresponding to (e). (g-h) The performance of our control scheme
under an incomplete network where a fraction of links are randomly removed in
the detection process.


# Audio Adversarial Examples: Attacks Using Vocal Masks

Tay Kai Yuan, Ng Hui Xian Lynnette, Chua Wei Han,
Lucerne Loke, Ye Danqi, Chua Wan Jun Melissa

###### Abstract

We construct audio adversarial examples on automatic Speech-To-Text systems .
Given any audio waveform, we produce an another by overlaying an audio vocal
mask generated from the original audio. We apply our audio adversarial attack
to five SOTA STT systems: DeepSpeech, Julius, Kaldi, wav2letter@anywhere and
CMUSphinx. In addition, we engaged human annotators to transcribe the
adversarial audio. Our experiments show that these adversarial examples fool
State-Of-The-Art Speech-To-Text systems, yet humans are able to consistently
pick out the speech. The feasibility of this attack introduces a new domain to
study machine and human perception of speech.

##  1 Introduction

With the advent of virtual assistants like Google Assistant, Apple’s Siri and
Amazon’s Alexa, more attention has been brought to the space of Speech-To-Text
(STT), where natural language commands are converted into computer texts.
These texts carry information from the natural language command,
authenticating the virtual assistant to execute an action, such as delivering
news, even order products from an online store [Mari, 2019].

Adversarial machine learning is a sub-field of machine learning that has
gained much attention in recent years. In adversarial machine learning,
malicious inputs exploit weaknesses in a trained model or training regime to
produce undesired and unforeseen behaviors. Goodfellow et. al. [Goodfellow et
al., 2014] proposed the Fast Gradient Sign Method for generating adversarial
inputs on image classifiers. Jang et. al. [Jang et al., 2017] further
perfected gradient based techniques by proposing a technique that could
generate minimal perturbations on images in relation to its features,
demonstrating the ease of exploitation of image classifiers.

Most of the research of adversarial machine learning has been centered around
the domain of computer vision, and very few techniques are available to create
adversarial examples in STT. Carlini and Wagner [Carlini and Wagner, 2018]
proposed a gradient based technique adopted from image classification domain
and applied it on audio samples, creating adversarial examples by iteratively
minimising perturbations introduced into the audio waveform, while ensuring
that the waveforms are transcribed as another message. Schoenherr et al.
[Schönherr et al., 2019] uses of psychoacoustic hearing properties to generate
perturbations to the audio, to create audio below the hearing threshold such
that the attack is inaudible to the listener. The generated audio renders the
transcription through Kaldi ineffective. Zhang et al. [Zhang et al., 2017]
introduced the dolphin attack that modulates voice on ultrasonic carriers to
achieve inaudible attack vectors. While it is validated on popular speech
recognition signals, this attack is costly to construct.

###  1.1 Perception of Speech

Just like how our brains fill in the gaps in our vision, our brains formulate
speech to allow us to understand the speaker better [King, 2007]. In a noisy
environment, if we pay enough attention to a speaker and his speech, it is
relatively easy for our brains to follow the words. Following this concept, as
long as any background noise is below a certain threshold in decibels, humans
should not fail to formulate a speaker’s speech[Darwin, 2007]. On the other
hand, STT systems often rely largely on several quantitative qualities of
speech in order to accurately transcribe speech. Lacking the language ability
to fill in words that were not heard clearly in an audio, STT systems may fail
to transcribe accurately in noisy situations.

Human speech sounds are produced by the different shapes of the vocal tract.
Mel-Frequency Cepstral Coefficients (MFCCs) [Muda et al., 2010] and vocal
masks are frequently used to represent the shape of the vocal tracts, which
manifests itself in the sort time power spectrum, generated by performing
Short-Time Fourier Transform (STFT) on the audio signal. STT systems learn
these quantitative representation of words on these audio waveforms to
transcribe subsequent audio files. Recent work on convolutional neural
networks on mel-frequency spectrograms have shown remarkable accuracy in voice
separation [Simpson et al., 2015][Ikemiya et al., 2016]. This is largely due
to extensive learning of features of vocal masks from Mel-Frequency
spectrograms[Lin et al., 2018], and how they differ from other noise, such as
background music. However, these properties allow adversaries to employ
specific attacks against these deep learning systems.

###  1.2 Contributions

In this paper, we propose a novel method of creating adversarial examples on
audio signals that attacks five State-Of-The-Art (SOTA) STT systems. These
adversarial examples are transcribed as a different message by Speech-To-Text
systems, while humans are able to decipher the speech in the signal, rendering
STT systems inadequate. We are able to achieve this by overlaying a vocal mask
on top of the original audio, making use of the inability of these neural
networks to differentiate a vocal mask from the original speech, resulting in
an average Word Error Rate (WER) of 0.64. In comparison, the same adversarial
audio can be transcribed by human annotators with an average WER of 0.28. This
is an end-to-end attack, that operates directly on the raw samples that are
used as inputs to the neural networks.

The tasks that this paper attempts are:

  1. 1.

Generation of adversarial audio examples at different decibel levels using
Mel-Frequency Cepstral Coefficients (MFCC) properties

  2. 2.

Generation of targeted adversarial audio examples using the Carlini-Wagner
audio attack [Carlini and Wagner, 2018]

  3. 3.

Transcription of adversarial audio examples using five SOTA speech-to-text
transcription neural networks.

  4. 4.

Comparing transcription output from neural networks with human transcription

The State-Of-The-Art Speech-To-Text Systems attacked in this paper are:

  1. 1.

DeepSpeech[Hannun et al., 2014]. An end-to-end speech recognition system built
upon Baidu’s DeepSpeech architecture. To enhance its capabilities, DeepSpeech
learns a function that is robust to background noise, reverberation and
speaker variation.

  2. 2.

Kaldi[Povey et al., 2011]. One of the oldest toolkit for Automatic Speech
Recognition (ASR), which integrates with Finite State Transducers and provide
extensive linear algebra support built upon the OpenBLAS library. Kaldi
supports Hidden Markov Models, Gaussian Mixture Models, and neural-network
based acoustic modelling.

  3. 3.

Wav2letter@anywhere[Pratap et al., 2020]. Uses ArrayFire tensor library and
flashlight machine learning library, enabling training of the LibriSpeech
dataset to be completed within minutes. The language model formation uses
Time-Depth Separable Convolutions, while the decoder module is a beam-search
decoder.

  4. 4.

Julius[Lee and Kawahara, 2009a]. A large vocabulary continuous speech
recognition decoder software based on n-gram and context-dependent hidden
markov models developed by the Japanese since 1997[Radenen and Artieres,
2012]. The decoding algorithm is based on a two-pass tree-trellis search that
incorporates major decoding techniques including tree lexicons, n-gram
factoring, enveloped beam search, and deep neural networks.

  5. 5.

CMUSphinx[Huggins Daines et al., 2006]. A collection of speech recognition
tools aiming to overcome the constraints of isolated words and small
vocabularies. Its acoustic model is based on Hidden Markov Models, and speech
constraints overcome through function-word-dependent phone models and
generalised triphone models[Kai-Fu, 1989].

To facilitate future work, we make our dataset available. We encourage the
reader to listen to our audio adversarial
examples111https://drive.google.com/open?id=1ixwuvPrk1H-hveX5HNSWWYGSzx-lWYnC.

##  2 Methodology

This transcription performance comparison of audio adversarial examples
between SOTA neural network-based STT models and human annotators is
empirically evaluated with a series of experiments. This section describes the
generation of audio adversarial examples and defines the threat model and
evaluation metric, which will subsequently be used for performance comparison
between the produced transcription and the original transcription.

###  2.1 Threat Model

Given an audio waveform $x$, target transcription $y$, and SOTA STT model $C$,
and the human transcription $D$, our task is to construct another audio
waveform $x^{\prime}=x+\delta$ so that $D(x^{\prime})=y$ but
$C(x^{\prime})\neq y$.

###  2.2 Distortion Metric

To quantify the distortion introduced by an adversarial audio, we use the
Decibels (dB) measure. dB is a logarithmic scale that measures the relative
loudness of an audio sample. This measure is a relative measure, hence we use
the original waveform $x$ as the reference point for the adversarial audio
$x_{1}$.

$dB(x_{1})=10\log\frac{S(x_{1})}{S(x)},$

where $S$ is the function that maps $x$ to the intensity of the soundwaves.

In this paper, most of the distortion is quieter than the original signal, the
distortion is thus a negative number.

This metric may not be a perfect measure of distortion, however the audio
distortion may be imperceptible to the humans, as we will show in our
experiments, where human annotators fare better than SOTA STT systems.

####  2.2.1 Accuracy Metric

In calculation of the accuracy of the transcription, we calculate the Word
Error Rate (WER) of the produced transcription against the original
transcription provided by the TIMIT dataset.

The WER is a common measure of accuracy in performance measurement in STT
systems. It is derived from the Levenshtein distance, which measures the
difference between a sequence of characters through the minimum number of
edits. By measuring the difference in words, transcribed word sequence can
have different lengths from the ground truth. However, this measure does not
detail the nature of transcription errors. In our experiments, we calculate
the Levenshtein distance using the Wagner-Fisher algorithm [Wagner and
Fischer, 1974]. It should be noted that the lower the WER, the better the
transcription.

$WER=\frac{S+D+I}{N},$

where $S$ is the number of substitutions, $D$ is the number of deletions, $I$
is the number of insertions, and $N$ is the number of words in the reference.

###  2.3 Dataset

We used the DARPA-TIMIT Acoustic-Phonetic Continuous Speech Corpus [Garofolo
et al., 1993] for construction of adversarial audio and experiments. This
dataset contains time-aligned transcriptions of phonetically rich spoken
American English sentences. A hand-verified transcription of the corpus is
made available, which serves during the calculation of the model accuracy.
This dataset is suitable for our studies as we are studying the spoken speech
and our human annotators are well-versed in the English language.

####  2.3.1 Generation of obfuscated audio

We generate obfuscated audio with the python library Pydub [Jiaaro, 2016].
Pydub’s AudioSegment object provides several methods that allow easy
manipulation of audio, including reading and writing audio files, as well as
reversing audio files and overlaying them on other audio files.

Let $x$ be the original audio waveform, and $\delta$ be the adversarial audio
signal generated using AudioSegment as an attack to the original waveform. We
generate $\delta$ by reversing $x$. We then overlay the audio $\delta$ on to
$x$, that is, $x^{\prime}=x+\delta$.

As decibels is a measure of the intensity of the sound, $dB(\delta)=dB(x)$. We
generate five adversarial audio waveforms by varying the intensity of the
sound, each time decreasing the intensity of $\delta$. The waveforms shall be
represented as: $dB(\delta_{p})=dB(\delta)-p$, where $p$ is the amount of
decibels that $\delta_{p}$ differs from $\delta$. For example, if the
adversarial audio was decreased by 5 decibels, $dB(\delta_{-5})=dB(\delta)-5$.

####  2.3.2 Model set up

We constructed the experimental set up of the neural network models as
follows:

  1. 1.

DeepSpeech: Used the provided pre-trained English model, an n-gram language
model trained from a corpus of 220 million phrases with a vocabulary of
495,000 words. This was trained on the LibriSpeech[Panayotov et al., 2015],
Fisher[Cieri et al., 2004], Switchboard[Godfrey et al., 1992] and Common Voice
English[Ardila et al., 2019] datasets, for 233,784 steps where the best
validation loss were selected at the end of 75 epochs.

  2. 2.

Julius: Used the provided pre-trained n-gram language model, in a hybrid Deep
Neural Network with Hidden Markov Model based architecture (LM+DNN-HMM). The
model with a 262,000 word dictionary and 32 bit Language Model[Lüscher et al.,
2019][Lee and Kawahara, 2009b].

  3. 3.

Kaldi: Used the Kaldi pre-trained Aspire Chain Model[Kaldi, 2016] with already
compiled HCLG sequence of phoneme decoding graph for inference, trained on
Fisher English[Cieri et al., 2004]. HCLG is a hidden-markov finite state
transducer representing the lexicon, grammar and phonetic contexts.

  4. 4.

Wav2letter@anywhere: Used the provided inference platform that pre-trains
BERT[Devlin et al., 2018] models with Librispeech dataset[Panayotov et al.,
2015].

  5. 5.

CMUSphinx: Used the provided inference platform on python bindings
PocketSphinx [Huggins Daines et al., 2006], with the pre-trained US English
constructed from Wall Street Journal data using hidden-markov finite state
transducers.

####  2.3.3 Baseline Adversarial Attack

We used the Carlini-Wagner (CW) audio adversarial attack as the baseline
[Carlini and Wagner, 2018] audio adversarial attack. This attack adds a small
perturbation that are quieter than the original signal to the original audio
which changes the transcribed result when passed through DeepSpeech. We show
that our audio attacks surpass the CW attack.

####  2.3.4 Human Audio Transcription

We engaged seven people from ages 25-30 to transcribe the audio for us. Each
person transcribed 34 audio files of a single type of obfuscation. These audio
files were randomly chosen from 2 dialect groups of the DARPA-TIMIT corpus’
test directory. As the audio files were primarily in English, to reduce
problems with understanding the text, we engaged people whose first spoken and
written language is English. We asked the annotators to listen to the samples
via headphones. The task was to type all words of the audio sample into a
blank text field without assistance from auto-complete or grammar and spell-
checking. Annotators often repeated the audio samples in order to enter a
complete sentence. In a post-processing phase, we removed symbols and new
lines from the transcript before calculating the WER.

###  2.4 Results and Analysis

We present the mean WER for the various transcription systems in Table 1.

We note that DeepSpeech has a WER that is greater than 1.0 because there are
more words than the number of words in the reference speech. This is likely
due to the fact that DeepSpeech was trained on a dataset that has more words
than the TIMIT dataset. In terms of audio adversarial attacks, we note that
the attack of $x+\delta_{-15}$ produces the best balance between fooling
neural STT systems and human identification of the audio. At the same time,
$x+\delta{-15}$ has a higher WER than $x+\text{CW}$, proving our attack to be
more effective at fooling neural network systems as compared to the baseline
Carlini-Wagner attack.

Audio Files |  DeepSpeech |  Julius |  Kaldi |  wav2letter@
anywhere |  CMUSphinx |  Humans
---|---|---|---|---|---|---
$x$ |  0.13 (0.15) |  0.76 (0.31) |  0.32 (0.20) |  0.16 (0.17) |  0.35 (0.31) |  0.07 (0.10)
$x+\delta_{0}$ |  1.03 (0.26) |  0.99 (0.19) |  0.90 (0.18) |  0.87 (0.22) |  1.26 (0.40) |  0.77 (0.36)
$x+\delta_{-5}$ |  0.82 (0.27) |  0.93 (0.17) |  0.77 (0.22) |  0.65 (0.22) |  1.02 (0.40) |  0.36 (0.30)
$x+\delta_{-10}$ |  0.63 (0.35) |  0.85 (0.26) |  0.62 (0.27) |  0.42 (0.23) |  0.94 (0.41) |  0.28 (0.29)
$x+\delta_{-15}$ |  0.35 (0.28) |  0.78 (0.29) |  0.48 (0.28) |  0.26 (0.20) |  0.68 (0.32) |  0.10 (0.11)
$x+\delta_{-20}$ |  0.23 (0.24) |  0.79 (0.29) |  0.37 (0.23) |  0.18 (0.18) |  0.63 (0.33) |  0.08 (0.11)
$x+\text{CW}$ |  0.49 (0.31) |  0.84 (0.26) |  0.49 (0.26) |  0.28 (0.25) |  0.85 (0.36) |  0.10 (0.14)
Table 1: Mean (Standard Deviation) WER of transcriptions for our experiments

##  3 Discussion

Upon investigation, SOTA STT systems fail to transcribe our attacks
accurately, which can be represented in terms of the attack audio’s similarity
with the original, and audio signal properties.

####  3.0.1 Cosine Similarity Analysis

We posit that our attack can successfully fool STT systems due to the
similarity of the attack to its original audio. We note that the mean
normalised cosine similarity of our attacked audio and the original is very
close to 1.0, surpassing the CW attacked audio. Table 2 shows the similarities
of our attacks.

Audio File |  Mean |  Standard Deviation
---|---|---
$x+\delta_{0}$ |  0.999930 |  0.0000337
$x+\delta_{-5}$ |  0.999962 |  0.0000216
$x+\delta_{-10}$ |  0.999979 |  0.0000142
$x+\delta_{-15}$ |  0.999989 |  0.0000095
$x+\delta_{-20}$ |  0.999994 |  0.0000063
$x+\text{CW}$ |  0.999961 |  0.0000229
Table 2: Cosine similarities between the original audio and the adversarial
audio

####  3.0.2 Audio Signal Analysis

We analyse the original signal and our adversarial audio signal using audio
signal analysis methods: Fast Fourier Transform (FFT) and Spectrograms.

Figure 1 presents the spectrograms and FFT plots of our audio adversarial
examples. We note that from the spectrograms generated from $x+\text{CW}$, the
CW attack possess clear vocal masks of the original audio, which makes it
possible to retrieve some semblance of the original audio using a vocal mask
technique, built upon computer vision principles. Our attacks obfuscate the
original audio by appearing as the original audio, yet in the opposite
direction, hence are able to fool techniques that employ vocal mask analyses.
In the case of the FFT plots, the plot generated from $x+\text{CW}$ is hugely
similar to the plot of $x$, which means it is possible to retrieve $x$ from
the adversarial audio. However, our adversarial audio has different FFT forms
compared to $x$, since our attack introduces additional audio signals, thereby
preventing attempts to recover the original audio.

Original audio $x$

$x+\text{CW}$

$x+\delta_{0}$

$x+\delta_{-15}$

Figure 1: Mel-frequency spectrograms and FFT overlays of audio samples. We
observe that the original vocal masks are preserved in the CW attacks, but our
attacks generate additional vocal masks that can effectively fool vocal mask
analysis methods.

####  3.0.3 Perception of Speech

Our results reflect that humans are better at the transcription task,
regardless of obfuscation. Humans have innate experience with language and are
able to predict words in a sentence even if they did not hear it from the
audio. This phenomenon is known as auditory perceptual restoration, where the
brain fills in missing information in areas where noise obstructs portions of
sounds [Bidelman and Patro, 2016][King, 2007]. Commonly observed in a
conversation with loud background noise, this phonemic restoration effect
occurs where the brain restores sounds missing from a speech signal. This
effect is typically observed when missing phonemes in an auditory signal are
replace with noises that masks the original phonemes, creating an auditory
ambiguity[Repp, 1992][Groppe et al., 2010]. In the case of our experiments,
our attack audio masks some original audio, which causes perceptual
restoration to kick in, where the humans perform better transcriptions due to
this phenomenon.

####  3.0.4 Future Work

From this work, we acknowledge that humans possess an innate language ability
which allowed them to perform better in the transcription task. We posit that
future STT systems can include predictive abilities, such as introducing
generative models with attention gates that govern which audio features have
more importance, thereby allowing STT systems to predict transcriptions and at
the same time inference them directly from the audio files.

##  4 Conclusion

We demonstrate a novel method of creating audio adversarial examples by
reversing the audio signal and overlaying it on the original speech. We
present evidence that these adversarial examples render vocal masks obsolete
due to the inability to identify the reversed audio from the true audio. Our
experiments show that these adversarial examples fool State-Of-The-Art Speech-
To-Text systems, yet humans are able to consistently pick out the speech. We
hope that future work will continue to investigate audio adversarial examples,
and improve STT systems with predictive language abilities that humans
possess.

## References

  * [Ardila et al., 2019]  Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber.  2019\.  Common voice: A massively-multilingual speech corpus.
  * [Bidelman and Patro, 2016]  Gavin M. Bidelman and Chhayakanta Patro.  2016\.  Auditory perceptual restoration and illusory continuity correlates in the human brainstem.  Brain Research, 1646:84 – 90.
  * [Carlini and Wagner, 2018]  Nicholas Carlini and David A. Wagner.  2018\.  Audio adversarial examples: Targeted attacks on speech-to-text.  2018 IEEE Security and Privacy Workshops (SPW), pages 1–7.
  * [Cieri et al., 2004]  Christopher Cieri, David Miller, and Kevin Walker.  2004\.  The fisher corpus: a resource for the next generations of speech-to-text.  In LREC.
  * [Darwin, 2007]  C. J. Darwin.  2007\.  Listening to speech in the presence of other sounds.  Philosophical Transactions of the Royal Society B: Biological Sciences, 363:1011 – 1021.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * [Garofolo et al., 1993]  J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren.  1993\.  Darpa timit acoustic phonetic continuous speech corpus cdrom.
  * [Godfrey et al., 1992]  J. J. Godfrey, E. C. Holliman, and J. McDaniel.  1992\.  Switchboard: telephone speech corpus for research and development.  In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517–520 vol.1, March.
  * [Goodfellow et al., 2014]  Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  2014\.  Explaining and harnessing adversarial examples.
  * [Groppe et al., 2010]  David Groppe, Marvin Choi, Tiffany Huang, Joseph Schilz, Ben Topkins, Thomas Urbach, and Marta Kutas.  2010\.  The phonemic restoration effect reveals pre-n400 effect of supportive sentence context in speech perception.  Brain research, 1361:54–66, 11.
  * [Hannun et al., 2014]  Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng.  2014\.  Deep speech: Scaling up end-to-end speech recognition.  ArXiv, abs/1412.5567.
  * [Huggins Daines et al., 2006]  David Huggins Daines, M. Kumar, A. Chan, A.W. Black, M. Ravishankar, and Alexander Rudnicky.  2006\.  Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices.  volume 1, pages I – I, 06.
  * [Ikemiya et al., 2016]  Yukara Ikemiya, Katsutoshi Itoyama, and Kazuyoshi Yoshii.  2016\.  Singing voice separation and vocal f0 estimation based on mutual combination of robust principal component analysis and subharmonic summation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(11):2084–2095, Nov.
  * [Jang et al., 2017]  Uyeong Jang, Xi Wu, and Somesh Jha.  2017\.  Objective metrics and gradient descent algorithms for adversarial examples in machine learning.  In Proceedings of the 33rd Annual Computer Security Applications Conference, ACSAC 2017, page 262–277, New York, NY, USA. Association for Computing Machinery.
  * [Jiaaro, 2016]  Jiaaro.  2016\.  Pydub by jiaaro.  Last Accessed: 15 January 2020.
  * [Kai-Fu, 1989]  Lee Kai-Fu.  1989\.  The development of the sphinx system.  In Automatic Speech Recognition, volume 62. Springer US.
  * [Kaldi, 2016]  Kaldi.  2016\.  Aspire chain model.  https://kaldi-asr.org/models/m1.
  * [King, 2007]  Andrew King.  2007\.  Auditory neuroscience: Filling in the gaps.  Current biology : CB, 17:R799–801, 10.
  * [Lee and Kawahara, 2009a]  Akinobu Lee and Tatsuya Kawahara.  2009a.  Recent development of open-source speech recognition engine julius.  Proceedings of the 2009 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 01.
  * [Lee and Kawahara, 2009b]  Akinobu Lee and Tatsuya Kawahara.  2009b.  Recent development of open-source speech recognition engine julius.
  * [Lin et al., 2018]  Kin Wah Edward Lin, T. BalamuraliB., Enyan Koh, Simon Lui, and Dorien Herremans.  2018\.  Singing voice separation using a deep convolutional neural network trained by ideal binary mask and cross entropy.  Neural Computing and Applications, pages 1–14.
  * [Lüscher et al., 2019]  Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney.  2019\.  Rwth asr systems for librispeech: Hybrid vs attention.  Interspeech 2019, Sep.
  * [Mari, 2019]  Alex Mari.  2019\.  Voice commerce: Understanding shopping-related voice assistants and their effect on brands.  10\.
  * [Muda et al., 2010]  Lindasalwa Muda, Mumtaj Begam, and Irraivan Elamvazuthi.  2010\.  Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques.  J Comput, 2, 03.
  * [Panayotov et al., 2015]  V. Panayotov, G. Chen, D. Povey, and S. Khudanpur.  2015\.  Librispeech: An asr corpus based on public domain audio books.  In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, April.
  * [Povey et al., 2011]  Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely.  2011\.  The kaldi speech recognition toolkit.  In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. IEEE Signal Processing Society, December.  IEEE Catalog No.: CFP11SRW-USB.
  * [Pratap et al., 2020]  Vineel Pratap, Qiantong Xu, Jacob Kahn, Gilad Avidov, Tatiana Likhomanenko, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronon Collobert.  2020\.  Scaling up online speech recognition using convnets.  https://research.fb.com/wp-content/uploads/2020/01/Scaling-up-online-speech-recognition-using-ConvNets.pdf.
  * [Radenen and Artieres, 2012]  Mathieu Radenen and Thierry Artieres.  2012\.  Contextual hidden markov models.  pages 2113–2116, 03.
  * [Repp, 1992]  Bruno H. Repp.  1992\.  Perceptual restoration of a “missing” speech sound: Auditory induction or illusion?  volume 51, page 14–32, January.
  * [Schönherr et al., 2019]  Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa.  2019\.  Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding.  In Network and Distributed System Security Symposium (NDSS).
  * [Simpson et al., 2015]  Andrew J. R. Simpson, Gerard Roma, and Mark D. Plumbley.  2015\.  Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network.
  * [Wagner and Fischer, 1974]  Robert A. Wagner and Michael J. Fischer.  1974\.  The string-to-string correction problem.  J. ACM, 21(1):168–173, January.
  * [Zhang et al., 2017]  Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu.  2017\.  Dolphinattack: Inaudible voice commands.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, page 103–117, New York, NY, USA. Association for Computing Machinery.