# Quantum (dis)charge of black hole interiors

Christiane Klein<EMAIL_ADDRESS>Jochen Zahn<EMAIL_ADDRESS>leipzig.de Stefan Hollands<EMAIL_ADDRESS>Institut für
Theoretische Physik, Universität Leipzig,
Brüderstraße 16, 04103 Leipzig, Germany

###### Abstract

We analyze the “vacuum” polarization induced by a quantum charged scalar field
near the inner horizon of a charged (Reissner-Nordström-de Sitter) black hole
in quantum states that start out as regular states near an initial Cauchy
surface. Contrary to the outer (i.e. event-) horizon, where polarization
effects lead to a discharge, we find that near an inner horizon, the
transversal component of the expected current density can have either sign
depending on the black hole and field parameters. Thus, the inner horizon can
be charged or discharged. But we find that it is always discharged close to
extremality thus driving the black hole interior away from this critical
point. Furthermore, we find that quantum effects dominate in that the strength
of the blow up of the quantum current at the inner horizon is state-
independent and stronger than that of the current of a classical solution.

##  I Introduction

It is well-known that black hole (BH) interiors, such as e.g. in the Kerr- or
Reissner-Nordström (-deSitter) spacetimes, pose interesting questions
regarding determinism. In fact, a common feature of these spacetimes is the
existence of an “inner” or “Cauchy” horizon $\mathcal{CH}$, beyond which the
evolution of classical or quantum fields cannot be predicted from their
initial data on a Cauchy surface $\Sigma$ at early times, see Fig. 1.

The strong cosmic censorship (sCC) conjecture [1] proposes that the issue is
academic because local observables are expected to diverge at $\mathcal{CH}$.
A more precise formulation of this idea due to [2] states that sCC holds if
the stress tensor of a classical field, such as a scalar or a linearized
gravitational field, is locally not integrable near $\mathcal{CH}$ for generic
smooth initial data on $\Sigma$, thus preventing one from continuing the
solution beyond $\mathcal{CH}$. Thus, one must understand the strength of the
singularity of physical fields near $\mathcal{CH}$ to decide whether sCC saves
determinism or not.

Figure 1: Penrose diagram for Reissner-Nordström-de Sitter spacetime.
${\mathrm{I}}$ and ${\mathrm{III}}$ constitute the exterior region, with
${\mathrm{III}}$ the region beyond the cosmological horizon $\mathcal{H}_{c}$,
i.e., the region out of causal contact with the BH. Regions ${\mathrm{II}}$
and ${\mathrm{IV}}$ are the BH interior, which are separated from the exterior
by the event horizon $\mathcal{H}^{R}$ and from each other by the Cauchy
horizon $\mathcal{CH}^{R}$. The wiggled line represents the curvature
singularity. The green line indicates a Cauchy surface $\Sigma$ for the region
${\mathrm{I}}\cup{\mathrm{II}}\cup{\mathrm{III}}$, whereas the red lines
indicate the direction of the electric field on time-slices in the interior
region. The blue arrows indicate the range of the coordinates $u$, $v$
introduced below.

Estimates for the degree of divergence of an (uncharged) classical Klein-
Gordon (KG) field $\Phi$ near the Cauchy horizon of a subextremal Reissner-
Nordström-deSitter (RNdS) spacetime were suggested already in the 1990’s [3,
4, 5] and have been revisited in recent years [6, 7, 8, 9, 10, 11]. In
particular, [9] have characterized the singular behavior of an arbitrary
solution $\Phi$ arising from smooth initial data on $\Sigma$ by its membership
in the Sobolev space $H^{\beta+\frac{1}{2}}$ near the Cauchy horizon
$\mathcal{CH}$. Here, $\beta=\frac{\alpha}{\kappa_{-}}$ where $\alpha$ is the
spectral gap of quasinormal modes and $\kappa_{-}$ is the surface gravity of
$\mathcal{CH}$. Interestingly, by tracing the dependency of $\beta$ on the BH
mass $M$, charge $Q$, and cosmological constant $\Lambda$, it was found that
$\beta$ can become $>\frac{1}{2}$ [11]. This implies in particular that the
stress tensor of the classical field $\Phi$ is locally integrable (meaning
roughly that $T_{VV}\sim V^{-2+2\beta}$ in terms of a Kruskal coordinate
locating $\mathcal{CH}^{R}$ at $V=0$, see Fig. 1), constituting a violation of
sCC.

Given this state of affairs it is of interest to understand whether
expectation values of quantum fields diverge near $\mathcal{CH}$ and how the
degree of divergence compares to classical fields. In fact, it was already
argued heuristically in the 1970’s [12] that the expected stress tensor in a
generic quantum state should diverge at the Cauchy horizon. This was recently
proven [13] for the real KG field in RNdS, in the following sense: In any
state $\Psi$ which is regular (i.e. “Hadamard”) in a neighborhood of a Cauchy
surface $\Sigma$ as in Fig. 1, one has, for $\beta>\frac{1}{2}$,

${\langle T_{VV}\rangle}_{\Psi}\sim CV^{-2},$ (1)

with $C$ a constant which does only depends on the parameters $M,Q,\Lambda$ of
the spacetime and the mass $\mu$ of the field, but not on the quantum state
$\Psi$. The state dependence only enters through sub-leading terms, which are
no more singular than the stress tensor for the classical field. The constant
$C$ has to be determined numerically and was found to be generically non-zero
[14] (see also previous work [15] on Reissner-Nordström and special states).
Interestingly, both signs of $C$ can occur. If we assume that the expected
stress tensor backreacts onto the metric via the semi-classical Einstein
equation

$G_{\nu\rho}+\Lambda g_{\nu\rho}=8\pi{\langle T_{\nu\rho}\rangle}_{\Psi}\,,$
(2)

then the two signs of $C$ correspond to distinct behavior (infinite stretching
resp. squeezing) of freely falling observers crossing $\mathcal{CH}$ [15, 13].

As the formation of charged BHs is only possible in the presence of charged
matter, it is actually more natural to consider a charged scalar field $\Phi$
with the usual minimal coupling $\nabla_{\nu}\to
D_{\nu}=\nabla_{\nu}-iqA_{\nu}$, where $q$ is the charge of the field. It is
known [16, 17, 18] that for certain values of $(M,Q,\Lambda)$ and $(\mu,q)$,
with $\mu$ the mass of the scalar field, there are classical instabilities
(exponentially growing modes) already in the exterior region I. Excluding such
unstable spacetimes, we have a positive spectral gap $\alpha>0$ and again
$\Phi\in H^{\beta+\frac{1}{2}}$ locally near $\mathcal{CH}$ [19]. The
dependence of $\beta=\alpha/\kappa_{-}$ on the parameters is influenced by the
field charge $q$. There is nevertheless a parameter range–though considerably
smaller than for the uncharged scalar field–for which $\beta>\frac{1}{2}$ [18,
19].

In the presence of a charged quantum field, backreaction should not only take
place via the semi-classical Einstein equation (2), but also via the semi-
classical Maxwell equation

$\nabla^{\rho}F_{\rho\nu}=-4\pi{\langle j_{\nu}\rangle}_{\Psi},$ (3)

due to vacuum polarization, i.e., an expectation value for the current
$j_{\nu}$. At the event horizon ${\mathcal{H}}^{R}$, this current is
responsible for the discharge of the BH via Hawking radiation [20, 21, 22].
While vacuum polarization always discharges the event horizon, its influence
on the Cauchy horizon is not obvious. As indicated in Fig. 1, the two
singularities have opposite charges $\pm Q$. It has been argued invoking the
Schwinger process [23] that pair creation in the interior (region
${\mathrm{II}}$) should discharge the Cauchy horizon, i.e., there should be a
net current from the left Cauchy horizon $(\mathcal{CH}^{L})$ to the right one
$(\mathcal{CH}^{R})$ [24, 25].

The arguments put forward are however not completely convincing: The BH
interior is not stationary, so the very notion of particle is ambiguous there.
Furthermore, already at the classical level the behavior of fields at the
$\mathcal{CH}$ is due to a very non-local effect: a competition between decay
(in region I) and blue shift (region II). Hence, Schwinger’s formula for pair
creation seems not applicable/relevant and a first principle calculation of
${\langle j_{\nu}\rangle}_{\Psi}$ at the Cauchy horizon is needed to settle
this important question. This is the main novelty presented in our letter.
Most interestingly, we find that the relevant component of the current,
${\langle j_{V}\rangle}_{\Psi}$, can have either sign at the Cauchy horizon,
depending on the parameters $Q,M,\Lambda$ of the spacetime and $\mu,q$ of the
field. It follows that, via back-reaction, the Cauchy horizon can also be
charged by quantum effects, contrary to naive expectations [24, 25].

More precisely, we argue that, in any quantum state $\Psi$ which is initially
Hadamard near $\Sigma$, the expectation value of the current diverges at
$\mathcal{CH}^{R}$, with leading divergence

${\langle j_{V}\rangle}_{\Psi}\sim C^{\prime}V^{-1},$ (4)

where $C^{\prime}$ is independent of $\Psi$ and can have either sign. The
state dependence again enters through subleading terms which behave as the
current of a classical field i.e. roughly as $j_{V}\sim V^{-1+2\beta}$. Hence,
as for the stress tensor, quantum effects dominate over classical effects
close to the Cauchy horizon.

##  II Geometric setup and backreaction

The metric $g$ and vector potential $A$ of the RNdS spacetime (Fig. 1) are
given in natural units $\hbar=c=G_{N}=4\pi\epsilon_{0}=1$ by

$\displaystyle g$
$\displaystyle=-f(r)\text{d}t^{2}+f(r)^{-1}\text{d}r^{2}+r^{2}\text{d}\Omega^{2}$
(5a) $\displaystyle f(r)$
$\displaystyle=-\frac{\Lambda}{3}r^{2}+1-\frac{2M}{r}+\frac{Q^{2}}{r^{2}}$
(5b) $\displaystyle A$ $\displaystyle=-\frac{Q}{r}\text{d}t,$ (5c)

with $\text{d}\Omega^{2}$ the area element of the unit sphere. The function
$f(r)$ has three positive roots $r_{c}>r_{+}>r_{-}$, corresponding to the
cosmological ($\mathcal{H}_{c}$), event ($\mathcal{H}$) and inner, or Cauchy,
horizon ($\mathcal{CH}$) of the BH.

A useful alternative radial coordinate is the tortoise coordinate $r_{*}$,
defined by $f(r)\text{d}r_{*}=\text{d}r$. One defines radial null coordinates
$v\equiv t+r_{*}$ and $u\equiv t-r_{*}$. Their ranges are indicated by blue
arrows in Fig. 1, pointing from $-\infty$ to $+\infty$. In order to extend the
metric smoothly across the horizons, we use the Kruskal coordinates
$U\equiv\mp e^{-\kappa_{+}u}$, $V_{c}\equiv-e^{-\kappa_{c}v}$, and
$V\equiv-e^{-\kappa_{-}v}$, where $\kappa_{i}=\tfrac{1}{2}|f^{\prime}(r_{i})|$
are the surface gravities on the corresponding horizons. For $U$, we use the
$-$ sign in the exterior, and the $+$ sign in the interior, and the
$V$-coordinates are defined in the exterior ($V_{c}$) and interior ($V$)
respectively. Note that $V=0$ on $\mathcal{CH}^{R}$ and that, by the tensor
transformation law, the constants $C,C^{\prime}$ in (1) and (4) which govern
the singular behavior of the stress tensor and the current near the inner
horizon are given by $C=\kappa_{-}^{-2}{\langle T_{vv}\rangle}_{\Psi}$ and
$C^{\prime}=-\kappa_{-}^{-1}{\langle j_{v}\rangle}_{\Psi}$, where the
expectation values are computed on the inner horizon. In the following, we
present our results in terms of ${\langle T_{vv}\rangle}_{\Psi}$ and ${\langle
j_{v}\rangle}_{\Psi}$.

Before presenting our arguments in favor of (4), let us briefly discuss
backreaction effects. We proceed heuristically and assume that the main
contribution to backreaction is spherically symmetric, as the leading
divergence in (4). Thus, for the metric $g$ and the field strength tensor $F$,
involving backreaction, we make the ansatz

$\displaystyle g$
$\displaystyle=-e^{\sigma}\text{d}u\text{d}v+r^{2}\text{d}\Omega^{2},$
$\displaystyle F$
$\displaystyle=-\frac{Q}{2r^{2}}e^{\sigma}\text{d}u\wedge\text{d}v,$ (6)

where $\sigma$, $r$, and $Q$ are functions of $u$ and $v$. By Gauß’ law,
$Q(u,v)$ is the charge contained within the sphere at $u,v=\text{const}$,
which has area $4\pi r^{2}$.

In a first approximation, we now use the expectation value of the current
obtained on the RNdS background and obtain, from the $v$ component of (3),

${\partial}_{v}Q=-4\pi r^{2}{\langle j_{v}\rangle}_{\Psi}.$ (7)

Hence, the sign of ${\langle j_{v}\rangle}_{\Psi}$, evaluated on the Cauchy
horizon, determines whether quantum effects discharge (${\langle
j_{v}\rangle}_{\Psi}>0$) or charge (${\langle j_{v}\rangle}_{\Psi}<0$) the BH
interior. Furthermore, using results of [15] on the influence of the
backreaction on $r$, we obtain for the behavior of the field strength
$\frac{Q}{r^{2}}$ near the Cauchy horizon

$\partial_{v}\frac{Q}{r^{2}}=-4\pi{\langle
j_{v}\rangle}_{\Psi}+8\pi\frac{Q}{r_{-}^{2}\kappa_{-}}{\langle
T_{vv}\rangle}_{\Psi},$ (8)

where in the second term on the r.h.s. we set $r$ and $Q$ to their background
value, corresponding to the weak backreaction regime. We see that even in the
case of discharge through vacuum polarization, ${\langle
j_{v}\rangle}_{\Psi}>0$, the field strength near the inner horizon might still
increase if ${\langle T_{vv}\rangle}_{\Psi}$ is large enough, so that the area
$4\pi r^{2}$ decreases sufficiently quickly.

##  III The scalar field and its current

The charged scalar field $\Phi$ obeys the KG equation

$\displaystyle\left[D_{\nu}D^{\nu}-\mu^{2}\right]\Phi$ $\displaystyle=0,$ (9)

The mode ansatz is

$\displaystyle\Phi_{\ell m}=(4\pi|\omega|)^{-1/2}r^{-1}Y_{\ell
m}(\theta,\phi)e^{-i\omega t}H_{\omega\ell}(r)\,,$ (10)

with $Y_{\ell m}(\theta,\phi)$ the spherical harmonics. Then, (9) reduces to a
one-dimensional problem

$\displaystyle\left[\partial_{r_{*}}^{2}+\left(\omega-\frac{qQ}{r}\right)^{2}-W\right]H_{\omega\ell}(r_{*})=0$
(11a) $\displaystyle
W=f(r)\left(\frac{\partial_{r}f(r)}{r}+\frac{\ell(\ell+1)}{r^{2}}+\mu^{2}\right)\,.$
(11b)

With gauge transformations of the form

$\displaystyle A$ $\displaystyle\to A+\frac{Q}{r_{0}}\,\text{d}t$
$\displaystyle\Phi$ $\displaystyle\to e^{i\tfrac{qQ}{r_{0}}t}\Phi$ (12)

we can set the potential to zero at any chosen $r_{0}$. If
$r_{0}\in\\{r_{+},r_{-},r_{c}\\}$, the solutions
$h_{\omega\ell}(r_{*},t)=e^{-i\omega t}H_{\omega\ell}(r_{*})$ will behave as
free waves in that gauge when approaching the corresponding horizon. We will
denote the gauge where $A=0$ on $\mathcal{H}_{c}$ by a $(c)$-, the gauge with
$A=0$ on $\mathcal{H}$ by a $(+)$\- and the gauge with $A=0$ on $\mathcal{CH}$
by a $(-)$-superscript .

Following standard procedures [26], we quantize the field $\Phi$ by choosing a
set of positive frequency mode solutions. It is convenient to choose these as
corresponding to the Unruh vacuum, which in turn corresponds to certain
initial conditions on
$\mathcal{H}_{c}^{-}\cup\mathcal{H}^{-}\cup\mathcal{H}^{L}$. There are two
types of modes, $h^{(c)\mathrm{in}}_{\omega\ell}\sim e^{-i\omega V_{c}}$ on
$\mathcal{H}_{c}^{-}$, and $\sim 0$ on $\mathcal{H}^{-}\cup\mathcal{H}^{L}$,
and $h^{(+)\mathrm{up}}_{\omega\ell}\sim e^{-i\omega U}$ on
$\mathcal{H}^{-}\cup\mathcal{H}^{L}$ and $\sim 0$ on $\mathcal{H}_{c}^{-}$.
The field operator is then given by an expansion in the positive frequency
solutions,

$\displaystyle\Phi(x)$
$\displaystyle=\int\limits_{0}^{\infty}\text{d}\omega\sum_{\lambda,\ell,m}\left(\Phi_{\omega\ell
m}^{\lambda}(x)a_{\omega\ell m}^{\lambda}+\Phi_{-\omega\ell
m}^{\lambda}(x)b_{\omega\ell m}^{\lambda\dagger}\right)\,,$ (13)

where the coefficients $a_{\omega\ell m}^{\lambda}$ and $b_{\omega\ell
m}^{\lambda}$ act as annihilation operators on the Unruh vacuum,
$a_{\omega\ell m}^{\lambda}|0\rangle_{\mathrm{U}}=b_{\omega\ell
m}^{\lambda}|0\rangle_{\mathrm{U}}=0$, and $\lambda$ runs over the types of
modes, $\mathrm{in}$ and $\mathrm{up}$. That this defines a proper quantum
field/state in the absence of classical instabilities, i.e., $\alpha>0$, can
be shown as in [13].

The observable driving backreaction onto the electromagnetic field is the
current density given by

$\displaystyle
j_{\nu}(x)=iq\left(\Phi(x)D_{\nu}\Phi^{*}(x)-\Phi^{*}(x)D_{\nu}\Phi(x)\right)\,.$
(14)

It is local and non-linear in the field, so it requires renormalization in
quantum field theory. To evaluate its expectation value in the state $\Psi$,
we proceed analogously to recent treatments of the stress-energy-tensor: As in
[13], the expectation value in the Unruh state ${\langle
j_{V}\rangle}_{\mathrm{U}}$ is already found to give the leading contribution
to (4), because the $\Psi$-dependent contributions are sub-leading. This can
be shown rigorously for $\beta>\frac{1}{2}$, analogously to [13], but is
expected to hold also for $\beta<\frac{1}{2}$ in a large class of states
(including all coherent states in the Unruh Fock space).

To evaluate ${\langle j_{v}\rangle}_{\mathrm{U}}$ we use a Hadamard point-
split renormalization, similar to the one performed in [27, 28, 15, 13] for
the stress tensor. By a suitable point-split prescription, all singularities
of the coinciding point limit vanish, so that Hadamard point-split
renormalization amounts to the subtraction of a finite part, which actually
vanishes on the horizons [29]. We then calculate the $v$-component of the
current on $\mathcal{CH}^{L}$, which coincides with the value on
$\mathcal{CH}^{R}$ because the state is stationary.

In order to evaluate the single-mode contributions, we express the Unruh modes
in terms of “Boulware modes”,
$\tilde{h}^{\mathrm{in},\mathrm{N}}_{\omega\ell}$ and
$\tilde{h}^{\mathrm{up},\mathrm{N}}_{\omega\ell}$. Here
$\mathrm{N}\in\\{{\mathrm{I}},{\mathrm{II}}\\}$ denotes the spacetime region
on which the modes are defined, c.f. Fig. 1. The modes are defined by
asymptotic data on their past characteristic surfaces, i.e.,
$\mathcal{H}^{-}\cup\mathcal{H}_{c}^{-}$ for region ${\mathrm{I}}$ and
$\mathcal{H}^{L}\cup\mathcal{H}^{R}$ for region ${\mathrm{II}}$. Concretely,
$\tilde{h}^{(c)\mathrm{in},{\mathrm{I}}}_{\omega\ell}\sim e^{-i\omega v}$ on
$\mathcal{H}_{c}^{-}$ and
$\tilde{h}^{(+)\mathrm{up},{\mathrm{I}}}_{\omega\ell}\sim e^{-i\omega u}$ on
$\mathcal{H}^{-}$ for region ${\mathrm{I}}$ and
$\tilde{h}^{(+)\mathrm{in},{\mathrm{II}}}_{\omega\ell}\sim e^{-i\omega v}$ on
$\mathcal{H}^{R}$ and
$\tilde{h}^{(+)\mathrm{up},{\mathrm{II}}}_{\omega\ell}\sim e^{-i\omega u}$ on
$\mathcal{H}^{L}$ for region ${\mathrm{II}}$ (with vanishing on the other
component of the characteristic surface implied). From the differential
equation (11a), and the relation of the modes, it follows that these modes
behave as

$\displaystyle\tilde{h}^{(+)\mathrm{in},{\mathrm{I}}}_{\omega\ell}$
$\displaystyle=\mathcal{T}^{\mathrm{I}}_{(\omega-\omega_{{\mathrm{I}}})\ell}\frac{\omega}{\omega-\omega_{{\mathrm{I}}}}e^{-i(\omega-\omega_{\mathrm{I}})v}\quad\text{
on }\mathcal{H}^{R}$ (15a)
$\displaystyle\tilde{h}^{(+)\mathrm{up},{\mathrm{I}}}_{\omega\ell m}$
$\displaystyle=\mathcal{R}^{\mathrm{I}}_{\omega\ell}e^{-i\omega
v}\qquad\qquad\qquad\text{ on }\mathcal{H}^{R}$ (15b)
$\displaystyle\tilde{h}^{(-)\mathrm{in},{\mathrm{II}}}_{\omega\ell m}$
$\displaystyle=\overline{\mathcal{T}^{{\mathrm{II}}}_{\omega\ell}}e^{-i(\omega-\omega_{{\mathrm{II}}})v}\qquad\qquad\text{
on }\mathcal{CH}^{L}$ (15c)
$\displaystyle\tilde{h}^{(-)\mathrm{up},{\mathrm{II}}}_{\omega\ell m}$
$\displaystyle=\mathcal{R}^{{\mathrm{II}}}_{\omega\ell}e^{-i(\omega-\omega_{{\mathrm{II}}})v}\qquad\qquad\text{
on }\mathcal{CH}^{L}\,,$ (15d)

where $\omega_{{\mathrm{I}}}=qQ(r_{+}^{-1}-r_{c}^{-1})$ and
$\omega_{{\mathrm{II}}}=qQ(r_{-}^{-1}-r_{+}^{-1})$. Here,
$\mathcal{T}^{\mathrm{N}}_{\omega\ell}$ and
$\mathcal{R}^{\mathrm{N}}_{\omega\ell}$ are the transmission and reflection
coefficients of the $\tilde{h}^{\mathrm{up},\mathrm{N}}_{\omega\ell}$-modes.
The scattering coefficients of the $\mathrm{in}$-type modes have been
expressed in terms of these by comparing the behaviour of the modes for
$r_{*}\to\pm\infty$.

The expansion of the Unruh modes in terms of the Boulware modes is perfomed as
in [27]. By taking the coinciding point limit before the limit onto the
horizon, one obtains expressions for the contributions of modes of fixed
$\ell,m$ to the current. Combining these, one finds

$\displaystyle{\langle j_{v}\rangle}_{\mathrm{U}}$
$\displaystyle=-\sum\limits_{\ell=0}^{\infty}\frac{q(2\ell+1)}{16\pi^{2}r_{-}^{2}}\int\limits_{0}^{\infty}\text{d}\omega\left[F_{\ell}(\omega)+F_{\ell}(-\omega)\right],$
(16a) $\displaystyle F_{\ell}(\omega)$
$\displaystyle=\frac{\omega(\omega^{+}+\omega_{{\mathrm{I}}})}{(\omega^{+})^{2}}\coth\left(\pi\tfrac{\omega^{+}+\omega_{{\mathrm{I}}}}{\kappa_{c}}\right)\left|\mathcal{T}^{\mathrm{I}}_{\omega^{+}\ell}\right|^{2}\left|\mathcal{T}^{{\mathrm{II}}}_{\omega^{+}\ell}\right|^{2}$
$\displaystyle+\frac{\omega\coth\left(\pi\tfrac{\omega^{+}}{\kappa_{+}}\right)}{\omega^{+}}\left[\left|\mathcal{R}^{{\mathrm{II}}}_{\omega^{+}\ell}\right|^{2}+\left|\mathcal{R}^{\mathrm{I}}_{\omega^{+}\ell}\right|^{2}\left|\mathcal{T}^{{\mathrm{II}}}_{\omega^{+}\ell}\right|^{2}\right]$
$\displaystyle+\frac{2\omega\operatorname{csch}\left(\pi\tfrac{\omega^{+}}{\kappa_{+}}\right)}{\omega^{+}}\mathrm{Re}\left(\overline{\mathcal{R}^{\mathrm{I}}_{\omega^{+}\ell}}\mathcal{T}^{{\mathrm{II}}}_{\omega^{+}\ell}\mathcal{R}^{{\mathrm{II}}}_{\omega^{+}\ell}\right),$
(16b)

with $\omega^{+}=\omega+\omega_{{\mathrm{II}}}$, and the scattering
coefficients as defined in (15). Proceeding as in [13, 15] one also finds

${\langle
T_{vv}\rangle}_{\mathrm{U}}=\sum\limits_{\ell=0}^{\infty}\frac{2\ell+1}{32\pi^{2}r_{-}^{2}}\int\limits_{0}^{\infty}\text{d}\omega\
\omega\left[F^{\mathrm{a}}_{\ell}(\omega)-2\coth\tfrac{\pi\omega}{\kappa_{-}}\right]$
(17)

with $F^{\mathrm{a}}_{\ell}(\omega)=F_{\ell}(\omega)-F_{\ell}(-\omega)$ for
the stress tensor on the inner horizon.

##  IV Numerical results

To compute the scattering coefficients, (11a) has to be solved numerically.
Following [30], in the case of a conformal mass $\mu^{2}=\frac{2\Lambda}{3}$,
one can bring the equation into the form of a Heun differential equation by a
special choice of ansatz. The solutions to the Heun equation have been
implemented in MATHEMATICA as special functions [31]. For a general $\mu^{2}$,
the same ansatz yields a slightly more general equation, that can however be
solved, near any of the horizons, by a power series as described in [14]. By
comparing the solutions around different horizons, one can calculate the
scattering coefficients, again as in [14].

For small enough $\mu$, the integrand in (16a) is rapidly decreasing both in
$\omega$ and $\ell$. We thus calculate the scattering coefficients and the
corresponding $F_{\ell}(\omega)$, c.f. (16b), for different values of $\omega$
and $\ell$ up to some maximal values beyond which the contribution to (16a) is
negligible. Then, the integral in (16a) is estimated by a Riemann sum, which
is also used to estimate the errors of this procedure.

Figure 2: ${\langle j_{v}\rangle}_{\mathrm{U}}$ evaluated on
$\mathcal{CH}^{R}$ as a function of $Q/M$ for different values of $qQ$ and
$\mu^{2}=2\Lambda/3$, $\Lambda=0.14M^{-2}$.

Fig. 2 shows the results for the current component $j_{v}$ on
$\mathcal{CH}^{R}$ as a function of $Q/M$ (nearly up to extremality) for
different charges of the scalar field. The most remarkable feature is that the
current can have either sign, depending on the charge of the BH and of the
field. As discussed above, ${\langle j_{v}\rangle}_{\mathrm{U}}>0$ corresponds
to the discharge of the Cauchy horizon, while for ${\langle
j_{v}\rangle}_{\mathrm{U}}<0$ backreaction effects increase its charge. We see
that, close enough to extremality, backreaction effects discharge the Cauchy
horizon, so that it is driven away from extremality. However, away from
extremality, a sign change may occur, so that quantum effects can tend to
charge Cauchy horizons of BHs which are far enough away from extremality. One
may also check that, for charges $q$ small enough and all other parameters
fixed, the current is proportional to $q^{2}$, i.e., the fine-structure
constant, compatible with expectations.

The possibility of a sign change of the current at the inner horizon is in
stark contrast to the situation at the outer horizon, where vacuum
polarization always tends to discharge the BH [29, 21] in physically
reasonable states.

Evaluation of the stress tensor according to (17) yields results compatible
with those obtained for the real scalar field in [15, 14], with a mild $q$
dependence, at least for small $q$. Combining the results for the current and
the stress tensor according to (8), we find that, for the parameters
considered in Fig. 2 and in the weak backreaction regime, backreaction effects
increase the field strength, ${\partial}_{v}(Q/r^{2})>0$, as one approaches
the inner horizon, even for parameters for which backreaction discharges the
inner horizon.

We note that the parameter range considered in Fig. 2 does not capture semi-
realistic BH and field parameters. In order to achieve comparability with [14,
11], a reasonable performance of the numerical code, and to avoid the
classical instability [18], $\Lambda$ was chosen unrealistically high, so that
the event horizon $r_{+}$ of the BH is of the same order of magnitude as the
cosmological horizon $r_{c}$. For our fixed value of $\Lambda$, the bound
$Q/M>0.755$ is necessary to achieve $r_{+}<r_{c}$. But astrophysical BHs are
expected to be only weakly charged [21], i.e., $Q\ll M$. Furthermore, if we
identify $q$ with the elementary charge $e$, we find that for near-extremal
BHs $qQ\sim 10^{36}M/M_{\odot}$, which is beyond the regime accessible to our
numerical code for realistic BH masses. Similarly, the assumption of a
conformal mass for $\Phi$ is unrealistic, but semi-realistic masses $\mu$ are
not accessible by our code (however, we checked stability of our results under
deviations from the conformal mass). Hence, our results unfortunately cannot
be used to infer the fate of the Cauchy horizon of semi-realistic BHs.
However, they are sufficient demonstrate that the naive expectation that
quantum effects will always discharge the BH interior is false. It would be
interesting to see whether a similar effect occurs for rotating (Kerr) black
holes where (dis-)charging would now correspond to (down-)upspinning of the
Cauchy horizon and $j_{v}$ would correspond to $T_{v\phi}$.

Acknowledgements: SH thanks Ted Jacobson for stimulating discussions at an
early stage of this project. This work has been funded by the Deutsche
Forschungsgemeinschaft (DFG) under the Grant No. 406116891 within the Research
Training Group RTG 2522/1.

## References

  * Penrose [1974] R. Penrose, Gravitational radiation and gravitational collapse (Springer, Heidelberg, 1974) Chap. Gravitational collapse.
  * Christodoulou [2009] D. Christodoulou,  _The Formation of Black Holes in General Relativity_ (European Mathematical Society Publishing House, Zürich, 2009) arXiv:0805.3880 [gr-qc] .
  * Mellor and Moss [1990] F. Mellor and I. Moss, Stability of Black Holes in De Sitter Space, Phys. Rev. D 41, 403 (1990).
  * Mellor and Moss [1992] F. Mellor and I. Moss, A reassessment of the stability of the cauchy horizon in de sitter space, Classical and Quantum Gravity 9, L43 (1992).
  * Brady  _et al._ [1998] P. R. Brady, I. G. Moss, and R. C. Myers, Cosmic censorship: As strong as ever, Phys. Rev. Lett. 80, 3432 (1998), arXiv:gr-qc/9801032 .
  * Costa  _et al._ [2015a] J. L. Costa, P. M. Girão, J. Natário, and J. D. Silva, On the global uniqueness for the Einstein-Maxwell-scalar field system with a cosmological constant: I. Well posedness and breakdown criterion, Class. Quant. Grav. 32, 015017 (2015a), arXiv:1406.7245 [gr-qc] .
  * Costa  _et al._ [2015b] J. L. Costa, P. M. Girão, J. Natário, and J. D. Silva, On the global uniqueness for the Einstein-Maxwell-scalar field system with a cosmological constant.: Part 2: Structure of the solutions and stability of the Cauchy horizon, Commun. Math. Phys. 339, 903 (2015b), arXiv:1406.7253 [gr-qc] .
  * Costa  _et al._ [2017] J. L. Costa, P. M. Girão, J. Natário, and J. D. Silva, On the global uniqueness for the Einstein-Maxwell-scalar field system with a cosmological constant. Part 3: Mass inflation and extendibility of the solutions, Annals of PDE 3, 10.1007/s40818-017-0028-6 (2017), arXiv:1406.7261 [gr-qc] .
  * Hintz and Vasy [2017] P. Hintz and A. Vasy, Analysis of linear waves near the Cauchy horizon of cosmological black holes, J. Math. Phys. 58, 081509 (2017), arXiv:1512.08004 [math.AP] .
  * Costa and Franzen [2017] J. L. Costa and A. T. Franzen, Bounded energy waves on the black hole interior of Reissner-Nordström-de Sitter, Annales Henri Poincare 18, 3371 (2017), arXiv:1607.01018 [gr-qc] .
  * Cardoso  _et al._ [2018a] V. Cardoso, J. a. L. Costa, K. Destounis, P. Hintz, and A. Jansen, Quasinormal modes and Strong Cosmic Censorship, Phys. Rev. Lett. 120, 031103 (2018a), arXiv:1711.10502 [gr-qc] .
  * Birrell and Davies [1978] N. Birrell and P. Davies, On falling through a black hole into another universe, Nature 272, 35 (1978).
  * Hollands  _et al._ [2020a] S. Hollands, R. M. Wald, and J. Zahn, Quantum instability of the Cauchy horizon in Reissner–Nordström–deSitter spacetime, Class. Quant. Grav. 37, 115009 (2020a), arXiv:1912.06047 [gr-qc] .
  * Hollands  _et al._ [2020b] S. Hollands, C. Klein, and J. Zahn, Quantum stress tensor at the Cauchy horizon of the Reissner–Nordström–de Sitter spacetime, Phys. Rev. D 102, 085004 (2020b), arXiv:2006.10991 [gr-qc] .
  * Zilberman  _et al._ [2020] N. Zilberman, A. Levi, and A. Ori, Quantum fluxes at the inner horizon of a spherical charged black hole, Phys. Rev. Lett. 124, 171302 (2020), arXiv:1906.11303 [gr-qc] .
  * Zhu  _et al._ [2014] Z. Zhu, S.-J. Zhang, C. E. Pellicer, B. Wang, and E. Abdalla, Stability of Reissner-Nordström black hole in de Sitter background under charged scalar perturbation, Phys. Rev. D 90, 044042 (2014), [Addendum: Phys.Rev.D 90, 049904 (2014)], arXiv:1405.4931 [hep-th] .
  * Konoplya and Zhidenko [2014] R. A. Konoplya and A. Zhidenko, Charged scalar field instability between the event and cosmological horizons, Phys. Rev. D 90, 064048 (2014), arXiv:1406.0019 [hep-th] .
  * Dias  _et al._ [2019] O. J. Dias, H. S. Reall, and J. E. Santos, Strong cosmic censorship for charged de Sitter black holes with a charged scalar field, Class. Quant. Grav. 36, 045005 (2019), arXiv:1808.04832 [gr-qc] .
  * Cardoso  _et al._ [2018b] V. Cardoso, J. L. Costa, K. Destounis, P. Hintz, and A. Jansen, Strong cosmic censorship in charged black-hole spacetimes: still subtle, Phys. Rev. D 98, 104007 (2018b), arXiv:1808.03631 [gr-qc] .
  * Hawking [1975] S. Hawking, Particle Creation by Black Holes, Commun. Math. Phys. 43, 199 (1975), [Erratum: Commun.Math.Phys. 46, 206 (1976)].
  * Gibbons [1975] G. Gibbons, Vacuum Polarization and the Spontaneous Loss of Charge by Black Holes, Commun. Math. Phys. 44, 245 (1975).
  * Unruh [1976] W. Unruh, Notes on black hole evaporation, Phys. Rev. D 14, 870 (1976).
  * Schwinger [1951] J. S. Schwinger, On gauge invariance and vacuum polarization, Phys. Rev. 82, 664 (1951).
  * Herman and Hiscock [1994] R. Herman and W. Hiscock, Discharge of a Reissner-Nordstroem black hole within the event horizon, Phys. Rev. D 49, 3946 (1994).
  * Sorkin and Piran [2001] E. Sorkin and T. Piran, The Effects of pair creation on charged gravitational collapse, Phys. Rev. D 63, 084006 (2001), arXiv:gr-qc/0009095 .
  * Hollands and Wald [2015] S. Hollands and R. M. Wald, Quantum fields in curved spacetime, Phys. Rept. 574, 1 (2015), arXiv:1401.2026 [gr-qc] .
  * Lanir  _et al._ [2018] A. Lanir, A. Levi, A. Ori, and O. Sela, Two-point function of a quantum scalar field in the interior region of a Reissner-Nordstrom black hole, Phys. Rev. D 97, 024033 (2018), arXiv:1710.07267 [gr-qc] .
  * Sela [2018] O. Sela, Quantum effects near the Cauchy horizon of a Reissner-Nordström black hole, Phys. Rev. D 98, 024025 (2018), arXiv:1803.06747 [gr-qc] .
  * Klein and Zahn [2021] C. Klein and J. Zahn,  _The renormalised charged scalar current in the Reissner-Nordström-de Sitter spacetime. In preparation._ (2021).
  * Suzuki  _et al._ [1999] H. Suzuki, E. Takasugi, and H. Umetsu, Analytic solutions of Teukolsky equation in Kerr-de Sitter and Kerr-Newman-de Sitter geometries, Prog. Theor. Phys. 102, 253 (1999), arXiv:gr-qc/9905040 .
  * Hatsuda [2020] Y. Hatsuda, Quasinormal modes of Kerr-de Sitter black holes via the Heun function, Class. Quant. Grav. 38, 025015 (2020), arXiv:2006.08957 [gr-qc] .


# Using Angle of Arrival for Improving Indoor Localization

Sai Koppula
Department of Computer Science
University of Texas at Austin
<EMAIL_ADDRESS>
&Shivang Singh
Department of Computer Science
University of Texas at Austin
<EMAIL_ADDRESS>

###### Abstract

In this paper, we primarily explore the improvement of single stream audio
systems using Angle of Arrival calculations in both simulation and real life
gathered data. We wanted to learn how to discern the direction of an audio
source from gathered signal data to ultimately incorporate into a multi modal
security system. We focused on the MUSIC algorithm for the estimation of the
angle of arrival but briefly experimented with other techniques such as
Bartlett and Capo. We were able to implement our own MUSIC algorithm on
stimulated data from Cornell. In addition, we demonstrated how we are able to
calculate the angle of arrival over time in a real life scene. Finally, we are
able to detect the direction of arrival for two separate and simultaneous
audio sources in a real life scene. Eventually, we could incorporate this
tracking into a multi modal system combined with video. Overall, we are able
to produce compelling results for angle of arrival calculations that could be
the stepping stones for a better system to detect events in a scene.

##  1 Introduction

Security systems in modern day systems typically rely on a single source type
to determine an action. For example, the Ring doorbell system uses a video
feed but no audio. The Simplisafe Glass Break detector uses a simple audio
feed that is refined to identify the unique sound of a window breaking. We
hypothesize that a combination of common monitoring mediums, such as audio,
video, and RF, in a multi-modal system can better detect a variety of
intrusion types. To this end, we will improve the single stream audio system
by using various AoA estimation techniques to localize the source of audio. We
implemented the MUSIC algorithm on simulated data to identify the direction of
arrival of multiple sources at once. Furthermore, we tried to use the
eigenvalues to determine the number of sources present in a simulated
environment. Finally, we used the library MUSIC function on captured audio
data to track the change of source angles over time. We also explored how we
can use machine learning models to further refine our audio system to properly
weight the contributions of each of the data sources. Our final product was a
working MUSIC implementation and improved audio system that can detect the
angles of multiple audio sources. We did not quite integrate the improved
audio system into our deficient visual system during the scope of this
project.

##  2 Related Works

There are some previous works in localization and anomaly detection. Other
works in the field of indoor localization use triangulation techniques to
isolate the location of sound. One recent paper, Improving AoA Localization
Accuracy, did indoor localization using 5 microphones. They calculated a
probability density for each x,y coordinate in the room that they used for
their experiment. Our work tries to do something similar with less intrusions
to the indoor setup (and having only 1 access point for determining the
angle). Furthermore, our work proposes to use indoor cameras to determine the
depth of the object. This sacrifices some of the accuracy, but makes the
installation and adoption of the system easier to do. Thoen et al. (2019).
Another paper in this domain is Three-Dimensional Empirical AoA Localization
Technique for Indoor Applications. This paper uses factors other than angle of
arrival in addition to angle of arrival to do localization. Our work does
something similar to this work. However, our work also examines how to do
angle of arrival for more than one source. The evaluation in this paper
doesn’t look at the multipath case. Additionally, the paper looks at RFID
signals, while we look at acoustic signals, which are more ubiquitous but also
prone to more noise. Alma’aitah et al. (2019). Another paper looks at doing
acoustic based angle of arrival estimation for robotics. This paper is similar
to our paper. However, there is no distinction made for the multipath case.
They do examine the error in the calculation of the real angles versus the
projected angle to do evaluation. Karanam et al. (2018)

##  3 Background

The core principal we used throughout our project was calculating the angle of
arrival (AoA) of a sound source to determine which direction the audio signal
was coming from. We envisioned we could project a visualization onto an
accompanying video feed to illustrate where the sound was originating from.
For example, in an instance when the subject is just waving their hand, the
video feed would show changes but the projected audio visualization would not.
However, when the subject is snapping their fingers, both the audio and video
visualizations would show activity. By checking for overlapping audio and
video disturbances, we hypothesized that we could create a better security
system than one that just relies on disjoint audio and video feeds. One
example of where a multi modal system would excel is where you have multiple
sounds in the scene at once. If multiple audio sources are present, an audio
only system will tend to alert the user continuously once the audio is past a
threshold. However, by mapping the audio onto the scene and correlating it
with the various visual disturbances, we can create a more selective system.

###  3.1 AoA

To understand how to achieve an audio visualization, we need to first
calculate the angle of arrival. By using an array of antenna elements, we can
gather enough data to determine the direction of an audio source. For the same
source, various antennas in an array will receive slightly phased version of
similar waves. By constructing a linear and equally spaced array of antennas,
we can use the expected phase perturbation to correlate what we expect the
angle that would have caused such a set of received signals to be. We can
accomplish this by leveraging the fact that we can decompose a received signal
x as a linear composition of a steering vector S, an amplitude a, and some
noise n.

$x=Sa+n$ (1)

If we can estimate the steering vector, a function of the angle of the signal
hitting an antenna element, we can estimate the angle of arrival.

###  3.2 MUSIC

The music algorithm leverages the fact that we can decompose the set of linear
equations that arise from Equation (1) into their eigenvalues and
eigenvectors. The lowest values of these eigenvalues correspond to the noise
eigenvectors. It is also known that the noise eigenvectors are orthogonal to
the signal sources. We can therefore compose the two across a sweep of angles
to judge the orthogonality of the resulting intersection. If the values are
close to 0 for a given value of theta, then we must have a good estimate for
the angle of arrival. Typically, we take the inverse of this composition to
create a pseudospectrum graph with peaks at each sources angles of arrival.
Dav (2013)

###  3.3 Finding the Number of Sources

The Music algorithm does have a limitation that it needs to know in advance
the number of audio sources. This is so we can properly divide the noise
eigenvalues from the signal eigenvalues. However, it is observed that the
noise eigenvalues are often quite close to each other while the signal
eigenvalues diverge significantly from the noise eigenvalues. Various
techniques can be used to dynamically predict the number of sources from
analyzing the clustering of eigenvaluesDav (2013). We explored one such
approach in our simulated experiments, but we did not succeed.

##  4 Single Stream Visual System

###  4.1 Summary

In order to determine how multimodal input can help improve the performance of
the wireless input, the results and the performance of models for audio and
visual detection were studied in isolation. We explored two different
approaches for our single stream systems. First, we did a real time stream
analysis for a visual system to discern if there is a change in the observable
scene.

###  4.2 Results

For the purposes of the single stream video system, we originally planned to
utilize a remote Arduino chip, wired with a camera module. This Arduino set up
would have a wireless component which would transmit audio and visual data to
a server setup. However, the latency and throughput of this configuration was
quite poor. For the sake of getting meaningful results, we transitioned to
using the webcam on our computer. Here, we were able to stream our computer’s
webcam into a python application and manipulate the data as needed to
determine whether an event occurred. Our algorithm was fairly simple. We kept
track of a small history of frames and averaged them together to create a
composite frame. We then compared the current frame to the composite frame to
see if it differed enough to flag an event. There are a few tunable knobs that
can help calibrate the system such as number of frames that contribute to the
composite image, the tolerance of a pixel to be different between the
composite frame and current frame and the percentage of pixels that can be
above that tolerance before an event is flagged. You can see from Figure 1
that our composite image is quite noisy.

Figure 1:  Left is the real time camera input. Right is the composite image of
the past 5 frames. As you can see, the composite frame is quite noisy and
error prone.

Overall, our stand alone video system showed quite a few deficiencies. First,
it was a noisy image that caused gradients to be present even when nothing in
the scene was changing. The noisy image above shows areas where the picture is
changing despite nothing occurring in the scene. Second, the visual system
suffers from a limited field of view. Any intruder would need to simply avoid
the field of view of the camera to avoid detection. Finally, system was
sensitive to how it was calibrated. We could get a high number of false
positives due to the noisy image and calculation metrics we chose. Whether we
chose to take the number of pixels that changed or the average color of the
image as a whole, there wasn’t a reliable metric to make a robust system from
just a video stream. Furthermore, the system could only detect changes but it
could not differentiate between different types of changes. In fairness, the
audio system by itself, without any machine learning enhancements, suffers
from this as well.

##  5 Audio System

The audio system was the main focus of our project. Here we explored how to
refine an audio system using AoA techniques including MUSIC to get a better
sense of what was happening in the scene over time. We first refined our
techniques on simulated data and then moved onto our actual physical setup.We
used various python libraries to read our .wav files and transform them into
numpy arrays that could be manipulated and calculated on. This results in our
current implementation being unable to handle audio in real time. While the
algorithm’s themselves are real time, we need a post processing step that is
not yet automated to bring the data into a manipulative form.

###  5.1 Simulation Setup

Our simulation setup used data harvested from Cornell University. Here we
procured data simulated on a system of a linear array of antennas that then
recorded a varying number of signals. We were able to implement our own music
algorithm and plot the data against a sweep of angles to find the angle of
arrival.

###  5.2 Simulation Results

For implementing the MUSIC algorithm, we did the standard approach of first
creating a correlation matrix and then partitioning the noise eigenvalues. We
then created pseudospectrum sweeps of the possible angles on the inverse of
the orthogonality of the signals and noise eigenvectors. The results are plots
below which show the angles from where the sources are from. We tried our
algorithm on simulation data for both three sources and two sources. The
received signals were from eight antennas in a linear equally spaced array.
From the plots below you can see our algorithm partially works for 3 sources.
We have peaks near the ground truth values of 18, 54, and 126 degrees.
However, we also have extraneous peaks above our 3 source angles which
suggests our implementation is not robust for 3 sources. We got much better
results for the two source set of simulation data. Here we have two clear
peaks near the ground truth values of 54 and 126 degrees. This demonstrated
that our MUSIC algorithm was working for at least two sources as we were able
to get no extraneous peaks above the ground truth peaks. The results of our
experiments from simulation data were good first steps in understanding the
MUSIC algorithm for AoA calculation.

Figure 2:  3 Source Simulation Data Plot Figure 3:  2 Source Simulation Data
Plot Figure 4:  This figure illustrates the localization setup that was used
for data gathering. The rooms dimensions are listed. Additionally, the AP’s
location relative to the room is also described. The AP used was the ReSpeaker
Mic Array 2.0, which carried an array of 4 microphones positioned in a square
formation on the ship. The illustration also shows how the algorithm deployed
may project the angle of arrival in the real environment.  Figure 5:  This
figure illustrates the pseudo spectrum generated by the MUSIC algorithm which
went to calculate the angle of arrival for a particular time point. In this
case the time point occurs at second 18 (highlighted by the green box). One
may notice that the pseudo-spectrum may be having three peaks instead of two.
However, due to the scaling of the figure, the difference between the first
and third peak looks negligible. The third peak is larger. Figure 6:  This
figure illustrates the evaluation of the system that was developed as part of
this project. This figure in contrast to the earlier figure is tracking a
single source. This is one of the experiments that was carried out. The
location of the subject was estimated by manually examining the location of
the subject in the video. In this experiment, the subject was clapping and so
one can see that the angles estimated are very discrete (as sound was created
for a roughly small duration of time). Regardless, the angles still follow the
location of the subject relative to the camera. In order to see what activity
is going on please see the video in our presentation.

###  5.3 Physical World Setup

Our experimental setup uses a ReSpeaker Mic Array 2.0 . It is a microphone
array that can connect to a computer using USB and it has four microphones in
a two by two grid. We recorded with all four microphone channels at once. We
tried a variety of different techniques for recording from this component
including using various python modules. However, we ultimately settled on
using the Audacity program to record all channels at once and then post
processing each microphone’s audio into a separate .wav file. This allowed to
keep the signals separated for when we applied our MUSIC algorithm’s on top of
them.

Figure 7: This figure illustrates the use of the multi path MUSIC algorithm
for determining the angle of arrival for multiple sources. There are two
sources of sound in our scenario: the phone, which is playing audio and the
subject themselves, who is talking out loud. The images show what the subject
was doing at that particular moment. The graph illustrates the angles of
arrival with the highest magnitude determined by the MUSIC algorithm. It is
important to note that it only illustrates the two angles at each time step
but does not determine if the angle is of the subject or the phone. In the
first interval there seems to be a lot of noise, with the subject picking up
his phone and orienting himself. In the second interval the subject is
talking, but is hard to hear over the sound of the phone. Both the phone and
the subject are stationary (hence why it seems to just be predicting the angle
of the phone). In the third step, the subject is moving the phone as well as
moving his chair. This can be seen by the orange angle shifting from the an
angle smaller than the blue to larger. At the last step, the subject quickly
moves the phone to the right of him, which is represented by a quick movement
in the blue and orange lines and the angle of the blue line points heavily to
the right (probably the angle of the phone).

###  5.4 Physical World Methodology

Each of these input channels has thousands of samples, which correspond to the
amplitude and the duration of a single wavelength of sound. Using the input
channels, the angle of arrival for each second was calculated. This was done
by dividing the total number of samples by the sampling rate, which in this
case was 48 kHz. This gave us blocks of samples that were of the size of the
sampling rate. Each of these blocks (which were of the size 48,000) was
analyzed to determine the angle of arrival of the sound at that location. The
number of sources of audio in each interval was set to the number of sources
that were used in the demonstration. For example, in our second experiment
where we use a phone and a subject, we set the number of audio sources or the
number of multi-path for the MUSIC algorithm to 2. We experimented with
calculating the number of sources, however the results were not reliable. The
output of the MUSIC algorithm is a pseudo-spectrum in which the signal
strength of each angle is projected. A series of signal strengths, one for
each steering vector corresponding to each theta, is returned. We examined the
output of the algorithm and took the angle corresponding to the highest
magnitude signal as the angle of arrival. When we had more than 2 sources of
audio, we looked at the highest and the second highest power in the MUSIC
spectrum instead of looking at the maximum power.

###  5.5 Physical World Results

In order to evaluate the effectiveness of the localization, experiments were
done for both the single source audio case and the multi-path audio case.
Figure 7 illustrates the experiment that was conducted for the multi-source
case. In that experiment, a phone and a subject both talked at the same time.
The MUSIC algorithm was used to determine the angles of arrival. The two most
prominent angles are plotted in the figure. Another visualization of our
system is presented in Figure 5, where we can examine the individual
pseudospectrum at a given time step (in this case the 18th sample in the time
series). This example shows how the angle of arrival in the multi path setting
was derived by looking at the signal with the most power. These figures can
give us a qualitative measure of how well the system was working. Figure 6
evaluates how well the MUSIC algorithm and our implementation works in the
real world. It shows the location of the subject over time as well as the
angle of arrival (single source in this case). We see that that the location
of the subject is well tracked. The RMSE measure would be a good way of
evaluating the system. However, since there was no baseline system to compare
it to, it is not presented here.

###  5.6 Extending Beyond Audio (making it multi modal)

In order to make the system more precise with regards to localization, we
propose to use depth as a tool for gauging where the subject or the source of
sound is. Figure 3 can be seen as an illustration of this concept. So far, our
system only suggests angles of arrival. However, by using two cameras our
system could sense the depth of the object. Given information on depth and
using the $sin$ trigonometric function, it would be possible to evaluate the x
and y positions of the sound.

##  6 Leveraging Neural Networks

###  6.1 Event Classification

After the event detection, a classification model can be used to evaluate if
the event was significant or not. This measure is to further decrease the
amount of false positive detection of the system. It does this by adding using
AI to further filter out positive event detection generated by the averaging
of the frames. The classification system would run after the event detection
system runs. This is because polling and running intervals of sound and visual
inputs is relatively expensive (as you have to constantly be making passes
through the network). This pretrained deep learning model was a Convolutional
Neural Network (CNN), trained by researchers at Google Pete Warden (2017). The
network takes in an audio stream and outputs a probability distribution over
the tasks it believes took place in the audio. These predefined tasks will
then be curated for the project to map to two output classes (Abnormal or
Normal). The Normal class was for the normal background sounds normal to
systems such as human speech. Abnormal class was for anomalies and tasks seen
as non-background. Similar to the audio input processing, a deep learning mode
to detect the task being done in a specific video interval can be used but we
did not implement in time for the writing of this paper. Predefined tasks
would be curated to the same categories as the audio model. The results of the
audio and visual models can be used as a baseline off of which multimodal
systems can be analyzed from.

Figure 8:  This diagram illustrates how adding classification to the audio
input, the model is able to be more robust about detecting less false
positives in event detection. It is able to be more precise about filtering
out data points

###  6.2 Future Work

Our work was primarily focused on AoA and understanding the theory and
application of it. We were able to get some compelling results which showed
that we could track the change of angle of arrival over time in our audio
system. Given time, we could make a few key enhancements to make a better
system. First, we can make our audio processing real time. Instead of
recording to audacity, we could go to a lower layer in our computer’s firmware
and read the individual channels, synchronize them, and pass them through to
our computational system. We actually do have most of this working already but
the problem is we couldn’t figure out how to split out the channels in real
time so we couldn’t put the data in a form our algorithms could understand.
Second, we could take the angle of arrival calculations in real time and
project onto the screen a visualization of where the audio is coming from.
This would help us combine our audio system with our visual system to create
more informed decisions on intrusion detection. It would also be further
visual confirmation that our AoA calculations have merit. We did compare our
results to ground truth values in our simulation exercise but it is more
pleasing to visually see the result projected onto real world data that we
gathered. Third, we can combine our audio-visual system with the ML model we
experimented with to not only detect an event but also classify it. A
classification of an audio event combined with a detection of an event in the
multi modal system can further inform a security system on what to do and
whether or not to alert the user.

###  6.3 Challenges

Our primary challenge stemmed from the fact that we wanted to do a project
that had a hardware component. We wanted to do our own data gathering and that
required research into various hardware components. Our initial attempts
failed as the Arduino cam and board did not provide sufficient data throughput
for our visual system. Our audio system ran into a different set of issues as
we needed to find a way to get an array of microphones to record
simultaneously. Eventually, we found the ReSpeaker hardware component that
allowed us to do that simply. It was an interesting challenge figuring out how
to sample from the hardware component properly. We had control over variables
such as the sample rate and it demonstrated that there was more to consider
than just pressing record when dealing with signal gathering in the field.

###  6.4 Conclusions

Overall, the project was a good learning experience in primarily implementing
an algorithm to detect Angle of Arrival of multiple signals in a real world
scenario. We were able to implement our own Music algorithm on simulation data
for multiple sources. We were also able to perform angle of arrival
calculation over time for our real life gathered data. The project opened us
up to a variety of techniques and challenges in signal based research and was
rewarding.

## References

  * Dav (2013) 2013\.  Direction of arrival estimation.  _University of Toronto_.
  * Alma’aitah et al. (2019) A. Alma’aitah, Baha’ A. Alsaify, and Raed Bani-Hani. 2019.  Three-dimensional empirical aoa localization technique for indoor applications.  _Sensors (Basel, Switzerland)_ , 19.
  * Karanam et al. (2018) Chitra R. Karanam, B. Korany, and Y. Mostofi. 2018.  Magnitude-based angle-of-arrival estimation, localization, and target tracking.  _2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)_ , pages 254–265.
  * Pete Warden (2017) Google Brain Team Pete Warden, Software Engineer. 2017.  Launching the speech commands dataset.  _Google AI Blog_.
  * Thoen et al. (2019) Bart Thoen, S. Wielandt, and L. D. Strycker. 2019.  Improving aoa localization accuracy in wireless acoustic sensor networks with angular probability density functions.  _Sensors (Basel, Switzerland)_ , 19.