# Black hole quasinormal modes and isospectrality in Deser-Woodard nonlocal
gravity

Che-Yu Chen<EMAIL_ADDRESS>Institute of Physics, Academia Sinica,
Taipei, Taiwan 11529  Sohyun Park<EMAIL_ADDRESS>Theoretical Physics
Department, CERN, CH-1211 Genève 23, Switzerland

###### Abstract

We investigate the gravitational perturbations of the Schwarzschild black hole
in the nonlocal gravity model recently proposed by Deser and Woodard (DW-II).
The analysis is performed in the localized version in which the nonlocal
corrections are represented by some auxiliary scalar fields. We find that the
nonlocal corrections do not affect the axial gravitational perturbations, and
hence the axial modes are completely identical to those in General Relativity
(GR). However, the polar modes get different from their GR counterparts when
the scalar fields are excited at the background level. In such a case the
polar modes are sourced by an additional massless scalar mode and, as a
result, the isospectrality between the axial and the polar modes breaks down.
We also perform a similar analysis for the predecessor of this model (DW-I)
and arrive at the same conclusion for it.

††preprint: CERN-TH-2021-011

##  I Introduction

The direct detection of gravitational waves emitted from binary black hole
mergers Abbott:2016blz ; LIGOScientific:2018mvr ; Abbott:2020niy  ushers in an
exciting era of gravitational wave astronomy. More surprisingly, the current
observing sensitivity already allows the detection of roughly one candidate
event per week in average Abbott:2020niy , not to mention the amount of data
available with the further upgrades and improvements of the detectors in the
future. The gravitational waves emitted from these events carry useful
information about the spacetime under strong gravitational fields. Therefore,
such a detection facilitates us to probe the physics in the extreme
environments such as black holes, and to test whether our current
understanding of black holes and gravity based on general relativity (GR) is
correct or not.

Essentially, the gravitational wave signals associated with a binary black
hole merger consist of three stages. The first is the inspiral stage during
which the binary black holes rotate around each other. At this stage, the
gravitational waves can be modeled well by the post-Newtonian approaches. The
second is the merger stage, in which the gravitational fields are so strong
that purely numerical analysis is inevitable. The third stage, which is called
the ringdown, corresponds to the post-merger phase of the event. At this
stage, the binary black holes have coalesced, and the distortion of the final
black hole is gradually settling. The black hole during this ringdown stage
loses its energy by emitting gravitational waves, with frequencies governed by
a superposition of decaying oscillations, i.e., the quasinormal modes (QNMs)
Kokkotas:1999bd ; Berti:2009kk ; Konoplya:2011qq . The ringing black hole and
the associated QNMs can be described well by black hole perturbation theories.
An important realization is that the QNM spectrum only depends on the
parameters quantifying the black hole, such as mass, spin, and whatever
parameters that characterize the black hole in a particular gravitational
theory. For instance, the QNM spectrum of a Kerr black hole in GR is
completely determined by its mass and spin. The spectrum does not depend on
what initially drives the ringings. This means that by examining the QNM
frequencies one can test the no-hair theorem in the underlying theory of
gravity. That is, the QNM spectrum is an observable to distinguish between GR
and other gravitational theories. In this respect, the investigation of black
hole perturbations and the associated QNM spectra in the context of theories
beyond GR has been an intensive field of research, see for instance Refs.
Kobayashi:2012kh ; Kobayashi:2014wsa ; Blazquez-Salcedo:2016enn ;
Bhattacharyya:2017tyc ; Glampedakis:2017dvb ; Chen:2018mkf ; Chen:2018vuw ;
Chen:2019iuo ; Moulin:2019ekf ; Chen:2020evr .

In addition to the no-hair theorem, there are other fundamental properties
which, by examining black hole QNMs, can be used to discriminate GR from other
theories. For example, it is well-known that in the eikonal approximation, the
QNM frequencies for stationary, axisymmetric, and asymptotically flat black
holes in GR are tightly related to the properties of the unstable photon
orbits around the black hole Cardoso:2008bp ; Dolan:2010wr ; Yang:2012he . The
eikonal QNMs are also related to the apparent size of black hole shadows
Cuadros-Melgar:2020kqn . Therefore, apart from extracting the mode frequencies
and comparing the spectra with those in GR, one can test gravitational
theories by examining these properties. Any breaking of them would be a
smoking-gun for different physics beyond Einstein’s gravity Konoplya:2017wot ;
Churilova:2019jqx ; Glampedakis:2019dqh ; Chen:2019dip .

Another important feature of QNMs in GR is the isospectrality. For the
Schwarzschild black hole in GR, the gravitational perturbations of the axial
(parity-odd) Regge:1957td  and the polar (parity-even) modes Zerilli:1970se
share an identical spectrum, even though their master equations are completely
different from each other Chandrabook . This interesting property also holds
for the Reissner-Nordström, Kerr, as well as Kerr-Newman (to linear-order in
the spin) black holes Pani:2013ija ; Pani:2013wsa . It was discovered also by
Chandrasekhar Chandrabook , in particular for Schwarzschild black holes, that
the master equations governing these two modes are related by a specific
transformation. It was not realized until very recently Glampedakis:2017rar
that the transformation introduced in Ref. Chandrabook  is actually a
particular subclass of the Darboux transformation Darboux:1882 . The
identification of this relation and its generalization Glampedakis:2017rar ;
Yurov:2018ynn  helps to investigate the isospectrality of more complex
systems, such as the Kerr black hole Teukolsky:1973ha ; Sasaki:1981sx . In
fact, the isospectrality of black hole QNMs is a longstanding issue
Moulin:2019bfh . For example, the physical origin of this property is unclear.
In addition, although the isospectrality holds for most of the astrophysically
relevant black holes in GR, not all black hole solutions in GR share this
property Ferrari:2000ep ; Cardoso:2001bb ; Berti:2003ud ; Blazquez-
Salcedo:2019nwd  and the range of its viability is still an open question. In
many theories beyond GR, such as $f(R)$ gravity Bhattacharyya:2017tyc ;
Bhattacharyya:2018qbe ; Datta:2019npq , dynamical Chern-Simons gravity
Bhattacharyya:2018hsj , and scalar-tensor theories Kobayashi:2012kh ;
Kobayashi:2014wsa , the isospectrality is generically broken. Furthermore, it
has been shown recently using a parameterization approach that the
isospectrality is actually very fragile Cardoso:2019mqo . Therefore, any
observational indication of the isospectrality breaking would strongly hint
toward new physics beyond GR Shankaranarayanan:2019yjx .

In this work, we will focus on the possibility of testing gravitational
theories by examining the isospectrality of black hole QNMs. In particular, we
will investigate the axial and the polar gravitational perturbations of the
Schwarzschild black hole in the Deser-Woodard nonlocal gravity models I and II
(DW-I Deser:2007jk  and DW-II Deser:2019lmm ). This class of nonlocal gravity
models was proposed by Deser and Woodard Deser:2007jk ; Deser:2019lmm  with
the motivation that quantum loop effects of infrared gravitons abundant in the
early universe are essentially nonlocal and the effects might persist in the
late universe Woodard:2014iga ; Woodard:2018gfj . Deser and Woodard
particularly aimed at a phenomenological model that can generate the current
phase of cosmic acceleration without invoking dark energy.111Other classes of
nonlocal gravity models with the same goal of reproducing the late-time
acceleration without dark energy have been proposed and studied extensively
Barvinsky:2011hd ; Maggiore:2013mea ; Maggiore:2014sia ; Vardanyan:2017kal ;
Amendola:2017qge ; Tian:2018bmn . Their first model DW-I Deser:2007jk  had
been very successful in various aspects. It reproduced the background
expansion identical to that of the $\Lambda$CDM expansion (without $\Lambda$)
in GR Deffayet:2009ca . In addition, once the background was fixed, the large
scale structure formation predicted by the model was comparable to the one in
GR Park:2012cp ; Dodelson:2013sma ; Park:2016jym ; Nersisyan:2017mgj ;
Park:2017zls ; Amendola:2019fhc . Even though the localized version
Nojiri:2007uq  of this model possesses a scalar ghost Zhang:2016ykx ;
Nojiri:2010pw ; Zhang:2011uv , the constraint of the model precludes explosive
growth of the ghost Park:2019btx . However a recent analysis showed that DW-I
violates the solar system constraints Belgacem:2018wtb . Deser and Woodard
then proposed an improved model (DW-II) Deser:2019lmm  that might pass the
solar system constraints. Therefore, in this paper we focus on the DW-II
gravity and briefly remark for the DW-I gravity at the end.

For the Schwarzschild QNMs in the DW-II model, we first find that the axial
modes are not affected by the nonlocal corrections. Furthermore, due to the
special role played by the nonlocal term that whether the auxiliary scalar
fields are excited or not depends on the boundary conditions, we find that the
master equation of the polar modes is sourced by an additional scalar mode
corresponding to the auxiliary scalar fields. However, this scalar mode is not
excited as long as the auxiliary scalar fields are quiescent at the
unperturbed level. If the scalar mode is not excited, the QNMs of the axial
and the polar modes reduce to those in GR and the isospectrality is preserved,
otherwise the isospectrality gets broken. As a remark, due to the fact that
the DW-I and DW-II models are similar in their mathematical structures, the
same conclusion can be drawn for the DW-I model.

The rest of this paper is organized as follows. In Section II, we briefly
review the DW-II nonlocal gravity including its action, equations of motion,
and localization. In Section III, we discuss the Schwarzschild black hole in
the DW-II nonlocal gravity. Then, we perturb the Schwarzschild black hole in
Section IV, and derive the master equations of the axial and polar
perturbations. The results will be compared with those in GR. In Section V, we
perform a similar analysis in the framework of DW-I nonlocal gravity. Finally,
Section VI draws our conclusions and discussions.

##  II Deser-Woodard-II nonlocal gravity

The action of the DW-II nonlocal gravity is Deser:2019lmm

$\mathcal{S}_{\textrm{DW-II}}=\frac{1}{16\pi}\int
d^{4}x\sqrt{-g}R\left[1+f\left(\zeta\right)\right]+\mathcal{S}_{m}\,,$ (1)

where $\mathcal{S}_{m}$ is the matter action. The nonlocal distortion function
$f$ is an arbitrary function of the inverse scalar d’Alembertian acting on
$(\partial\phi)^{2}\equiv g^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi$,
where $\phi$ is the inverse scalar d’Alembertian acting on the Ricci scalar
$R$. More precisely, the scalar fields $\zeta$ and $\phi$ can be written
explicitly as follows:

$\displaystyle\zeta$
$\displaystyle\equiv\Box^{-1}\left(\partial\phi\right)^{2}\,,$ (2)
$\displaystyle\phi$ $\displaystyle\equiv\Box^{-1}R\,.$ (3)

The distortion function $f$ is thus a function of $\zeta$. The presence of the
distortion function adds nonlocal corrections to the Einstein-Hilbert action.
Note that we have assumed $c=G=1$ in this paper, where $c$ is the speed of
light and $G$ is the gravitational constant.

As has been shown in Ref. Deser:2019lmm , the DW-II model can be localized by
introducing two additional Lagrange multipliers $\Xi$ and $\psi$ as
follows222The notations for the auxiliary scalar fields used in Ref.
Deser:2019lmm  ($X,Y,U,V$) and ours ($\phi,\zeta,\Xi,\psi$) are different.
They can be directly converted as follows: $X\rightarrow\phi$,
$Y\rightarrow\zeta$, $U\rightarrow-\Xi$, and $V\rightarrow-\psi$.

$\displaystyle\mathcal{S}_{\textrm{DW-II}}$ $\displaystyle=\frac{1}{16\pi}\int
d^{4}x\sqrt{-g}\Big{[}R\left(1+f-\Xi\right)-\partial_{\mu}\Xi\partial^{\mu}\phi$
$\displaystyle-\psi\left(\partial\phi\right)^{2}-\partial_{\mu}\psi\partial^{\mu}\zeta\Big{]}+\mathcal{S}_{m}\,.$
(4)

The equations of motion are derived by varying the action (4) with respect to
$\Xi$, $\psi$, $\phi$, $\zeta$, and the metric $g_{\mu\nu}$. Varying the
action with respect to the Lagrange multipliers $\Xi$ and $\psi$ gives

$\displaystyle\Box\phi$ $\displaystyle=R\,,$ (5) $\displaystyle\Box\zeta$
$\displaystyle=\left(\partial\phi\right)^{2}\,,$ (6)

which correspond to the definition of the scalar fields $\phi$ and $\zeta$
given by Eqs. (3) and (2), respectively. Then, the variation with respect to
the scalar fields $\phi$ and $\zeta$ gives

$\displaystyle\Box\Xi$
$\displaystyle=-2\nabla_{\mu}\left(\psi\nabla^{\mu}\phi\right)\,,$ (7)
$\displaystyle\Box\psi$ $\displaystyle=-R\frac{df}{d\zeta}\,.$ (8)

Finally, the equation of motion derived by varying the action with respect to
the metric reads

$\displaystyle\left(G_{\mu\nu}+g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu}\right)\left(1+f-\Xi\right)$
$\displaystyle+\frac{1}{2}g_{\mu\nu}\left[\partial_{\alpha}\Xi\partial^{\alpha}\phi+\partial_{\alpha}\psi\partial^{\alpha}\zeta+\psi\left(\partial\phi\right)^{2}\right]$
$\displaystyle-\partial_{(\mu}\Xi\partial_{\nu)}\phi-\partial_{(\mu}\psi\partial_{\nu)}\zeta-\psi\partial_{\mu}\phi\partial_{\nu}\phi=8\pi
T_{\mu\nu}\,,$ (9)

where $G_{\mu\nu}$ and $T_{\mu\nu}$ stand for the Einstein tensor and the
energy-momentum tensor, respectively.

##  III Schwarzschild spacetime in DW-II gravity

Before investigating black hole perturbations in the DW-II nonlocal gravity,
it is necessary to specify the background spacetime that is going to be
perturbed. In this paper, we are going to consider the perturbations of the
Schwarzschild black hole in the DW-II nonlocal gravity. In this section, we
will use the requirement that the Schwarzschild metric is an exact vacuum
solution ($T_{\mu\nu}=0$) of the DW-II nonlocal gravity to constrain the
expressions of the auxiliary scalar fields. By doing so, we thus exhibit that
the Schwarzschild solution does satisfy the field equations of the DW-II
nonlocal gravity.

We consider the Schwarzschild metric

$ds^{2}=-e^{2\bar{\nu}}dt^{2}+e^{-2\bar{\nu}}dr^{2}+r^{2}d\Omega_{2}^{2}\,,$
(10)

where $e^{2\bar{\nu}}=1-2m/r$ with $m$ being the mass of the black hole. From
now on, quantities with a bar represent the quantities at the background
level, in order to distinguish them from their linear perturbations. Since the
Ricci scalar $R$ is identically zero, the scalar field equations (5) and (8)
can be written as

$\displaystyle\bar{\phi}_{,rr}+2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\bar{\phi}_{,r}$
$\displaystyle=0\,,$ (11)
$\displaystyle\bar{\psi}_{,rr}+2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\bar{\psi}_{,r}$
$\displaystyle=0\,.$ (12)

Note that we have assumed that the scalar fields are functions of $r$ only at
the background level. The above equations (11) and (12) can be solved to get

$\bar{\phi}_{,r}=C_{\phi}\frac{e^{-2\bar{\nu}}}{r^{2}}\,,\qquad\bar{\psi}_{,r}=C_{\psi}\frac{e^{-2\bar{\nu}}}{r^{2}}\,,$
(13)

where $C_{\phi}$ and $C_{\psi}$ are integration constants. Furthermore, using
Eqs. (6) and (7), the other two scalar fields can be solved as

$\displaystyle\bar{\zeta}_{,r}$
$\displaystyle=\left(C_{\phi}\bar{\phi}+C_{\zeta}\right)\frac{e^{-2\bar{\nu}}}{r^{2}}\,,$
(14) $\displaystyle\bar{\Xi}_{,r}$
$\displaystyle=\left(-2C_{\phi}\bar{\psi}+C_{\Xi}\right)\frac{e^{-2\bar{\nu}}}{r^{2}}\,,$
(15)

where $C_{\zeta}$ and $C_{\Xi}$ are integration constants as well.

To proceed, we define $K\equiv f-\Xi$ and $\bar{K}\equiv\bar{f}-\bar{\Xi}$.
Using the gravitational equation (9), we find that

$\displaystyle\bar{K}=\textrm{constant}\,,$ (16) $\displaystyle-
C_{\phi}^{2}\bar{\psi}+C_{\phi}C_{\psi}\bar{\phi}$
$\displaystyle+C_{\phi}C_{\Xi}+C_{\psi}C_{\zeta}=0\,,$ (17)

in which the later can be treated as a constraint to be satisfied by the
integration constants. Since $\bar{\phi}$ and $\bar{\psi}$ satisfy Eq. (13),
the constraint Eq. (17) can be rewritten as

$C_{\phi}C_{\Xi}+C_{\psi}C_{\zeta}=C_{\infty}\,,$ (18)

where $C_{\infty}$ is a constant, whose value is determined by the asymptotic
value of $\bar{\phi}$ and $\bar{\psi}$.333In fact, the asymptotic values of
the scalar fields may be set to zero in order to satisfy the asymptotic
flatness condition. Note that at the weak-field regime of the DW-I model, the
values of the scalar fields, as well as that of the distortion function
$\bar{f}$, are supposed to approach zero asymptotically Chu:2018mld . We
expect this conclusion is also true for the DW-II model.

Essentially, if any of the scalar fields ($\phi$, $\psi$, $\Xi$, $\zeta$) is
excited at the background level, it could be a varying function of $r$.
Whether the field is excited or not is determined by the value of the
associated integration constants ($C_{\phi}$, $C_{\psi}$, $C_{\Xi}$,
$C_{\zeta}$). We will show explicitly later how the excited fields are related
to the black hole perturbations within the DW-II nonlocal gravity.

##  IV Black hole perturbations

In this section, we will investigate the gravitational perturbations of the
Schwarzschild black hole in the DW-II nonlocal gravity. Without loss of
generality, the perturbed spacetime can be described by a non-stationary and
axisymmetric metric in which the symmetrical axis is turned in such a way that
no $\varphi$ (the azimuthal angle) dependence appears in the metric functions.
In general, the metric can be written as follows Chandrabook :

$\displaystyle ds^{2}=$
$\displaystyle-e^{2\nu}\left(dx^{0}\right)^{2}+e^{2\mu_{1}}\left(dx^{1}-\sigma
dx^{0}-q_{2}dx^{2}-q_{3}dx^{3}\right)^{2}$
$\displaystyle+e^{2\mu_{2}}\left(dx^{2}\right)^{2}+e^{2\mu_{3}}\left(dx^{3}\right)^{2}\,.$
(19)

Up to their first-order perturbations, the metric functions

$\displaystyle e^{2\nu}$
$\displaystyle=e^{2\bar{\nu}}\left(1+2\delta\nu\right)\,,$ $\displaystyle
e^{2\mu_{1}}$
$\displaystyle=r^{2}\sin^{2}\theta\left(1+2\delta\mu_{1}\right)\,,$
$\displaystyle e^{2\mu_{2}}$
$\displaystyle=e^{-2\bar{\nu}}\left(1+2\delta\mu_{2}\right)\,,$ $\displaystyle
e^{2\mu_{3}}$ $\displaystyle=r^{2}\left(1+2\delta\mu_{3}\right)\,,$ (20)

are functions of time $t$ ($t=x^{0}$), radial coordinate $r$ ($r=x^{2}$), and
polar angle $\theta$ ($\theta=x^{3}$). As we have mentioned, the perturbed
metric is axisymmetric, therefore, we will assume that the metric functions
are independent of the azimuthal angle $\varphi$ ($\varphi=x^{1}$). The
quantities with a delta denote the corresponding first-order terms of the
fields, and contribute to the polar perturbations in general. On the other
hand, the functions $\sigma$, $q_{2}$, and $q_{3}$ are also functions of $t$,
$r$, and $\theta$, and they correspond to the axial perturbations of the
metric. We will discuss these two types (the polar and axial types) of
perturbations in more detail later. In addition, the fields that induce
nonlocal corrections should be perturbed, hence we have $K=\bar{K}+\delta K$,
$\phi=\bar{\phi}+\delta\phi$, $\psi=\bar{\psi}+\delta\psi$,
$\Xi=\bar{\Xi}+\delta\Xi$, and $\zeta=\bar{\zeta}+\delta\zeta$. We would like
to emphasize again that the background spacetime that we are considering is
the Schwarzschild metric where $e^{2\bar{\nu}}=1-2m/r$, and the scalar fields
at the background level are functions of $r$ only. The field $\bar{K}$ is a
constant, as shown in Eq. (16).

###  IV.1 Tetrad formalism

To study the perturbations of the spacetime metric (19), we use the tetrad
formalism in which one defines a basis associated with the metric (19)
Chandrabook :

$\displaystyle e^{\mu}_{(0)}$ $\displaystyle=\left(e^{-\nu},\quad\sigma
e^{-\nu},\quad 0,\quad 0\right)\,,$ $\displaystyle e^{\mu}_{(1)}$
$\displaystyle=\left(0,\quad e^{-\mu_{1}},\quad 0,\quad 0\right)\,,$
$\displaystyle e^{\mu}_{(2)}$ $\displaystyle=\left(0,\quad
q_{2}e^{-\mu_{2}},\quad e^{-\mu_{2}},\quad 0\right)\,,$ $\displaystyle
e^{\mu}_{(3)}$ $\displaystyle=\left(0,\quad q_{3}e^{-\mu_{3}},\quad 0,\quad
e^{-\mu_{3}}\right)\,,$ (21)

and

$\displaystyle e_{\mu}^{(0)}$ $\displaystyle=\left(e^{\nu},\quad 0,\quad
0,\quad 0\right)\,,$ $\displaystyle e_{\mu}^{(1)}$
$\displaystyle=\left(-\sigma e^{\mu_{1}},\quad e^{\mu_{1}},\quad-
q_{2}e^{\mu_{1}},\quad-q_{3}e^{\mu_{1}}\right)\,,$ $\displaystyle
e_{\mu}^{(2)}$ $\displaystyle=\left(0,\quad 0,\quad e^{\mu_{2}},\quad
0\right)\,,$ $\displaystyle e_{\mu}^{(3)}$ $\displaystyle=\left(0,\quad
0,\quad 0,\quad e^{\mu_{3}}\right)\,,$ (22)

where the tetrad indices are enclosed in parentheses to distinguish them from
the tensor indices. The tetrad basis then satisfies

$\displaystyle e_{\mu}^{(a)}e^{\mu}_{(b)}$
$\displaystyle=\delta^{(a)}_{(b)}\,,\quad
e_{\mu}^{(a)}e^{\nu}_{(a)}=\delta^{\nu}_{\mu}\,,$ $\displaystyle
e_{\mu}^{(a)}$ $\displaystyle=g_{\mu\nu}\eta^{(a)(b)}e^{\nu}_{(b)}\,,$
$\displaystyle g_{\mu\nu}$
$\displaystyle=\eta_{(a)(b)}e_{\mu}^{(a)}e_{\nu}^{(b)}\equiv
e_{(a)\mu}e_{\nu}^{(a)}\,.$ (23)

Essentially, in the tetrad formalism we project the relevant quantities
defined on the coordinate basis of $g_{\mu\nu}$ onto a chosen basis of
$\eta_{(a)(b)}$ by constructing the tetrad basis correspondingly. Usually,
$\eta_{(a)(b)}$ is assumed to be the Minkowskian matrix

$\eta_{(a)(b)}=\eta^{(a)(b)}=\textrm{diag}\left(-1,1,1,1\right)\,.$ (24)

In this regard, any vector or tensor field can be projected onto the tetrad
frame, in which the field can be expressed through its tetrad components:

$\displaystyle A_{\mu}$ $\displaystyle=e_{\mu}^{(a)}A_{(a)}\,,\quad
A_{(a)}=e_{(a)}^{\mu}A_{\mu}\,,$ $\displaystyle B_{\mu\nu}$
$\displaystyle=e_{\mu}^{(a)}e_{\nu}^{(b)}B_{(a)(b)}\,,\quad
B_{(a)(b)}=e_{(a)}^{\mu}e_{(b)}^{\nu}B_{\mu\nu}\,.$ (25)

One should notice that in the tetrad formalism, the covariant (partial)
derivative in the original coordinate frame is replaced by the intrinsic
(directional) derivative in the tetrad frame. For instance, the derivatives of
an arbitrary rank-two object $H_{\mu\nu}$ in the two frames are related as
follows Chandrabook

$\displaystyle\,H_{(a)(b)|(c)}\equiv
e^{\lambda}_{(c)}H_{\mu\nu;\lambda}e_{(a)}^{\mu}e_{(b)}^{\nu}$
$\displaystyle=$ $\displaystyle\,H_{(a)(b),(c)}$
$\displaystyle-\eta^{(m)(n)}\left(\gamma_{(n)(a)(c)}H_{(m)(b)}+\gamma_{(n)(b)(c)}H_{(a)(m)}\right)\,,$
(26)

where a vertical rule and a comma denote the intrinsic and directional
derivative with respect to the tetrad indices, respectively. A semicolon
denotes a covariant derivative with respect to the tensor indices.
Furthermore, the Ricci rotation coefficients are defined by

$\gamma_{(c)(a)(b)}\equiv e_{(b)}^{\mu}e_{(a)\nu;\mu}e_{(c)}^{\nu}\,,$ (27)

and their components corresponding to the metric (19) are given in Ref.
Chandrabook .

In the tetrad frame, the gravitational equation (9) can be written as

$\displaystyle
R_{(a)(b)}\left(1+K\right)-e_{(a)}^{\mu}\left(K_{,(b)}\right)_{,\mu}+\gamma_{(c)(b)(a)}K_{,(d)}\eta^{(c)(d)}$
$\displaystyle+\eta_{(a)(b)}\left\\{\Box
K-\frac{R}{2}\left(1+K\right)+\frac{1}{2}\left[\partial_{\rho}\Xi\partial^{\rho}\phi+\partial_{\rho}\psi\partial^{\rho}\zeta+\psi\left(\partial\phi\right)^{2}\right]\right\\}$
$\displaystyle-e^{\rho}_{((a)}e^{\lambda}_{(b))}\left(\partial_{\rho}\Xi\partial_{\lambda}\phi+\partial_{\rho}\psi\partial_{\lambda}\zeta+\psi\partial_{\rho}\phi\partial_{\lambda}\phi\right)=0\,.$
(28)

Note that we have assumed a vacuum spacetime: $T_{\mu\nu}=0$.

###  IV.2 Axial perturbations

According to how they react with respect to the parity change, the
gravitational perturbations of a spherically symmetric black hole can be
classified into two parts: the axial perturbations and the polar
perturbations. The master equation governing the axial perturbations can be
derived by perturbing Eq. (28) and considering its ($1,3$) and $(1,2)$
components (or equivalently the ($\varphi,\theta$) and $(\varphi,r)$
components).

Since the perturbed metric is axisymmetric and $\bar{K}$ is a constant, one
can easily verify that the master equation of the axial perturbations
$\Psi_{\textrm{aixal}}$ is described by

$R_{(1)(2)}=R_{(1)(3)}=0\,.$ (29)

Therefore, the nonlocal terms do not contribute to the axial perturbations,
and the master equation is the same as the one for the Schwarzschild black
hole in GR:

$\left(\frac{d^{2}}{dr_{*}^{2}}+\omega^{2}\right)\Psi_{\textrm{aixal}}=V_{RW}\Psi_{\textrm{aixal}}\,,$
(30)

where $r_{*}$ is the tortoise radius and $\omega$ is the quasinormal
frequencies. The effective potential $V_{RW}$, or the so-called Regge-Wheeler
potential, is defined as

${V_{RW}=\frac{e^{2\bar{\nu}}}{r^{2}}\left[l(l+1)-\frac{6m}{r}\right]}\,,\quad
e^{2\bar{\nu}}=1-\frac{2m}{r}\,,$ (31)

where $l$ is the multipole number. The master equation (30) is equivalent to
the Regge-Wheeler equation in GR Regge:1957td , which describes the axial
perturbations of the Schwarzschild black hole. Therefore, we conclude that the
nonlocal terms do not change the axial modes.

###  IV.3 Polar perturbations

In GR, the polar mode $\Psi_{\textrm{polar}}$ of the spacetime metric (19) is
essentially a linear combination of the metric perturbations $\delta\nu$,
$\delta\mu_{1}$, $\delta\mu_{2}$, and $\delta\mu_{3}$. In the DW-II nonlocal
gravity, the polar mode may also depend on the perturbations of the nonlocal
terms, such as $\delta K$, $\delta\phi$, $\delta\psi$, $\delta\Xi$, and
$\delta\zeta$. In this subsection, we will use the tetrad formalism to derive
the master equation of the polar perturbations for the Schwarzschild black
hole in the DW-II nonlocal gravity. After obtaining the master equation, one
can see how the nonlocal terms affect the polar modes. Regarding the
derivation of the master equations of the Schwarzschild black hole in GR using
the tetrad formalism, we refer the readers to Ref. Chandrabook .

First, the $(0,2)$ component of the linearized Eq. (28) can be written as

$\displaystyle\left(1+\bar{K}\right)\left[\left(\delta\mu_{1}+\delta\mu_{3}\right)_{,r}+\left(\frac{1}{r}-\bar{\nu}_{,r}\right)\left(\delta\mu_{1}+\delta\mu_{3}\right)-\frac{2}{r}\delta\mu_{2}\right]$
$\displaystyle+\,e^{\bar{\nu}}\left(e^{-\bar{\nu}}\delta K\right)_{,r}$
$\displaystyle+\frac{1}{2}\left(\bar{\phi}_{,r}\delta\Xi+\bar{\Xi}_{,r}\delta\phi+\bar{\psi}_{,r}\delta\zeta+\bar{\zeta}_{,r}\delta\psi+2\bar{\psi}\bar{\phi}_{,r}\delta\phi\right)=0\,.$
(32)

Note that we can drop the derivatives with respect to $x^{0}$ in the Fourier
space.

The $(0,3)$, $(2,3)$, and $(1,1)$ components of the linearized Eq. (28) read

$\displaystyle\left(1+\bar{K}\right)$
$\displaystyle\,\left[\left(\delta\mu_{1}+\delta\mu_{2}\right)_{,\theta}+\left(\delta\mu_{1}-\delta\mu_{3}\right)\cot\theta\right]=-\delta
K_{,\theta}\,,$ (33) $\displaystyle\left(1+\bar{K}\right)\,$
$\displaystyle\Bigg{[}\left(\delta\mu_{1}+\delta\nu\right)_{,r\theta}+\left(\delta\mu_{1}-\delta\mu_{3}\right)_{,r}\cot\theta$
$\displaystyle+\left(\bar{\nu}_{,r}-\frac{1}{r}\right)\delta\nu_{,\theta}-\left(\bar{\nu}_{,r}+\frac{1}{r}\right)\delta\mu_{2,\theta}\Bigg{]}$
$\displaystyle=$ $\displaystyle-\delta K_{,r\theta}+\frac{1}{r}\delta
K_{,\theta}-\frac{1}{2}\big{(}\bar{\phi}_{,r}\delta\Xi_{,\theta}+\bar{\Xi}_{,r}\delta\phi_{,\theta}$
$\displaystyle+\bar{\psi}_{,r}\delta\zeta_{,\theta}+\bar{\zeta}_{,r}\delta\psi_{,\theta}+2\bar{\psi}\bar{\phi}_{,r}\delta\phi_{,\theta}\big{)}\,,$
(34)

and

$\displaystyle
e^{2\bar{\nu}}\,\Bigg{[}\delta\mu_{1,rr}+2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\delta\mu_{1,r}$
$\displaystyle+\frac{1}{r}\left(\delta\mu_{1}+\delta\nu+\delta\mu_{3}-\delta\mu_{2}\right)_{,r}-\frac{2\delta\mu_{2}}{r}\left(\frac{1}{r}+2\bar{\nu}_{,r}\right)\Bigg{]}$
$\displaystyle+$
$\displaystyle\,\frac{1}{r^{2}}\left[\delta\mu_{1,\theta\theta}+\cot\theta\left(2\delta\mu_{1}+\delta\nu+\delta\mu_{2}-\delta\mu_{3}\right)_{,\theta}+2\delta\mu_{3}\right]$
$\displaystyle-e^{-2\bar{\nu}}\delta\mu_{1,tt}$ $\displaystyle=$
$\displaystyle\,\frac{-1}{1+\bar{K}}\left(\frac{\Box\delta
K}{2}+\frac{e^{2\bar{\nu}}}{r}\delta K_{,r}+\frac{\cot\theta}{r^{2}}\delta
K_{,\theta}\right)\,,$ (35)

respectively. Finally, the $(2,2)$ component of linearized Eq. (28) can be
written as

$\displaystyle e^{2\bar{\nu}}$
$\displaystyle\left[\frac{2}{r}\delta\nu_{,r}+\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\left(\delta\mu_{1}+\delta\mu_{3}\right)_{,r}-2\delta\mu_{2}\left(\frac{1}{r^{2}}+\frac{2\bar{\nu}_{,r}}{r}\right)\right]$
$\displaystyle+\frac{1}{r^{2}}\left[\left(\delta\mu_{1}+\delta\nu\right)_{,\theta\theta}+\left(2\delta\mu_{1}+\delta\nu-\delta\mu_{3}\right)_{,\theta}\cot\theta+2\delta\mu_{3}\right]$
$\displaystyle-e^{-2\bar{\nu}}\left(\delta\mu_{1}+\delta\mu_{3}\right)_{,tt}$
$\displaystyle=$ $\displaystyle\,\frac{-1}{1+\bar{K}}\Bigg{[}\Box\delta
K-e^{\bar{\nu}}\left(\delta
K_{,r}e^{\bar{\nu}}\right)_{,r}-\frac{e^{2\bar{\nu}}}{2}\big{(}\bar{\phi}_{,r}\delta\Xi_{,r}+\delta\phi_{,r}\bar{\Xi}_{,r}$
$\displaystyle+\bar{\psi}_{,r}\delta\zeta_{,r}+\delta\psi_{,r}\bar{\zeta}_{,r}+\delta\psi\bar{\phi}_{,r}^{2}+2\bar{\psi}\bar{\phi}_{,r}\delta\phi_{,r}\big{)}\Bigg{]}\,.$
(36)

To proceed further, we consider the following field decompositions Chandrabook
:

$\displaystyle\delta\nu$ $\displaystyle=N(r)P_{l}e^{i\omega t}\,,$
$\displaystyle\delta\mu_{2}$ $\displaystyle=L(r)P_{l}e^{i\omega t}\,,$
$\displaystyle\delta\mu_{3}$
$\displaystyle=\left[T(r)P_{l}+V(r)P_{l,\theta\theta}\right]e^{i\omega t}\,,$
$\displaystyle\delta\mu_{1}$
$\displaystyle=\left[T(r)P_{l}+V(r)P_{l,\theta}\cot\theta\right]e^{i\omega
t}\,,$ $\displaystyle\delta\mu_{1}+\delta\mu_{3}$
$\displaystyle=\left[2T-l(l+1)V\right]P_{l}e^{i\omega t}\,,$
$\displaystyle\delta K$
$\displaystyle=(1+\bar{K})\delta\tilde{K}(r)P_{l}e^{i\omega t}\,,$ (37)

where $P_{l}$ is the Legendre polynomials. Furthermore, according to Eqs.
(13), (14), and (15), we find it convenient to redefine a new scalar field and
make the following decomposition

$\displaystyle U$ $\displaystyle=\frac{\tilde{U}(r)}{r}P_{l}e^{i\omega t}$
$\displaystyle\equiv\frac{C_{\phi}\delta\Xi+C_{\Xi}\delta\phi+C_{\psi}\delta\zeta+\left(C_{\phi}\bar{\phi}+C_{\zeta}\right)\delta\psi}{2\left(1+\bar{K}\right)}\,.$
(38)

Then, Eqs. (32), (33), (34), (36) can be written as

$\displaystyle\left[\frac{d}{dr}+\left(\frac{1}{r}-\bar{\nu}_{,r}\right)\right]$
$\displaystyle\left[2T-l(l+1)V\right]-\frac{2}{r}L$ $\displaystyle=$
$\displaystyle-e^{\bar{\nu}}\left(e^{-\bar{\nu}}\delta\tilde{K}\right)_{,r}-\frac{e^{-2\bar{\nu}}}{r^{3}}\tilde{U}\,,$
(39) $\displaystyle T-V+L=$ $\displaystyle-\delta\tilde{K}\,,$ (40)
$\displaystyle\left(T-V+N\right)_{,r}$
$\displaystyle-\left(\frac{1}{r}-\bar{\nu}_{,r}\right)N-\left(\frac{1}{r}+\bar{\nu}_{,r}\right)L$
$\displaystyle=$
$\displaystyle-\delta\tilde{K}_{,r}+\frac{1}{r}\delta\tilde{K}-\frac{e^{-2\bar{\nu}}}{r^{3}}\tilde{U}\,,$
(41)

and

$\displaystyle
e^{2\bar{\nu}}\Bigg{[}\frac{2}{r}N_{,r}+\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\left(2T-l(l+1)V\right)_{,r}$
$\displaystyle-\frac{2}{r}\left(\frac{1}{r}+2\bar{\nu}_{,r}\right)L\Bigg{]}-\frac{l(l+1)N}{r^{2}}-\frac{(l+2)(l-1)}{r^{2}}T$
$\displaystyle+e^{-2\bar{\nu}}\omega^{2}\left(2T-l(l+1)V\right)$
$\displaystyle=$
$\displaystyle\,-e^{-2\bar{\nu}}\omega^{2}\delta\tilde{K}-\left(\frac{2}{r}+\bar{\nu}_{,r}\right)e^{2\bar{\nu}}\delta\tilde{K}_{,r}$
$\displaystyle+\frac{l(l+1)}{r^{2}}\delta\tilde{K}+\frac{1}{r^{2}}\left({\frac{\tilde{U}}{r}}\right)_{,r}\,,$
(42)

respectively.

Defining $X\equiv nV$ where $n\equiv(l+2)(l-1)/2$, one gets

$2T-l(l+1)V=-2(L+X+\delta\tilde{K})\,,$ (43)

where we have used Eq. (40). Using Eq. (43), one can rewrite Eqs. (39) and
(41) as

$\displaystyle\left(L+X+\frac{\delta\tilde{K}}{2}\right)_{,r}=$
$\displaystyle-\left(\frac{1}{r}-\bar{\nu}_{,r}\right)\left(L+X+\frac{\delta\tilde{K}}{2}\right)$
$\displaystyle-\frac{L}{r}$
$\displaystyle-\frac{1}{2}\left(\frac{\delta\tilde{K}}{r}-\frac{e^{-2\bar{\nu}}}{r^{3}}\tilde{U}\right)\,,$
(44) $\displaystyle N_{,r}-L_{,r}=\left(\frac{1}{r}-\bar{\nu}_{,r}\right)N$
$\displaystyle+\left(\frac{1}{r}+\bar{\nu}_{,r}\right)L$
$\displaystyle+\frac{1}{r}\delta\tilde{K}-\frac{e^{-2\bar{\nu}}}{r^{3}}\tilde{U}\,.$
(45)

Then, subtracting Eqs. (35) from (36), and taking only the terms proportional
to $P_{l,\theta}\cot\theta$, one gets

$\displaystyle V_{,rr}$
$\displaystyle+2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)V_{,r}$
$\displaystyle+\frac{e^{-2\bar{\nu}}}{r^{2}}\left(N+L+\delta\tilde{K}\right)+\omega^{2}e^{-4\bar{\nu}}V=0\,.$
(46)

Furthermore, one can remove all the $\delta\tilde{K}$ terms by making the
following field redefinitions:

$\tilde{L}\equiv L+\frac{\delta\tilde{K}}{2}\,,\qquad\tilde{N}\equiv
N+\frac{\delta\tilde{K}}{2}\,,$ (47)

and the above equations can be rewritten as

$\displaystyle\left(\tilde{L}+X\right)_{,r}=-\left(\frac{1}{r}-\bar{\nu}_{,r}\right)\left(\tilde{L}+X\right)-\frac{\tilde{L}}{r}+\frac{e^{-2\bar{\nu}}}{2r^{3}}\tilde{U}\,,$
(48)
$\displaystyle\tilde{N}_{,r}-\tilde{L}_{,r}=\left(\frac{1}{r}-\bar{\nu}_{,r}\right)\tilde{N}+\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\tilde{L}-\frac{e^{-2\bar{\nu}}}{r^{3}}\tilde{U}\,,$
(49) $\displaystyle
V_{,rr}+2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)V_{,r}+\frac{e^{-2\bar{\nu}}}{r^{2}}\left(\tilde{N}+\tilde{L}\right)+\omega^{2}e^{-4\bar{\nu}}V=0\,.$
(50)

Also, Eq. (42) can be written as

$\displaystyle\frac{2}{r}\tilde{N}_{,r}-2\left(\frac{1}{r}+\bar{\nu}_{,r}\right)\left(\tilde{L}+X\right)_{,r}-\frac{2}{r}\left(\frac{1}{r}+2\bar{\nu}_{,r}\right)\tilde{L}$
$\displaystyle-$
$\displaystyle\,\frac{l(l+1)}{r^{2}}e^{-2\bar{\nu}}\tilde{N}-\frac{2n}{r^{2}}\left(V-\tilde{L}\right)e^{-2\bar{\nu}}-2\omega^{2}e^{-4\bar{\nu}}\left(\tilde{L}+X\right)$
$\displaystyle=$
$\displaystyle\,\left(\frac{e^{-2\bar{\nu}}}{r^{2}}-\frac{1}{r^{2}}-\frac{2\bar{\nu}_{,r}}{r}\right)\delta\tilde{K}+\frac{e^{-2\bar{\nu}}}{r^{2}}\left({\frac{\tilde{U}}{r}}\right)_{,r}$
$\displaystyle=$
$\displaystyle\,\frac{e^{-2\bar{\nu}}}{r^{2}}\left({\frac{\tilde{U}}{r}}\right)_{,r}\,,$
(51)

where we have used $e^{2\bar{\nu}}=1-2m/r$ in the last equality.

Then, using Eqs. (48), (49), and (51), one can obtain

$\displaystyle\tilde{N}_{,r}=$
$\displaystyle\,\alpha\tilde{N}+\beta\tilde{L}+\gamma X$
$\displaystyle+\frac{e^{-2\bar{\nu}}}{2r^{3}}\left[\left(1+r\bar{\nu}_{,r}\right)\tilde{U}+r^{2}\left(\frac{\tilde{U}}{r}\right)_{,r}\right]\,,$
(52) $\displaystyle\tilde{L}_{,r}=$
$\displaystyle\,\left(\alpha-\frac{1}{r}+\bar{\nu}_{,r}\right)\tilde{N}+\left(\beta-\frac{1}{r}-\bar{\nu}_{,r}\right)\tilde{L}+\gamma
X$
$\displaystyle+\frac{e^{-2\bar{\nu}}}{2r^{3}}\left[\left(3+r\bar{\nu}_{,r}\right)\tilde{U}+r^{2}\left(\frac{\tilde{U}}{r}\right)_{,r}\right]\,,$
(53) $\displaystyle X_{,r}=$
$\displaystyle-\left(\alpha-\frac{1}{r}+\bar{\nu}_{,r}\right)\tilde{N}-\left(\beta-2\bar{\nu}_{,r}+\frac{1}{r}\right)\tilde{L}$
$\displaystyle-\left(\gamma+\frac{1}{r}-\bar{\nu}_{,r}\right)X$
$\displaystyle-\frac{e^{-2\bar{\nu}}}{2r^{3}}\left[\left(2+r\bar{\nu}_{,r}\right)\tilde{U}+r^{2}\left(\frac{\tilde{U}}{r}\right)_{,r}\right]$
$\displaystyle=$ $\displaystyle\,\frac{\lambda
e^{-2\bar{\nu}}}{r^{2}}\tilde{N}-\left(\gamma+\frac{1}{r}-\bar{\nu}_{,r}\right)\left(\tilde{L}+X\right)+\alpha\tilde{L}$
$\displaystyle-\frac{e^{-2\bar{\nu}}}{2r^{3}}\left[\left(2+r\bar{\nu}_{,r}\right)\tilde{U}+r^{2}\left(\frac{\tilde{U}}{r}\right)_{,r}\right]\,,$
(54)

where

$\displaystyle\alpha$ $\displaystyle=\frac{l(l+1)}{2r}e^{-2\bar{\nu}}\,,$
$\displaystyle\beta$
$\displaystyle=-\frac{1}{r}+r\left(\bar{\nu}_{,r}\right)^{2}+\bar{\nu}_{,r}-\frac{ne^{-2\bar{\nu}}}{r}+re^{-4\bar{\nu}}\omega^{2}\,,$
$\displaystyle\gamma$
$\displaystyle=-\frac{1}{r}+r\left(\bar{\nu}_{,r}\right)^{2}+\frac{e^{-2\bar{\nu}}}{r}+re^{-4\bar{\nu}}\omega^{2}\,,$
$\displaystyle\lambda$ $\displaystyle=-nr-3m\,.$ (55)

Defining

$\Psi_{\textrm{polar}}\equiv\frac{r}{n}X+\frac{r^{2}}{\lambda}\left(\tilde{L}+X\right)\,,$
(56)

and differentiating it with respect to the tortoise radius $r_{*}$ twice, we
get the master equation

$\displaystyle\left(\frac{d^{2}}{dr_{*}^{2}}+\omega^{2}\right)\Psi_{\textrm{polar}}=$
$\displaystyle\,V_{Z}\Psi_{\textrm{polar}}$ $\displaystyle+\,e^{2\bar{\nu}}$
$\displaystyle\,\left[\frac{3m+2nr}{r^{2}\left(3m+nr\right)^{2}}\right]\tilde{U}\,,$
(57)

where the Zerilli potential $V_{Z}$ is Zerilli:1970se

$V_{Z}=\frac{2e^{2\bar{\nu}}\left[n^{2}(n+1)r^{3}+3mn^{2}r^{2}+9m^{2}nr+9m^{3}\right]}{r^{3}\left(nr+3m\right)^{2}}\,.$
(58)

Therefore, one can see that the polar modes are sourced by the scalar mode
$\tilde{U}$. If the scalar mode is not excited, the master equation reduces to
the Zerilli equation in GR Zerilli:1970se .

From the linearized scalar field equations (5), (6), (7), and (8), one finds
that the scalar mode $U$ satisfies a massless Klein-Gordon equation444It
should be mentioned that the constraint Eq. (17) has to be used in order to
prove Eq. (59).:

$\Box U=0\,,$ (59)

which, using the decomposition given by the first line of Eq. (38), can be
written as

$\left(\frac{d^{2}}{dr_{*}^{2}}+\omega^{2}\right)\tilde{U}=V_{s}\tilde{U}\,,$
(60)

where the effective potential is

$V_{s}=\frac{e^{2\bar{\nu}}}{r^{2}}\left[l(l+1)+\frac{2m}{r}\right]\,.$ (61)

It should be noted that even if the scalar mode $U$ satisfies the massless
Klein-Gordon equation, it is excited only when at least one of the scalar
fields ($\bar{\phi},\bar{\Xi},\bar{\psi},\bar{\zeta}$) is excited already at
the background level. More precisely, as one can see from Eq. (38), the scalar
mode is excited when there exists at least one non-zero integration constant
in the set ($C_{\phi},C_{\Xi},C_{\psi},C_{\zeta}$). If so, the scalar mode
$\tilde{U}$ is dynamical and it would source the polar modes. As a result, the
polar modes will be affected by the scalar mode and the isospectrality breaks
down. On the other hand, if all the scalar fields are quiescent at the
background level ($C_{\phi}=C_{\Xi}=C_{\psi}=C_{\zeta}=0$), then the scalar
mode vanishes and the Schwarzschild black hole in the DW-II nonlocal gravity
is completely indistinguishable from its GR counterpart even at the
perturbative level.

##  V Remarks for DW-I nonlocal model

In this section, we will briefly exhibit that in the DW-I nonlocal gravity,
the master equations of the gravitational perturbations share the similar
property to those in the DW-II gravity. The resemblance between the two models
results from the structural similarity of their equations of motion. In short,
the conclusions that the axial gravitational perturbations are the same as in
GR, and that the polar gravitational perturbations are sourced by a massless
scalar mode, which depends on whether the scalar fields are excited at the
background level or not, remain true for the DW-I model.

The action of the DW-I nonlocal gravity is Deser:2007jk

$\mathcal{S}=\frac{1}{16\pi}\int
d^{4}x\sqrt{-g}R\left[1+f\left(\Box^{-1}R\right)\right]+\mathcal{S}_{m}\,,$
(62)

Following the procedure in Ref. Nojiri:2007uq , one can localize the
gravitational action (62) by introducing an auxiliary scalar field $\phi$, as
in Eq. (3), and a Lagrange multiplier $\xi$, such that the action can be
written as

$\mathcal{S}=\frac{1}{16\pi}\int
d^{4}x\sqrt{-g}\left[R\left(1+f\right)-\partial_{\mu}\xi\partial^{\mu}\phi-\xi
R\right]+\mathcal{S}_{m}\,.$ (63)

The equations of motion are derived by varying the action (63) with respect to
$\xi$, $\phi$, and the metric $g_{\mu\nu}$. Varying the action with respect to
the Lagrange multiplier $\xi$ essentially gives Eq. (5). Then, after varying
the action with respect to the auxiliary scalar field $\phi$, one can obtain

$\Box\xi=-R\frac{df}{d\phi}\,.$ (64)

Finally, the variation of the action (63) with respect to the metric
$g_{\mu\nu}$ leads to the modified Einstein field equation

$G_{\mu\nu}+\Delta G_{\mu\nu}=8\pi T_{\mu\nu}\,,$ (65)

where

$\displaystyle\Delta G_{\mu\nu}=$
$\displaystyle\,\left(G_{\mu\nu}+g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu}\right)\left(f-\xi\right)$
$\displaystyle+\frac{1}{2}g_{\mu\nu}\partial_{\rho}\xi\partial^{\rho}\phi-\partial_{(\mu}\xi\partial_{\nu)}\phi\,.$
(66)

For the non-perturbed Schwarzschild metric with the Ricci scalar being zero,
one gets

$\bar{\phi}_{,r}=C_{\phi}\frac{e^{-2\bar{\nu}}}{r^{2}}\,,\qquad\bar{\xi}_{,r}=C_{\xi}\frac{e^{-2\bar{\nu}}}{r^{2}}\,,$
(67)

where $C_{\xi}$ is an integration constant. Furthermore, at the background
level, the gravitational equation (65) implies that

$\bar{F}\equiv\bar{f}-\bar{\xi}=\textrm{constant}\,,\quad
C_{\phi}C_{\xi}=0\,.$ (68)

Therefore, in the DW-I model, at least one of the scalar fields ($\bar{\phi}$
and $\bar{\xi}$) has to be a constant.

For the perturbations of the Schwarzschild black hole, it is easy to find that
the master equation of the axial perturbations is again the Regge-Wheeler
equation

$\left(\frac{d^{2}}{dr_{*}^{2}}+\omega^{2}\right)\Psi_{\textrm{aixal}}=V_{RW}\Psi_{\textrm{aixal}}\,,$
(69)

where the effective potential $V_{RW}$ is

${V_{RW}=\frac{e^{2\bar{\nu}}}{r^{2}}\left[l(l+1)-\frac{6m}{r}\right]}\,,\quad
e^{2\bar{\nu}}=1-\frac{2m}{r}\,.$ (70)

As for the polar perturbations, we find that upon defining the following
scalar mode

$W\equiv\frac{\tilde{W}(r)}{r}P_{l}e^{i\omega
t}\equiv\frac{1}{2\left(1+\bar{F}\right)}\left(C_{\phi}\delta\xi+C_{\xi}\delta\phi\right)\,,$
(71)

where we have defined $F\equiv f-\xi$ and decompose $F$ as $F=\bar{F}+\delta
F$, the master equation of the polar perturbations in the DW-I gravity can be
derived under the recipe of the DW-II gravity, with the replacement
$(\bar{K},\delta K,\tilde{U})\rightarrow(\bar{F},\delta F,\tilde{W})$.
Therefore, the master equation of the polar perturbations takes the following
form:

$\displaystyle\left(\frac{d^{2}}{dr_{*}^{2}}+\omega^{2}\right)\Psi_{\textrm{polar}}=$
$\displaystyle\,V_{Z}\Psi_{\textrm{polar}}$ $\displaystyle+\,e^{2\bar{\nu}}$
$\displaystyle\,\left[\frac{3m+2nr}{r^{2}\left(3m+nr\right)^{2}}\right]\tilde{W}\,,$
(72)

where the Zerilli potential $V_{Z}$ is given by

$V_{Z}=\frac{2e^{2\bar{\nu}}\left[n^{2}(n+1)r^{3}+3mn^{2}r^{2}+9m^{2}nr+9m^{3}\right]}{r^{3}\left(nr+3m\right)^{2}}\,.$
(73)

Similar to what we have done in the DW-II nonlocal gravity, by using the
linearized Eqs. (5), (64), and the constraint $C_{\phi}C_{\xi}=0$, it can be
proven that $\Box W=0$. Therefore, the polar modes are sourced by a massless
scalar mode $W$, which is defined by Eq. (71). One can see that whether the
scalar mode is excited or not depends on whether the scalar fields
$(\phi,\xi)$ are excited or not at the background level. More specifically, if
none of them is excited at the background level ($C_{\phi}=C_{\xi}=0$), the
polar gravitational perturbations are completely the same as those in GR. On
the other hand, if one of the integration constants ($C_{\phi},C_{\xi}$) is
not zero, the polar modes would be sourced by the massless scalar mode $W$,
and the isospectrality would break.

##  VI Conclusions

The DW-I nonlocal gravity was originally proposed in order to explain the
late-time acceleration of the universe without resorting to any mysterious
dark energy. The recent discovery Belgacem:2018wtb  that the DW-I is conflict
with the solar system tests has prompted the authors of Ref. Deser:2019lmm  to
improve the model and propose the DW-II nonlocal gravity. These two theories
share an interesting property, which is that the auxiliary scalar fields in
the localized framework are not completely dynamical, but subject to the
initial data. It is thus interesting to see whether these nonlocal corrections
would alter the QNMs or not, as compared with those in GR.

In this paper, we investigated the gravitational perturbations of the
Schwarzschild black hole in the DW-II nonlocal gravity. The gravitational
perturbations of the Schwarzschild black hole in GR are governed by the Regge-
Wheeler equation (axial modes) and the Zerilli equation (polar modes), and the
two corresponding modes share the same QNM spectrum. In fact, the
isospectrality between the axial and the polar modes have been shown to be
very fragile Cardoso:2019mqo , in the sense that it easily breaks when the
underlying theory is slightly changed. For instance, the isospectrality is no
longer satisfied in the presence of a dynamical scalar field, such as in the
$f(R)$ gravity Bhattacharyya:2017tyc ; Datta:2019npq .

Essentially, in this work we showed that the axial gravitational perturbations
in the DW-II gravity are completely identical with those in GR. On the other
hand, the polar modes are sourced by a massless scalar mode, if there is at
least one auxiliary scalar field being excited at the background level.
However, if all the auxiliary scalar fields stay quiescent at the background
level, the scalar mode disappears and the polar modes reduce to those of the
Schwarzschild black hole in GR. Whether the auxiliary fields are excited or
not depends on the values of their corresponding integration constants
($C_{\phi},C_{\psi},C_{\Xi},C_{\zeta}$). We also performed a similar analysis
in the DW-I gravity and, thanks to their structural resemblance, a similar
conclusion was drawn.

Remarkably, although the excitation of the auxiliary scalar fields does break
the isospectrality, meaning that the observations of isospectrality breaking
can constrain the integration constants of the auxiliary scalar fields,
however, the question about whether one can really falsify the DW-II gravity
via isospectrality tests should be answered with great care. The reason is
that what one really constrains, through isospectrality tests, are just some
integration constants which do not directly characterize the theory. This is
different from the cases in some other models (e.g. $f(R)$ gravity or
dynamical Chern-Simons gravity) in which the isospectrality tests directly
constrain the coupling constants of the theories. This thus makes the DW-II
gravity more intrinsically tenacious against isospectrality tests. Therefore,
at this point, since DW-I model has been ruled out, we want to point out that
the DW-II nonlocal gravity seems not only successful in describing the
accelerating expansion of our universe in the cosmological scale, but also
theoretically consistent down to small scales as such of black holes. From an
astrophysical point of view, to confirm the validity of the DW model on a more
concrete basis, one should then consider rotating black holes since
astrophysical black holes are usually spinning. It would be interesting to see
in such cases, whether the DW model still gives predictions identical with
those made in GR, or some distinctive features would emerge. We leave these
issues for our future works.

## Acknowledgments

CYC is supported by the Institute of Physics of Academia Sinica.

## References

## References

  * (1) B. P. Abbott et al. [LIGO Scientific and Virgo], Phys. Rev. Lett. 116 (2016) no.6, 061102.
  * (2) B. P. Abbott et al. [LIGO Scientific and Virgo], Phys. Rev. X 9 (2019) no.3, 031040.
  * (3) R. Abbott et al. [LIGO Scientific and Virgo], [arXiv:2010.14527 [gr-qc]].
  * (4) K. D. Kokkotas and B. G. Schmidt, Living Rev. Rel. 2 (1999), 2.
  * (5) E. Berti, V. Cardoso and A. O. Starinets, Class. Quant. Grav. 26, 163001 (2009).
  * (6) R. A. Konoplya and A. Zhidenko, Rev. Mod. Phys. 83 (2011), 793-836.
  * (7) T. Kobayashi, H. Motohashi and T. Suyama, Phys. Rev. D 85, 084025 (2012) [erratum: Phys. Rev. D 96, no.10, 109903 (2017)].
  * (8) T. Kobayashi, H. Motohashi and T. Suyama, Phys. Rev. D 89, no.8, 084042 (2014).
  * (9) J. L. Blázquez-Salcedo, C. F. B. Macedo, V. Cardoso, V. Ferrari, L. Gualtieri, F. S. Khoo, J. Kunz and P. Pani, Phys. Rev. D 94, no.10, 104024 (2016).
  * (10) S. Bhattacharyya and S. Shankaranarayanan, Phys. Rev. D 96 (2017) no.6, 064044.
  * (11) K. Glampedakis, G. Pappas, H. O. Silva and E. Berti, Phys. Rev. D 96, no.6, 064054 (2017).
  * (12) C. Y. Chen and P. Chen, Phys. Rev. D 98, no.4, 044042 (2018).
  * (13) C. Y. Chen, M. Bouhmadi-López and P. Chen, Eur. Phys. J. C 79, no.1, 63 (2019).
  * (14) C. Y. Chen and P. Chen, Phys. Rev. D 99, no.10, 104003 (2019).
  * (15) F. Moulin, A. Barrau and K. Martineau, Universe 5, no.9, 202 (2019).
  * (16) C. Y. Chen, Y. H. Kung and P. Chen, Phys. Rev. D 102, 124033 (2020).
  * (17) V. Cardoso, A. S. Miranda, E. Berti, H. Witek and V. T. Zanchin, Phys. Rev. D 79, 064016 (2009).
  * (18) S. R. Dolan, Phys. Rev. D 82, 104003 (2010).
  * (19) H. Yang, D. A. Nichols, F. Zhang, A. Zimmerman, Z. Zhang and Y. Chen, Phys. Rev. D 86, 104006 (2012).
  * (20) B. Cuadros-Melgar, R. D. B. Fontana and J. de Oliveira, Phys. Lett. B 811, 135966 (2020).
  * (21) R. A. Konoplya and Z. Stuchlík, Phys. Lett. B 771, 597-602 (2017).
  * (22) M. S. Churilova, Eur. Phys. J. C 79, no.7, 629 (2019).
  * (23) K. Glampedakis and H. O. Silva, Phys. Rev. D 100, no.4, 044040 (2019).
  * (24) C. Y. Chen and P. Chen, Phys. Rev. D 101, no.6, 064021 (2020).
  * (25) S. Chandrasekhar(ed.): The Mathematical Theory of Black Holes. Oxford University Press, Oxford (1992).
  * (26) P. Pani, E. Berti and L. Gualtieri, Phys. Rev. Lett. 110, no.24, 241103 (2013).
  * (27) P. Pani, E. Berti and L. Gualtieri, Phys. Rev. D 88, 064048 (2013).
  * (28) K. Glampedakis, A. D. Johnson and D. Kennefick, Phys. Rev. D 96, no.2, 024036 (2017).
  * (29) G. Darboux, C. R. Academy Sci. (Paris) 94, 1456 (1882).
  * (30) A. V. Yurov and V. A. Yurov, Phys. Lett. A 383, no.22, 2571-2578 (2019).
  * (31) S. A. Teukolsky, Astrophys. J. 185, 635-647 (1973).
  * (32) M. Sasaki and T. Nakamura, Prog. Theor. Phys. 67, 1788 (1982).
  * (33) F. Moulin and A. Barrau, Gen. Rel. Grav. 52, no.8, 82 (2020).
  * (34) V. Ferrari, M. Pauri and F. Piazza, Phys. Rev. D 63, 064009 (2001).
  * (35) V. Cardoso and J. P. S. Lemos, Phys. Rev. D 64, 084017 (2001).
  * (36) E. Berti and K. D. Kokkotas, Phys. Rev. D 67, 064020 (2003).
  * (37) J. L. Blázquez-Salcedo, S. Kahlen and J. Kunz, Eur. Phys. J. C 79, no.12, 1021 (2019).
  * (38) S. Bhattacharyya and S. Shankaranarayanan, Eur. Phys. J. C 78 (2018) no.9, 737.
  * (39) S. Datta and S. Bose, Eur. Phys. J. C 80 (2020) no.1, 14.
  * (40) S. Bhattacharyya and S. Shankaranarayanan, Phys. Rev. D 100, no.2, 024022 (2019).
  * (41) V. Cardoso, M. Kimura, A. Maselli, E. Berti, C. F. B. Macedo and R. McManus, Phys. Rev. D 99, no.10, 104077 (2019).
  * (42) S. Shankaranarayanan, Int. J. Mod. Phys. D 28, no.14, 1944020 (2019).
  * (43) S. Deser and R. P. Woodard, Phys. Rev. Lett. 99 (2007) 111301.
  * (44) S. Deser and R. P. Woodard, JCAP 06, 034 (2019).
  * (45) R. P. Woodard, Found. Phys. 44 (2014), 213.
  * (46) R. P. Woodard, Universe 4 (2018), 88.
  * (47) A. O. Barvinsky, Phys. Lett. B 710 (2012) 12.
  * (48) M. Maggiore, Phys. Rev. D 89 (2014) 043008.
  * (49) M. Maggiore and M. Mancarella, Phys. Rev. D 90 (2014) 023005.
  * (50) V. Vardanyan, Y. Akrami, L. Amendola and A. Silvestri, JCAP 1803 (2018) 048.
  * (51) L. Amendola, N. Burzilla and H. Nersisyan, Phys. Rev. D 96 (2017) 084031.
  * (52) S. Tian, Phys. Rev. D 98 (2018) 084040.
  * (53) C. Deffayet and R. P. Woodard, JCAP 08 (2009), 023.
  * (54) S. Park and S. Dodelson, Phys. Rev. D 87 (2013) 024003.
  * (55) S. Dodelson and S. Park, Phys. Rev. D 90 (2014) 043535, Erratum: [Phys. Rev. D 98 (2018) 029904].
  * (56) S. Park and A. Shafieloo, Phys. Rev. D 95 (2017) 064061.
  * (57) H. Nersisyan, A. F. Cid and L. Amendola, JCAP 04 (2017), 046.
  * (58) S. Park, Phys. Rev. D 97 (2018) 044006.
  * (59) L. Amendola, Y. Dirian, H. Nersisyan and S. Park, JCAP 03 (2019), 045.
  * (60) S. Nojiri and S. D. Odintsov, Phys. Lett. B 659, 821-826 (2008).
  * (61) Y. l. Zhang, K. Koyama, M. Sasaki and G. B. Zhao, JHEP 1603 (2016) 039.
  * (62) S. Nojiri, S. D. Odintsov, M. Sasaki and Y. l. Zhang, Phys. Lett. B 696 (2011) 278.
  * (63) Y. l. Zhang and M. Sasaki, Int. J. Mod. Phys. D 21 (2012) 1250006.
  * (64) S. Park and R. P. Woodard, Phys. Rev. D 99 (2019) 024014.
  * (65) E. Belgacem, A. Finke, A. Frassino and M. Maggiore, JCAP 1902 (2019) 035.
  * (66) Y. Z. Chu and S. Park, Phys. Rev. D 99 (2019) 044052.
  * (67) T. Regge and J. A. Wheeler, Phys. Rev. 108 (1957) 1063.
  * (68) F. J. Zerilli, Phys. Rev. Lett. 24, 737-738 (1970).


# Extreme Face Inpainting with Sketch-Guided Conditional GAN

Nilesh Pandey and Andreas Savakis; Rochester Institute of Technology;
Rochester, NY, USA

###### Abstract

Recovering badly damaged face images is a useful yet challenging task,
especially in extreme cases where the masked or damaged region is very large.
One of the major challenges is the ability of the system to generalize on
faces outside the training dataset. We propose to tackle this extreme
inpainting task with a conditional Generative Adversarial Network (GAN) that
utilizes structural information, such as edges, as a prior condition. Edge
information can be obtained from the partially masked image and a structurally
similar image or a hand drawing. In our proposed conditional GAN, we pass the
conditional input in every layer of the encoder while maintaining consistency
in the distributions between the learned weights and the incoming conditional
input. We demonstrate the effectiveness of our method with badly damaged face
examples.

## Introduction

Image inpainting is a popular and challenging problem in computer vision with
applications in image editing and restoring of damaged image regions [1], [2],
[3], [4], [5], [6]. The damaged image regions may range from rectangular
patches of various sizes to irregular regions or entire objects. In image
editing, some regions are tactically damaged so that they can be inpainted
with different content.

In this paper we deal with the problem of extreme inpainting, where large
regions of faces have been damaged with the application of an irregular mask.
Initial works dealt with small square patches, while more recent approaches
take on the more challenging problem of inpainting large irregular regions
[3], [4]. Our face inpainting method, called FIpoly-GAN, is based on the Poly-
GAN conditional Generative Adversarial Network (GAN) architecture [7], that
was initially proposed for fashion synthesis and is suitable for various
tasks, e.g. geometric transformations, image stitching and inpainting. To
recover the facial structure that is lost after significant image regions have
been masked, we utilize an edge sketch as a condition that guides the
inpainting process by providing structure in the damaged regions. The proposed
FIpoly-GAN has shown ability to generalize on faces obtained in the wild from
various sources unrelated to the training data. A representative result is
shown in in Fig. 1. The main contributions of this paper can be summarized as
follows:

  * •

We propose FIpoly-GAN, a conditional GAN architecture that leverages edge
information for extreme face inpainting.

  * •

Our FIpoly-GAN architecture uses the edge map, from a sketch or a similar
image, as a condition that is fed at various levels of the network to learn
the structure of the image for inpainting color and texture in the missing
region.

  * •

FIpoly-GAN achieves state of the art results on standard face datasets and
outperforms other methods on images that are unrelated to the training
dataset.

Figure 1: Example of sketch-conditioned face inpainting with the proposed
FIpoly-GAN network. From left to right: original, damaged image with large
irregular mask, hand-drawn sketch, inpainted result.  Figure 2: The FIpoly-GAN
architecture used for edge-guided face inpainting. The left side block shows
the overall architecture and the right side shows more details of the encoder
and decoder layers.

## Related Work

Early inpainting methods operated under the assumption that the damaged region
is a standard shape, i.e. square or rectangle. Free form inpainting, where the
damaged regions are large and irregular, poses a challenge to many methods due
to lack of relevant information that can be used from nearby regions. A
popular approach is the Patch-based image inpainting network [1], which
progressively learns to associate nearby image contect to the missing region.
However, this approach is not as effective in the case of free form
inpainting.

Image Inpainting using Partial Convolution [2] is one of the first methods to
tackle free form inpainting on images with irregular damage. It utilizes a
neural network with a partial convolution layer and performs an automatic
update on the binary mask. Given a binary mask, the partial convolution
utilizes non-damaged information near damaged region to estimate missing
content. The next step in the method is to update the binary mask according to
the output of the partial convolution, so that if the output is positive for
the damaged region then the mask is updated accordingly.

Free-form image inpainting with Gated Convolution (SN-patchGAN) [3] is a
state-of-the-art method for inpainting images with irregular damage. This
method introduces a new convolution layer named gated convolution. The input
features in the model are first used to compute the gating value,
$g=sigmoid(W_{g}*x)$, where $x$ denotes the input features. The final output
of the layer is multiplication of learned features and gating values,
$y=\phi(W*x)\odot g$. The architecture includes a contextual attention module
with the refinement network, with both having a gated convolution layer. Gated
Convolution has inspired several other methods for image inpainting.

Image inpainting via Generative Multi-column Convolutional Neural Networks
(GMCNN) [4] The GMCNN network consists of three sub-networks, one generator,
one global and one local discriminator for adversarial training. Only the
generator is deployed when testing on a new image.

SC-FEGAN [5] and Face-Shop [6] are state-of-the-art image inpainting methods
based on image editing. They work by providing the network a simple sketch of
the face. Both methods use a similar approach to generate sketch images of the
face, which can be edited and passed to the model as conditional input. The
EdgeConnect method [8] uses an edge generator to estimate structural
information in the missing regions.

The SC-FEGAN trains both the generator and discriminator to generate a
(512x512) image. The SC-FEGAN generator uses gated convolution in all layers,
except the initial input layer in the encoder and final output layer in the
decoder, followed by a local signal normalization layer, as in [10]. The SC-
FEGAN discriminator is similar to that in SN-patchGAN [3].

## Proposed Inpainting GAN Architecture

### Generator

In the proposed edge-guided inpainting approach, we use the FIpoly-GAN
architecture shown in Figure 2, which is based on the Poly-GAN network
initially used for fashion synthesis [7]. The main differences between the
FIpoly-GAN inpainting architecture and the original Poly-GAN architecture are
due to different inputs and training methodology. FIpoly-GAN is a conditional
GAN based on encoder-decoder network configuration with two additional
modules: the Conv-Module and the Conv-Norm Module, illustrated in Fig. 2. We
present the architecture, training methodology and results in the following
sections.

#### Encoder

The FIpoly-GAN encoder is an extension of deep learning conditional GAN
encoders with some variations in the network. The most important aspect of the
FIpoly-GAN encoder is that the conditional input is fed to multiple layers. We
found that if the conditional input is provided only in the first layer of the
encoder, then its effect diminishes as the features propagate deeper in the
network.

In our encoder, each layer is like a ResNet module [9] consisting of two
spectral normalization layers followed by a ReLU activation function. The
encoder in Figure 2 has modules to handle the incoming conditional inputs in
every layer using a Conv Module and a Conv-Norm Module, shown in Fig. 2. In
the Conv Module, we pass the conditional inputs to every convolution layer
starting with the third layer, followed by a ReLU activation function. The
learned features from the previous layer have different statistics than the
learned features from Conv Module, so we concatenate the features for input
through the Conv-Norm Module. The Conv-Norm Module consists of two
convolutions each followed by an instance normalization and an activation
function. It is important to note that the initial input to the encoder
consists of conditional input with Gaussian noise.

#### Decoder

The FIpoly-GAN decoder, shown in Figure 2 is the same as the one in Poly-GAN
[7]. The layers in the decoder consist of a ResNet module with two spectral
normalization layers followed by a ReLU and a transposed convolution to
upsample the learned features. The decoder learns to generate the desired
output using the input from the encoder. To increase the effectiveness of the
decoder, we utilize skip connections from low resolution layers of the encoder
to the corresponding resolution layers of the decoder, as illustrated in Fig.
2. It is known from previous works, such as Progressive GAN [10], that the
coarse layers learn structural information which is easy to manipulate through
the conditional input. Our experiments determined that only three low
resolution skip connections are needed. These skip connections help the
decoder enforce the structure of the conditional inputs fed to consecutive
layers of the encoder.

h

Figure 3: Discriminator network using in FIpoly-GAN.

### Discriminator

The FIpoly-GAN discriminator network shown in Figure 3 is the same as the one
in [7]. The discriminator architecture is similar to the one in Cycle GAN
[11], as it has been found to be effective in various applications.

### Loss

The loss function used for training consists of three components: adversarial
loss $L_{adv}$, GAN loss $L_{gan}$ and identity loss $L_{id}$. The total loss
and its components are presented below, where $D$ is the discriminator, $G$ is
the generator, $x_{i},i=1,...,N$ represent $N$ input samples from different
distributions $p_{i}(x_{i}),i=1,...N$, $t$ is the target, $F$ are fake labels
for generated images and $R$ are real labels for ground truth images.

$\begin{split}\underset{G}{\mathit{min}}\mathit{L_{GAN}}={}&\lambda_{3}E_{x_{1}\sim
p_{1}(x_{1}),..,x_{N}\sim p_{N}(x_{N})}\\\
&{\left\|D(G(x_{1},..,x_{N})-R\right\|_{2}^{2}}\end{split}$ (1)

$\begin{split}\underset{D}{\mathit{min}}\mathit{L_{Adv}}={}&\lambda_{1}E_{t\sim
p_{d}(t)}{\left\|D(t)-R\right\|_{2}^{2}}+\\\ &\lambda_{2}E_{x_{1}\sim
p_{1}(x_{1}),..,x_{N}\sim p_{N}(x_{N})}\\\
&{\left\|D(G(x_{1},..,x_{N})-F\right\|_{2}^{2}}\end{split}$ (2)

$\begin{split}L_{Id}={}&\lambda_{4}E_{t\sim p_{d}(t),x_{1}\sim
p_{1}(x_{1}),..,x_{N}\sim p_{N}(x_{N})}\\\
&{\left\|G(x_{1},..,x_{N})-t\right\|_{1}}\end{split}$ (3)

In the above equations, $\lambda_{1}$ to $\lambda_{4}$ are hyperparamters that
are tuned during training. Similarly to [12], we use the $L_{2}$ loss as the
adversarial loss of our GAN in Eqs. (1) and (2). We use the $L_{1}$ loss for
the identity loss, which helps reduce texture and color shift between the
generated image and the ground truth. We considered adding other loss
functions, such as the perceptual loss [13] and SSIM loss, but they did not
improve our results.

For each epoch, we first train the generator with the GAN loss and identity
loss in Eqs. (1) and (3) respectively. Then we train the discriminator with
the adversarial loss in Eq. (2). This process was repeated for several epochs
until a satisfactory result was obtained.

## Experiments and Results

For our experiments, we used standard datasets, specifically Labeled Faces in
the Wild (LFW) [14] and Large-scale CelebFaces Attributes (CelebA) [15]. The
LFW dataset contains more than 13,000 images of faces collected from the web.
Each face is labeled with the name of the person pictured and 1680 people have
two or more distinct photos in the dataset. The version of LFW dataset we uses
has been aligned using the deep funneling method. The CelebA dataset has more
than 200k celebrities faces in different scenes, poses, and with and without
jewelry. The dataset is pre-processed and aligned. A key similarity between
the datasets is the alignment of facial features, e.g. all images are aligned
so that the tip of the nose is at the center of the image. Training of FIpoly-
GAN was done using free form irregular masks.

We used publicly released code or online demos for comparisons with other
methods. We experimented with image inpainting for irregular holes using the
method of image inpainting via Generative Multi-column Convolutional Neural
Networks [4], image inpainting Using Gated Convolution [3], and image
inpainting using SC-FEGAN [5], due to the availability of code or online demo.

Figure 4: Results inpainting with FIpoly-GAN when passing the edge map as
conditional input. These examples use images from the internet to illustrate
that FIpoly-GAN is able to generalize to unknown images from random sources.
Figure 5: Results comparison of edge-based FIpoly-GAN with Generative Multi-
column convolutinal nets (GMCNN) [4] and Gated Convolution method [3]. Left to
right columns show: Original, masked, GMCNN result, masked, GatedConv result,
masked, edge-based FIpoly-GAN result. Best viewed in color and 4$\times$ zoom.

We next present our experiments with conditional inputs to our network,
passing the structural information such as edges, edges of similar-looking
images, and edited edges.

### Edge-based Face Inpainting

Edge-based face inpainting operates under the assumption that the edges of the
masked regions are available. In our experiments we obtained the edge map from
the original image, operating under a best case scenario. Alternatively, the
edges can be obtained from a similar image or a sketch drawing. We utilized
the processing pipeline as an encoder-decoder style conditional GAN
illustrated in Figure 2. For the training data, we randomly sampled nearly 30
thousand images from the CelebA [15] dataset. We created edge maps for the
sampled images using the Sobel filter and thresholded the filter response to
obtain the edges that are used as conditions. We used an NVIDIA 1080TI GPU for
training.

We display representative results from our edge-based inpainting in Figure 4
and comparisons with other methods in Figure 5. Our method is trained using
free-form irregular masks on CelebA, while GMCNN is pre-trained on rectangular
shaped mask on CelebA. The test images are randomly sampled from the LFW
dataset. From the results of Fig. 5 it is evident that our edge-conditioned
FIpoly-GAN significantly outperforms the other methods when using images from
unknown sources. In contrast to the other methods, our network has the ability
to generalize and recover the masked image content consistently close to the
original image.

For quantitative comparison with other methods, we trained on the LFW dataset.
We used three quantitative metrics to compare our results against state of the
art methods: i) Structural Similarity Index (SSIM); ii) Mean Squared Error
(MSE); iii) Signal to Noise Ratio (SNR). SSIM should be high for similar
images, while MSE should be low for similar images. Higher SNR values
represent better quality of image recovery. The results indicate that the
edge-based FIpoly-GAN scores highest in two of the metrics. Most importantly,
our method is able to generalize much better than the other methods on
previously unseen images, as illustrated in Fig. 5.

Method | SSIM | MSE | SNR
---|---|---|---
Edge-based FIpoly-GAN | 0.824 | 27.533 | 1.678
GMCNN [4] | 0.845 | 36.041 | 1.568
Gated Conv [3] | 0.834 | 37.261 | 1.540
Table 1: Quantitative results comparison of edge-based FIpoly-GAN with other
methods on randomly sampled images from the LFW dataset. Bold shows best
performance.

### Sketch-based Face Inpainting

Our edge-based inpainting experiments demonstrated that even if the image is
badly damaged, we can recover the original content by conditioning on the edge
information. However, edge information is not readily available when
inpainting. As an alternative, we consider a hand-drawn sketch that provides
edge information for the damaged image regions. We created a dataset
consisting of sketches of CelebA faces following the instructions provided in
FaceShop [6]. We used Potrace to generate a vectorized representation of the
image edges, which is used for training purposes. We randomly sampled 150k
images for training and testing purposes from CelebA. The model was trained on
Ubuntu 16.04 with an NVIDIA 1080TI GPU for nearly 18 hours.

Representative results generated by our sketch-based FIpoly-GAN method are
shown in Figure 6. The results illustrate that the network can generate
realistic images by inpainting conditioned on a rough sketch.

Figure 6: Results obtained with sketch-based FIpoly-GAN. From left to right:
original, damaged, sketch, inpainted image.

## Conclusion

In this paper, we presented FIpoly-GAN, a conditional GAN for extreme face
inpainting. Our work demonstrated the importance of edge information in the
recovery of badly damaged regions for image restoration or editing. The
FIpoly-GAN approach performs equally well or better than state-of-the-art
methods on standard datasets and outperforms other methods on images unrelated
to the training data.

## Acknowledgments

This research was supported in part by the Air Force Office of Scientific
Research (AFOSR) under Dynamic Data Driven Applications Systems (DDDAS) grant
FA9550-18-1-0121 and the Center for Emerging and Innovative Sciences (CEIS),
an Empire State Development-designated Center for Advanced Technology..

## References

  * [1] H. Zhang, Z. Hu, C. Luo, W. Zuo, M. Wang, “Semantic image inpainting with progressive generative networks,” In Proceedings of the 26th ACM International Conference on Multimedia, pp. 1939-1947, 2018.
  * [2] G. Liu, F.A. Reda, K.J. Shih, T.C. Wang, A. Tao, B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” In Proceedings of the European Conference on Computer Vision (ECCV), pp. 85-100, 2018.
  * [3] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, T.S. Huang, “Free-form image inpainting with gated convolution,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 4471-4480, 2019.
  * [4] Y. Wang, X. Tao, X. Qi, X. Shen, J. Jia, “Image inpainting via generative multi-column convolutional neural networks,” In Advances in Neural Information Processing Systems, pp. 331-340, 2018.
  * [5] Y. Jo, J. Park, “SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 1745-1753, 2019.
  * [6] T. Portenier, Q. Hu, A. Szabo, S.A. Bigdeli, P. Favaro, M. Zwicker, “Faceshop: Deep sketch-based face image editing,” ACM Transactions On Graphics, 2018.
  * [7] N. Pandey, A. Savakis, “Poly-GAN: Multi-conditioned GAN for fashion synthesis,” Neurocomputing, 414, pp.356-364, 2020.
  * [8] K. Nazeri, E. Ng, T. Joseph, F. Qureshi, M. Ebrahimi, ”Edge-Connect: Structure guided image inpainting using edge prediction,” International Conference on Computer Vision Workshops (ICCVW), 2019.
  * [9] K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
  * [10] T. Karras, T. Aila, S. Laine, J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196, 2017.
  * [11] J.Y. Zhu, T. Park, P. Isola, A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
  * [12] X. Mao, Q. Li, H. Xie, R.Y. Lau, Z. Wang, S. P, Smolley, “Least squares generative adversarial networks,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2017.
  * [13] J. Johnson, A. Alahi, L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” In European Conference on Computer Vision, pp. 694-711, 2016.
  * [14] G.B. Huang, M. Mattar, T. Berg, E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” In Workshop on faces in ’Real-Life’ Images: detection, alignment and recognition, 2008.
  * [15] Z. Liu, P. Luo, X. Wang, X. Tang, “Deep learning face attributes in the wild,” In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.

## Author Biography

Nilesh Pandey completed his Bachelor in Electrical Engineering at Ramrao Adik
Institute of Technology (2016) and received his Master in Computer Engineering
from Rochester Institute of Technology (2019). His research interests include
Generative Adversarial Networks (GAN), and applications for tracking and pose
estimation architectures.

Andreas Savakis is Professor of Computer Engineering and Director of the
Center for Human-aware Artificial Intelligence (CHAI) at Rochester Institute
of Technology (RIT). He received his Ph.D. in Electrical and Computer
Engineering from North Carolina State University. Prior to joining RIT, he was
Senior Research Scientist at Kodak Research Labs. His research interests
include computer vision, deep learning, machine learning, domain adaptation,
object tracking, human pose estimation, and scene analysis. Dr. Savakis has
coauthored over 120 publications and is co-inventor on 12 U.S. patents.