# Quadratic-in-spin interactions at fifth post-Newtonian order probe new
physics

Jung-Wook Kim  Queen Mary University of London, London E1 4NS, United Kingdom
Michèle Levi<EMAIL_ADDRESS>University of Oxford, Oxford OX2 6GG, United
Kingdom  Queen Mary University of London, London E1 4NS, United Kingdom  Niels
Bohr Institute, University of Copenhagen, 2100 Copenhagen, Denmark  Zhewei Yin
Uppsala University, 75108 Uppsala, Sweden

###### Abstract

We obtain for the first time all quadratic-in-spin interactions in spinning
binaries at the third subleading order in post-Newtonian (PN) gravity, and
provide their observable binding energies and their gauge-invariant relations
to the angular momentum. Our results are valid for generic compact objects,
orbits, and spin orientations, and enter at the fifth PN order for maximally-
rotating objects, thus pushing the state of the art. This is accomplished
through an extension of the effective field theory of spinning gravitating
objects, and of its computational application. We also discover a new finite-
size effect which is unique to spinning objects, with a new “Spin Love number”
as its characteristic coefficient, that is a new probe for gravity and QCD.

††preprint: QMUL-PH-21-52, UUITP-61/21

The success of ground-based experiments in measuring gravitational waves (GWs)
since the first detection from a black-hole (BH) binary merger [1] by the
Advanced LIGO [2] and Advanced VIRGO [3] collaboration has exceeded most
expectations. By now there is a worldwide network of GW detectors of second-
generation technology, as the twin Advanced LIGO detectors in the US have been
joined by the Advanced Virgo detector in Europe [3], and then by the KAGRA
detector in Japan [4]. These experiments have been continually reaching higher
sensitivities, which yield more frequent detections, and the influx of data
has been steeply growing [5, 6, 7].

Since 2017 these detections also include neutron stars (NSs) as individual
components of the binaries, in NS binaries [8] or mixed NS-BH binaries [9].
The study of these various GW sources in the inspiral phase, when the
components of the binary are still orbiting in non-relativistic velocities, is
carried out analytically via the post-Newtonian (PN) approximation of General
Relativity (GR) [10]. This enables to model theoretical gravitational
waveforms, which inform us on a wealth of astrophysical and cosmological
scenarios that were previously unthought-of, and uniquely also on gravity in
the strong-field regime, and QCD in extreme conditions for NSs [11, 12]. To
this end, it is crucial to study spin and finite-size effects, as all
components of the binaries are in fact spinning gravitating objects [13].

In both traditional GR and modern HEP approaches to study these GW sources,
the compact objects that make up the binaries are essentially captured by an
effective description of a point particle that is endowed with characteristic
coefficients which encapsulate the physics at the small scales of its internal
structure [14, 15, 16]. These effective characteristic coefficients are
generally referred to as “Wilson coefficients” in the language of effective
field theory (EFT) from QFT. Determining the numerical values of these
coefficients in the low-energy or large-scale approximation constitutes the
final and often most challenging piece of fixing the effective theory. This
task, commonly referred to as “matching” in EFT parlance, is tackled in an
indefinite variety of ways, e.g. via analytical studies of specific
observables in the full theory, numerical simulations of the full theory, or
by simply matching the unknown coefficients to experimental data.

In the non-spinning case the effective description of a point particle remains
trivial till high PN orders, where the simple point-mass alone is sufficient
up to the fifth PN (5PN) order, that has been approached only recently after
decades of studies in PN theory. Finite-size effects thus enter only at the
5PN order in this simplified case, preceded by effective coefficients that
correspond to the so-called “Love numbers”. These have been introduced more
than a century ago in Newtonian theory for planetary bodies as the parameters
that measure their rigidity, and thus their response to tidal forces. In the
context of BH physics the related numbers have been studied for almost 4
decades already, see e.g. [17] for reference, where general studies in GR have
been carried out mainly in the last 15 years, pioneered by e.g. [18, 19, 17].

For the real spinning case the physics gets dramatically more complicated. To
begin with, the spin induces higher multipoles to all orders, and the
associated finite-size effects enter already as of the 2PN order with the
spin-induced quadrupole [20]. These finite-size effects are characterized by
coefficients commonly referred to in GR as “multipole deformation parameters”,
see e.g. [21], corresponding to analogous Wilson coefficients in the EFT
description [22], see (EFT of spinning gravitating objects.) below. The
“multipole deformation parameters” are not to be confused with the
aforementioned “Love numbers”, see (7a) below. For example, whereas “Love
numbers”, which can also be studied for spinning objects, see e.g. a recent
surge of studies [23, 24, 25, 26, 27, 28], have been shown in virtually all
studies to date, to vanish for BHs in GR (in 4 dimensions), the spin-induced
“multipole deformation parameters” in contrast equal 1 for BHs [22, 16].

In recent years impressive progress has been made in the state of the art of
PN theory for the conservative dynamics of an inspiraling binary. In
particular, the point-mass interaction at the 5PN order has been recently
accomplished via a combined exploitation of traditional GR methods [29, 30,
31], with crucial ingredients taken from self-force theory, and the effective-
one-body approach [32]. Shortly after, this sector was also confirmed via an
EFT computation [33]. However, in order to attain any PN accuracy (beyond
1PN), the spinning case must be tackled. [34, 35] then followed the footsteps
of [29, 30] in implementing a similar approach to the sector that is linear in
the spins at 4.5PN order, and for a limited simplified configuration of
circular orbits with aligned spins – to the piece that is linear in the spins
at 5PN order [35].

It is critical to note however that while it is important and illustrative to
target specific new PN sectors via the capitalization on available results
from existing elementary methods, such an ad-hoc approach is essentially
limited. It does not provide a conceptual framework to generally tackle the
various sectors required to a certain accuracy, nor does it provide an
independent framework to study PN theory, and thus it is also prone to the
propagation of errors from the combined inputs of the various ingredient
methods.

In this letter we tackle the 5PN order with spins in the most generic
settings, obtaining for the first time all the interactions that are quadratic
in the spins. Our derivation builds on the EFT of spinning gravitating objects
introduced in [22], see also [16, 36], and its extensions [37, 38, 39, 40, 41,
42, 43, 44, 45, 46]. It is the most formidable undertaking in PN theory with
spins as yet. The cutting-edge calculation of the present sectors outlined
here also serves as a unique computational experiment, to eventually discover
a new feature in the theory of a spinning particle: a new type of finite-size
effect, which is unique to spinning objects, with new “Spin Love numbers”.
These do not exist in the non-spinning case, and thus they provide a new
unique probe for gravity and QCD.

## EFT of spinning gravitating objects.

We build on the EFT of spinning gravitating objects introduced in [22]. To
obtain all the quadratic-in-spin interactions, we need to start from the two-
particle effective action for the compact binary [14, 16]:

$S_{\text{eff}}=S_{\text{g}}[g_{\mu\nu}(x)]+\sum_{a=1}^{2}S_{\text{pp}}[(\lambda_{a})],$
(1)

and then carefully consider the effective action of the spinning particle,
$S_{\text{pp}}$, localized on the worldline parametrized by $\lambda_{a}$ for
each of the two components of the binary.

First, in the non-relativistic approximation it is useful to employ a Kaluza-
Klein time+space decomposition of the field [47, 48], which was first tested
in sectors with spins in [49, 50]. The spatial dimension, $d$, must be kept
generic throughout, as dimensional regularization will be used to evaluate the
Feynman integrals, with the modified minimal subtraction
($\overline{\text{MS}}$) prescription applied through the $d$-dimensional
gravitational constant [44]:

$G_{d}\equiv G_{N}\left(\sqrt{4\pi e^{\gamma_{E}}}\,R_{0}\right)^{d-3},$ (2)

in which $G_{N}\equiv G$ is Newton’s gravitational constant in three-
dimensional space, $\gamma_{E}$ is Euler’s constant, and $R_{0}$ is some fixed
renormalization scale.

The quadratic-in-spin sectors include finite-size effects in addition to the
minimal coupling of spinning objects to gravity, so we need to consider the
following extended effective action for each of the two spinning particles
[22, 16]:

$\displaystyle S_{\text{pp}}[(\lambda)]=\int d\lambda\bigg{[}-m\sqrt{u^{2}}-$
$\displaystyle\frac{1}{2}\hat{S}_{\mu\nu}\hat{\Omega}^{\mu\nu}-\frac{\hat{S}^{\mu\nu}p_{\nu}}{p^{2}}\frac{Dp_{\mu}}{D\lambda}$
$\displaystyle+L_{\text{NMC}}\Big{[}g_{\mu\nu},u^{\mu},S^{\mu}\Big{]}\bigg{]},$
(3)

where the non-minimal coupling of gravity to spin, $L_{\text{NMC}}$, is
formulated in terms of the definite-parity classical analogue of the Pauli-
Lubanski pseudovector, $S_{\mu}$, as defined in [39, 22, 43]. We note that
another treatment of spin utilizing EFT techniques was approached in [51],
where it was applied to low PN orders.

The non-minimal coupling of gravity to all orders in spin, that is linear in
the curvature, is given in the following compact form [22]:

$\displaystyle L_{\text{NMC(R)}}$
$\displaystyle=\sum_{n=1}^{\infty}\frac{(-1)^{n}}{(2n)!}\frac{C_{ES^{2n}}}{m^{2n-1}}D_{\mu_{2n}}\cdots
D_{\mu_{3}}\frac{E_{\mu_{1}\mu_{2}}}{\sqrt{u^{2}}}$
$\displaystyle\qquad\qquad\qquad\quad\bullet S^{\mu_{1}}S^{\mu_{2}}\cdots
S^{\mu_{2n-1}}S^{\mu_{2n}}$
$\displaystyle+\sum_{n=1}^{\infty}\frac{(-1)^{n}}{(2n+1)!}\frac{C_{BS^{2n+1}}}{m^{2n}}D_{\mu_{2n+1}}\cdots
D_{\mu_{3}}\frac{B_{\mu_{1}\mu_{2}}}{\sqrt{u^{2}}}$
$\displaystyle\qquad\qquad\qquad\quad\bullet S^{\mu_{1}}S^{\mu_{2}}\cdots
S^{\mu_{2n}}S^{\mu_{2n+1}},$ (4)

with the definite-parity electric and magnetic components of the curvature:

$\displaystyle E_{\mu\nu}$ $\displaystyle\equiv$ $\displaystyle
R_{\mu\alpha\nu\beta}u^{\alpha}u^{\beta},$ (5) $\displaystyle B_{\mu\nu}$
$\displaystyle\equiv$
$\displaystyle\frac{1}{2}\epsilon_{\alpha\beta\gamma\mu}R^{\alpha\beta}_{\,\,\,\,\,\,\,\delta\nu}u^{\gamma}u^{\delta},$
(6)

and their covariant derivatives, $D_{\mu}$. In this infinite series we
introduced an infinite set of Wilson coefficients, which correspond to the
aforementioned “multipole deformation parameters”. The only term from this
series that contributes to our present sectors is the first electric term
preceded by a Wilson coefficient, which corresponds to the quadrupolar
deformation constant, similar to [20].

Yet, at this high PN order the effective action of a spinning particle needs
to be extended beyond linear order in the curvature, namely beyond (EFT of
spinning gravitating objects.). Such extension was briefly suggested very
recently in [45] from basic symmetry considerations and reasoning [22, 16].
From dimensional analysis and power counting only the following terms may
enter at the 5PN order to quadratic order in the spins:

$\displaystyle L_{\text{NMC(R${}^{2}$)}}$
$\displaystyle=C_{E^{2}}\frac{E_{\alpha\beta}E^{\alpha\beta}}{\sqrt{u^{2}}^{\,3}}+C_{B^{2}}\frac{B_{\alpha\beta}B^{\alpha\beta}}{\sqrt{u^{2}}^{\,3}}$
(7a) $\displaystyle+\,C_{E^{2}S^{2}}$ $\displaystyle
S^{\mu}S^{\nu}\frac{E_{\mu\alpha}E_{\nu}^{\,\alpha}}{\sqrt{u^{2}}^{\,3}}+C_{B^{2}S^{2}}S^{\mu}S^{\nu}\frac{B_{\mu\alpha}B_{\nu}^{\alpha}}{\sqrt{u^{2}}^{\,3}}.$
(7b)

Interestingly, non-minimal couplings that are only linear in the spins, enter
only at higher PN orders. The terms in (7) involve coefficients that at this
point absorb all numerical and mass factors (unlike those in (EFT of spinning
gravitating objects.)). Yet further scrutiny of (7a), (7b) reveals that their
dimensionless Wilson coefficients should be defined as

$\displaystyle C_{[E/B]^{2}}$
$\displaystyle\to+\frac{1}{2}G^{4}m^{5}\,C_{[E/B]^{2}},$ (8) $\displaystyle
C_{[E/B]^{2}S^{2}}$
$\displaystyle\to+\frac{1}{2}G^{2}m\,\,C_{[E/B]^{2}S^{2}}.$ (9)

Notably these are the first terms that exhibit an additional scaling in $G$ in
their coefficients. This is unlike any PN contributions previously
encountered, where the order in $G$ of the leading field couplings has always
been identical to the order of their overall contributions.

The coefficients in (7a), (8) correspond to the aforementioned generic “Love
numbers”. The terms in (7b) however seem to represent a new type of effects
that would be relevant only for spinning objects, preceded by a new type of
coefficients in (9). Yet, at this high perturbative order, in which quadratic-
in-spin effects have already entered at many subleading corrections, these
seemingly new terms may not correspond to a real physical effect, but rather
could be possibly removed by virtue of subleading equations of motion, or more
formally via some complicated subleading redefinitions of the field and
worldline variables. Such spurious terms in the theory are referred to as
“redundant operators” in EFT parlance, and simply vanish from physical
observables [16]. Indeed, due to the high complexity of the present sectors it
is virtually impossible to identify such a redundancy by any means other than
actually computing the total observables, and therefore we must press on with
the full-scale evaluation of the sectors to discover the nature of the terms
in (7b).

## From EFT formulation through to observables.

To proceed towards the physics of the present sectors, we need to obtain first
the effective action of the quadratic-in-spin interactions via an evaluation
of the diagrammatic expansion of the two-particle action in (1) in terms of
Feynman graphs. To that end, we build on the EFTofPNG – a unique public code
for Feynman computation in PN theory [36, 52]. First we need to extend the
code to generate the required Feynman rules in a generic number of spatial
dimensions $d$, including a recursive implementation of the gauge of the
rotational variables, where we use the “canonical gauge” which was introduced
in [22]. Notably, there are now spin couplings up to quadratic order in the
curvature, in addition to a proliferation of time derivatives on the spin
couplings, and from the gravitational self-interaction.

The Feynman graphs that contribute to these sectors are then generated through
another extension of EFTofPNG, and we find that there are 1122 graphs that
make up the complete next-to-next-to-next-to-leading order (N3LO) quadratic-
in-spin sectors, and are of the highest complexity ever tackled in sectors
with spins as yet. This large volume of intricate graphs also carries a higher
tensorial load than ever, due to the derivative coupling of spins, on top of
the high PN order. It was thus essential to streamline new code for the
projection of integrals, due to the high rank of their numerators [53], and
for algorithmic integration by parts (IBP) to reduce the integrals that show
up to basic master integrals [54, 55, 45]. In general, to verify the
reliability of our new codes, we carried out independent code development in
parallel, and new results were compared at crucial check points along the
elaborate derivation.

Figure 1: The unique graph at the N3LO quadratic-in-spin sectors, which arises
from the quadratic-in-curvature coupling that is also quadratic in the spin,
and is preceded by an unknown coefficient, see (7b), (9), (10). The coupling
is denoted by a black square labeled EE.

An expansion of the terms in (7b), (9), reveals that only the term with the
electric curvature component actually enters at this PN order. This term gives
rise to a unique graph of a two-graviton exchange depicted in Fig. 1 [56, 57],
whose value is given by

$\text{Fig.~{}1}=-\frac{1}{2}C_{1(\text{E}^{2}\text{S}^{2})}\frac{G^{4}m_{1}m_{2}^{2}}{r^{6}}\left[S_{1}^{2}+3\big{(}\vec{S}_{1}\cdot\vec{n}\big{)}^{2}\right].$
(10)

As noted the Feynman graphs are evaluated using dimensional regularization
with the $\overline{\text{MS}}$ scheme in (2), and similar to [45] the values
of the individual graphs contain poles in the dimensional parameter,
$\epsilon_{d}\equiv d-3$, in conjunction with logarithms in $r/R_{0}$.
However, whereas in the piece which was approached in [45], all the poles and
logarithms conspired to cancel out from the total sum, summing over all the
graphs in our sectors does leave such divergent and logarithmic terms, which
do not cancel out from the total sum. In fact, the expansion of results in
terms of $\epsilon_{d}$ that is required here, makes for one of the most
computationally demanding tasks in the evaluation of the present sectors.

Summing up all of the graphs’ values we get an initial action with many large
pieces that contain terms with higher-order time derivatives – up to 6th order
in the total number of derivatives. To remove these terms, we need to extend
the formulation of a rigorous procedure that was uniquely introduced in [38],
for subleading redefinitions of both position and rotational variables. This
elaborate procedure is critical here since such redefinitions, e.g.
$\vec{x}\to\vec{x}+\Delta\vec{x}$, need to be applied beyond linear order,
e.g. beyond ${\cal{O}}(\Delta\vec{x})$, in sectors that are beyond linear in
the spins, and at high PN orders. The redefinitions of rotational variables,
the Lorentz matrices $\Lambda^{ij}$ and the spins $S^{ij}$, are parametrized
by $\omega^{ij}$, the anti-symmetric generator of rotations on the Lorentz
matrices,

$\Lambda^{ij}\equiv\Lambda^{ik}\left(e^{\omega}\right)^{kj}.$ (11)

We then require here for the first time redefinitions of position which depend
on the spins, in addition to redefinitions of the rotational variables, that
both scale as:

$|\Delta\vec{x}|\sim\epsilon_{d}^{-1},\quad|\omega_{ij}|\sim\epsilon_{d}^{-1}.$
(12)

Further, to reduce the action to a standard action without higher-order time
derivatives, all sectors up to quadratic-in-spin and to this PN order should
be treated consistently, i.e. from LO to N3LO of point-mass sectors (namely
Newtonian through to 3PN order), and further through spin-orbit, to quadratic-
in-spin sectors. We get contributions to the present sectors from the
application of redefinitions in all these sectors. Notably, further
logarithmic terms arise, preceded by the dimensional parameter,
$\epsilon_{d}$, and thus the Newtonian and the LO quadratic-in-spin
interactions should all be expanded here in $\epsilon_{d}$, as these yield
contributions to the present sectors.

Altogether after this arduous reduction procedure, all the divergent and
logarithmic terms vanish when going to observables or they can be removed from
the reduced action already through the addition of a total time derivative as
we also verified. At this stage the transition to a generic Hamiltonian, as
well as to various observables in simplified binary configurations, of e.g.
circular orbits and aligned spins, is straightforward [38, 22, 40]. The total
binding energy of the N3LO quadratic-in-spin interactions can be written as
the following sum of pieces:

$\displaystyle(e)_{\text{S${}^{2}$}}^{\text{N${}^{3}$LO}}=(e)_{\text{S${}_{1}$S${}_{2}$}}^{\text{N${}^{3}$LO}}+$
$\displaystyle\bigg{[}(e)_{\text{S${}_{1}^{2}$}}^{\text{N${}^{3}$LO}}+(e)_{\text{C${}_{1(ES^{2})}$S${}_{1}^{2}$}}^{\text{N${}^{3}$LO}}$
$\displaystyle+(e)_{\text{C${}_{1(E^{2}S^{2})}$S${}_{1}^{2}$}}^{\text{N${}^{3}$LO}}+\left(1\leftrightarrow
2\right)\bigg{]}.$ (13)

As a function of the orbital frequency parameter $x$, these various pieces are
given by

$\displaystyle(e$
$\displaystyle)^{\text{N}^{3}\text{LO}}_{\text{S}_{1}\text{S}_{2}}(x)=\tilde{S}_{1}\tilde{S}_{2}x^{6}\nu\left[\frac{243}{16}-\left(\frac{2107}{16}-\frac{123}{32}\pi^{2}\right)\nu+\frac{147}{8}\nu^{2}+\frac{13}{16}\nu^{3}\right],$
(14) $\displaystyle(e$
$\displaystyle)^{\text{N}^{3}\text{LO}}_{\text{S}_{1}^{2}}(x)=\tilde{S}_{1}^{2}x^{6}\nu\left[\left(\frac{1947}{112}\nu-\frac{48357}{560}\nu^{2}+\frac{159}{16}\nu^{3}\right)-q^{-1}\left(\frac{243}{16}-\left(\frac{747}{16}-\frac{189}{2048}\pi^{2}\right)\nu+\frac{13731}{280}\nu^{2}-\frac{153}{16}\nu^{3}\right)\right],$
(15) $\displaystyle(e$
$\displaystyle)^{\text{N}^{3}\text{LO}}_{\text{C${}_{1(ES^{2})}$S${}_{1}^{2}$}}(x)=C_{1(\text{ES}^{2})}\tilde{S}_{1}^{2}x^{6}\nu\left[\left(\frac{789}{28}\nu-\frac{156}{7}\nu^{2}+\frac{5}{8}\nu^{3}\right)+q^{-1}\left(\frac{405}{32}-\left(\frac{2389}{32}-\frac{3747}{2048}\pi^{2}\right)\nu-\frac{555}{56}\nu^{2}+\frac{21}{32}\nu^{3}\right)\right],$
(16)
$\displaystyle(e)^{\text{N}^{3}\text{LO}}_{\text{C${}_{1(E^{2}S^{2})}$S${}_{1}^{2}$}}(x)=-\frac{3}{2}C_{1(\text{E}^{2}\text{S}^{2})}\tilde{S}_{1}^{2}x^{6}\nu\left[\nu^{2}(1+q^{-1})\right],\qquad$
(17)

where all definitions and notations are identical to [38]. These results were
obtained by going from our most general new results to the simplified specific
configuration of circular orbits with aligned spins. The simplest piece is
linear in the individual spins as shown in (14), and is in agreement with
[35], who obtained it only within a limited treatment of the simplified
specific case of circular orbits with aligned spins in the center-of-mass
frame, unlike our most generic treatment.

Importantly we also find that there is a new type of contribution in (17) due
to the new term from (7b), (9). This means that this new term is not a
“redundant operator” in the EFT, but rather represents a real new physical
effect which is unique to spinning objects. It can be verified that (17) is
always negative, and thus it increases the binding energy of the compact
binary, similar to the effects linked with the long-known “Love numbers” in
(7a). Thus, we also discovered here a new “Spin Love number”, which is unique
to spinning objects.

We also find the following pieces for the gauge-invariant relation of the
binding energy to the angular momentum:

$(e)^{\text{N}^{3}\text{LO}}_{\text{S}_{1}\text{S}_{2}}(\tilde{L})=-\tilde{S}_{1}\tilde{S}_{2}\frac{\nu}{\tilde{L}^{12}}\left[\frac{102897}{16}-\left(\frac{31653}{32}-\frac{369}{32}\pi^{2}\right)\nu-\frac{579}{16}\nu^{2}+\frac{209}{64}\nu^{3}\right],$
(18)
$\displaystyle(e)^{\text{N}^{3}\text{LO}}_{\text{S}_{1}^{2}}(\tilde{L})=-\tilde{S}_{1}^{2}\frac{\nu}{\tilde{L}^{12}}$
$\displaystyle\left[\left(\frac{2117357}{896}\nu-\frac{714891}{2240}\nu^{2}+\frac{201}{128}\nu^{3}\right)\right.$
$\displaystyle\left.+\,q^{-1}\left(\frac{211653}{128}+\left(\frac{21195}{16}-\frac{63}{2048}\pi^{2}\right)\nu-\frac{167739}{560}\nu^{2}+\frac{3}{32}\nu^{3}\right)\right],$
(19)
$\displaystyle(e)^{\text{N}^{3}\text{LO}}_{\text{C${}_{1(ES^{2})}$S${}_{1}^{2}$}}(\tilde{L})=-C_{1(\text{ES}^{2})}\tilde{S}_{1}^{2}\frac{\nu}{\tilde{L}^{12}}$
$\displaystyle\left[\left(\frac{2593}{14}\nu-\frac{319}{28}\nu^{2}+\frac{3}{8}\nu^{3}\right)\right.$
$\displaystyle\left.+\,q^{-1}\left(\frac{16065}{32}-\left(\frac{3061}{32}-\frac{11745}{2048}\pi^{2}\right)\nu-\frac{313}{28}\nu^{2}+\frac{17}{32}\nu^{3}\right)\right],$
(20)
$\displaystyle(e)^{\text{N}^{3}\text{LO}}_{\text{C${}_{1(E^{2}S^{2})}$S${}_{1}^{2}$}}(\tilde{L})=\frac{1}{2}C_{1(\text{E}^{2}\text{S}^{2})}\tilde{S}_{1}^{2}\frac{\nu}{\tilde{L}^{12}}\left[\nu^{2}\left(1+q^{-1}\right)\right].\qquad$
(21)

These relations provide a useful tool for evaluating different analytic and
numerical descriptions of the binary dynamics, see [38] and references
therein.

## State of the art and new physics.

The EFT of spinning gravitating objects introduced in [22] provides a self-
contained framework that allows for significant formal and technical
extensions which in turn enable to push the precision frontier, as
demonstrated in this letter. Our framework handles generic compact binaries,
and provides a host of useful mathematical and observable quantities, which
are not limited to simplified specific binary configurations, such as circular
orbits or no eccentricity, and the aligned-spins cases. In this letter we
pushed the state of the art of the conservative dynamics at the 5PN order in
these most generic settings for all quadratic-in-spin sectors. The push in PN
accuracy already greatly improves our ability to learn on the fundamental
physics that is encrypted in GW data. Yet, the state of the art accomplished
in this letter is even more crucial as it handles the real-world spinning
case, and moreover goes beyond linear order in spins, namely to finite-size
effects that provide unique information on gravity and QCD.

Our framework here also enabled to discover a new type of physical effect that
is unique to spinning objects, with a new “Spin Love number”. The new effect
enters at the 5PN interaction, and binding energy in (From EFT formulation
through to observables.), similar to the spinless terms in (7a) with
coefficients in (8) that correspond to the long-known “Love numbers”. As the
5PN frontier has been recently approached, there has been a surge of studies
on these long-known Love numbers – for rotating BHs and NSs [23, 24, 25, 26,
27, 28]. These studies by various groups arrived at several new intriguing
findings and insights did not reach unanimity in whether or not these Love
numbers vanish for rotating black holes, and seem to be far from being
concluded. It is thus vital to thoroughly tackle this challenging and rich
line of study on these coefficients for generic compact objects in various
approaches, and for various general theories (including in generic dimensions,
see e.g. [58]), as this is bound to uncover new physics. It remains for future
analytical and numerical studies, and analysis of GW data, to also uncover the
unique new physics, that is encapsulated in the new coefficients or “Spin Love
numbers” discovered here in this letter.

###### Acknowledgements.

## Acknowledgments.

We thank Johannes Blümlein and Alex Edison for pleasant discussions. J-WK was
supported by the Science and Technology Facilities Council (STFC) consolidated
grant ST/T000686/1 “Amplitudes, Strings and Duality”. ML received funding from
the European Union’s Horizon 2020 research and innovation programme under the
Marie Skłodowska-Curie grant 847523, and is supported by the STFC Rutherford
grant ST/V003895/1 “Harnessing QFT for Gravity”. ZY is supported by the Knut
and Alice Wallenberg Foundation under grants KAW 2018.0116 and KAW 2018.0162.

## References

  * Abbott  _et al._ [2016] B. P. Abbott _et al._ (LIGO, VIRGO), Observation of Gravitational Waves from a Binary Black Hole Merger, Phys. Rev. Lett. 116, 061102 (2016), arXiv:1602.03837 [gr-qc] .
  * Aasi  _et al._ [2015] J. Aasi _et al._ (LIGO Scientific), Advanced LIGO, Class. Quant. Grav. 32, 074001 (2015), arXiv:1411.4547 [gr-qc] .
  * Acernese  _et al._ [2015] F. Acernese _et al._ (VIRGO), Advanced Virgo: a second-generation interferometric gravitational wave detector, Class. Quant. Grav. 32, 024001 (2015), arXiv:1408.3978 [gr-qc] .
  * Akutsu  _et al._ [2020] T. Akutsu _et al._ (KAGRA), Overview of KAGRA: Detector design and construction history, (2020), arXiv:2005.05574 [physics.ins-det] .
  * Abbott  _et al._ [2019] B. P. Abbott _et al._ (LIGO Scientific, Virgo), GWTC-1: A Gravitational-Wave Transient Catalog of Compact Binary Mergers Observed by LIGO and Virgo during the First and Second Observing Runs, Phys. Rev. X9, 031040 (2019), arXiv:1811.12907 [astro-ph.HE] .
  * Abbott  _et al._ [2021a] R. Abbott _et al._ (LIGO Scientific, Virgo), GWTC-2: Compact Binary Coalescences Observed by LIGO and Virgo During the First Half of the Third Observing Run, Phys. Rev. X 11, 021053 (2021a), arXiv:2010.14527 [gr-qc] .
  * Abbott  _et al._ [2021b] R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo During the Second Part of the Third Observing Run, (2021b), arXiv:2111.03606 [gr-qc] .
  * Abbott  _et al._ [2017] B. P. Abbott _et al._ (LIGO Scientific, Virgo), GW170817: Observation of Gravitational Waves from a Binary Neutron Star Inspiral, Phys. Rev. Lett. 119, 161101 (2017), arXiv:1710.05832 [gr-qc] .
  * Abbott  _et al._ [2021c] R. Abbott _et al._ (LIGO Scientific, KAGRA, VIRGO), Observation of Gravitational Waves from Two Neutron Star–Black Hole Coalescences, Astrophys. J. Lett. 915, L5 (2021c), arXiv:2106.15163 [astro-ph.HE] .
  * Blanchet [2014] L. Blanchet, Gravitational Radiation from Post-Newtonian Sources and Inspiralling Compact Binaries, Living Rev. Rel. 17, 2 (2014), arXiv:1310.1528 [gr-qc] .
  * Abbott [2016a] B. P. a. o. Abbott (Virgo, LIGO Scientific), Tests of general relativity with GW150914, Phys. Rev. Lett. 116, 221101 (2016a), arXiv:1602.03841 [gr-qc] .
  * Abbott [2016b] B. P. a. o. Abbott (Virgo, LIGO Scientific), Properties of the Binary Black Hole Merger GW150914, Phys. Rev. Lett. 116, 241102 (2016b), arXiv:1602.03840 [gr-qc] .
  * Abbott [2016c] B. P. a. o. Abbott (Virgo, LIGO Scientific), Improved analysis of GW150914 using a fully spin-precessing waveform Model, Phys. Rev. X6, 041014 (2016c), arXiv:1606.01210 [gr-qc] .
  * Goldberger and Rothstein [2006] W. D. Goldberger and I. Z. Rothstein, An Effective field theory of gravity for extended objects, Phys.Rev. D73, 104029 (2006), arXiv:hep-th/0409156 [hep-th] .
  * Goldberger [2007] W. D. Goldberger, Les Houches lectures on effective field theories and gravitational radiation, in  _Les Houches Summer School - Session 86: Particle Physics and Cosmology: The Fabric of Spacetime Les Houches, France, July 31-August 25, 2006_ (2007) arXiv:hep-ph/0701129 [hep-ph] .
  * Levi [2020] M. Levi, Effective Field Theories of Post-Newtonian Gravity: A comprehensive review, Rept. Prog. Phys. 83, 075901 (2020), arXiv:1807.01699 [hep-th] .
  * Binnington and Poisson [2009] T. Binnington and E. Poisson, Relativistic theory of tidal Love numbers, Phys. Rev. D 80, 084018 (2009), arXiv:0906.1366 [gr-qc] .
  * Hinderer [2008] T. Hinderer, Tidal Love numbers of neutron stars, Astrophys. J. 677, 1216 (2008), arXiv:0711.2420 [astro-ph] .
  * Damour and Nagar [2009] T. Damour and A. Nagar, Relativistic tidal properties of neutron stars, Phys. Rev. D 80, 084035 (2009), arXiv:0906.0096 [gr-qc] .
  * Barker and O’Connell [1975] B. Barker and R. O’Connell, Gravitational Two-Body Problem with Arbitrary Masses, Spins, and Quadrupole Moments, Phys.Rev. D12, 329 (1975).
  * Poisson [1998] E. Poisson, Gravitational waves from inspiraling compact binaries: The Quadrupole moment term, Phys.Rev. D57, 5287 (1998), arXiv:gr-qc/9709032 [gr-qc] .
  * Levi and Steinhoff [2015a] M. Levi and J. Steinhoff, Spinning gravitating objects in the effective field theory in the post-Newtonian scheme, JHEP 09, 219, arXiv:1501.04956 [gr-qc] .
  * Le Tiec and Casals [2021] A. Le Tiec and M. Casals, Spinning Black Holes Fall in Love, Phys. Rev. Lett. 126, 131102 (2021), arXiv:2007.00214 [gr-qc] .
  * Le Tiec  _et al._ [2021] A. Le Tiec, M. Casals, and E. Franzin, Tidal Love Numbers of Kerr Black Holes, Phys. Rev. D 103, 084021 (2021), arXiv:2010.15795 [gr-qc] .
  * Chia [2021] H. S. Chia, Tidal deformation and dissipation of rotating black holes, Phys. Rev. D 104, 024013 (2021), arXiv:2010.07300 [gr-qc] .
  * Charalambous  _et al._ [2021a] P. Charalambous, S. Dubovsky, and M. M. Ivanov, On the Vanishing of Love Numbers for Kerr Black Holes, JHEP 05, 038, arXiv:2102.08917 [hep-th] .
  * Charalambous  _et al._ [2021b] P. Charalambous, S. Dubovsky, and M. M. Ivanov, Hidden Symmetry of Vanishing Love Numbers, Phys. Rev. Lett. 127, 101101 (2021b), arXiv:2103.01234 [hep-th] .
  * Castro  _et al._ [2021] G. Castro, L. Gualtieri, and P. Pani, Hidden symmetry between rotational tidal Love numbers of spinning neutron stars, Phys. Rev. D 104, 044052 (2021), arXiv:2103.16595 [gr-qc] .
  * Bini  _et al._ [2019] D. Bini, T. Damour, and A. Geralico, Novel approach to binary dynamics: application to the fifth post-Newtonian level, Phys. Rev. Lett. 123, 231104 (2019), arXiv:1909.02375 [gr-qc] .
  * Bini  _et al._ [2020a] D. Bini, T. Damour, and A. Geralico, Binary dynamics at the fifth and fifth-and-a-half post-Newtonian orders, Phys. Rev. D 102, 024062 (2020a), arXiv:2003.11891 [gr-qc] .
  * Bini  _et al._ [2020b] D. Bini, T. Damour, A. Geralico, S. Laporta, and P. Mastrolia, Gravitational dynamics at $O(G^{6})$: perturbative gravitational scattering meets experimental mathematics, (2020b), arXiv:2008.09389 [gr-qc] .
  * Buonanno and Damour [1999] A. Buonanno and T. Damour, Effective one-body approach to general relativistic two-body dynamics, Phys.Rev. D59, 084006 (1999), arXiv:gr-qc/9811091 [gr-qc] .
  * Blümlein  _et al._ [2021] J. Blümlein, A. Maier, P. Marquard, and G. Schäfer, The fifth-order post-Newtonian Hamiltonian dynamics of two-body systems from an effective field theory approach: potential contributions, Nucl. Phys. B 965, 115352 (2021), arXiv:2010.13672 [gr-qc] .
  * Antonelli  _et al._ [2020a] A. Antonelli, C. Kavanagh, M. Khalil, J. Steinhoff, and J. Vines, Gravitational spin-orbit coupling through third-subleading post-Newtonian order: from first-order self-force to arbitrary mass ratios, Phys. Rev. Lett. 125, 011103 (2020a), arXiv:2003.11391 [gr-qc] .
  * Antonelli  _et al._ [2020b] A. Antonelli, C. Kavanagh, M. Khalil, J. Steinhoff, and J. Vines, Gravitational spin-orbit and aligned spin1-spin2 couplings through third-subleading post-Newtonian orders, Phys. Rev. D 102, 124024 (2020b), arXiv:2010.02018 [gr-qc] .
  * Levi and Steinhoff [2017] M. Levi and J. Steinhoff, EFTofPNG: A package for high precision computation with the Effective Field Theory of Post-Newtonian Gravity, Class. Quant. Grav. 34, 244001 (2017), arXiv:1705.06309 [gr-qc] .
  * Levi [2012] M. Levi, Binary dynamics from spin1-spin2 coupling at fourth post-Newtonian order, Phys.Rev. D85, 064043 (2012), arXiv:1107.4322 [gr-qc] .
  * Levi and Steinhoff [2014] M. Levi and J. Steinhoff, Equivalence of ADM Hamiltonian and Effective Field Theory approaches at next-to-next-to-leading order spin1-spin2 coupling of binary inspirals, JCAP 1412, 003, arXiv:1408.5762 [gr-qc] .
  * Levi and Steinhoff [2015b] M. Levi and J. Steinhoff, Leading order finite size effects with spins for inspiralling compact binaries, JHEP 06, 059, arXiv:1410.2601 [gr-qc] .
  * Levi and Steinhoff [2016a] M. Levi and J. Steinhoff, Next-to-next-to-leading order gravitational spin-orbit coupling via the effective field theory for spinning objects in the post-Newtonian scheme, JCAP 1601, 011, arXiv:1506.05056 [gr-qc] .
  * Levi and Steinhoff [2016b] M. Levi and J. Steinhoff, Next-to-next-to-leading order gravitational spin-squared potential via the effective field theory for spinning objects in the post-Newtonian scheme, JCAP 1601, 008, arXiv:1506.05794 [gr-qc] .
  * Levi and Steinhoff [2021] M. Levi and J. Steinhoff, Complete conservative dynamics for inspiralling compact binaries with spins at the fourth post-Newtonian order, JCAP 09, 029, arXiv:1607.04252 [gr-qc] .
  * Levi  _et al._ [2021a] M. Levi, S. Mougiakakos, and M. Vieira, Gravitational cubic-in-spin interaction at the next-to-leading post-Newtonian order, JHEP 01, 036, arXiv:1912.06276 [hep-th] .
  * Levi  _et al._ [2021b] M. Levi, A. J. Mcleod, and M. Von Hippel, N3LO gravitational spin-orbit coupling at order $G^{4}$, JHEP 07, 115, arXiv:2003.02827 [hep-th] .
  * Levi  _et al._ [2021c] M. Levi, A. J. Mcleod, and M. Von Hippel, N3LO gravitational quadratic-in-spin interactions at G4, JHEP 07, 116, arXiv:2003.07890 [hep-th] .
  * Levi and Teng [2021] M. Levi and F. Teng, NLO gravitational quartic-in-spin interaction, JHEP 01, 066, arXiv:2008.12280 [hep-th] .
  * Kol and Smolkin [2008] B. Kol and M. Smolkin, Non-Relativistic Gravitation: From Newton to Einstein and Back, Class.Quant.Grav. 25, 145011 (2008), arXiv:0712.4116 [hep-th] .
  * Kol  _et al._ [2011] B. Kol, M. Levi, and M. Smolkin, Comparing space+time decompositions in the post-Newtonian limit, Class.Quant.Grav. 28, 145021 (2011), arXiv:1011.6024 [gr-qc] .
  * Levi [2010a] M. Levi, Next to Leading Order gravitational Spin1-Spin2 coupling with Kaluza-Klein reduction, Phys.Rev. D82, 064029 (2010a), arXiv:0802.1508 [gr-qc] .
  * Levi [2010b] M. Levi, Next to Leading Order gravitational Spin-Orbit coupling in an Effective Field Theory approach, Phys.Rev. D82, 104004 (2010b), arXiv:1006.4139 [gr-qc] .
  * Porto [2006] R. A. Porto, Post-Newtonian corrections to the motion of spinning bodies in NRGR, Phys.Rev. D73, 104031 (2006), arXiv:gr-qc/0511061 [gr-qc] .
  * Levi [2018] M. Levi, A public framework for Feynman calculations and post-Newtonian gravity, (2018), arXiv:1811.12401 [gr-qc] .
  * Passarino and Veltman [1979] G. Passarino and M. J. G. Veltman, One Loop Corrections for e+ e- Annihilation Into mu+ mu- in the Weinberg Model, Nucl. Phys. B160, 151 (1979).
  * Laporta [2000] S. Laporta, High precision calculation of multiloop Feynman integrals by difference equations, Int. J. Mod. Phys. A15, 5087 (2000), arXiv:hep-ph/0102033 [hep-ph] .
  * Smirnov [2012] V. A. Smirnov,  _Analytic tools for Feynman integrals_, Vol. 250 (2012).
  * Vermaseren [1994] J. A. M. Vermaseren, Axodraw, Comput. Phys. Commun. 83, 45 (1994).
  * Binosi  _et al._ [2009] D. Binosi, J. Collins, C. Kaufhold, and L. Theussl, JaxoDraw: A Graphical user interface for drawing Feynman diagrams. Version 2.0 release notes, Comput. Phys. Commun. 180, 1709 (2009), arXiv:0811.4113 [hep-ph] .
  * Kol and Smolkin [2012] B. Kol and M. Smolkin, Black hole stereotyping: Induced gravito-static polarization, JHEP 02, 010, arXiv:1110.3764 [hep-th] .


# Gradient-Based Adversarial and Out-of-Distribution Detection

Jinsol Lee  Mohit Prabhushankar  Ghassan Alregib

###### Abstract

We propose to utilize gradients for detecting adversarial and out-of-
distribution samples. We introduce confounding labels—labels that differ from
normal labels seen during training—in gradient generation to probe the
effective expressivity of neural networks. Gradients depict the amount of
change required for a model to properly represent given inputs, providing
insight into the representational power of the model established by network
architectural properties as well as training data. By introducing a label of
different design, we remove the dependency on ground truth labels for gradient
generation during inference. We show that our gradient-based approach allows
for capturing the anomaly in inputs based on the effective expressivity of the
models with no hyperparameter tuning or additional processing, and outperforms
state-of-the-art methods for adversarial and out-of-distribution detection.

Machine Learning, ICML

  * Citation

J. Lee, M. Prabhushankar, and G. AlRegib, “Gradient-Based Adversarial and Out-
of-Distribution Detection,” in International Conference on Machine Learning
(ICML) Workshop on New Frontiers in Adversarial Machine Learning, Baltimore,
MD, USA, July 2022.

  * Review

Date of Acceptance: June 13, 2022

  * Bib

@inproceedings{lee2022adversarial,
title={Gradient-Based Adversarial and Out-of-Distribution Detection},
author={Lee, Jinsol, and Prabhushankar, Mohit, and AlRegib, Ghassan},
booktitle={International Conference on Machine Learning (ICML) Workshop on New
Frontiers in Adversarial Machine Learning},
year={2022}}

  * Contact

{jinsol.lee<EMAIL_ADDRESS>
https://ghassanalregib.info/

##  1 Introduction

Deep neural networks (DNNs) have achieved significant advancements over the
past decade, and they are increasingly utilized in cybersecurity applications
such as malware detection and intrusion detection (Mahdavifar & Ghorbani,
2019). However, many recent studies pointed out that DNNs are prone to failure
when deployed in real-world environments where they often encounter data that
diverge from training conditions (Temel et al., 2017, 2018, 2019). Neural
networks rely heavily on the implicit assumption that any given input during
inference is drawn from the same distribution as the training data (_i.e._ ,
in-distribution), thereby making overconfident predictions (Guo et al., 2017).
In addition, the discovery of adversarial attacks by Szegedy  _et al._
(Szegedy et al., 2013) highlighted the vulnerability of DNNs to inputs with
small perturbations specifically designed to deceive the models. To ensure
reliable performance of practical applications of a neural network, the model
must be capable of distinguishing inputs that differ from training data and
cannot be handled properly.

Figure 1: Overview of our approach. We argue that the necessary change
observed in gradients invoked by a confounding label is smaller for an input
similar to training data (_i.e._ , image a) than a more anomalous input from
the model’s perspective (_i.e._ , image b).

In this paper, we examine the ability of trained neural networks to handle
inputs based on its effective expressivity. We define effective expressivity
as the representational power of a model based on the training data. We
propose to utilize backpropagated gradients to analyze the effective
expressivity of a trained model. Gradients correspond to the amount of changes
that a model requires to accurately represent a given sample. We argue that
with gradients, we can characterize the anomaly in inputs based on what the
model is unfamiliar with and thus incapable of representing properly. We
describe the motivation of our approach in Fig. 1. A classifier is trained
with handwritten digit images of 0 through 8 to capture their characteristics
with its weights, visualized for some layers of the trained classifier in Fig.
1. During inference, we observe the necessary changes captured in gradients
invoked by a confounding label with respect to an image similar to training
data (_i.e._ , a handwritten digit 9) and highly dissimilar (_i.e._ , dog).
Our hypothesis is that the necessary change to effective expressivity would be
larger for the dissimilar input since the model weights learned on digit 0 to
8 would not be able to capture the characteristics of a dog. One problem in
utilizing gradients to observe the necessary change is that we do not have
access to the labels for given inputs or any information regarding their
distribution. To remove dependency on ground truth labels in gradient
generation during inference, we introduce a confounding label—label that the
network has not seen during training. While many studies proposed to
manipulate the design of labels for various purposes (Tokozume et al., 2018;
Zhang et al., 2017b; Yun et al., 2019; Durand et al., 2019; Duarte et al.,
2021), they only analyze the effect of different label designs in training
with the statistics of training dataset. In comparison, a confounding label
does not require the knowledge of training data and can be used during
inference with no access to ground truth labels. Our methodology allows for
capturing anomaly in inputs from the perspective the model based on its
effective expressivity. The contributions of this paper are three-fold:

  * •

We propose to utilize backpropagated gradients to characterize anomaly in
inputs seen during inference from the perspective of the model.

  * •

We introduce confounding labels as a tool to elicit model response that can be
utilized to probe the effective expressivity of trained neural networks.

  * •

We validate our approach on the applications of detecting anomalous inputs
including adversarial and out-of-distribution (OOD) samples, and achieve
state-of-the-art performance with no hyperparameter tuning or additional
processing.

##  2 Related Work and Preliminary

###  2.1 Adversarial & Out-of-Distribution Detection

There are many recent studies of detecting anomalous samples, whether they are
adversarially generated or due to statistical shift in data distribution.
Hendrycks & Gimpel (2017) introduced a baseline method to threshold samples
based on the predicted softmax distributions. Liang et al. (2018) proposed
additional input and output processing to further improve the baseline method.
DeVries & Taylor (2018) utilized prediction confidence obtained from an
augmented confidence estimation branch on a pre-trained classifier. Ma et al.
(2018) proposed to characterize the dimensional properties of adversarial
regions with local intrinsic dimensionality. Lee et al. (2018) proposed a
confidence metric using Mahalanobis distance. Liu et al. (2020) utilized
energy score to capture the likelihood of occurrence during inference and
training time. Most approaches utilize activation-based representations to
characterize inputs via learned features, _i.e._ , what the network knows
about given inputs. However, for anomalous inputs, the overconfident nature of
neural networks (Guo et al., 2017) makes it counterintuitive to characterize
them solely with the learned features. Rather, we argue that the anomaly in
inputs should be established from the perspective of the model based on what
the model is unfamiliar with and thus incapable of representing properly.

(a) Gradient-based representation generation (b) Detection Figure 2: Overall
framework for gradient-based anomalous input detection.

###  2.2 Gradients as Features

At the core of training neural networks lies gradient-based optimization
techniques (Ruder, 2016). Apart from their utility as a tool to search for a
converged solution, backpropagated gradients have been utilized for various
purposes. We discuss the recent studies based on how the relevant gradients
are generated. First category is by backpropagating model outputs (_i.e._ ,
logits) directly. This is widely observed in visualization techniques
(Selvaraju et al., 2017; Chattopadhay et al., 2018) to highlight pixels based
on classification scores. Zinkevich et al. (2017) also utilizes logits to
extract holographic representation of the network. These approaches assume
that the test data is drawn from the same distribution as the training data,
and they observe the model response by further enforcing the output of the
model. Another category of approaches is by backpropagating loss between
logits and some target labels. Gradient-based adversarial attack generation
fits under this category. Goodfellow et al. (2015) was first to demonstrate
that a small but intentional perturbation can affect models to completely
change their predictions with high confidence. Others proposed adversarial
attacks that also utilize gradients in an iterative manner (Kurakin et al.,
2017; Madry et al., 2017) with different initialization constraints. Another
relevant work include contrastive visual explanation (Prabhushankar et al.,
2020) where model classification is contrasted with other possible classes.
Oberdiek et al. (2018); Lee & AlRegib (2020, 2021) also utilize loss-based
approaches to generate gradients to quantify the uncertainty of neural
networks. Kwon et al. (2019, 2020) employ the directional information of
gradients for representation learning. These approaches utilize gradients in a
similar way to model training by incorporating some forms of class labels seen
during training. It is natural to utilize these in-distribution class labels
when the considered test data is safe to be assumed in-distribution and it
allows for inciting the model response relevant for the classes of interest.
However, this becomes irrelevant when the data during inference is not drawn
from the same distribution as the training data. We argue that to characterize
anomaly in inputs, unseen labels should be introduced for gradient generation
instead of seen class labels.

###  2.3 Manipulating Label Designs

One-hot encoding is the most widely used approach in formulating labels to
handle nominal data (_i.e._ , categorical data without quantitative
relationships between categories) for many applications including single-label
image classification. For multi-label image classification, a combination of
multiple one-hot encoded vectors is used to indicate information about
multiple classes in each image. Based on these two, many recent studies
analyze different formulations of labels for various purposes. Some work
(Tokozume et al., 2018; Zhang et al., 2017b; Yun et al., 2019) proposed to mix
two input images and their labels with pre-computed ratio as a data
augmentation technique to improve generalizability as well as robustness of
the models. Prabhushankar & AlRegib (2021) explored combinations of binary
classification labels in extracting causal visual features for
interpretability. Another type of manipulated label design is partial labels.
Durand et al. (2019) tackled the problem of missing labels per image in multi-
label classification and proposed to exploit the proportion of known labels
per sample to weigh the contribution of the images and labels. Duarte et al.
(2021) proposed to mask parts of labels based on the ratio of positive samples
to negative samples to handle imbalanced dataset. These approaches explore
different designs of labels only in model training and require the knowledge
of training data. We propose to manipulate label designs to characterize
inputs from the perspective of models with gradients during inference.
Contrary to existing studies, our approach does not depend on the availability
of training data statistics or ground truth labels of inference data.

###  2.4 Capacity and Expressivity

We discuss the ability of a trained neural network to handle given inputs
based on two main concepts: capacity and expressivity. Capacity of a neural
network is mainly expressed by its size or the number of parameters (Neyshabur
et al., 2017; Sun et al., 2017). On the other hand, expressivity describes how
different architectural properties of a network (depth, width, layer type)
affects its ability to approximate functions (Raghu et al., 2017; Zhang et
al., 2017a). Many studies discuss expressivity by analyzing different classes
of neural networks with the same number of parameters on the functions which
the networks are capable or incapable of representing. In essence, capacity
can be seen as the maximum utilization of the network’s representational power
whereas expressivity is the observed utilization based on the connection of
network parameters.

##  3 Method

In this section, we discuss our framework in two stages as shown in Fig. 2a:
generating gradient-based representations with confounding labels and
detecting anomalous inputs. We also demonstrate the effectiveness of gradients
generated with confounding labels in comparison with activations.

###  3.1 Gradients with Confounding Labels

(a)  (a) Gradients

(b) Activations

Figure 3: Comparison between layer-wise gradients $\nabla_{\ell}J$ and
activations in $L2$-norms. The magnitude values are obtained from a
convolutional layer from each residual block of a ResNet-18 model trained on
CIFAR-10 dataset.

Based on the concepts of capacity and expressivity, we define effective
expressivity as the expressivity that is effectively utilized for learning
given training data, which establishes the knowledge base of the model.
Consider two models of the same architecture but with different training
datasets of MNIST (LeCun et al., 1998) and ImageNet (Deng et al., 2009). The
capacity and expressivity would be the same for both models since the network
architecture remains fixed and these characteristics are independent of data.
However, the effective expressivity would be less for the model trained with a
simpler dataset of MNIST than more complex dataset of ImageNet. We propose to
utilize gradients generated with a confounding label to examine the effective
expressivity of a classifier during inference. Our approach focuses on the
lack of knowledge that the model should offset in order to fully grasp the
anomaly in the data. A confounding label provides an unsupervised way to
elicit gradient responses relevant for the effective expressivity of a trained
network that is necessary to improve its representational power and get closer
to its capacity.

We discuss the first stage of our framework, which is the generation of
gradient-based representations with confounding labels, shown in Fig. 2a. An
input image is passed through a trained image classifier, and the model yields
logits. We utilize binary cross entropy loss $J(\theta)$ between the logits
and a confounding label, as shown in Eq. (1), where $\hat{y_{i}}$ is the
predicted probability for class $i$, $y_{c,i}$ is the true probability
represented by the confounding label, and $N$ is the number of classes. For
this paper, we utilize the confounding label that combines one-hot encodings
of all classes (_i.e._ , a vector of all 1’s).

$J(\theta)=\frac{1}{N}\sum^{N-1}_{i=0}\left(y_{c,i}\cdot\log\left(\hat{y_{i}}\right)+\left(1-y_{c,i}\right)\cdot\log\left(1-\hat{y_{i}}\right)\right),$
(1)

The loss is then backpropagated through the model, generating gradients for
each network layer. While any form of the gradients that preserve their
magnitude would be valid, we measure the squared $L_{2}$-norm of the gradients
for each model parameter set (_i.e._ , weight and bias parameters of network
layers) and concatenate them to construct a gradient-based representation for
each input. The obtained representation is of the following form,

$\begin{gathered}d_{\nabla\theta}=\big{[}~{}\|\nabla
J_{\theta_{0}}(\theta)\|^{2}_{2}~{},~{}\cdots~{},~{}\|\nabla
J_{\theta_{L-1}}(\theta)\|^{2}_{2}~{}\big{]},\end{gathered}$ (2)

where $L$ is the number of layers in a given network.

Validation of gradients with confounding labels We demonstrate the
effectiveness of gradients obtained with confounding labels in comparison with
activations. In an adversarial detection setup, we show the disparity between
effective expressivity established with training data and the necessary
adjustments to represent original image samples and adversarial samples. The
key idea is that the model will need larger updates to handle adversarial
samples than original images, captured by the magnitudes of the generated
gradients. For this experiment, we first train a ResNet-18 (He et al., 2016)
classifier trained with original CIFAR-10 (Krizhevsky & Hinton, 2009) train
set. The test set of CIFAR-10 is utilized to generate adversarial attacks of
the following: fast gradient sign method (FGSM) (Goodfellow et al., 2015),
basic iterative method (BIM) (Kurakin et al., 2017), Carlini & Wagner attack
(C&W) (Carlini & Wagner, 2017), projected gradient descent (PGD) (Madry et
al., 2017), iterative least-likely class method (IterLL) (Kurakin et al.,
2017), and semantic attack (Hosseini et al., 2017). We collect the L2 norm of
layer-wise gradients and activations and visualize their distribution in Fig.
3. Due to the space limitation, we select a convolutional layer from each
residual block of the ResNet architecture for visualization. The separation in
the ranges of gradient magnitudes, shown in Fig. 3a, between original images
of CIFAR-10 test set and their adversarial versions is more evident in some
parts of the network than others because each layer captures information about
different aspects of given inputs. But overall, we observe the smaller
magnitude ranges for original images and significantly larger magnitude ranges
for adversarial attack images. On the other hand, the activations throughout
the network, shown in Fig. 3b, do not show as much variation in the magnitude
ranges for different image sets. It supports our intuition that gradients can
characterize the anomaly better than activations based on the unfamiliar
aspects of given inputs from the model’s perspective.

Table 1: Adversarial detection results for CIFAR-10 in AUROC. For Mahalanobis
method (denoted as M), we report vanilla results (V), _i.e._ , without the
input pre-processing or feature ensemble, input pre-processing only (P),
feature ensemble only (FE), and with both (P+FE). All values are in
percentages and the best results are highlighted in bold.

Model | Attacks | Baseline | LID | M(V) | M(P) | M(FE) | M(P+FE) | Ours
---|---|---|---|---|---|---|---|---
ResNet | FGSM | 51.20 | 90.06 | 81.69 | 84.25 | 99.95 | 99.95 | 93.45
BIM | 49.94 | 99.21 | 87.09 | 89.20 | 100.0 | 100.0 | 96.19
C&W | 53.40 | 76.47 | 74.51 | 75.71 | 92.78 | 92.79 | 97.07
PGD | 50.03 | 67.48 | 56.27 | 57.57 | 65.23 | 75.98 | 95.82
IterLL | 60.40 | 85.17 | 62.32 | 64.10 | 85.10 | 92.10 | 98.17
Semantic | 52.29 | 86.25 | 64.18 | 65.79 | 83.95 | 84.38 | 90.15
DenseNet | FGSM | 52.76 | 98.23 | 86.88 | 87.24 | 99.98 | 99.97 | 96.83
BIM | 49.67 | 100.0 | 89.19 | 89.17 | 100.0 | 100.0 | 96.85
C&W | 54.53 | 80.58 | 75.77 | 76.16 | 90.83 | 90.76 | 97.05
PGD | 49.87 | 83.01 | 70.39 | 66.52 | 86.94 | 83.61 | 96.77
IterLL | 55.43 | 83.16 | 70.17 | 66.61 | 83.20 | 77.84 | 98.53
Semantic | 53.54 | 81.41 | 62.16 | 62.15 | 67.98 | 67.29 | 89.55

###  3.2 Anomalous Input Detection

We discuss the second stage of our framework in Fig. 2b, which is the
detection of anomalous inputs with gradient-based representations. The
gradient-based representations are collected for both normal and anomalous
images with confounding labels, and they are split into train, validation, and
test sets (40%-40%-20% split). Each representation is labeled with the binary
information regarding its corresponding input image (positive for anomalous
inputs, negative for normal inputs). A multi-layer perceptrons (MLP) of 2
layers is trained as an anomalous input detector using the train and
validation sets, and the trained MLP model is applied on the test set to
report the detection accuracy.

##  4 Experiments

In this section, we utilize gradient-based representations generated in
response to confounding labels for detecting anomalous inputs: adversarial
detection and out-of-distribution detection.

###  4.1 Adversarial Detection

For adversarial detection, we train ResNet and DenseNet (Huang et al., 2017)
architectures for a classification task on CIFAR-10 training set. The test set
of CIFAR-10 is utilized to generate adversarial attacks mentioned in the
validation portion of Section 3.1. We utilize the pristine CIFAR-10 test set
as negative samples and the adversarial images as positive samples. For
comparison, we employ the Baseline method (Hendrycks & Gimpel, 2017), local
intrinsic dimensionality (LID) scores (Ma et al., 2018), and the Mahalanobis
method (Lee et al., 2018). Following the setup detailed in Section 3.2, the
performance is measured in AUROC and reported in Table 1. For the Mahalanobis
method, we report vanilla results (V), _i.e._ , without input pre-processing
or feature ensemble, results with input pre-processing only (P), results with
feature ensemble only (FE), results with both (P+FE) to demonstrate its
dependency on delicate calibration of hyperparameters.

The Mahalanobis method shows the best performance for the adversarial attack
types of FGSM and BIM. On the other hand, our approach outperforms all state-
of-the-art methods for the C&W, PGD, IterLL and Semantic attacks. Note that
the Mahalanobis method requires fine-tuning of hyperparameters. For all
attacks, we utilize the range of hyperparameter values that they have explored
in their paper and report the performance with the best parameter values.
Interestingly enough, the Mahalanobis approach shows less saturated
performance for the types of attacks that were not included in their work.
These attack types also correspond to the ones for which our method
outperforms all the state-of-the-art approaches. This indicates the need for
further hyper parameter tuning outside the suggested parameter values,
depending on the types of considered adversarial attacks. It highlights the
benefit of our approach in its simplicity of obtaining the gradient-based
representations with no hyperparameter tuning or any pre-/post-processing. We
remark that our method can characterize the adversarial attack samples
effectively, even for the attack types that are widely considered to be highly
challenging to detect: C&W and PGD attacks.

###  4.2 Out-of-distribution Detection

Table 2: Out-of-distribution detection results in detection accuracy, AUROC,
and AUPR. All values are in percentages and the best results are highlighted
in bold.

Dataset Distribution | Baseline / ODIN / Mahalanobis / Energy / Ours
---|---
In | Out | Detection Accuracy | AUROC | AUPR
CIFAR-10 | SVHN |  83.36 / 88.81 / 91.95 / - / 98.04 |  88.30 / 94.93 / 97.10 / 95.72 / 99.84 |  88.26 / 95.45 / 96.12 / 97.26 / 99.98
(ResNet) | ImageNet |  84.01 / 85.21 / 97.45 / - / 91.28 |  90.06 / 91.86 / 99.68 / 84.95 / 96.92 |  89.26 / 91.60 / 99.60 / 84.60 / 97.21
| LSUN |  87.34 / 88.42 / 98.60 / - / 98.37 |  92.79 / 94.48 / 99.86 / 92.39 / 99.86 |  92.30 / 94.22 / 99.82 / 95.40 / 99.87
SVHN | CIFAR-10 |  79.98 / 80.12 / 88.84 / - / 97.90 |  81.50 / 81.49 / 95.05 / 98.85 / 99.79 |  81.01 / 80.95 / 90.25 / 99.57 / 98.11
(ResNet) | ImageNet |  81.70 / 81.92 / 96.17 / - / 97.74 |  83.69 / 83.82 / 99.23 / 98.18 / 99.77 |  82.54 / 82.60 / 98.17 / 99.30 / 97.93
| LSUN |  80.96 / 81.15 / 97.50 / - / 99.04 |  82.85 / 82.98 / 99.54 / 97.72 / 99.93 |  81.97 / 82.01 / 98.84 / 99.09 / 99.21
CIFAR-10 | SVHN |  87.71 / 86.80 / 95.75 / - / 98.92 |  93.51 / 94.51 / 98.96 / 89.33 / 99.94 |  94.60 / 95.20 / 97.21 / 94.01 / 99.99
(DenseNet) | ImageNet |  84.93 / 91.16 / 96.83 / - / 90.07 |  91.50 / 96.93 / 99.45 / 86.17 / 96.44 |  91.02 / 96.85 / 99.29 / 87.20 / 96.41
| LSUN |  85.58 / 91.91 / 98.08 / - / 99.20 |  92.09 / 97.51 / 99.74 / 92.77 / 99.96 |  91.71 / 97.51 / 99.72 / 95.57 / 99.96
SVHN | CIFAR-10 |  86.61 / 86.25 / 96.50 / - / 98.34 |  91.90 / 91.76 / 99.02 / 90.63 / 99.82 |  92.12 / 92.38 / 94.40 / 94.30 / 98.36
(DenseNet) | ImageNet |  90.22 / 90.31 / 98.89 / - / 98.21 |  94.78 / 95.08 / 99.84 / 87.51 / 99.81 |  94.76 / 95.53 / 99.37 / 92.15 / 98.16
| LSUN |  89.14 / 89.13 / 99.49 / - / 99.52 |  94.12 / 94.52 / 99.92 / 83.45 / 99.96 |  94.32 / 95.20 / 99.54 / 89.33 / 99.70

We employ various image classification datasets for OOD detection: CIFAR-10
and SVHN (Netzer et al., 2011) as in-distribution, and additional TinyImageNet
(Deng et al., 2009) and LSUN (Yu et al., 2015) as OOD. We demonstrate our
approaches using ResNet-18 and DenseNet. For evaluation, we measure detection
accuracy, area under receiver operating characteristic curve (AUROC), and area
under precision-recall curve (AUPR). The results of OOD detection are reported
in Table 2 with other state-of-the-art methods, including Baseline (Hendrycks
& Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018),
and Energy (Liu et al., 2020). For Energy method, detection accuracy is
omitted as they do not specifically determine the threshold values for energy
scores for OOD detection.

We observe that our method outperforms other state-of-the-art methods in AUROC
when CIFAR-10 is used as in-distribution and SVHN as OOD, and SVHN as in-
distribution and all others as OOD for ResNet. For DenseNet, our method
performs better in every experiment except with TinyImageNet as OOD. Our
method is especially effective when there is a larger difference in the
complexity of in-distribution and OOD datasets. The results support our
intuition behind the effective expressivity of a trained network based on its
training data. The models trained on CIFAR-10 have better effective
expressivity than those trained on the simpler SVHN, and they are capable of
extracting more diverse features. The models trained on SVHN would require
larger updates to handle inputs of higher complexity, exhibiting more apparent
gap in the model responses and resulting in the better performance in OOD
detection.

It is interesting to note the performance of our gradient-based approach when
TinyImageNet is used as OOD for both architectures. With SVHN as the in-
distribution dataset and TinyImageNet as OOD, our approach achieves the best
performance, if not a very close second. However, with CIFAR-10 as in-
distribution, our approach shows inferior performance to the Mahalanobis
method. We provide samples images of each dataset for visual analysis in Fig.
4. With CIFAR-10 as in-distibution, TinyImageNet shows the most visual
similarity with CIFAR-10 in the types and scales of objects in the images
among the three OOD datasets shown in Fig. 4 LABEL:sub@sfig:dataset_svhn-
LABEL:sub@sfig:dataset_lsun. SVHN is a street-view house number dataset, which
is vastly different from CIFAR-10 of natural images. LSUN is a scene
understanding dataset with scene categories as classifications. Our experiment
results show that the more similar the in-distribution and OOD datasets, the
smaller the gap in the spreads of gradient magnitudes, which makes it more
challenging to differentiate OOD samples from in-distribution samples.
Overall, we remark that our gradient-based method outperforms all activation-
based methods in most cases.

(a) CIFAR-10

(b) SVHN

(c) ImageNet

(d) LSUN

Figure 4: Sample dataset images.

##  5 Conclusion

In this paper, we propose to examine the effective expressivity of neural
networks with backpropagated gradients for adversarial and out-of-distribution
detection. With gradients, we characterize the anomaly in inputs seen during
inference based on the lack of knowledge of the model, rather than with its
learned features. We introduce confounding labels as a tool to elicit gradient
response to remove the dependency on ground truth labels during inference. We
show that our approach outperforms state-of-the-art methods for adversarial
and out-of-distribution detection with no need for hyperparameter tuning or
additional processing.

## References

  * Carlini & Wagner (2017) Carlini, N. and Wagner, D.  Towards evaluating the robustness of neural networks.  In _IEEE Symposium on Security and Privacy_ , pp. 39–57, 2017.
  * Chattopadhay et al. (2018) Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE winter conference on applications of computer vision (WACV)_ , pp. 839–847. IEEE, 2018.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.
  * DeVries & Taylor (2018) DeVries, T. and Taylor, G. W.  Learning confidence for out-of-distribution detection in neural networks.  _arXiv preprint arXiv:1802.04865_ , 2018.
  * Duarte et al. (2021) Duarte, K., Rawat, Y., and Shah, M.  Plm: Partial label masking for imbalanced multi-label classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2739–2748, 2021.
  * Durand et al. (2019) Durand, T., Mehrasa, N., and Mori, G.  Learning a deep convnet for multi-label classification with partial labels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 647–657, 2019.
  * Goodfellow et al. (2015) Goodfellow, I., Shlens, J., and Szegedy, C.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations_ , 2015.
  * Guo et al. (2017) Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q.  On calibration of modern neural networks.  In _Proceedings of the International Conference on Machine Learning-Volume 70_ , pp. 1321–1330. JMLR. org, 2017.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 770–778, 2016.
  * Hendrycks & Gimpel (2017) Hendrycks, D. and Gimpel, K.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _Proceedings of the International Conference on Learning Representations_ , 2017.
  * Hosseini et al. (2017) Hosseini, H., Xiao, B., Jaiswal, M., and Poovendran, R.  On the limitation of convolutional neural networks in recognizing negative images.  In _IEEE International Conference on Machine Learning and Applications_ , pp. 352–358, 2017.
  * Huang et al. (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.  Densely connected convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4700–4708, 2017.
  * Krizhevsky & Hinton (2009) Krizhevsky, A. and Hinton, G.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Kurakin et al. (2017) Kurakin, A., Goodfellow, I., and Bengio, S.  Adversarial examples in the physical world.  _International Conference on Learning Representations Workshop_ , 2017\.
  * Kwon et al. (2019) Kwon, G., Prabhushankar, M., Temel, D., and AlRegib, G.  Distorted representation space characterization through backpropagated gradients.  In _2019 IEEE International Conference on Image Processing (ICIP)_ , pp. 2651–2655. IEEE, 2019.
  * Kwon et al. (2020) Kwon, G., Prabhushankar, M., Temel, D., and AlRegib, G.  Backpropagated gradient representations for anomaly detection.  _arXiv preprint arXiv:2007.09507_ , 2020.
  * LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee & AlRegib (2020) Lee, J. and AlRegib, G.  Gradients as a measure of uncertainty in neural networks.  In _IEEE International Conference on Image Processing_ , 2020.
  * Lee & AlRegib (2021) Lee, J. and AlRegib, G.  Open-set recognition with gradient-based representations.  In _IEEE International Conference on Image Processing_ , 2021.
  * Lee et al. (2018) Lee, K., Lee, K., Lee, H., and Shin, J.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Advances in Neural Information Processing Systems_ , pp. 7167–7177, 2018.
  * Liang et al. (2018) Liang, S., Li, Y., and Srikant, R.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _Proceedings of International Conference on Learning Representations_ , 2018.
  * Liu et al. (2020) Liu, W., Wang, X., Owens, J., and Li, Y.  Energy-based out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Ma et al. (2018) Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S., Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J.  Characterizing adversarial subspaces using local intrinsic dimensionality.  In _Proceedings of the International Conference on Learning Representations_ , 2018.
  * Madry et al. (2017) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  In _Proceedings of the International Conference on Learning Representations_ , 2017.
  * Mahdavifar & Ghorbani (2019) Mahdavifar, S. and Ghorbani, A. A.  Application of deep learning to cybersecurity: A survey.  _Neurocomputing_ , 347:149–176, 2019.
  * Netzer et al. (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y.  Reading digits in natural images with unsupervised feature learning.  _NeurlIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2011.
  * Neyshabur et al. (2017) Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N.  Exploring generalization in deep learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Oberdiek et al. (2018) Oberdiek, P., Rottmann, M., and Gottschalk, H.  Classification uncertainty of deep neural networks based on gradient information.  In _IAPR Workshop on Artificial Neural Networks in Pattern Recognition_ , pp. 113–125. Springer, 2018.
  * Prabhushankar & AlRegib (2021) Prabhushankar, M. and AlRegib, G.  Extracting causal visual features for limited label classification.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pp. 3697–3701. IEEE, 2021.
  * Prabhushankar et al. (2020) Prabhushankar, M., Kwon, G., Temel, D., and AlRegib, G.  Contrastive explanations in neural networks.  In _2020 IEEE International Conference on Image Processing (ICIP)_ , pp. 3289–3293. IEEE, 2020.
  * Raghu et al. (2017) Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J.  On the expressive power of deep neural networks.  In _international conference on machine learning_ , pp. 2847–2854. PMLR, 2017.
  * Ruder (2016) Ruder, S.  An overview of gradient descent optimization algorithms.  _arXiv preprint arXiv:1609.04747_ , 2016.
  * Selvaraju et al. (2017) Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Sun et al. (2017) Sun, C., Shrivastava, A., Singh, S., and Gupta, A.  Revisiting unreasonable effectiveness of data in deep learning era.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 843–852, 2017.
  * Szegedy et al. (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Temel et al. (2017) Temel, D., Kwon, G., Prabhushankar, M., and AlRegib, G.  Cure-tsr: Challenging unreal and real environments for traffic sign recognition.  _NeurIPS Workshop on Machine Learning for Intelligent Transportation Systems_ , 2017.
  * Temel et al. (2018) Temel, D., Lee, J., and AlRegib, G.  Cure-or: Challenging unreal and real environments for object recognition.  In _IEEE International Conference on Machine Learning and Applications_ , pp. 137–144, 2018.
  * Temel et al. (2019) Temel, D., Lee, J., and AlRegib, G.  Object recognition under multifarious conditions: A reliability analysis and a feature similarity-based performance estimation.  In _IEEE International Conference on Image Processing_ , pp. 3033–3037, 2019.
  * Tokozume et al. (2018) Tokozume, Y., Ushiku, Y., and Harada, T.  Between-class learning for image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 5486–5494, 2018.
  * Yu et al. (2015) Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 6023–6032, 2019.
  * Zhang et al. (2017a) Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O.  Understanding deep learning requires rethinking generalization.  _Proceedings of the International Conference on Learning Representations_ , 2017a.
  * Zhang et al. (2017b) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017b.
  * Zinkevich et al. (2017) Zinkevich, M. A., Davies, A., and Schuurmans, D.  Holographic feature representations of deep networks.  In _UAI_ , 2017.