# A test of Einstein’s equivalence principle in future VLBI observations

Joseph P Johnson  0000-0003-4618-2092<EMAIL_ADDRESS>Department of
Physics, Indian Institute of Technology Bombay, Mumbai 400076, India  IISER
Mohali, Knowledge City, Sector 81, SAS Nagar, Manauli PO 140306, India
Susmita Jana  0000-0001-7681-7533<EMAIL_ADDRESS>Department of
Physics, Indian Institute of Technology Bombay, Mumbai 400076, India  S.
Shankaranarayanan  0000-0002-7666-4116<EMAIL_ADDRESS>Department of
Physics, Indian Institute of Technology Bombay, Mumbai 400076, India

###### Abstract

We show that very-long-baseline-interferometry (VLBI) observations of
supermassive black holes will allow us to test the fundamental principles of
General Relativity (GR). GR is based on the universality of gravity and
Einstein’s equivalence principle (EEP). However, EEP is not a basic principle
of physics but an empirical fact. Non-minimal coupling (NMC) of
electromagnetic fields violates EEP, and their effects manifest in the strong-
gravity regime. Hence, VLBI observations of black holes provide an opportunity
to test NMC in the strong-gravity regime. To the leading order in the spin
parameter, we explicitly show that the NMC of the electromagnetic field
introduces observable modifications to the black hole image. In addition, we
find that the size of the photon rings varies by $\sim 3r_{H}$, which
corresponds to $\sim 30\mu as$ for Sagittarius $A^{*}$ and $\sim 23\mu as$ for
M87. VLBI telescopes are expected to attain a resolution of $\sim 5\mu as$ in
the near future. However, direct detection of photon ring will require the
resolution of $\sim 1\mu as$ for M87, which can potentially be probed by the
space-based Event Horizon Explorer.

_Introduction:_ Einstein’s general relativity (GR) has been tested at solar
system scales and in the weak field limit [1, 2, 3]. Until recently, strong-
gravity regime observations were difficult. The detection of gravitational
waves by the LIGO-VIRGO-KAGRA collaboration [4, 5, 6] and the imaging of the
black hole shadow by the Event Horizon Telescope (EHT) using Very-Long-
Baseline Interferometry (VLBI) [7, 8, 9] offer fresh possibilities to search
for deviations from GR in the strong-gravity regime.

GR is based on the universality of gravity and Einstein’s equivalence
principle (EEP) [3, 10]. EEP specifies the interaction between gravity and all
other matter fields [11, 12, 13]. EEP is divided into three sub-principles:
the weak EP (the universality of free fall), local Lorentz invariance, and
local position invariance. EEP is satisfied if all these three sub-principles
are satisfied [3].

However, EEP is not a basic principle of physics but an empirical fact [14].
Hence, one way to assess the validity of GR is to test EEP. In the weak field
limit, the weak EP has been tested in several parts in $10^{16}$ using the
torsion balance, lunar-laser ranging, and space missions like MicroSCOPE [15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. Experiments using laser-
cooled trapped atoms to look for variations in the relative frequencies of
different types of atoms as the Earth rotates around the mean rest frame of
the universe have placed very strict constraints on local Lorentz invariance
[28, 29, 1, 30]. Additionally, gravitational redshift experiments and tests of
variations in fundamental constants have been used to examine local position
invariance [31].

While the tests of EP have improved over the years, contamination by complex
physics is one of the primary challenges of testing GR in the strong-gravity
regime. This work aims to develop an observational test of GR by investigating
the _potential violation_ of EEP in future VLBI observations. Although the
black hole image depends on the details of the accretion flow, two signatures
of the image are unique [32]: First, regardless of the black hole’s spin, the
horizon casts a shadow on the image of the source at around
$\sqrt{27}GM/c^{2}$ [33]. Second, due to the high velocity of the accreting
plasma and the effects of gravitational lensing, the image brightness of the
accretion flow is highly non-uniform [34]. We show that non-minimal coupling
(NMC) of the electromagnetic field distinctly modifies these two signatures
[35, 36, 37, 38, 39].

EEP excludes products between curvature and matter fields (aka NMC) in action
[40]. One way EEP violation occurs is due to NMC [35, 36, 37]. Interestingly,
NMC is required for consistency of the standard model in curved space-time
[38, 39]. NMC is unique if we demand at most second derivatives of
$g_{\mu\nu}$, and the electromagnetic potential ($A_{\mu}$) appears in the
field equations [37]. The effects due to NMC manifest in the strong-gravity
regime, such as in the cores of neutron stars or close to the black hole
horizon. The curvature effects cannot be ruled out despite the extreme
precision of current experiments on the electromagnetic field [41, 30, 42].

VLBI observations of black holes allow for testing NMC as their effects are
substantial near black hole horizons [43]. EHT image shows glowing gas
circling the black hole at high speed and a black hole shadow caused by
lensing and photon capture [7, 9]. Future VLBI telescopes, including the
space-based projects [44] with the help of advanced data analysis techniques,
and imaging techniques such as super-resolution are expected to detect and
observe the unstable photon circular orbit as a photon ring [45, 46, 47, 48].
Furthermore, as we show explicitly, NMC will lead to modified dispersion
relations leading to _different propagation_ for the two polarization modes.
We also analyze the difference in the resulting black hole shadow for the two
modes compared to GR for Schwarzschild and slowly rotating Kerr.

First, we analyze the photon trajectories in the Schwarzschild black hole and
observe photon flux distribution for each mode that determines the properties
of the black hole image. We show that, for one polarization mode, the horizon
casts a shadow of radius _greater than_ $\sqrt{27}GM/c^{2}$ on the source
image. For the other polarization mode, it is _smaller than_
$\sqrt{27}GM/c^{2}$. Later, in the case of the Kerr black hole, we show that
the size of the observed black hole shadow is largely independent of the spin
of the black hole. The effect of NMC on the shadow size follows a similar
trend as in the case of Schwarzschild. [By shadow, we mean the interior region
of the observed photon ring.] Finally, we discuss the possibility of
constraining the NMC constant using future VLBI observations.

[Variables, like $({r},t,{s},{a})$, _used below are dimensionless_. They are
related to dimensionful variables $(\tilde{r},\tilde{t},\tilde{s},\tilde{a})$
by a factor $1/{r}_{H}$ where ${r}_{H}=2GM/c^{2}$ is the Schwarzschild radius.
The rotational parameter ($a$) is twice the dimensionless rotational parameter
used in the literature. Over-dot denotes derivative w.r.t. dimensionless
affine parameter $s$ and prime denotes derivative w.r.t. $r$.]

_Implications of NMC for Schwarzschild:_ As mentioned above, NMC is unique if
we demand at most second derivatives of $g_{\mu\nu}$ and $A_{\mu}$ appear in
the field equations [37]. For the Ricci-flat space-times, NMC electromagnetic
(test) field action is [35, 49]

$\mathrm{S}=\int
d^{4}x\sqrt{-g}\left[-F_{\mu\nu}F^{\mu\nu}+2{\tilde{\lambda}}R^{\mu\nu\alpha\beta}F_{\mu\nu}F_{\alpha\beta}\right]/4\,,$
(1)

where $\tilde{\lambda}$ is the dimensionful NMC constant. The resulting
equation of motion is:

$\nabla_{\nu}F^{\mu\nu}=2\tilde{\lambda}\left[R^{\mu\nu\rho\sigma}\left(\nabla_{\nu}F_{\rho\sigma}\right)+\left(\nabla_{\rho}R_{\sigma}^{\mu}\right)F^{\rho\sigma}\right]\,.$
(2)

In the local inertial (tetrad) frame basis, the above equation leads to the
following dispersion relation [49]:

$\left[p^{(\mu)}p_{(\mu)}\delta^{(j)}_{(k)}+4\tilde{\lambda}\left[\frac{p^{(j)}}{p_{(0)}}\epsilon^{(0)}_{(k)}+\epsilon^{(j)}_{(k)}\right]\right]F^{(0)(k)}=0,$
(3)

where $p_{(\mu)}$ is the photon momenta in the tetrad basis and

$\epsilon^{(\alpha)}_{(\beta)}\equiv
R^{(\alpha)(\mu)(\nu)}\,_{(\beta)}\,p_{(\mu)}p_{(\nu)}\,.$ (4)

Note that the above expression is valid for any Ricci-flat space-time.
However, the tetrad basis depends on the background geometry, leading to
modified dispersion relations for different space-times.

To discern the effects of NMC on the photon orbit and the flux, we first
consider the Schwarzschild background:

$ds^{2}=-f(r)\,dt^{2}+dr^{2}/f(r)+r^{2}d\Omega^{2}\,,$ (5)

where, $f(r):=1-1/r$ and $d\Omega^{2}$ is the metric on unit $\mathbb{S}^{2}$.
In these coordinates, the conserved quantities — energy ($E$) and angular
momentum ($L$) — are:

$E=f(r)\,\dot{t}\,,\quad L=r^{2}\,\dot{\phi}\,.$ (6)

Substituting the above metric in Eq. (3) and using the tetrad basis defined
(in Eq.(31.4a)) in Ref. [10], the modified dispersion relation for the two
polarization modes of the photon (denoted by ${}^{\prime}+^{\prime}$ and
${}^{\prime}-^{\prime}$) is:

$\\!\\!p^{2}=\mathscr{C}_{\pm}\,p_{(\phi)}^{2},\leavevmode\nobreak\
\mathscr{C}_{+}=\dfrac{6{\lambda}}{2{\lambda}+{r}^{3}},\leavevmode\nobreak\
\mathscr{C}_{-}=\dfrac{6{\lambda}}{4{\lambda}-{r}^{3}},$ (7)

where $p^{2}=p^{\mu}p_{\mu}$, and ${\lambda}=\tilde{\lambda}/r_{H}^{2}$ is
dimensionless. In the equatorial plane, Eq. (7) reduces to:

$\dot{r}^{2}={E}^{2}\,\left[1-{{b}^{2}}\left(1-\mathscr{C}_{\pm}\right)f(r)/r^{2}\right],$
(8)

where ${b}={L}/{E}$ denotes the impact parameter. The photon circular orbit
satisfy two independent conditions [50, 51, 52]: $\dot{{r}}=0$ and
$\ddot{{r}}=0$. It is not easy to obtain an exact analytical solution for
these equations; hence, we numerically obtain the radius (${r}_{c}$) of the
circular orbits and the corresponding impact parameter ${b}_{c}$. (Using the
perturbative approach, one can obtain $r_{c}$ and $b_{c}$. These are
consistent with numerical results. For details, see the supplementary material
[53].)

1 contains the plot of $r$ (solid line) and $b$ (dotted line) corresponding to
the circular orbit for the $+,-$ modes as functions of $\lambda$. For a
Schwarzchild black hole, $b_{c}$ denotes the position of the photon ring as
seen by a distant observer. As mentioned, we consider the interior of the
photon ring as the shadow of the black hole. However, in general scenarios,
depending on the characteristics of the accretion process that acts as the
light source around the black hole, the observed shadow consists of a region
larger than the photon ring. The black hole shadow with NMC for Novikov-Thorne
accretion model [54] (with low numerical accuracy) is discussed in the
supplementary material [53]. Figs 2 and 3 in [53] contain black hole image for
the two accretion flows.

Figure 1: Position of the photon ring $r_{\pm}$ and corresponding impact
parameter $b_{\pm}$ as functions of the coupling constant $\lambda$ along with
the values corresponding to the minimally coupled (MC) mode ($r_{0},b_{0}$)
and the horizon radius $r_{H}$.

From the above figure, we infer the following: First, for both
(${}^{\prime}+^{\prime}$ and ${}^{\prime}-^{\prime}$) modes, $r_{c}$ and
$b_{c}$ intersect at $\lambda=0$. Second, there is no observable photon ring
($r_{c}>r_{H}$) for ${}^{\prime}+^{\prime}$ mode in the case of $\lambda<-0.5$
and ${}^{\prime}-^{\prime}$ mode in the case of $\lambda>0.25$. Third, for
${}^{\prime}+^{\prime}$ mode, when ${\lambda}>0$ radius of the photon sphere
and the corresponding impact parameter are larger than the minimally coupled
case, resulting in a larger observed shadow. When ${\lambda}<0$,
${}^{\prime}+^{\prime}$ mode forms a smaller shadow. This trend is reversed in
the case of ${}^{\prime}-^{\prime}$ mode, where ${\lambda}>0$ leads to the
formation of a smaller shadow, and ${\lambda}<0$ results in a larger shadow.
Lastly, the perturbative approach to obtain $r_{c}$ and $b_{c}$ matches with
numerical results for $|\lambda|<0.2$ (See supplementary material [53]).
Later, we use this approach for Kerr to obtain $r_{c}$ and $b_{c}$. As we show
below, the above results for Schwarzschild continue to hold for Kerr.

The probability of photons from the unstable circular orbit reaching the
distant observer, hence forming a bright ring in the observer’s image plane
with its position given by $b_{c}$ [51, 50]. We now obtain the flux for the
NMC photons by considering radially free-falling optically thin light-emitting
accretion flow [33]. For such a system, photon flux as observed at a point
$(x,y)$ in the observer’s image plane is [52]:

$F(x,y)\propto\int dr\leavevmode\nobreak\ g^{3}{p_{t}}/[|p_{r}|r^{2}],$ (9)

where $g\equiv{p_{\alpha}\,u^{\alpha}_{\rm obs}}/({p_{\beta}u^{\beta}_{e}})$
is the red-shift factor. $u_{\rm obs}$ and $u_{e}$ are the four velocities of
distant observer and accreting gas, respectively, and are given by:

$u_{\rm obs}=(1,0,0,0),\leavevmode\nobreak\ \leavevmode\nobreak\
u_{e}=\left(1/f(r),-{1}/{\sqrt{{r}}},0,0\right)\,.$ (10)

The impact parameter of the photon trajectory is related to the image plane
coordinates by ${b}^{2}=x^{2}+y^{2}$.

Figure 2: Observed flux as a function of impact parameter $b$ for
${}^{\prime}\pm^{\prime}({\cal C}_{\pm})$, and minimally coupled $({\cal
C}_{0})$ modes. The dashed lines are the position of the photon ring (cf. 1).

In 2, we plot the observed flux as a function of $b$ for two specific values
of $\lambda$. From the figure, we infer the following: First, the spikes in
the observed flux (scaled by a constant) correspond to the observed photon
ring for the two NMC modes and minimally coupled photons. Second, in the case
of minimally coupled photons, it is known that the horizon casts a _shadow_ on
the image of the source at $\sqrt{27}\leavevmode\nobreak\
GM/c^{2}=\sqrt{27/4}\,r_{H}$ [33]. However, for ${\lambda}>0$ and
${}^{\prime}+^{\prime}$ mode, the shadow radius is greater than
$\sqrt{27/4}\,r_{H}$ and for ${}^{\prime}-^{\prime}$ mode, the shadow radius
is smaller. This trend is reversed for ${\lambda}<0$. Third, we have not
imposed the circular orbit conditions to obtain the flux distribution. These
results, including the observed position of the photon ring, are consistent
with 1. Lastly, we also see that the brightness and the position of the
lensing ring are affected by the non-minimal coupling [34]. For ${\lambda}<0$
and ${}^{\prime}+^{\prime}$ mode (and ${\lambda}>0$ and
${}^{\prime}-^{\prime}$ mode), the lensing ring is more prominent as compared
to other cases. Hence, the NMC leads to two distinct observable effects in
Schwarzschild — shadow radius changes and lensing ring characteristics are
modified.

An attentive reader might ask: How do the above results translate for Kerr
space-time, describing the super-massive black holes at the galactic centers?
As mentioned earlier, regardless of the black hole’s spin, the horizon casts a
shadow on the image of the source at around $\sqrt{27/4}\leavevmode\nobreak\
r_{H}$ [33]. Noting that the black hole spin is not tightly constrained [55,
56], we show that the shadow radius of NMC photons in a slowly rotating black
hole shows the same trend as Schwarzschild space-time.

_Implications of NMC for Kerr:_ In dimensionless rational polynomial
coordinates $(t,r,\zeta[=\cos\theta],\phi)$, the Kerr space-time is [57]:

$\displaystyle\mathrm{d}{s}^{2}$ $\displaystyle=$
$\displaystyle-\dfrac{\Delta}{\rho^{2}}\left[\mathrm{d}{t}-a(1-\zeta^{2})\mathrm{d}{\phi}\right]^{2}+\frac{\rho^{2}}{\Delta}\mathrm{\leavevmode\nobreak\
d}{r}^{2}+\frac{\rho^{2}}{1-\zeta^{2}}\mathrm{d}\zeta^{2}$ (11)
$\displaystyle+$
$\displaystyle\dfrac{(1-\zeta^{2})}{\rho^{2}}\left[(r^{2}+a^{2})\mathrm{d}{\phi}-a\mathrm{d}{t}\right]^{2}\,,$

where ${a}$ is dimensionless spin parameter, $\Delta={a}^{2}+{r}^{2}-{r}$, and
$\rho^{2}={r}^{2}+{a}^{2}\zeta^{2}$. Substituting the above metric in Eq. (3)
and using the tetrad basis defined in Eq. (2.9) in Ref. [58], the modified
dispersion relation for the two modes of the photon, for arbitrary $a$, is:

${p^{2}}=\dfrac{6(p_{(\zeta)}^{2}+p_{(\phi)}^{2}){\lambda}\rho^{3}}{3{\lambda}\rho^{3}\mp\sqrt{\rho^{12}+9{\lambda}^{2}\rho^{6}+2{\lambda}r\Gamma[\rho^{2}-4{\lambda}{r}\Gamma]}}\,,$
(12)

where $\Gamma=(3\rho^{2}-4{r}^{2})$ and the relation between momenta in tetrad
and coordinate bases is:

$\displaystyle
p^{2}=p_{{r}}^{2}\dfrac{\Delta}{\rho^{2}}-\dfrac{({a}^{2}{E}-{a}{L}+{E}{r}^{2})^{2}}{\Delta\rho^{2}}+p_{(\phi)}^{2}+p_{(\zeta)}^{2},$
$\displaystyle
p_{(\phi)}^{2}+p_{(\zeta)}^{2}=p_{\zeta}^{2}\dfrac{1-\zeta^{2}}{\rho^{2}}+\dfrac{({L}-{a}{E}(1-\zeta^{2}))^{2}}{(1-\zeta^{2})\rho^{2}}.$
(13)

Unlike the minimally coupled case, the above dispersion relation lacks
separability in $r$ and $\zeta$. However, this does not affect the analysis of
the photon trajectory in the equatorial plane. For the
${}^{\prime}\pm^{\prime}$ photon modes, photon trajectory in the equatorial
plane is determined by

$\\!\\!\\!\\!\frac{\dot{{r}}_{\pm}^{2}}{E^{2}}=1+\dfrac{{a}^{2}-{b}^{2}}{{r}^{2}}+\dfrac{({a}-{b})^{2}}{{r}^{2}}\left[\frac{1}{r}+\mathscr{C}_{\pm}\left[\dfrac{{a}^{2}}{{r^{2}}}-f(r)\right]\right].$
(14)

Like in the Schwarzschild, the above equations can not be solved analytically
for the photon circular orbit. However, we can perturbatively solve the same
by expanding the variables _to any order_ in $\lambda$ and $a$. As shown in In
the ${a}\rightarrow 0$ limit, the perturbative solutions are consistent with
the above results for Schwarzschild (see supplementary material [53]). In
${\lambda}\rightarrow 0$ limit, the perturbative solutions reduce to Kerr
results for minimal coupling [50].

_Black hole shadow by a slowly-rotating Kerr:_ One must look at the photon
trajectory around the black hole to analyze the shadow properties. Since the
modified dispersion relation for Kerr (13) is not variable separable,
obtaining analytical solutions for the photon trajectory is difficult.
Interestingly, for a slowly rotating black hole at linear order in the spin
parameter ($a$), the dispersion relation (13) can be separated and are given
by:

$p_{\theta}^{2}+\dfrac{{b}^{2}E^{2}}{\sin^{2}\theta}=\dfrac{E^{2}\mathscr{C}_{\mp}}{f(r)}\left[2ab\left[1-\dfrac{1}{r\mathscr{C}_{\mp}}\right]-\dfrac{r^{2}}{\mathscr{C}_{\pm}}\right]\\\
+p_{{r}}^{2}\dfrac{\mathscr{C}_{\mp}}{\mathscr{C}_{\pm}}{r}^{2}f(r)=k_{\pm}{E}^{2},$
(15)

where ${\cal C}_{\pm}$ are defined in Eq. (7) and $k_{\pm}$ are the separation
constants. Corresponding radial motion of the non-minimally coupled photon
(${}^{\prime}\pm^{\prime}$ modes) is:

$\\!\\!\dot{{r}}_{\rm\pm}^{2}=E^{2}\left[1+\dfrac{\mathscr{C}_{\pm}}{r^{2}\mathscr{C}_{\mp}}\left[k_{\pm}f(r)-2ab\left(\mathscr{C}_{\mp}-\dfrac{1}{r}\right)\right]\right].$
(16)

As in Schwarzschild space-time, we can obtain the circular orbit of the photon
by imposing the conditions $\dot{r}=0$ and $\ddot{r}=0$. For the two modes, we
get the following:

$\displaystyle\scalebox{1.0}{$k_{\pm}=\dfrac{r^{2}[r\mathscr{C}_{\pm}\mathscr{C}_{\mp}^{\prime}+r^{2}f(r)\mathscr{C}_{\mp}\mathscr{C}_{\pm}^{\prime}+\mathscr{C}_{\mp}\mathscr{C}_{\pm}[3-2r\mathscr{C}_{\mp}]]}{{\mathscr{C}_{\pm}}^{2}(\mathscr{C}_{\mp}-r^{2}f(r)\mathscr{{C}_{\mp}^{\prime}}-1)}\Bigr{\rvert}_{r=r_{c}}$}\\!\\!\\!\\!\\!\\!.$
(17)

The above values of ${b}$ and $k$ (in terms of the radius $r_{c}$ of the
circular orbits) determine the shape of the black hole shadow on the image
plane of a distant observer. Before we proceed with the black hole shadow, we
want to mention the following points about the validity of the slowly-rotating
approximation for the supermassive black holes: First, the spin of the
supermassive black holes observed by EHT is not tightly constrained [55, 56].
Second, for minimal coupling, the horizon casts a _shadow_ on the image of the
source at around $\sqrt{27/4}\leavevmode\nobreak\ r_{H}$ [33], regardless of
the black hole’s spin.

Image of the photon ring seen by a distant observer can be described by the
celestial coordinates $(x,y)$ [51, 50]. These celestial coordinates are
defined using the momentum components of photon measured by a distant locally
non-rotating observer (LNRO) [51, 50]:

$x=-{\left[r{p^{(\phi)}}/{p^{(t)}}\right]}_{r\rightarrow\infty},\quad
y={\left[r{p^{(\theta)}}/{p^{(t)}}\right]}_{r\rightarrow\infty}.$ (18)

Using the relation between the non-zero components of the inverse tetrad in
Boyer-Linquist coordinate and LNRO frames (cf. Eq.(3.2) in Ref. [51]), the
celestial coordinates in the distant LNRO’s image plane are:

$x=-{b}/{\sin{\theta_{0}}},\quad y=\pm\sqrt{k-b^{2}/\sin^{2}\theta_{0}}\,,$
(19)

Figure 3: Outline of the shadow of a slowly-rotating black hole (${a}=0.2$)
observed by a distant LNRO for ${}^{\prime}\pm^{\prime}$ modes for
$\lambda=-0.4999,0.2499,$ and $0$ (MC).

where $\theta_{0}$ is the inclination angle of the observer w.r.t. the
symmetry axis of the black hole. Using Eq. (17), one can evaluate the
celestial coordinates $(x,y)$ corresponding to photon circular orbit with
radius ${r}_{c}$. Collection of points $(x({r}_{c}),y({r}_{c}))$ for all
possible values of $r_{c}$ will form the outline of the black hole shadow seen
by a distant observer. The values of all possible ${r}_{c}$ are given by the
conditions ${\rm Im}(y({r}_{c}))=0$ and ${r}_{c}\geq{r}_{h}$, where ${r}_{h}$
is the radius of the outer horizon of the black hole.

3 contains the outline of the shadow of a slowly rotating black hole observed
by a distant LNRO for different photon modes for different $\lambda$ values.
This is the key result of this work, regarding which we want to mention the
following points: First, w.r.t ${\lambda}$, we see a similar trend as in the
case of the Schwarzschild space-time. Observable photon circular orbits
(${r}_{c}>{r}_{h}$) do not form if ${\lambda}\leq-0.5$ for
${}^{\prime}+^{\prime}$ mode and ${\lambda}\geq 0.25$ for
${}^{\prime}-^{\prime}$ mode. Second, for ${\lambda}>0$ the shadow radius is
greater than $\sqrt{27/4}\leavevmode\nobreak\ r_{H}$ for the
${}^{\prime}+^{\prime}$ mode. The ${}^{\prime}+^{\prime}$ mode forms a smaller
shadow when ${\lambda}<0$. This trend is reversed in the case of
${}^{\prime}-^{\prime}$ mode; for ${\lambda}>0$ leads to the formation of a
smaller shadow, and ${\lambda}<0$ results in the shadow radius greater than
$\sqrt{27/4}\leavevmode\nobreak\ r_{H}$. Third, repeating the analysis for
${a}=0.1$ and ${a}=0.15$ shows that the spin ${a}$ does not have a significant
effect on the shadow radius, even though there is a visible effect on the
shadow shape (see Fig. 4 in supplementary material [53]). This is similar to
the results of the minimally coupled case for Kerr space-time [33]. Hence we
expect that the predicted deviation in the shadow radius due to the non-
minimal coupling will be similar for higher-spin black holes. However, this
needs to be verified by numerics.

Fourth, the current resolution of EHT — one of the popular VLBI telescopes —
has a resolution of $24\mu as$ [7]. This translates to $2.4r_{H}$ for
Sagittarius $A^{*}$ and $3.2r_{H}$ for M87, in the image plane of the observer
[8]. This is not good enough for the direct detection of the photon ring in
the case of an equatorial accretion disk with the inner radius of
$r_{ISCO}=3r_{H}$. However, as discussed below, the resolution is expected to
improve in future missions.

Lastly, the difference between the shadow widths for the non-minimally coupled
modes and minimally coupled case, as well as the difference between the
${}^{\prime}+^{\prime}$ and ${}^{\prime}-^{\prime}$ modes for the two nearly
limiting values of ${\lambda}$, are:
1\. Deviation of shadow size for $\lambda=-0.4999$

$\Delta x_{-,0}=1.55,\quad\Delta x_{+,0}=-3.03,\quad\Delta x_{+,-}=-4.58.$

2\. Deviation of shadow size for $\lambda=0.2499$

$\Delta x_{-,0}=-2.16,\quad\Delta x_{+,0}=0.98,\quad\Delta x_{+,-}=3.14.$

Thus, we see that even though the current resolution of EHT is insufficient to
directly detect the photon ring, it is expected that the future VLBI missions
will be able to constrain the non-minimal coupling. Proposed projects include
the Event Horizon Explorer, which aims to add a space-based node to the EHT to
detect and study the photon ring [44]. Advancements in imaging techniques such
as super-resolution and hybrid imaging are expected to improve the resolution
to 5 $\mu as$ by the current estimates [48]. This, along with the higher-
frequency observations (such as 345 GHz), can improve the coverage, along with
the improvements in the data analysis techniques can enhance the instrumental
resolution in the near future [59, 46, 45, 60, 47].

Analyzing the difference between the shadows corresponding to the
${}^{\prime}+^{\prime}$ and ${}^{\prime}-^{\prime}$ modes requires more
information regarding their polarization, which we plan to analyze in a future
work, along with the effect of NMC on the black hole shadow for a variety of
accretion flow models. For instance, the black hole shadow with NMC for
Novikov-Thorne accretion model [54] (with low numerical accuracy) is discussed
in Ref. [53].

_Conclusions:_ EEP is not a basic principle of physics but an empirical fact
[14]. Therefore, testing EEP will assess the validity of GR. NMC of
electromagnetic fields violates EEP and their effects manifest in the strong-
gravity regime. Hence, VLBI observations of black hole image provide an
opportunity to test NMC in the strong-gravity regime. We explicitly showed
that NMC of the electromagnetic field modifies the black hole image in two
ways: First, in linear order in the spin parameter $a$, the horizon casts a
_shadow_ of radius _greater than_ $\sqrt{27/4}\leavevmode\nobreak\ r_{H}$ on
the image of the source for one mode. For the other mode, it is _smaller than_
$\sqrt{27/4}\leavevmode\nobreak\ r_{H}$. Second, the brightness and the
position of the lensing ring are affected by the non-minimal coupling. For
${\lambda}<0$ and ${}^{\prime}+^{\prime}$ mode (and ${\lambda}>0$ and
${}^{\prime}-^{\prime}$ mode), the lensing ring is more prominent as compared
to other cases. Interestingly, the black hole image observations can provide a
weak bound on $\lambda$. The observable photon ring does not form for
${\lambda}<-0.5$ for ${}^{\prime}+^{\prime}$ mode and for ${\lambda}>0.25$ for
the ${}^{\prime}-^{\prime}$ mode. EEP constraint we predict is different from
the one reported Gravity collaboration [61]. Here our interest is to constrain
NMC constant while the authors constrain the violation of EEP via atomic
transitions.

For Kerr space-time, the dispersion relation of the NMC photon is more
complicated and lacks separability in $r$ and $\zeta$. Hence, in the current
work, we considered slowly rotating black holes at linear order in the
rotational parameter $a$. Our analysis is valid for $a\leq 0.20$
($\tilde{a}\leq 0.4M$). In order to compare with ngEHT observation for higher
$a$, we need to solve Eq. (13) numerically for higher spins. This is currently
in progress.

EHT has detected high linear polarization fractions $(2–15\%)$ and large
rotation measures ($>10^{3}$) for Sgr $A^{*}$ and M87 [62, 63]. EHT has
associated linear polarization due to the strong magnetic field. Our analysis
shows that the non-minimal coupling of the electromagnetic field affects the
two polarization modes of the photon distinctly. Thus, our analysis suggests
that some percentage of the linear polarization seen in EHT observations can
be due to NMC. Note that the synchrotron emission can contribute to the
polarized image, which can complicate the proposed measurements [64]. Next, we
plan to do a detailed numerical analysis to investigate the polarization
effects of NMC and possible constraints from future VLBI projects such as
ngEHT and Event Horizon Explorer.

Since NMC introduces direct interaction between curvature and electromagnetic
field, our analysis is independent of the accretion flow. Hence, we have
considered radially free-falling optically thin light-emitting accretion flow
mode. However, to establish the robustness of the NMC signatures on the black
hole image, it is imperative to extend the analysis for other types of
accretion models, like Keplerian accretion disk, thin infinite accretion disk,
and Ion torus [65, 66]. This is currently under investigation.

_Acknowledgments:_ The authors thank M. Johnson, V. Marthi, K. Moriyama, D.
Palumbo, and K. Sarkar for insights regarding the current and future VLBI
observations. The authors thank A. Choudhury, K. Chandra, S. M. Chandran, A.
Kushwaha, K. Rajeev, and S. Xavier for comments. JPJ was funded by IITB-IoE
grant. The work is supported by the SERB-Core Research Grant.

## References

  * Will [2014] C. M. Will, Living Rev. Rel. 17, 4 (2014).
  * Berti  _et al._ [2015] E. Berti _et al._ , Class. Quant. Grav. 32, 243001 (2015).
  * Will [2018] C. M. Will,  _Theory and Experiment in Gravitational Physics_ (Cambridge University Press, 2018).
  * Abbott  _et al._ [2019] B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D 100, 104036 (2019), arXiv:1903.04467 [gr-qc] .
  * Abbott  _et al._ [2021a] R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D 103, 122002 (2021a), arXiv:2010.14529 [gr-qc] .
  * Abbott  _et al._ [2021b] R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), (2021b), arXiv:2112.06861 [gr-qc] .
  * Akiyama  _et al._ [2019] K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 875, L1 (2019), arXiv:1906.11238 [astro-ph.GA] .
  * Akiyama  _et al._ [2022a] K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 930, L12 (2022a).
  * Akiyama  _et al._ [2022b] K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 930, L17 (2022b).
  * Misner  _et al._ [2017] C. W. Misner, K. S. Thorne, and J. A. Wheeler,  _Gravitation_ (Princeton University Press, 2017).
  * Schiff [1960] L. I. Schiff, Am. J. of Phys. 28, 340 (1960).
  * Dicke [2019] R. H. Dicke, Gen. Rel. and Grav. 51, 57 (2019).
  * Thorne  _et al._ [1973] K. S. Thorne, D. L. Lee, and A. P. Lightman, Phys. Rev. D 7, 3563 (1973).
  * Damour [2001] T. Damour, Compt. Rend. Acad. Sci. Ser. IV Phys. Astrophys. 2, 1249 (2001), arXiv:gr-qc/0109063 .
  * Damour and Schaefer [1991] T. Damour and G. Schaefer, Phys. Rev. Lett. 66, 2549 (1991).
  * Williams  _et al._ [1996] J. G. Williams, X. X. Newhall, and J. O. Dickey, Phys. Rev. D 53, 6730 (1996).
  * Baeßler  _et al._ [1999] S. Baeßler, B. R. Heckel, E. G. Adelberger, J. H. Gundlach, U. Schmidt, and H. E. Swanson, Phys. Rev. Lett. 83, 3585 (1999).
  * Barkovich  _et al._ [2001] M. Barkovich, H. Casini, J. C. D’Olivo, and R. Montemayor, Phys. Lett. B 506, 20 (2001).
  * Kesden and Kamionkowski [2006] M. Kesden and M. Kamionkowski, Phys. Rev. D 74, 083007 (2006), arXiv:astro-ph/0608095 .
  * Schlamminger  _et al._ [2008] S. Schlamminger, K. Y. Choi, T. A. Wagner, J. H. Gundlach, and E. G. Adelberger, Phys. Rev. Lett. 100, 041101 (2008), arXiv:0712.0607 [gr-qc] .
  * Keselman  _et al._ [2010] J. A. Keselman, A. Nusser, and P. J. E. Peebles, Phys. Rev. D 81 (2010), 10.1103/physrevd.81.063521.
  * Wagner  _et al._ [2012] T. A. Wagner, S. Schlamminger, J. H. Gundlach, and E. G. Adelberger, Class. Quant. Grav. 29, 184002 (2012), arXiv:1207.2442 [gr-qc] .
  * Overduin  _et al._ [2012] J. Overduin, F. Everitt, P. Worden, and J. Mester, Class. Quant. Grav. 29, 184012 (2012), arXiv:1401.4784 [gr-qc] .
  * Overduin  _et al._ [2014] J. Overduin, J. Mitcham, and Z. Warecki, Class. Quant. Grav. 31, 015001 (2014), arXiv:1307.1202 [gr-qc] .
  * Esmaili  _et al._ [2014] A. Esmaili, D. R. Gratieri, M. M. Guzzo, P. C. de Holanda, O. L. G. Peres, and G. A. Valdiviesso, Phys. Rev. D 89, 113003 (2014), arXiv:1404.3608 [hep-ph] .
  * Wang  _et al._ [2016] Z.-Y. Wang, R.-Y. Liu, and X.-Y. Wang, Phys. Rev. Lett. 116, 151101 (2016), arXiv:1602.06805 .
  * Touboul  _et al._ [2017] P. Touboul _et al._ , Phys. Rev. Lett. 119, 231101 (2017), arXiv:1712.01176 [astro-ph.IM] .
  * Mattingly [2005] D. Mattingly, Living Rev. Rel. 8, 5 (2005).
  * Liberati [2013] S. Liberati, Class. Quant. Grav. 30, 133001 (2013).
  * Kislat and Krawczynski [2015] F. Kislat and H. Krawczynski, Phys. Rev. D 92, 045016 (2015), arXiv:1505.02669 [astro-ph.HE] .
  * Uzan [2011] J.-P. Uzan, Living Rev. Rel. 14, 2 (2011).
  * Psaltis [2008] D. Psaltis, Living Rev. Rel. 11, 9 (2008).
  * Falcke  _et al._ [2000] H. Falcke, F. Melia, and E. Agol, AP. J. Letts 528, L13 (2000), arXiv:astro-ph/9912263 [astro-ph] .
  * Gralla  _et al._ [2019] S. E. Gralla, D. E. Holz, and R. M. Wald, Phys. Rev. D 100, 024018 (2019), arXiv:1906.00873 [astro-ph.HE] .
  * Prasanna [1971] A. Prasanna, Physics Letters A 37, 331 (1971).
  * Deser and van Nieuwenhuizen [1974] S. Deser and P. van Nieuwenhuizen, Phys. Rev. D 10, 401 (1974).
  * Horndeski [1976] G. W. Horndeski, J. Math. Phys. 17, 1980 (1976).
  * Drummond and Hathrell [1980] I. T. Drummond and S. J. Hathrell, Phys. Rev. D 22, 343 (1980).
  * Feynman [2002] R. P. Feynman,  _Feynman lectures on gravitation_, edited by F. B. Morinigo and et al (CRC Press, Baton Rouge, 2002).
  * Shankaranarayanan and Johnson [2022] S. Shankaranarayanan and J. P. Johnson, Gen. Rel. Grav. 54, 44 (2022), arXiv:2204.06533 [gr-qc] .
  * Hellings and Nordtvedt [1973] R. W. Hellings and K. Nordtvedt, Phys. Rev. D 7, 3593 (1973).
  * Kostelecky and Mewes [2002] V. A. Kostelecky and M. Mewes, Phys. Rev. D 66, 056005 (2002), arXiv:hep-ph/0205211 .
  * Vagnozzi  _et al._ [2023] S. Vagnozzi _et al._ , Class. Quant. Grav. 40, 165007 (2023), arXiv:2205.07787 [gr-qc] .
  * Kurczynski and Others [2022] P. Kurczynski and Others, in  _Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave_, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, edited by L. E. Coyle, S. Matsuura, and M. D. Perrin (2022) p. 121800M.
  * Broderick  _et al._ [2020] A. E. Broderick, D. W. Pesce, P. Tiede, H.-Y. Pu, and R. Gold, Astrophys. J. 898, 9 (2020), arXiv:2208.09003 .
  * Johnson  _et al._ [2020] M. D. Johnson _et al._ , Sci. Adv. 6, eaaz1310 (2020), arXiv:1907.04329 [astro-ph.IM] .
  * Hadar  _et al._ [2021] S. Hadar, M. D. Johnson, A. Lupsasca, and G. N. Wong, Phys. Rev. D 103, 104038 (2021), arXiv:2010.03683 .
  * Roelofs  _et al._ [2023] F. Roelofs _et al._ , Galaxies 11, 12 (2023), arXiv:2212.11355 [astro-ph.IM] .
  * Prasanna and Mohanty [2003] A. R. Prasanna and S. Mohanty, Class. Quant. Grav. 20, 3023 (2003), arXiv:gr-qc/0306021 .
  * Chandrasekhar [1983] S. Chandrasekhar,  _The Mathematical Theory of Black Holes_, International series of monographs on physics (Clarendon Press, 1983).
  * Bardeen  _et al._ [1972] J. M. Bardeen, W. H. Press, and S. A. Teukolsky, Astrophys. J. 178, 347 (1972).
  * Bambi [2013] C. Bambi, Phys. Rev. D 87, 107501 (2013), arXiv:1304.5691 [gr-qc] .
  * Johnson  _et al._ [2023] J. P. Johnson, S. Jana, and S. Shankaranarayanan, “A test of Einstein’s equivalence principle in ngEHT observations: Supplementary Material,” (2023).
  * Novikov and Thorne [1973] I. D. Novikov and K. S. Thorne, in  _Black Holes (Les Astres Occlus)_ (1973) pp. 343–450.
  * Fragione and Loeb [2020] G. Fragione and A. Loeb, Astrophys. J. Lett. 901, L32 (2020), arXiv:2008.11734 [astro-ph.GA] .
  * Fragione and Loeb [2022] G. Fragione and A. Loeb, Astrophys. J. Lett. 932, L17 (2022), arXiv:2205.12274 [astro-ph.GA] .
  * Visser [2007] M. Visser, in  _Kerr Fest: Black Holes in Astrophysics, General Relativity and Quantum Gravity_ (2007) arXiv:0706.0622 [gr-qc] .
  * Saijo  _et al._ [1998] M. Saijo, K.-i. Maeda, M. Shibata, and Y. Mino, Phys. Rev. D 58, 064005 (1998).
  * Doeleman  _et al._ [2019] S. Doeleman _et al._ , in  _Bulletin of the American Astronomical Society_, Vol. 51 (2019) p. 256, arXiv:1909.01411 .
  * Raymond  _et al._ [2021] A. W. Raymond _et al._ , Ap. J. Supplement 253, 5 (2021), arXiv:2102.05482 [astro-ph.IM] .
  * Amorim  _et al._ [2019] A. Amorim _et al._ (GRAVITY), Phys. Rev. Lett. 122, 101102 (2019), arXiv:1902.04193 [astro-ph.GA] .
  * Goddi  _et al._ [2021] C. Goddi _et al._ (ALMA), Astrophys. J. Lett. 910, L14 (2021), arXiv:2105.02272 [astro-ph.GA] .
  * Akiyama  _et al._ [2021a] K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 910, L12 (2021a), arXiv:2105.01169 [astro-ph.HE] .
  * Akiyama  _et al._ [2021b] K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. 912, 35 (2021b), arXiv:2105.01804 [astro-ph.HE] .
  * Beckwith and Done [2005] K. Beckwith and C. Done, Mon. Not. Roy. Astron. Soc. 359, 1217 (2005), arXiv:astro-ph/0411339 .
  * Vincent  _et al._ [2011] F. H. Vincent, T. Paumard, E. Gourgoulhon, and G. Perrin, Class. Quant. Grav. 28, 225011 (2011).

## Supplementary material

This document contains details and plots related to the calculations presented
in the main text.

###  .1 Photon circular orbit in the Schwarzschild space-time

The two conditions for the circular orbit of the photon ($\dot{r}=0$,
$\ddot{r}=0$) leads to the following equations:

$\displaystyle{E}^{2}\,\left[1-\frac{{b}^{2}}{{r}^{2}}\left(1-\frac{1}{{r}}\right)\,\left(1-\,\mathscr{C}_{\pm}\right)\right]$
$\displaystyle=0,$ (20)
$\displaystyle\dfrac{b^{2}E^{2}}{2r^{4}}\left[(2r-3)(1-\mathscr{C}_{\pm})+r(r-1)\dfrac{\partial\mathscr{C}_{\pm}}{\partial
r}\right]$ $\displaystyle=0.$ (21)

To find the analytical solutions for the radius of the photon circular orbit
and the corresponding impact parameter, one can use the perturbative approach.
We expand the coordinate ${r}$, and the parameters ${E}$ and ${b}$ in the
powers of the coupling constant ${\lambda}$ given by

${r}({s},{\lambda})=\sum_{i=0}^{n}{r}_{i}({s}){\lambda}^{i},\quad{b}({s},{\lambda})=\sum_{i=0}^{n}{b}_{i}({s}){\lambda}^{i},\quad{E}({s},{\lambda})=\sum_{i=0}^{n}{E}_{i}({s}){\lambda}^{i}.$
(22)

We can then solve the conditions for the circular orbits order by order to
obtain the radius of photon sphere and the corresponding impact parameters as
a power series expansion given by

${r}_{c}^{\pm}({\lambda})=\sum_{i=0}^{n}{r}^{\pm}_{i}\lambda^{i},\quad{b}_{c}^{\pm}({\lambda})=\sum_{i=0}^{n}{b}^{\pm}_{i}\lambda^{i}.$
(23)

For $n=4$, the expansion is given below. For ${}^{\prime}+^{\prime}$ mode,

$\displaystyle
r_{c}^{+}(\lambda)=\dfrac{3}{2}+\dfrac{4}{3}\lambda+\dfrac{64}{81}\lambda^{2}-\lambda^{3}\dfrac{2816}{729}-\dfrac{102400}{59049}\lambda^{4},$
(24) $\displaystyle
b_{c}^{+}(\lambda)=\dfrac{3\sqrt{3}}{2}+\dfrac{4}{\sqrt{3}}\lambda-\dfrac{64}{27\sqrt{3}}\lambda^{2}-\dfrac{1280}{729\sqrt{3}}\lambda^{3}+\dfrac{167936}{19683\sqrt{3}}\lambda^{4}.$

Similarly, for ${}^{\prime}-^{\prime}$ mode

$\displaystyle
r_{c}^{-}(\lambda)=\dfrac{3}{2}-\dfrac{4}{3}\lambda-\dfrac{64}{81}\lambda^{2}-\lambda^{3}\dfrac{256}{729}-\dfrac{45056}{59049}\lambda^{4},$
(25) $\displaystyle
b_{c}^{-}(\lambda)=\dfrac{3\sqrt{3}}{2}-\dfrac{4}{\sqrt{3}}\lambda-\dfrac{128}{27\sqrt{3}}\lambda^{2}-\dfrac{6400}{729\sqrt{3}}\lambda^{3}+\dfrac{389120}{19683\sqrt{3}}\lambda^{4}.$

Deviation of the analytical solution obtained using the perturbative approach
from the exact numerical solution is given in 4. For $|\lambda|<0.2$, we see
that the above perturbative expansion up to $n=4$ is consistent with the exact
numerical solution. The deviation is of the order of $\sim 10^{-2}$. For
higher values of $\lambda$, perturbative expansion matches with the numerical
results for higher values of $n$.

Figure 4: Deviation of the perturbed analytical solution from the exact
numerical solution for $r_{c}$ and $b_{c}$ for ${}^{\prime}\pm^{\prime}$
modes.

Figure 5:  $\tilde{\lambda}=-0.4$(top)$,\tilde{\lambda}=0.2$(bottom). Left:
$\tilde{\lambda}=0$, Center: ‘$+$’ mode, Right: ‘$-$’ mode.

###  .2 Observed image of accretion flow

For the radially infalling accretion flow described by eq. 10 in the
manuscript, the image of the black hole appears as given in 5. Similarly, for
a Novikov-Thorn accretion disk [54] with an inclination angle of $60^{o}$, the
images for different values of coupling constant are given in 6. Here the
light emitting particles forms a very thin disk and follows near-Keplerian
orbits till the inner most stable circular orbit, followed by plunging into
the black holes.

Here we see that for the radially infalling accretion flow, the shadow of the
black hole is bordered by the photon ring, which is not the case for the
Novikov-Thorn accretion disk with the inner radius $r_{i}=r_{ISCO}$

Note that for the Novikov-Thorne accretion disk in the equatorial plane, we
need to run the ray tracing code with a higher resolution to get the complete
image without any gaps. Images given here represents the observed shape of the
accretion disk. They do not contain any information regarding the observed
flux.

Figure 6:  $\tilde{\lambda}=-0.4999$(top)$,\tilde{\lambda}=0.2499$(bottom).
Left: $\tilde{\lambda}=0$, Center: ‘$+$’ mode, Right: ‘$-$’ mode.

###  .3 Photon circular orbit in the equatorial plane of a Kerr black hole

Radius of photon circular orbit and the corresponding impact parameter ${b}$
can be obtained by imposing the independent conditions $\dot{{r}}=0$ and
$\ddot{{r}}=0$. These equations can be solved numerically to obtain the radius
of the circular photon orbit and corresponding impact parameter. However, as
in the case of Schwarzschild black hole, here we use the analytical
perturbative approach to obtain the radius of the circular photon orbit and
the corresponding impact parameter as the function of the coupling constant
${\lambda}$ and the spin ${a}$ as given below. Here each coefficient of
${\lambda}$ is expanded in terms of ${a}$. Depending on the value of
${\lambda}$ and ${a}$ in consideration, one can use the expansion up to an
appropriate order $n$.

$\displaystyle{r}_{c}^{\pm}({a},{\lambda})=\sum_{i,j=0}^{n}{r}^{\pm}_{ij}{a}^{i}\lambda^{j},\quad{b}_{c}^{\pm}({a},{\lambda})=\sum_{i,j=0}^{n}{b}^{\pm}_{ij}{a}^{i}\lambda^{j}.$
(26)

The coefficients ${r}^{\pm}_{ij}$ and ${b}^{\pm}_{ij}$ for $n=3$ are the given
by Eqs.(28,29,30,31) .

Radius of the circular orbit of the non-minimally coupled photon in the
equatorial plane of a Kerr black hole and the corresponding impact parameter
can be expanded as

$\displaystyle{r}_{c}^{\pm}({a},{\lambda})=\sum_{i,j=0}^{4}{r}^{\pm}_{ij}{\lambda}^{i}{a}^{j},\quad{b}_{c}^{\pm}({a},{\lambda})=\sum_{i,j=0}^{4}{b}^{\pm}_{ij}{\lambda}^{i}{a}^{j}.$
(27)

For ${}^{\prime}+^{\prime}$ mode, the coefficients are given by

$\displaystyle r^{+}_{00}=\dfrac{3}{2},\quad
r^{+}_{01}=-\dfrac{2}{\sqrt{3}},\quad r^{+}_{02}=-\dfrac{4}{9},\quad
r^{+}_{03}=-\dfrac{20}{27\sqrt{3}},$ $\displaystyle
r^{+}_{10}=\dfrac{4}{3},\quad r^{+}_{11}=-\dfrac{32}{9\sqrt{3}},\quad
r^{+}_{12}=-\dfrac{16}{9},\quad r^{+}_{13}=-\dfrac{256}{81\sqrt{3}},$
$\displaystyle r^{+}_{20}=\dfrac{64}{81},\quad
r^{+}_{21}=\dfrac{1280}{81\sqrt{3}},\quad r^{+}_{22}=-\dfrac{8704}{2187},\quad
r^{+}_{23}=-\dfrac{248576}{6561\sqrt{3}},$ (28)
$\displaystyle{r^{+}_{30}=-\dfrac{2816}{729},\quad
r^{+}_{31}=\dfrac{57344}{6561\sqrt{3}},\quad
r^{+}_{32}=-\dfrac{7737344}{59049},\quad
r^{+}_{33}=-\dfrac{16310272}{59049\sqrt{3}}}.$ $\displaystyle
b^{+}_{00}=\dfrac{3\sqrt{3}}{2},\quad b^{+}_{01}=-2,\quad
b^{+}_{02}=-\dfrac{1}{\sqrt{3}},\quad b^{+}_{03}=-\dfrac{16}{27},$ (29)
$\displaystyle b^{+}_{10}=\dfrac{4}{\sqrt{3}},\quad
b^{+}_{11}=-\dfrac{8}{9\sqrt{3}},\quad b^{+}_{12}=-\dfrac{256}{243},\quad
b^{+}_{13}=-\dfrac{88}{27\sqrt{3}},$ $\displaystyle
b^{+}_{20}=-\dfrac{64}{27\sqrt{3}},\quad b^{+}_{21}=\dfrac{512}{81},\quad
b^{+}_{22}=\dfrac{3520}{243\sqrt{3}},\quad
b^{+}_{23}=\dfrac{5120}{729\sqrt{3}},$
$\displaystyle{b^{+}_{30}=-\dfrac{1280}{729\sqrt{3}},\quad
b^{+}_{31}=-\dfrac{40960}{2187},\quad
b^{+}_{32}=\dfrac{28672}{729\sqrt{3}},\quad
b^{+}_{33}=\dfrac{19382272}{177147}}.$

For ${}^{\prime}-^{\prime}$ mode

$\displaystyle r^{-}_{00}=\dfrac{3}{2},\quad
r^{-}_{01}=-\dfrac{2}{\sqrt{3}},\quad r^{-}_{02}=-\dfrac{4}{9},\quad
r^{-}_{03}=-\dfrac{20}{27\sqrt{3}},$ $\displaystyle
r^{-}_{10}=-\dfrac{4}{3},\quad r^{-}_{11}=\dfrac{32}{9\sqrt{3}},\quad
r^{-}_{12}=\dfrac{16}{9},\quad r^{-}_{13}=\dfrac{256}{81\sqrt{3}},$
$\displaystyle r^{-}_{20}=-\dfrac{64}{81},\quad
r^{-}_{21}=\dfrac{1024}{81\sqrt{3}},\quad
r^{-}_{22}=-\dfrac{11264}{2187},\quad
r^{-}_{23}=-\dfrac{239872}{6561\sqrt{3}},$ (30)
$\displaystyle{r^{-}_{30}=-\dfrac{256}{729},\quad
r^{-}_{31}=\dfrac{139264}{6561\sqrt{3}},\quad
r^{-}_{32}=-\dfrac{4911104}{59049},\quad
r^{-}_{33}=-\dfrac{9699328}{59049\sqrt{3}}}.$ $\displaystyle
b^{-}_{00}=\dfrac{3\sqrt{3}}{2},\quad b^{-}_{01}=-2,\quad
b^{-}_{02}=-\dfrac{1}{\sqrt{3}},\quad b^{-}_{03}=-\dfrac{16}{27},$ (31)
$\displaystyle b^{-}_{10}=-\dfrac{4}{\sqrt{3}},\quad b^{-}_{11}=0,\quad
b^{-}_{12}=\dfrac{8}{9\sqrt{3}},\quad b^{-}_{13}=\dfrac{256}{243},$
$\displaystyle b^{-}_{20}=-\dfrac{128}{27\sqrt{3}},\quad
b^{-}_{21}=\dfrac{256}{81},\quad b^{-}_{22}=\dfrac{1088}{243\sqrt{3}},\quad
b^{-}_{23}=-\dfrac{2048}{729},$
$\displaystyle{b^{-}_{30}=-\dfrac{6400}{729\sqrt{3}},\quad
b^{-}_{31}=\dfrac{28672}{2187},\quad
b^{-}_{32}=-\dfrac{7168}{243\sqrt{3}},\quad
b^{-}_{33}=\dfrac{13828096}{177147}}.$

###  .4 Observed shadow of the Kerr black hole

7 shows the outline of the observed shadow of the Kerr black hole for non-
minimally coupled modes and minimally coupled mode for different values of
spin ${a}$.

Figure 7: Outline of the shadow of a slowly rotating black hole observed by a
distant LNRO for different $\pm$ photon modes for near limiting values of
${\lambda}$ along with the minimally coupled case for ${a}=0.1$ (top) and
${a}=0.15$ (bottom).


# Person Re-identification Method Based on Color Attack and Joint Defence

Yunpeng Gong Liqing Huang Lifei Chen
College of Computer and Cyber Security, Fujian Normal University, P. R. China
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contributionCorresponding author

###### Abstract

The main challenges of ReID is the intra-class variations caused by color
deviation under different camera conditions. Simultaneously, we find that most
of the existing adversarial metric attacks are realized by interfering with
the color characteristics of the sample. Based on this observation, we first
propose a local transformation attack (LTA) based on color variation. It uses
more obvious color variation to randomly disturb the color of the retrieved
image, rather than adding random noise. Experiments show that the performance
of the proposed LTA method is better than the advanced attack methods.
Furthermore, considering that the contour feature is the main factor of the
robustness of adversarial training, and the color feature will directly affect
the success rate of attack. Therefore, we further propose joint adversarial
defense (JAD) method, which include proactive defense and passive defense.
Proactive defense fuse multi-modality images to enhance the contour feature
and color feature, and considers local homomorphic transformation to solve the
over-fitting problem. Passive defense exploits the invariance of contour
feature during image scaling to mitigate the adversarial disturbance on
contour feature. Finally, a series of experimental results show that the
proposed joint adversarial defense method is more competitive than a state-of-
the-art methods.

##  1 Introduction

Person re-identification (ReID) is matching the same person across diferent
cameras and scenes[1, 2, 3, 4]. This technology have been widely applied to
video surveillance[5, 6, 7], image retrieval[8, 9], criminal investigation[8],
target tracking[10] and others. ReID has been a challenging and hot problem
since illumination, complex environment, occlusion, image blur and other
factors. In recent years, many ReID works[5, 6, 7, 10, 8, 9] used deep-
learning module, and have made great progress. However, Szegedy et al.[11]
found the deep-learning models are susceptible to attacks from adversarial
samples, which will cause the network to completely change its prediction
results. The works of [12, 13, 14, 15, 16] have proved that the ReID systems
based on deep-learning have the same vulnerability. And these adversarial
samples have only added a slight disturbance, which is hidden enough for the
human visual system. It is very important to study the security of ReID
systems because the insecurity may cause severe losses, eg., criminals may use
adversarial disturbance to cheat the monitoring systems [14, 16].

Figure 1: (a) shows the retrieval results of clean example. (b) corresponds to
Meric-IFGSM attack [12], (c) corresponds to the SMA attack [13], (d)
corresponds to the proposed LTA attack. The numbers on the images indicate the
rank of similarity in the retrieval results, the red and green number denote
the wrong and correct results, respectively. Figure 2: (a) shows clean example
and (b) (c) (d) shows the adversarial noise generated after attacking three
different models using SMA [13], respectively. (b) corresponds to the normally
trained model, (c) corresponds to the model which using [17] to train with
better robustness to color variations, (d) corresponds to our proactive
defense model, (e) corresponds to our joint adversarial defense.

The adversarial metric attack usually requires additional push or pull
guidance to distort the distance between the attacked image and other images
with the same identity or class [12, 13, 14, 18, 19, 16, 20, 21, 22, 23], so
as to achieve the purpose of deception models. Therefore, most of the existing
studies [12, 13, 18, 23] of adversarial metric attack and defense revolve
around metric relationships. Generally, the optimization function of the
adversarial noise is designed for pulling the distance between negative pairs
and pushing the distance between positive pairs.

The intra-class variations caused by color deviations such as lighting,
chromatic aberration, etc., in the various camera conditions is one of the
main challenge for ReID [24, 25, 26], due to the training set solely
encompasses a limited portion of the intra-class variations of the color
domain, the model is easy to overfit. In the existing adversarial metric
attacks [12, 13, 14], it was observed that the attacks naturally perturb the
color feature of the samples, which consistent with the color feature is
important clue for image retrieval. The effect of the adversarial attack on
the color feature is visualized in Figure $\color[rgb]{1,0,0}1$. When
retrieving clean samples, the model was able to identify the retrieved
pedestrians with blue-gray striped top, grayish trousers, and backpack, even
though they looked a little off from different cameras. While under
adversarial perturbation, as shown in $\color[rgb]{1,0,0}(b)$,
$\color[rgb]{1,0,0}(c)$, $\color[rgb]{1,0,0}(d)$, the misjudgment of the model
in color becomes more and more serious, which include the colors of pedestrian
top, trousers and backpack. Two classic metric attacks include metric-IFGSM
[12] and SMA [13] attack. The metric-IFGSM [12] attack was realized by
maximizing the metric distance between the retreved image and the other images
with the same identity, and used reference images. The SMA [13] attack added
random noise to the retrieved image, and maximizing the metric distance from
the clean image. The SMA not require reference images and thus is more with
realistic scenario. Compared with metric-IFGSM, SMA only used the retrived
image to generate disturbance, so it performs better in making the model
misjudge color. Therefore, we further propose loacal transformation attack
(LTA), which does not add random noise, but use local gray transformation with
more obvious color variation to randomly disturb the color of the retrieved
image, so as to learn robust adversarial noise against the color variation,
and to strengthen the attack on color feature. Finally, experiments verify
that the performance of the proposed LTA method is better than the advanced
methods.

After exploring the vulnerability of ReID attack, we begin to research the
effectiveness of defense methods. Adversarial training is currently the main
adversarial metric defense method [12, 13, 18, 23]. Generally speaking, in
adversarial training, a defense model trained by adversarial examples of an
attack cannot defend against multiple attacks at the same time [27], and
extreme overfitting during training leads to obvious reduction in model
generalization capacity [28].

Considering that being better at capturing shape or contour features is the
main factor for the robustness of adversarial training [29], and color
features have a direct impact on the success rate of attacks. So there
speculate that color features and contour feature are inherently important
targets for attacker. To this end, we propose a corresponding joint
adversarial defense approach. Firstly, we consider increasing the robustness
of the model to color variations as a proactive defense. We speculate that
when the robustness of the model to color variations is increased, the
adversary will change the attack direction and strengthen the attack on the
contour feature. It can be seen from Figure $\color[rgb]{1,0,0}2(c)$ that the
contour feature have been more seriously damaged. In addition, we also fuse
sketch images during the model training process to strengthen the learning of
contour feature so as to enhance the defense against the two attack modes
(color and contour). From Figure $\color[rgb]{1,0,0}2(d)$, it can be seen that
the adversarial noise is significantly weakened on our proactive defense
model. And then, we further propose a passive defense strategy, which
utilizing the invariance of contour features in the circuitous scaling to
mitigate attack on contour feature. This strategically complementary to the
proactive defense. From Figure $\color[rgb]{1,0,0}2(e)$, it can be seen that
after implementing the joint adversarial defense proposed in this paper, the
adversarial noise becomes very sparse, and the contour feature are also well
protected. The proposed joint defense model is a lightweight method without
any additional parameter learning. It can be combined with various ReID models
without changing the learning strategy. Therefore, the main contributions of
this paper are summarized as follows:

$\bullet$ We propose a new attack method – local transformation attack (LTA)
for the first time, by using more obvious color variation to randomly disturb
the color of the retrieved image and without reference image.

$\bullet$ We propose a joint adversarial defense model based on feature-
invariant is to against adversarial metric attacks, which does not rely on
adversarial training. The proposed method improves the robustness of the
model, and performs well in cross-domain tests.

$\bullet$ Finally, the comparative experimental results with the state-of-the-
art algorithms further verified the effectiveness and the advanced nature of
the proposed method.

##  2 Related Work

In this section, the previous work on adversarial attacks and defenses of the
metric learning is described.

###  2.1 Adversarial Attacks

Adversarial attacks can be categorized into white-box [30, 12] and black-box
[31, 14] attacks. The black-box attack means that the attacker does not know
the structure and parameters of the target network, and the adversaries can
only resort to the query access to generate adversarial samples. White-box
attack assumes that the attacker has prior knowledge of the target networks,
including the structure and parameters of model, which means that the
adversarial examples are generated with and tested on the targe network. For
the same attack, the success rate of white-box attack is higher than black-box
attacks.

There are some metric attack methods proposed in ReID. Metric-FGSM [12]
extended some metric attacks by classification attacks, including fast
gradient sign method (FGSM) [32], iterative FGSM (IFGSM) and momentum IFGSM
(MIFGSM) [33]. Among the three attack methods, IFGSM delivers the strongest
white-box attacks [12]. Opposite-direction feature attack (ODFA) [15] exploits
feature-level adversarial gradients to generate adversarial examples to pull
the feature in the opposite direction with an artificial guide. Self metric
attack (SMA) [13] uses the image with added noise as the reference image and
obtains the adversarial examples by attacking the feature distance between the
original image and the reference image. This process does not require any
additional images, it is more in line with the actual situation that the
attacker usually lacks data. Furthest-negative attack (FNA) [13] combine hard
sample mining [34, 35] and triple loss to obtain pushing guides and pulling
guides to move image feature to head towards the least similar cluster of
features while moving away from the other similar features. Deep mis-ranking
(DMR) [14] proposed a learning-to-mis-rank formulation to perturb the ranking
of the system output, which used a multi-stage network architecture that
pyramids the features of different levels to extract general and transferable
features for the adversarial perturbations. The success attack rate of black-
box attacks is almost as high as that of white-box attack. At the same time,
it also showed that when applied to classification attacks, it has a higher
success attack rate than DeepFool [36], NewtonFool [37], and CW [38], and it
has successfully broken through many classical ReID models[39, 40, 41, 42, 43,
44, 45, 46, 47, 25].

###  2.2 Adversarial Defenses

Recently, a number of effective defense methods have been employed to against
adversarial classification attacks [48, 49, 50, 51, 52, 53, 54], such as
denoising methods, randomization-based schemes, adversarial training and
others. The defense methods based on denoising, such as Guo et al. [48] used
more diversified non-differentiable image transformation operations, which
include depth reduction, total variance minimization and image quilting. The
goal is to increase the difficulty of network gradient prediction, and then
achieve the purpose of defense. Noting that most of the training images are in
JPG format, Dziugaite [49] used JPG image compression method to reduce the
impact of adversarial disturbance. In terms of randomization, RRP (random
resizing and padding) [50] mitigates adversarial effects by combining random
resizing and random padding based on adversarial training. [51, 52, 53, 54]
showed that adversarial training is a robust way to defend against adversarial
attacks, which includes offline adversarial training and online adversarial
training.

The metric defense schemes employ by [12, 13, 23, 18] correspond to offline
adversarial training and online adversarial training respectively. The defense
method uses in [12] is offline adversarial training, which is based on a
generation of an adversarial version of the training set obtained with a
frozen version of the trained model. As a frozen model is used to generate
attacks, this method is referred to as offline adversarial training. The
defense method uses in [13, 23, 18] is online adversarial training, which
generates adversarial examples online while the defended model evolves by
triplet loss. However, adversarial training is prone to overfitting [28, 13]
because dependent on the training data results in reducing the generalization
capacity of the model. Enhancing the robustness towards adversarial examples
and maintaining the generalization capacity of the model is the important
issue of adversarial defense.

##  3 Proposed Methods

In this section, we propose the local transformation attack (LTA) based on
color features. In order to push the feature of the reference image away from
the original image, there constructs a reference image with local difference
from the original image in each basic iteration based on LGT [17]. As for the
proposed attack method (LTA), we further propose a joint defense method. In
proactive defense, we combine the three modal images of visible (RGB),
grayscale and sketch for random channel fusion. In passive defense, it
realizes by circuitous scaling of image. The specific attack and joint defende
method framework is showed in Figure $\color[rgb]{1,0,0}3$.

Figure 3: Framework diagram of our attack and joint adversarial defence. In
our Local Transformation Attack (LTA), It pushes the feature of the reference
image away from the original image by constructing a reference image with
local difference from the original image in each basic iteration based on LGT
[17]. In proactive defense, we combines the three modal images of visible
(RGB), grayscale and sketch for random channel fusion. In our passive defense,
it realizes by circuitous scaling of image.

###  3.1 Proposed Local Transformation Attack

In order to attack the color feature, we propose the local transformation
attack (LTA), which adopts local grayscale transformation (LGT) [17]
constructing the local color deviation of the input. And then, it randomly
selects a rectangular area in the image and replaces it with the pixels of the
same rectangular area in the corresponding grayscale image. As showed in
Figure $\color[rgb]{1,0,0}3$, the LGT makes the constructed reference image
have appropriate local differences from the original image.

The initialization of the proposed LTA method is defined as:

$x_{adv}^{(0)}=x$ (1)

where $x$ denote the attacked image. There using $x_{adv}^{(n)}$ denotes the
adversarial example at the $n$-th iteration. $\hat{x}^{(n)}$ is the reference
image with local variability constructed by LGT[17] at the $n$-th basic
iteration. So the proposed LTA is defined as the following iterative
optimization:

$\hat{x}^{(n)}=LGT(x)$ (2)
$x_{adv}^{(n+1)}=\Psi_{x}^{\varepsilon}(x_{adv}^{(n)}+\alpha\cdot
sign(grad^{(n+1)}))$ (3)

where $\epsilon$ is the adversarial bound and $\alpha$ is the iteration step
size, $\Psi_{x}^{\varepsilon}$ is the clip function, which ensures that
$\|x_{adv}^{(n+1)}-x\|_{\infty}<\epsilon$ and that adversarial noise
inconspicuousness. And the $grad^{(n)}$ is the accumulated gradient at the
$n$-th iteration:

$grad^{(n+1)}=\theta\cdot
grad^{(n)}+\frac{\Delta_{LTA}^{(n)}}{\|\Delta_{LTA}^{(n)}\|_{1}}$ (4)

where $\theta$ is the decay factor of the momentum term, in our experiments
$\theta$ is set to 1. And $\Delta_{LTA}^{(n)}$ is calculated as follows:

$\Delta_{LTA}^{(n)}=\frac{\partial D(f_{adv}^{(n)},\hat{f}^{(n)})}{\partial
f_{adv}^{(n)}}$ (5)
$D(f_{adv}^{(n)},\hat{f}^{(n)})=\|f_{adv}^{(n)}-\hat{f}^{(n)}\|_{2}^{2}$ (6)

$f_{adv}^{(n)}$ denotes the feature of the adversarial example. Specifically,
each iteration optimizes adversarial noise by attacking the feature distance
between the adversarial image generated from the result of previous iteration
and the new reference image.

###  3.2 Proposed Joint Adversarial Defence Method

In order to overcome the attack based on color features, we further propose
the joint adversarial defense method (JAD). The proposed method includes the
proactive and the passive defense. The proactive defense consists of channel
fusion (CF) and local homogeneous transformation (LHT), and the passive
defense consists of circuitous scaling (CS).

####  3.2.1 Proposed Proactive Defence

The proactive defense consists of channel fusion (CF) and local homogeneous
transformation (LHT).

Channel fusion (CF). Visible images, grayscale images, and sketch images are
homogeneous, which contain the same structural information. The results in
[55, 17] showed that using the homogeneous grayscale images to learn
structural information in training is effective in increasing the robustness
to color variations.

We add grayscale information and sketch information by channel fusing. The
operation (Grayscale(3)) in Pytorch is adopted to get the grayscale image for
each visible image, and sketch images can be obtained by inverting the
grayscale image and then Gaussian blurring it, finally blending it with the
grayscale image. As shown in Figure $\color[rgb]{1,0,0}3$, the RGB images are
randomly converted with a certain probability into 3-channel grayscale images
or sketch images in pre-processing stage, and then randomly merge the channels
of the grayscale image and the sketch image with the channels of the RGB image
to create a new homogeneous modal image.

In the process of CF, 1 or 2 channels are randomly selected from the R, G, and
B channels of the visible image. After the visible image channel and the
number of channels n is determined, the grayscale or sketch image channel is
randomly selected to reconstruct a new 3-channel image. In fact, a maximum of
60 homogeneous variations can be generated by combining the 5 image channels
types of R, G, B, grayscale, and sketch in random order.

In addition, from Figure $\color[rgb]{1,0,0}4a$, we can see that the
augmentation based on image transformation will further extend the diversity
of modes, such as Posterize, Equalize, Solarize, Contrast, Inversion and so
on. However, the multi-modal inputs will lead to overfitting of the model,
which affects the generalization capacity of the model as showed in Figure
$\color[rgb]{1,0,0}5a$.

Local homogeneous transformation (LHT). To solve this problem, we propose a
strategy method based on local homogeneous transformation (LHT), which extends
local grayscale transformation (LGT)[17]. Using LHT to guide the model to fit
the diversity of variation gradually from local variations. The difference of
LGT is that the proposed LHT replaces randomly selected regions with
homogeneous images. As showed in Figure $\color[rgb]{1,0,0}5b$, it positively
helps reduce the overfitting in training. Unless otherwise specified, the
diversity data learning in subsequent experiments combine with LHT by default.

Figure 4: Hyper-parameter sensitivity analysis: (a) the contribution of each
component of the joint defense to the defense and the impact of different
ratios on defense performance; (b) the effect of different image scaling
ratios on defense performance in passive defense.

The LHT for each visible image $x^{v}$ can be achieved by the following
equation:

$x^{h}=T(x^{v})$ (7)

and

$rect=RandPosition(x^{v})$ (8)

and

$x^{LHT}=LT(x^{v},x^{h},rect)$ (9)

and

$(x^{LHT}|y)=(x^{v}|y)$ (10)

where the $x^{h}$ is the homogeneous images, and $T(\bullet)$ is the
homogeneous transformation funtion. $RandPosition(\bullet)$ is used to
generate a random rectangle in the image, and the function of $LT(\bullet)$ is
to give the pixels in the rectangle corresponding to the $x^{h}$ image to the
$x^{v}$ image. $x^{LHT}$ is the sample after local homogeneous transformation,
and $y$ is the label of the transformed image.

Figure 5: The comparison of training curve without LHT and with LHT.

From Figure $\color[rgb]{1,0,0}4a$, we can see that the best defensive
performance is achieved when the components are in the ratio of 5% to 15%.
Therefore, the probability of the image using augmentation transformation is
set to 5%, and the probability of converting to a grayscale image is set to
5%. In addition, the probability of using CF transformation is set to 5%, and
the probability of using LHT is set to 10%.

####  3.2.2 Proposed Passive Defence

Since color features and contour feature are two important targets in the
attack, the increased robustness of the model to color variations will force
the adversary to change the direction of the attack to some extent, more
towards attacking contour feature. Therefore, we exploit the invariance of
contour features during image scaling to mitigate the adversarial disturbance
on contour feature.

The basic principle of image scaling is to calculate the pixel value of the
target image according to the pixel value of the original image by certain
rules, common image scaling algorithms such as linear interpolation [56]. In
the scaling process, some pixels are discarded or some new pixels are added.
[50] found that the adversarial noise structure can be effectively destroyed
by one-time image scaling. Circuitous scaling (CS) consists of multiple image
scaling to give full play to this advantage.

The passive defense is realizes by a series of image resizing. The scaling of
an image does not bring more information about the image, so the quality of
the image will inevitably be affected, which also has an impact on the
retrieval accuracy. Therefore, it is important to find a suitable scaling
ratio to trade-off the retrieval accuracy and the adversarial robustness. The
effect of scaling ratio on accuracy can be seen in Figure
$\color[rgb]{1,0,0}4b$. Taking the Market1501[2] dataset as an example, the
original size of the dataset image is [128, 64], and the size is uniformly
resized to [256, 128] when fed into the CNN. When using image resizing as
passive defense, we observe that the network performance hardly drops and gets
a satisfactory defense effect if we resize the image to [110, 50]
(Approximately 0.8 times the original image) to corrupt the adversarial noise
structure. Our passive defense consists of a series of resizing that reszie
the image to [110, 50] then to [220, 100], then to [110, 50] again (finally
uniformly to the [256, 128]), so it called circuitous scaling. The effect of
image scaling[50] (only once resizing) and CS on the adversarial noise can be
seen in Figure $\color[rgb]{1,0,0}6$, and it can be observed that the
adversarial noise at the contour feature is continuously weakened, and the
outline of the pedestrian is more clearer.

Figure 6: The effect of image scaling and CS on the adversarial noise, where
(a) shows the adversarial noise from the original adversarial example, and (b)
shows the adversarial noise after reszie the adversarial sample to [110, 50]
(then restored to [256, 128]). (c) shows the adversarial noise after CS.

##  4 Experiments

In this section, we evaluate our LTA by comparing with SMA[13] and then
evaluate the robustness of our approach using cross-domain tests. Finally, we
verify the effectiveness of our JAD under white-box attacks and black-box
attack.

###  4.1 Attack Evaluation and Cross-Domain Tests

Datasets. Experiments are conducted on Market1501 [2] and DukeMTMC [45]. The
Market1501 includes 1,501 pedestrians captured by six cameras (five HD cameras
and one low-definition camera). The DukeMTMC is a large-scale multi-target,
multi-camera tracking dataset, a HD video dataset recorded by 8 synchronous
cameras, with more than 2,700 individual pedestrians. The above two datasets
are widely used in ReID studies.

Evaluation criteria. Following existing works [2], Rank-k precision and mean
Average Precision (mAP) are adapted as evaluation metrics. Rank-1 denotes the
average accuracy of the first return result corresponding to each query image.
mAP denotes the mean of average accuracy, the query results are sorted
according to the similarity, the closer the correct result is to the top of
the list, the higher the score.

Implementation details. The proposed adversarial attack and defense algorithm
is development based on on PyTorch framework. In our baseline, ResNet50 [57]
and DenseNet [40] are used as the backbone network in experiments, and the
pre-trained ImageNet parameters are adopted for network initialization.
Specifically, the stride of the last convolutional block is set to 2. We adopt
the stochastic gradient descent (SGD) optimizer for optimization, and the
momentum parameter is set to 0.9. We set the initial learning rate as 0.1. The
learning rate is decayed by 0.1 every 40 iteration, with a total of 60
training epochs and a batch size of 32 for normal training on both datasets,
and 120 training epochs for our method as well as for adversarial
training[12].

Attack evaluation. The hyper-parameters are unified for fair comparison, the
adversarial boundary is set to 5 pixels, the iteration step size is set to 1,
and the number of basic iterations is set to 15. Note that in contrast to an
adversarial defense problem, lower precision indicates better attack
performance. It can be seen from Table $\color[rgb]{1,0,0}1$ that when only
one version of the reference image is used, the success attack rate of LTA* is
better than that of SMA. The comparison between LTA* and LTA shows that using
diverse versions of reference images has a higher attack success rate than
using only one version of reference images. The experimental results fully
demonstrate that the attack against color features are more aggressive
compared to the same type of SMA attack.

Table 1: Evaluation on Market1501[2] under a white-box attack on the query set. Where LTA* means that only one version with local differences is used as a reference image, and LTA generates image versions with different local differences in each basic iteration to conduct the metric attack. Attack | Rank-1 | Rank-5 | Rank-10 | mAP
---|---|---|---|---
No-attack | 88.4% | 95.5% | 97.1% | 72.1%
SMA[13] | 15.7% | 26.4% | 32.7% | 11.1%
LTA*(ours) | 15.7% | 26.2% | 32.1% | 11.0%
LTA(ours) | 13.3% | 22.4% |  28.1% | 9.6%
Table 2: The performance of different models is evaluated on cross-domain dataset. M→D means that we train the model on Market1501[2] and evaluate it on DukeMTMC[45]. Model | M→D | D→M
---|---|---
Rank-1 | mAP | Rank-1 | mAP
Baseline | 36.1% | 18.9% | 45.7% | 19.6%
GOAT[13] | 23.6% | 11.4% | 47.3% | 18.5%
JAD(w/o LHT)(ours) | 36.9% | 18.4% | 47.4% | 19.5%
JAD(ours) | 42.5% | 21.5% | 47.5% | 19.4%
Table 3: The performance of normally trained models (Baseline) and our JAD models on Market1501 and DukeMTMC. Methods | Market1501[2] | DukeMTMC[45]
---|---|---
Rank-1 | mAP | Rank-1 | mAP
Baseline | 88.4% | 72.2% | 78.7% | 62.3%
Baseline+RK [58] | 90.2% | 84.7% | 83.3% | 79.3%
JAD(ours) | 88.7% | 70.3% | 77.2% | 57.8%
JAD+RK(ours) | 91.0% | 85.0% | 82.7% | 77.0%
Table 4: The performance of normally trained models and our JAD models under white-box attack of the query. Dataset | Model | Rank-1/mAP(%)
---|---|---
M-IFGSM[12] |  SMA[13] | LTA
Market1501[2] | Baseline | 8.1/4.3 | 15.7/11.1 | 13.3/9.6
Baseline+RK | 13.2/13.0 | 17.6/20.0 | 14.2/16.1
JAD(ours) | 47.1/27.8 | 79.3/60.1 | 56.5/41.1
JAD+RK(ours) | 61.3/56.8 | 85.6/80.3 | 66.2/63.9
DukeMTMC[45] | Baseline | 10.1/5.8 | 15.0/10.4 | 13.0/9.2
Baseline+RK | 16.8/16.3 | 18.8/19.8 | 15.2/16.3
JAD(ours) | 30.5/16.3 | 56.7/39.5 | 41.8/27.6
JAD+RK(ours) | 48.1/43.4 | 69.3/64.5 | 52.4/49.7

Cross-domain tests. It is pointed out by[12] that the higher accuracy of the
model does not mean that it has better generalization capacity. The defense
capabilities of different baselines under the same attack would have been
greatly different, and the high accuracy model may even have worse defenses
capabilities due to overfit. In response to the above potential problems, we
suggest to use cross-domain tests and adversarial defense tests to verify the
robustness of the model. Experiments show that the proposed method effectively
enhances the generalization capacity of the model, and the Table
$\color[rgb]{1,0,0}2$ shows the cross-domain experiments of the proposed
method between two datasets, Market-1501[2] and DukeMTMC[45]. We use the
state-of-the-art defense model GOAT[13] for comparison.

In the cross-domain tests of Market1501$\rightarrow$DukeMTMC, it can be seen
that JAD (without LHT) enhances the Rank-1 by 3.1 percentage points compared
with the baseline, and further enhances by 4.8 percentage points after using
LHT. In the cross-domain tests of DukeMTMC$\rightarrow$Market1501, it can be
seen that the proposed method (without LHT) enhances the Rank-1 by 3.4
percentage points compared with the baseline, and further enhances by 0.2
percentage points after using LHT. The above shows that the proposed method
effectively enhances the generalization capacity of the model, and LHT further
enhances the generalization capacity.

Table 5: Comparison of baseline, channel fusion (CF), proactive defence (PD), and joint adversarial defense (JAD) in terms of defense accuracy under white-box attack on Market1501[2]. Model (with RK) | Rank-1/mAP(%)
---|---
No-attack |  M-IFGSM[12] |  SMA[13] | LTA
Baseline | 90.2/84.7 | 13.2/13.0 | 17.6/20.0 | 14.2/16.1
CF(ours) | 90.8/85.3 | 18.3/16.7 | 18.0/19.6 | 17.1/18.5
PD(ours) | 91.5/85.6 | 31.7/28.7 | 58.4/57.1 | 25.2/27.6
JAD(ours) | 91.0/85.0 | 61.3/56.8 | 85.6/80.3 | 66.2/63.9
Table 6: Comparison of different resizing combinations in terms of defense accuracy, where $P_{1}$ means the scaling pattern that resize image to [110, 50] and $P_{2}$ means resize to [220, 100]. $P_{1}$→$P_{2}$ means resize image to [110, 50] then to [220, 100]. Dateset | Model | No-attack | LTA Attack
---|---|---|---
Rank-1/mAP(%) | Rank-1/mAP(%)
Market1501[2] | Baseline | 90.2/84.7 | 14.2/16.1
$P_{1}$ | 90.0/84.7 | 25.5/27.5
$P_{2}$ | 90.3/84.9 | 15.9/17.9
$P_{1}$→$P_{2}$ | 90.1/84.8 | 25.8/27.8
$P_{1}$→$P_{2}$→$P_{1}$ | 89.8/84.3 | 31.4/33.1
Table 7: Comparison of different defense methods in terms of defense accuracy under white-box attacks on Market1501. Model | Rank-1/mAP(%)
---|---
No-attack |  M-IFGSM[12] |  SMA[13] | LTA
Baseline | 90.2/84.7 | 13.2/13.0 | 17.6/20.0 | 14.2/16.1
AT[12] | 86.4/76.9 | 46.8/41.3 | 48.3/47.4 | 49.1/48.4
AT +CS | 86.3/76.1 | 60.6/53.8 | 64.9/61.1 | 62.7/58.5
JAD(ours) | 90.6/84.3 | 66.9/62.1 | 86.5/80.7 | 73.5/69.5
Table 8: Comparison of different defense methods and other baselines in terms of defense accuracy under DMR black-box attack. Model (w/o RK) | Rank-1/mAP(%) |  Model (with RK) | Rank-1/mAP(%)
---|---|---|---
No-attack |  DMR[14] | No-attack | DMR
Baseline | 88.4/72.2 | 19.8/15.8 |  SB[59] | 95.4/94.2 | 6.2/4.8
GOAT[13] | 87.5/66.9 | 67.8/46.4 | SB+JAD(ours) | 95.1/94.0 | 93.3/91.2
GOAT+CS | 88.0/68.3 | 72.8/50.7 |  FR[60] | 96.8/95.3 | 24.8/25.9
JAD(ours) | 88.7/70.3 | 81.1/60.7 | FR+JAD(ours) | 96.3/94.9 | 91.6/90.1

###  4.2 Experiments of JAD

This subsection verifies the effectiveness of the proposed method from white-
box[12, 13] attack, black-box[14] attack and the other baselines[59, 60], and
shows the effect of each component of the proposed method in defense through
the ablation experiment. Then, the state-of-the-art black-box attack DMR[14]
is used to compare defence performance of our JAD and the state-of-the-art
defense method GOAT[13]. Finally, we give a visual analysis of our defense.

We tested our JAD with white-box attacks on Market1501[2] and DukeMTMC[45],
and the attacks include metric-IFGSM (M-IFGSM)[12], SMA[13] and the proposed
LTA. To be consistent with recent works, we follow the new training/testing
protocol to conduct our experiments by k-reciprocal re-ranking (RK) [58]. It
can be seen from Table $\color[rgb]{1,0,0}4$ that on the two datasets, our JAD
has enhanced the Rank-1 in all white-box attacks by more than 40 percentage
points after using re-ranking.

Figure 7: t-SNE [61] visualization of six randomly selected images with
different identities on Market1501[2]. Each image corresponds to an metric-
IFGSM[13] adversarial example and some randomly generated homogeneous
modalities images. The same color means that they are obtained by
transformation of the same image. Dots means adversarial examples.

Ablation studies. We studied the contributions of our channel fusion without
augmentation (CF) , proactive defence (PD) and joint adversarial defence
(JAD). From Table $\color[rgb]{1,0,0}5$, we can see that defense accuracy
increases with the increase of data diversity. As a passive defense, CS
further significantly enhances defense performance. Table
$\color[rgb]{1,0,0}6$ shows that CS effectively reduces the adversarial effect
and thus significantly enhances the defense with negligible performance
degradation.

Comparison of state-of-the-arts. AT[12] needs to be customized according to
attacks. Specifically, in order to defend an attack, it is necessary to add
corresponding adversarial examples to train the model. In Table
$\color[rgb]{1,0,0}7$, the original accuracy (no attack) of defense methods
based on AT[12] are the average accuracy of the defense models corresponding
to the three attacks. AT+CS is the defense model combining the AT[12] and our
CS, and it can be seen that the proactive defence and passive defence method
CS (that is, our JAD) exert a better combined effect. This shows that our
proactive and passive defenses can complement each other and have a better
gaining effect. As with RRP[50], even if the attacker is aware of the
existence of passive defenses, CS can still be effectively defended by a
randomization mechanism that allows the resize to fluctuate within a certain
range. Compared with AT[12], our JAD enhances Rank-1 by more than 14.5% in all
white-box attacks.

In the tests of DMR[14] black-box attack, we use adversarial examples
generated by Resnet50 [57] to attack the DenseNet [40] model. GOAT[13] model
is training based on the adversarial samples generated online by the FNA
attack[13] using triplet loss. GOAT+CS is the defense model combining the
GOAT[13] and our CS. It can be seen from Table $\color[rgb]{1,0,0}8$ that the
defense accuracy of our JAD is far better than GOAT[13]. In addition, the
experimental results show that the JAD is applicable to other baselines [59,
60] and performs well. The strong baseline (SB) [59] is implemented based on
the Resnet50 backbone network adding the batch normalization neck structure,
and FastReID (FR) [60] is implemented based on the IBN-ResNet101 [62] backbone
network. The method proposed in this paper has good defense effect in both
white-box attack and black-box attack.

Figure 8: Comparison of Grad-CAM [63] activation map between normally trained
model and our proactive defense model.

Visualization analysis. As the show in Figure $\color[rgb]{1,0,0}7$, our
proactive defense model with robust to color variations which is insensitive
to the variations in the adversarial examples relative to the original
examples. Therefore, we can observe that the features of adversarial example
and homogeneous examples exhibit clustering effects.

Grad-CAM [63] uses the gradient information flowing into the last
convolutional layer of the CNN to visualize the importance of each neuron in
the output layer for the final prediction, by which it is possible to
visualize which regions of the image have a significant impact on the
prediction of a model. As shown in Figure $\color[rgb]{1,0,0}8b$, we can see
that the adversarial example successfully distracts the attention of the
normally trained model and activates the opposite parts, while the our
proactive defense model is still effectively activating some important parts.

##  5 Conclusion

In this paper, we proposed a color attack method (LTA) based on the local
transformation, and further propose a joint adversarial defense method (JAD)
based on the feature-invariance mechanism to enhance the adversarial
robustness of ReID. Finally, we use different network structures and baselines
under different attack modes to conduct comparative experiments to verify the
effectiveness of proposed attack method and joint defense method. Our future
goal is to further enhance the stability of the proactive defense model,
because we experimentally observed that the limitations of proactive defense
are regular. Therefore, we will try to consider cross-datasets when training
the model, and update the parameters of the model when a model is improved on
the original and cross-domain dataset.

## Acknowledgement

This work was supported by the National Natural Science Foundation of China
under Grant Nos. U1805263, 61976053, 61672157.

## References

  * [1] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C.H. Hoi.  Deep learning for person re-identification: A survey and outlook.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [2] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jing dong Wang, , and Qi Tian.  Scalable person re-identification:a benchmark.  In ICCV, 2015.
  * [3] Zhedong Zheng, Liang Zheng, and Yi Yang.  A discriminatively learned cnn embedding for person reidentification.  IEEE Transactions on Information Forensics and Security, 16:728–739, 2017.
  * [4] Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu.  Diverse part discovery: Occluded person re-identification with part-aware transformer.  In CVPR, pages 2898–2907, 2021.
  * [5] Ruibing Hou, Hong Chang, Bingpeng Ma, Rui Huang, and Shiguang Shan.  Bicnet-tks: Learning efficient spatial-temporal representation for video person re-identification.  In CVPR, pages 2014–2023, 2021.
  * [6] Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu, Yuan Xie, and Lizhuang Ma.  Farewell to mutual information: Variational distillation for cross-modal person re-identification.  In CVPR, pages 1522–1531, 2021.
  * [7] Xuehu Liu, Pingping Zhang, Chenyang Yu, Huchuan Lu, and Xiaoyun Yang.  Watching you: Global-guided reciprocal learning for video-based person re-identification.  In CVPR, pages 13334–13343, 2021.
  * [8] Lu Pang, Yaowei Wang, YiZhe Song, Tiejun Huang, and Yonghong Tian.  Cross-domain adversarial feature learning for sketch re-identification.
  * [9] Yi Li, Timothy M. Hospedales, Yizhe Song, and Shaogang Gong.  Fine-grained sketch-based image retrieval by matching deformable part models.  BMVC, pages 1–12, 2014.
  * [10] Lucas Beyer, Stefan Breuers, Vitaly Kurin, and Bastian Leibe.  Towards a principled integration of multi-camera re-identification and tracking through optimal bayes filters.  In CVPRW, 2017.
  * [11] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv:1312.6199, 2014.
  * [12] Song Bai, Yingwei Li, Yuyin Zhou, Qizhu Li, and Philip H.S. Torr.  Metric attack and defense for person re-identification.  arXiv:1901.10650, 2019.
  * [13] Quentin Bouniot, Romaric Audigier, and Angelique Loesch.  Vulnerability of person re-identification models to metric adversarial attacks.  In CVPRW, 2020.
  * [14] Hongjun Wang, Guangrun Wang, Ya Li, Dongyu Zhang, and Liang Lin.  Transferable, controllable, and inconspicuous adversarial attacks on person re-identification with deep mis-ranking.  In CVPR, 2020.
  * [15] Zhedong Zheng, Liang Zheng, Zhilan Hu, and Yi Yang.  Open set adversarial examples.  arXiv:1809.02681v1, 2018.
  * [16] Zhibo Wang, Siyan Zheng, Mengkai Song, Qian Wang, Alireza Rahimpour, and Hairong Qi.  advpattern: physical-world attacks on deep person re-identification via adversarially transformable patterns.  In ICCV, 2019.
  * [17] Yunpeng Gong.  A general multi-modal data learning method for person re-identification.  arXiv:2101.08533, 2021.
  * [18] Mo Zhou, Zhenxing Niu, Le Wang, Qilin Zhang, and Gang Hua.  Adversarial ranking attack and defense.  In ECCV, 2020.
  * [19] Giorgos Tolias, Filip Radenovic, and Ondrej Chum.  Targeted mismatch adversarial attack: Query with a flower to retrieve the tower.  In ICCV, 2019.
  * [20] Mo Zhou, Le Wang, Zhenxing Niu, Qilin Zhang, Yinghui Xu, Nanning Zheng, and Gang Hua.  Practical relative order attack in deep ranking.  In ICCV, 2021.
  * [21] Xiaodan Li, Jinfeng Li, Yuefeng Chen, Shaokai Ye, Yuan He, Shuhui Wang, Hang Su, and Hui Xue.  Qair: Practical query-efficient black-box attacks for image retrieval.  In CVPR, 2021.
  * [22] Jie Li, Rongrong Ji, Hong Liu, Xiaopeng Hong, Yue Gao, and Qi Tian.  Universal perturbation attack against image retrieval.  In ICCV, 2019.
  * [23] Mo Zhou and Vishal M. Patel.  Enhancing adversarial robustness for deep metric learning.  arXiv:2203.01439, 2022.
  * [24] Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz.  Joint discriminative and generative learning for person re-identification.  In CVPR, 2019.
  * [25] Zhun Zhong, Liang Zheng, Zhedong Zheng, Shaozi Li, and Yi Yang.  Camera style adaptation for person re-identification.  In CVPR, 2018.
  * [26] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In CVPR, 2018.
  * [27] Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft.  Improving the generalization of adversarial training with domain adaptation.  In ICLR, 2019.
  * [28] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry.  Robustness may be at odds with accuracy.  In ICLR, 2019.
  * [29] Tianyuan Zhang and Zhanxing Zhu.  Interpreting adversarially trained convolutional neural networks.  In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 7502–7511, 2019.
  * [30] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In ICLR, 2017.
  * [31] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow.  Transferability in machine learning:from phenomena to black-box attacks using adversarial samples.  arXiv:1605.07277, 2016.
  * [32] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In ICLR, 2014.
  * [33] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.  Boosting adversarial attacks with momentum.  In CVPR, 2018.
  * [34] Qiqi Xiao, Hao Luo, and Chi Zhang.  Margin sample mining loss: A deep learning based method for person re-identification.  arXiv:1710.00478, 2017.
  * [35] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv:1703.07737, 2017.
  * [36] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In CVPR, 2016.
  * [37] Uyeong Jang, Xi Wu, and Somesh Jha.  Objective metrics and gradient descent algorithms for adversarial examples in machine learning.  In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 262–277, 2017.
  * [38] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  IEEE Symposium on Security and Privacy, pages 39–57, 2017.
  * [39] Liang Zheng, Yi Yang, and Alexander G. Hauptmann.  Person re-identification: Past, present and future.  arXiv:1610.02984, 2016.
  * [40] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [41] Xuelin Qian, Yanwei Fu, Yu-Gang Jiang, Tao Xiang, and Xiangyang Xue.  Multi-scale deep learning architectures for person re-identification.  In ICCV, 2017.
  * [42] Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun.  Alignedreid: Surpassing human-level performance in person re-identification.  arXiv:1711.08184, 2017.
  * [43] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In ECCV, 2018.
  * [44] Wei Li, Xiatian Zhu, and Shaogang Gong.  Harmonious attention network for person re-identification.  In CVPR, 2018.
  * [45] Zhedong Zheng, Liang Zheng, and Yi Yang.  Unlabeled samples generated by gan improve the person re-identification baseline in vitro.  In ICCV, 2017.
  * [46] Zhun Zhong, Liang Zheng, Shaozi Li, and Yi Yang.  Generalizing a person retrieval model hetero- and homogeneously.  In ECCV, 2018.
  * [47] Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, and Jianbin Jiao.  Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification.  In CVPR, 2018.
  * [48] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maatenn.  Countering adversarial images using input transformations.  In ICLR, 2018.
  * [49] Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M. Roy.  A study of the effect of jpg compression on adversarial images.  arXiv preprint arXiv:1608.00853, 2016.
  * [50] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille.  Mitigating adversarial effects through randomization.  In ICLR, 2018.
  * [51] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv:1706.06083, 2017.
  * [52] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L. Yuille, and Kaiming He.  Feature denoising for improving adversarial robustness.  In CVPR, pages 501–509, 2019.
  * [53] Nicholas Carlini, Guy Katz, Clark Barrett, and David L. Dill.  Ground-truth adversarial examples.  In ICLR, 2018.
  * [54] Papernot N, Faghri F, Carlini N, Goodfellow I, Feinman R, Kurakin A, and et al.  Technical report on the cleverhans v2.1.0 adversarial examples library.  arXiv:1610.00768v6, 2016.
  * [55] Mang Ye, Jianbing Shen, Senior Member, IEEE, and Ling Shao.  Visible-infrared person re-identification via homogeneous augmented tri-modal learning.  IEEE Transactions on Information Forensics and Security, 16:728–739, 2021.
  * [56] Thierry Blu, Philippe Thévenaz, and Michael Unser.  Linear interpolation revitalized.  IEEE Transactions on Image Processing, 13(5):710–719, 2004.
  * [57] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [58] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li.  Re-ranking person re-identification with k-reciprocal encoding.  In CVPR, 2017.
  * [59] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of tricks and a strong baseline for deep person re-identification.  In CVPRW, 2019.
  * [60] Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, and Tao Mei.  Fastreid: a pytorch toolbox for general instance re-identification.  arXiv:2006.02631, 2020.
  * [61] L Maaten and G Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, pages 2579–2605, 2008.
  * [62] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang.  Two at once: enhancing learning and generalization capacities via ibn-net.  In ECCV, 2018.
  * [63] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, pages 618–626, 2017.