# Analogue Hawking radiation as a tunneling in a two-level
$\mathcal{PT}$-symmetric system

Bijan Bagchi1,†,∗ Department of Physics, Shiv Nadar Institution of Eminence,
Gautam Buddha Nagar, Uttar Pradesh 203207, India  Rahul Ghosh2,† Department of
Physics, Shiv Nadar Institution of Eminence,
Gautam Buddha Nagar, Uttar Pradesh 203207, India  and Sauvik Sen3,† Department
of Physics, Shiv Nadar Institution of Eminence,
Gautam Buddha Nagar, Uttar Pradesh 203207, India

In the light of a general scenario of a two-level non-Hermitian
$\mathcal{PT}$-symmetric Hamiltonian we apply the tetrad-based method to
analyze the possibility of analogue Hawking radiation. It is done by making
use of the conventional null-geodesic approach wherein the associated Hawking
radiation is described as a quantum tunneling process across a classically
forbidden barrier which the event horizon imposes. An interesting aspect of
our result is that our estimate for the tunneling probability is independent
of the non-Hermitian parameter that defines the guiding Hamiltonian.

††e-mail : 1<EMAIL_ADDRESS>2<EMAIL_ADDRESS>3.
<EMAIL_ADDRESS>: corresponding author††$\dagger$ : These
authors contributed equally to this work.

Keywords: Hawking radiation, $\mathcal{PT}$-symmetry, tunneling probability,
exceptional points

##  I Introduction

The physics of black holes has continually aroused interest after Bekenstien-
Hawking’s pioneering works in the 1970s in trying to interpret them as
thermodynamical objects which release radiation outside their event horizon
[1, 2, 3, 4] (see, for a review of literature, [5]). The idea of Hawking
radiation exploits the concept of creation of pair production next to the
event horizon (out of the vacuum) with one of the particles running away to
the infinite space from the boundary while the other with negative energy
getting sucked into the black hole resulting in the decrease in its mass until
the whole black hole disappears in a cloud of radiation. A natural question
has been asked as to whether viable information could be gathered at
temperatures near the scale of Planckian mass when the quantum gravitational
effects become substantial [6].

In this paper our primary focus is to look at a suitable structure of a non-
Hermitian two-level effective Hamiltonian [7, 8] to illustrate the possibility
of artificial Hawking radiation. This is done by mapping to a coordinate
setting and making use of the tetrad-based method. The latter strategy is
frequently used for seeking solutions pertaining to a curved space. We will
demonstrate that black hole similarities emerge with the emission of Hawking-
like radiation when the event horizon causes separation of two distinct
topological regions [9, 10].

A non-interacting field theory is often sought to address Hawking radiation.
In fact, it was observed that in two dimensions Schwarzschild geometry,
interaction effects are minor and that a free particle theory is adequate for
the treatment of Hawking radiation, see for some detailed discussion [11, 12].
In the following, we adopt the procedure of Parikh and Wilczek to estimate the
tunneling probability by employing the standard classical approach of WKB
approximation[13]. In their formalism, effect of back-reaction was included to
ensure energy conservation while a particle was emitted through the process of
tunneling in going past the horizon. Noting that the need for a nonsingular
coordinate system is essential at the horizon, we adopt below the well known
Painleve-Gullstrand coordinates [14, 15] which are simply coordinate
transformations of the usual Schwarzschild solution. The corresponding metric
tensor, which involves an off-diagonal element, is regular at the
Schwarzschild radius but has a singularity only at the origin, see for example
[16]. It needs to be mentioned that prior to the work of [13], the idea of
Hawking radiation as tunneling was first investigated by Srinivasan and
Padmanabhan [17], including a treatment in different coordinate settings [18].

A remark is in order: The need to use Painlevé-Gullstrand coordinates for the
Schwarzschild metric arises due to the singularity in the usual Schwarzschild
coordinates. However, use of Kruskal-Szekeres coordinates was shown to give an
incorrect formulation [19, 20] implying that one should be able to use other
coordinates so long as one did not have a real singularity instead of
coordinate singularity.

##  II Biorthogonality and exceptional point

The most general $2\times 2$ Hermitian Hamiltonian (with Fermi velocity set to
unity) in terms of the linear node $\vec{k}=(k_{x},k_{y},k_{z})$, has the form
[21, 22] $\hat{h}_{0}=\chi(k)+\sum_{i=x,y,z}d_{i}(k)\sigma_{i}$, where $\chi$
and the $d_{i}$’s are suitable real coefficients, $k=|\vec{k}|$, and
$\sigma_{i}$’s are Pauli matrices. The Hamiltonian which characterizes
conduction and valence bands supports a pair of eigenvalues whose separation
is removed when the coefficients vanish simultaneously. This is at once
obvious from the energy dispersion providing the energy splitting

$\epsilon_{\pm}=\chi\pm\sqrt{d_{x}^{2}+d_{y}^{2}+d_{z}^{2}}$ (2.1)

We see that degeneracy requires all three $d_{i}$’s to become zero together
i.e. $d_{x}=d_{y}=d_{z}=0$, which is not symmetry protected. In Weyl
semimetals, in which the conduction and valence bands energy coincide over a
certain region of the Brillouin zone, a linear crossing of two bands takes
place at the forming of nondegenerate Dirac cones corresponding to the
conduction and valance bands.

Extension of topological phases from the Hermitian to the non-Hermitian sector
has been pursued in a variety of papers [23] and the presence of gains and
losses investigated [24, 25]. A point was made a few years ago about the
question of whether real black holes can emit Hawking radiation and whether
meaningful information can be gathered about Planckian physics [6]. Very
recently, De Beule et al [26] made an explicit analysis of the existence of
artificial event horizon in Weyl semimetal heterostructures. In their work,
the electronic analogs of stimulated Hawking emission was studied and physical
observables were identified. Sabsovich et al [27] examined black and white
hole analogs in Weyl semimetals subjected to inhomogeneous nodal tilts and
explored experimentally viable consequences and showed the general relativity
analogy of such Hamiltonians. Analogy was also drawn in some papers between
the low-energy Hamiltonian of tilted nodes and black hole metric [8, 28, 29].
In a somewhat similar context, the possibility of the emission of Hawking
radiation was looked into [30]. Further, imitation of black hole Hawking
radiation was found in a purely classical-mechanical system by employing a
coupled double chain model admitting frequency dispersion [31]. The tied up
issue of the tunneling probability across the event horizon points was
explored in the framework of a two-level non-Hermitian topologically insulated
Weyl-type Hamiltonian which contained titing in one of the directions [32].

A two-dimensional structure of a non-Hermitian dissipative Hamiltonian also
featured in an elaborate investigation of analogue Schwarzschild black holes
emitting Hawking radiation [33]. A typical situation is described by the
inclusion of a non-Hermitian term $i\vec{\tau}(k)\cdot\vec{\sigma}$, with
$\vec{\tau}=(\tau_{x},\tau_{y},\tau_{z})$, to $\hat{h}_{0}$ when the overall
Hamiltonian assumes the form
$\hat{h}=\hat{h}_{0}+i\vec{\tau}(k)\cdot\vec{\sigma}$. In such a situation the
eigenvalues turn out to be

$E_{\pm}(k)=\chi\pm\sqrt{\Delta}$ (2.2)

where
$\Delta=d_{x}^{2}+d_{y}^{2}+d_{z}^{2}-(\tau_{x}^{2}+\tau_{y}^{2}+\tau_{z}^{2})+2i(\tau_{x}d_{x}+\tau_{y}d_{y}+\tau_{z}d_{z})$.
A consequence is that these become equal when $\Delta=0$ pointing to the
presence of an exceptional point [34, 35, 36, 37, 38] where we have

$d_{x}^{2}+d_{y}^{2}+d_{z}^{2}=\tau_{x}^{2}+\tau_{y}^{2}+\tau_{z}^{2}\quad\mbox{and}\quad\tau_{x}d_{x}+\tau_{y}d_{y}+\tau_{z}d_{z}=0$
(2.3)

Defining the right and left eigenstates to be

$\displaystyle\hat{h}(k)|\psi_{\pm}^{R}\rangle=E_{\pm}|\psi_{\pm}^{R}\rangle$
(2.4)
$\displaystyle\hat{h}^{\dagger}(k)|\psi_{\pm}^{L}\rangle=E^{*}_{\pm}|\psi_{\pm}^{L}\rangle$
(2.5)

where $|\psi_{\pm}^{R}\rangle$ and $|\psi_{\pm}^{R}\rangle$ are explicitly

$\displaystyle|\psi_{\pm}^{R}\rangle=\frac{1}{\sqrt{2(E_{\pm}-\chi)(E_{\pm}-\chi+d_{z}+i\tau_{z})}}\left[E_{\pm}-\chi+d_{z}+i\tau_{z},\quad(d_{x}+id_{y})-(\tau_{y}-i\tau_{x})\right]^{T}$
(2.6)
$\displaystyle|\psi_{\pm}^{L}\rangle=\frac{1}{\sqrt{2(E^{*}_{\pm}-\chi)(E^{*}_{\pm}-\chi+d_{z}-i\tau_{z})}}\left[E^{*}_{\pm}-\chi+d_{z}-i\tau_{z},\quad(d_{x}+id_{y})+(\tau_{y}-i\tau_{x})\right]^{T}$
(2.7)

the biorthogonality relations are a ready outcome

$\displaystyle\langle\psi_{\alpha}^{L}|\psi_{\beta}^{R}\rangle=\delta_{\alpha\beta}\quad\alpha,\beta=\pm$
(2.8)

The self-orthogonality of the eigenstates can be worked out easily [39].

##  III A two-level PT-symmetric model

###  III.A The Hamiltonian

Consider the following arrangement of the $d$-coefficients to enquire into the
spectral phase transition as the system transits to exhibiting complex
eigenvalues from the real ones

$d_{x}=\rho(k)\cos\phi(k),\quad d_{y}=\rho(k)\sin\phi(k),\quad
d_{z}=0,\quad\tau_{x}=\tau_{y}=0,\quad\tau_{z}=\lambda\eta(k)$ (3.1)

where k is a real variable, $\rho(k),\eta(k)$ and $\phi(k)$ are a set of real,
nonzero periodic functions, and $\lambda\in{\Re}^{+}$ is a coupling parameter.
Inclusion of the latter implies introduction of gain and loss in the system
thereby signaling the possibility of the appearance of exceptional points
where abrupt phase transitions could occur. The class of representations (3.1)
have also been studied in [40, 41].

Expressed in the matrix form the Hamiltonian corresponding to (3.1) reads

$\hat{h}\rightarrow\hat{\mathcal{H}}=\begin{pmatrix}T(k)&S(k)\\\\[6.0pt]
S^{*}(k)&T^{*}(k)\end{pmatrix},\quad\hat{\mathcal{H}}\neq\hat{\mathcal{H}}^{\dagger}$
(3.2)

where $T=\chi(k)+i\lambda\eta(k)$ and $S=\rho(k)e^{i\phi(k)}$. The Hermitian
counterpart of $\hat{\mathcal{H}}(k)$ corresponds to $k=0$. Enforcing
$\mathcal{PT}$-symmetry [42] requires the diagonal elements of the matrix
generated by (3.2) to be complex conjugate of each other and similarly for the
off-diagonal elements too. $\hat{\mathcal{H}}(k)$ is easily seen to commute
with the joint operator $\mathcal{PT}$ i.e.
$[\hat{\mathcal{H}},\mathcal{PT}]=0$ with the $\mathcal{P}$ operator
represented by $\sigma_{x}$, and $\mathcal{T}$ standing for the usual complex
conjugation operation.

Apart from analytical evaluations [43], numerical algorithms for the
diagonalization of $\mathcal{PT}$-symmetric Hamiltonians have also been
carried out in the literature [44]. The possibility of $\mathcal{PT}$-symmetry
residing in quantum mechanical systems is a prominent forefront in research.
Their position was soon found out to be intermediate between open and closed
systems. While the role of non-Hermiticity in understanding stable phases has
been pursued in the literature for last several years [45, 46], the character
of $\mathcal{PT}$-symmetry for stable nodal points concerning gapped and
gapless semimetals [47, 22], where the invariants are constituted by Bloch
bands, is a somewhat recent realisation. Indeed due to such a symmetry
prevailing, one finds stable nodal points to exist in lesser dimensions [48].

The eigenvalues of $\hat{\mathcal{H}}$ are easily seen to satisfy the relation

$\mathcal{E}_{\pm}(k)=\chi(k)\pm\eta(k)\sqrt{\overline{\lambda}^{2}(k)-\lambda^{2}},\quad\overline{\lambda}(k)\equiv\frac{\rho(k)}{\eta(k)}$
(3.3)

Introducing $\tan\theta=\frac{\rho(k)}{i\lambda\eta(k)}$ where
$\theta=\theta(k)$, one can classify the accompanying right eigenvectors to be

$\left[\cos(\frac{\theta}{2}),\,\sin(\frac{\theta}{2})e^{-i\phi}\right]^{T},\;\left[\sin(\frac{\theta}{2}),\,-\cos(\frac{\theta}{2})e^{-i\phi}\right]^{T}$
(3.4)

while their left partners are

$\left[\cos^{*}(\frac{\theta}{2}),\,\sin^{*}(\frac{\theta}{2})e^{-i\phi}\right]^{T},\;\left[\sin^{*}(\frac{\theta}{2}),\,-\cos^{*}(\frac{\theta}{2})e^{-i\phi}\right]^{T}$
(3.5)

It can be readily checked that these obey the biorthogonal conditions (2.8).

The expression (3.3) clearly shows that the eigenvalues stay real when the
inequality $\lambda<\overline{\lambda}$ holds. This corresponds to the
situation when $\mathcal{PT}$ is unbroken. However, when opposite is the case,
i.e. $\lambda>\overline{\lambda}$, a broken $\mathcal{PT}$ phase is
encountered. At the critical value $\lambda=\overline{\lambda}$, the
exceptional points appear when both the eigenvalues $\mathcal{E}_{+}$ and
$\mathcal{E}_{-}$ coincide to become $\chi(k)$ and the associated eigenvectors
coalesce to form a single entity. In other words, at the exceptional points we
have $\rho=\pm\lambda\eta$.

It is worthwhile to mention that a simpler form of $\hat{\mathcal{H}}$,
proposed by Bender et al [49] a few years ago, discusses the type
$\hat{\mathcal{H}}=q\cos\phi\P+iq\sin\phi\sigma_{z}+s\sigma_{x}$ which
supports the eigenvalues
$\lambda_{\pm}=q\cos\phi\pm\sqrt{s^{2}-q^{2}\sin^{2}\phi}$. These remain
entirely real when the inequality $s^{2}>q^{2}\sin^{2}\phi$ is obeyed.

###  III.B The tetrad representation

Let us represent the Hamiltonian in the following tetrad basis

$\hat{\mathcal{H}}=e{{}^{\mu}}{{}_{a}}h_{\mu}\sigma^{a}+e{{}^{\mu}}{{}_{0}}h_{\mu}\P$
(3.6)

and $h_{\mu}$’s are suitable entities to be matched. The vielbiens
$e{{}^{\mu}}{{}_{a}}$ and $e{{}^{\mu}}{{}_{0}}$ satisfy the orthonormality
condition $e{{}^{a}}_{\mu}e{{}^{\mu}}_{b}=\delta{{}^{a}}_{b}$, $\mu=(0,x,y,z)$
and $a,b=(x,y,z)$ subject to the metric being expressed as the bilinear
combination of the tetrads

$g^{\mu\nu}=e{{}^{\mu}}{{}_{\alpha}}e{{}^{\nu}}{{}_{\beta}}\,\eta^{\alpha\beta}$
(3.7)

where $\eta^{\alpha\beta}=diag(-1,1,1,1)$ is the Minkowski metric of flat
spacetime.

To proceed with (3.7), let us first write the vielbiens $e_{a}$μ in terms of
four functions $f_{1},f_{2},g_{1},g_{2}$, and then seek to relate them with
the known functions at hand namely, $\chi(k),\rho(k),\eta(k),\phi(k)$. A
convenient set of vielbiens is given by [60]

$e_{0}{{}^{0}}=f_{1},\quad e_{1}{{}^{0}}=f_{2},\quad e_{0}{{}^{1}}=g_{2},\quad
e_{1}{{}^{1}}=g_{1},\quad e_{2}{{}^{2}}=\frac{1}{r},\quad
e_{3}{{}^{3}}=\frac{1}{r\sin\theta}$ (3.8)

with the respective inverses

$\displaystyle e^{0}{{}_{0}}=\frac{g_{1}}{f_{1}g_{1}-f_{2}g_{2}},\quad
e^{0}{{}_{1}}=-\frac{f_{2}}{f_{1}g_{1}-f_{2}g_{2}},\quad
e^{1}{{}_{0}}=-\frac{g_{2}}{f_{1}g_{1}-f_{2}g_{2}},\quad$ (3.9) $\displaystyle
e^{1}{{}_{1}}=\frac{f_{1}}{f_{1}g_{1}-f_{2}g_{2}},\quad e^{2}{{}_{2}}=r,\quad
e^{3}{{}_{3}}=r\sin\theta$

The line element then simplifies to the form

$ds^{2}=\frac{g_{1}^{2}-g_{2}^{2}}{f_{1}g_{1}^{2}}dt^{2}+\frac{2g_{2}}{f_{1}g_{1}^{2}}dtdr-\frac{1}{g_{1}^{2}}dr^{2}-r^{2}d\Omega^{2}$
(3.10)

The Schwarzschild gauge arises for the following choice of
$f_{1},f_{2},g_{1},g_{2}$ namely

$f_{1}=1,\quad f_{2}=0,\quad g_{1}=1,\quad
g_{2}=-\sqrt{\frac{2\mathcal{M}}{r}}$ (3.11)

and yields the following form of the black hole metric in Painlevé-Gullstrand
coordinates

$ds^{2}=\left(1-\frac{2\mathcal{M}}{r}\right)dt^{2}-2\sqrt{\frac{2\mathcal{M}}{r}}drdt-
dr^{2}-r^{2}d\Omega^{2}$ (3.12)

where $\mathcal{M}$ is the mass of the black hole and $t$ represents the
Painlevé time. The metric (3.12) is stationary (i.e. invariant under
translation of $t$) but not static (i.e. not invariant under time-reversal)
and is consistent with the transformation proposed originally in [14, 15].

With the help of (3.6) and (3.8), the general form of the Hamiltonian emerges
as

$\begin{split}\hat{\mathcal{H}}=\frac{g_{1}h_{0}-g_{2}h_{1}}{f_{1}g_{1}-f_{2}g_{2}}\P+\frac{-f_{2}h_{0}+f_{1}h_{1}}{f_{1}g_{1}-f_{2}g_{2}}\sigma_{x}+rh_{2}\sigma_{y}+r\sin\theta
h_{3}\sigma_{z}\end{split}$ (3.13)

Comparing with (3.10), we can easily derive the coordinate-space
correspondence

$\begin{split}\chi(k)\rightarrow\frac{g_{1}h_{0}-g_{2}h_{1}}{f_{1}g_{1}-f_{2}g_{2}},\quad\rho(k)\cos\phi\rightarrow\frac{-f_{2}h_{0}+f_{1}h_{1}}{f_{1}g_{1}-f_{2}g_{2}},\quad\rho(k)\sin\phi\rightarrow
r,\quad\lambda\eta(k)\rightarrow r\sin\theta\end{split}$ (3.14)

where we have specified $h_{0}=h_{1}=h_{2}=1$ and $h_{3}=i$. Using the values
in (3.11), we have the mapping correspondence to the $(r,\theta)$ variables

$\begin{split}\chi\rightarrow
1+\sqrt{\frac{2\mathcal{M}}{r}},\quad\rho\cos\phi\rightarrow
1,\quad\rho\sin\phi\rightarrow r,\quad\lambda\eta\rightarrow
r\sin\theta\end{split}$ (3.15)

which is consistent with $\rho\rightarrow\sqrt{r^{2}+1}$ and
$\phi\rightarrow\tan^{-1}(r)$. As a result, the Hamiltonian assumes the form

$\hat{\mathcal{H}}=\left(1+\sqrt{\frac{2\mathcal{M}}{r}}\right)\P+\sigma_{x}+r\sigma_{y}+ir\sin\theta\sigma_{z}$
(3.16)

In the following we estimate the probability transmission amplitude of the
analogue Hawking radiation by making use of the correspondence set up in
(3.15).

##  IV Analogue Hawking radiation and tunneling estimate

First of all, the energy eigenvalues acquire read

$\mathcal{E}_{\pm}=\left(1+\sqrt{\frac{2\mathcal{M}}{r}}\right)\pm\sqrt{1+r^{2}\cos^{2}\theta}$
(4.1)

on using (3.3). The exceptional points correspond to $r=\pm i\sec\theta$
located on the imaginary axis. In what follows we will adhere to the positive
sign. We then have

$d\mathcal{E}=\frac{d\mathcal{M}}{\sqrt{2\mathcal{M}r}}$ (4.2)

Before we calculate the tunneling probability, let us note that when the
particle escapes from the black hole with an energy $\omega$, the mass of
black hole decreases from $\mathcal{M}$ to $\mathcal{M}-\omega$. Indeed, as
the pair production takes place around the event horizon, the positive energy
particle when breaks free [13] has to transit the separating region defined
between $r_{in}$, which is the radius of the black hole before the emission of
the particle, and $r_{out}$, which is the radius of the black hole after the
emission of the particle, acting as a possible barrier wall. This is possible
if the particle can tunnel through such a barrier. Actually, a classically
inaccessible zone is replicated with the particle possessing energy below such
a resistance.

In dealing with the tunneling problem we observe that since the action $\zeta$
in the transmission region is imaginary, the probability of tunneling to take
place can be straightforwardly calculated by making use of the semiclassical
WKB approximation [13]. Here an s-wave particle is considered to go outwards
from $r_{in}$ to $r_{out}$ meaning that $\zeta$ can be cast in the form

$\mbox{Im}\zeta=\mbox{Im}\int_{r_{in}}^{r_{out}}p_{r}dr=\mbox{Im}\int_{r_{in}}^{r_{out}}\int_{0}^{p_{r}}dp_{r}dr$
(4.3)

where the equation of motion for the canonical momentum is imposed as
prescribed by the classical Hamilton’s equation. Noting that the Hamiltonian
assumes the respective values $\mathcal{M}$ and $\mathcal{M}-\omega$ for
$p_{r}=0$ and $p_{r}=p_{r}$, $\mbox{Im}\zeta$ can be transformed to [51]

$\mbox{Im}\zeta=-\mbox{Im}\int_{r_{in}}^{r_{out}}\int_{0}^{\omega}\displaystyle\frac{d\omega}{\dot{r}}dr$
(4.4)

where, since the emitted particle has a very negligible mass, we can
approximate
$\mathcal{M}=\mathcal{M}_{\mbox{in}}\approx\mathcal{M}_{\mbox{out}}$ and
$d\mathcal{M}\approx-d\omega$.

For the metric at hand, the presence of the horizon can be determined from the
radial null geodesic condition $ds^{2}=0$ corresponding to (3.12). The
resulting differential equation becomes

$\dot{r}^{2}+2\sqrt{\frac{2\mathcal{M}}{r}}\dot{r}-\left(1-\frac{2\mathcal{M}}{r}\right)=0$
(4.5)

which admits of the following acceptable solution

$\dot{r}=1-\sqrt{\frac{2\mathcal{M}}{r}}$ (4.6)

Substituting in (4.4) results in

$\mbox{Im}\zeta=-\mbox{Im}\int_{2\mathcal{M}}^{2(\mathcal{M}-\omega)}\int_{0}^{\omega}\frac{d\omega
dr}{1-\sqrt{\frac{2(\mathcal{M}-\omega)}{r}}}$ (4.7)

which can be reduced to a tractable form on using the residue theorem with the
help of the substitution $\alpha^{2}=\frac{2(\mathcal{M}-\omega)}{r}$. This
yields

$\mbox{Im}\zeta=-\mbox{Im}\int_{2\mathcal{M}}^{2(\mathcal{M}-\omega)}dr\int_{0}^{\omega}\frac{d\alpha}{1-\alpha}=4\pi\omega$
(4.8)

Extracting the imaginary part from the right side, we obtain for the tunneling
probability the result

$\Gamma\approx e^{-2\,\mbox{Im}\zeta}=e^{-8\pi\omega}$ (4.9)

A similar estimate was made in [13] on accounting for the global conservation
laws. It is to be noticed that $\mbox{Im}\zeta$ as given by (4.9) is
independent of the parameters of the model Hamiltonian. It is also important
to point out that we do not claim invariance under canonical transformation of
the tunneling rate as given by (3.12) as also follows from the works of [19,
20]. Furthermore, for the inclusion of the temporal contribution to the
tunneling rate we refer to the works in [52, 53].

In conclusion, concerning the observable that could be associated with the
Hawking radiation, a natural question that arises as to what is the observable
that would result analogous to the Hawking radiation and in particular, the
decay rate of complex eigenstates. Indeed, because of the near impossibility
to observe Hawking radiation in a real black hole [54], seeking black-hole
analogues has become an interesting alternative because these could reveal
properties akin to gravitational black holes, emission of Hawking-like
radiation being a specific one. In this regard, construction of experimental
set-ups has been undertaken to identify quantum fluctuations that might emerge
[55]. Also, very recently, observation of stimulated Hawking radiation was
reported which occurs in a regime of extreme nonlinear fiber optics [56].
Another point which requires a more detailed study is the relationship between
the decay rates addressed in [57, 58, 59] and the one estimated in (4.9). As a
final remark, let us state that we can formally extend our model to other
black hole metrics like the charged black holes in 2D dilaton gravity which
originates from the low energy effective theory of type 0A string theory [60]
as well as for the rotating and charged black hole background [61].

##  V Summary

Non-Hermitian Hamiltonians are found to play a central role in diverse
physical problems. In this paper we studied a generalized form of a two-level
$\mathcal{PT}$-symmetric system that depends on a real parameter k and
exhibits a period of $2\pi$. Adopting the approach of tetrad formalism a
correspondence was established between such a Hamiltonian and the one
constructed in terms of vielbeins. This enabled us to connect to the metric of
curved spacetime. By suitably writing the tetrad components in terms of four
unknown functions and making a specific choice of them such that the
Schwrarzschild metric could be described in Painlevé-Gullstrand coordinates,
we could put in a one to one correspondence with our chosen form of the
Hamiltonian. We computed the probability transmission amplitude of Hawking
radiation by looking at it as a tunneling process and making use of the
semiclassical WKB approximation. Our result turned out to be independent of
the non-Hermitian parameter $\lambda$ so that the nature of phase transitions
that the system supports does not influence the tunneling probability
estimate.

##  VI Data Availability

All data supporting the findings of this study are included in the article.

##  VII Acknowledgments

We thank all the anonymous referees for making a number of constructive
remarks. We also thank Andreas Fring for pointing a couple of errors in an
earlier version of the manuscript. Two of us (RG, SS) thank Shiv Nadar IoE for
the grant of research fellowships.

##  VIII Conflicts of Interest

The authors declare no conflict of interest.

## References

  * [1] Bekenstein, J.D. Black holes and entropy. Phys. Rev. D 1973, 7, 2333–2346.
  * [2] Bekenstein, J.D. Generalized second law of thermodynamics in black-hole physics. Phys. Rev. D 1974, 9, 3292–3300.
  * [3] Hawking, S.W. Black hole explosions?. Nature 1974, 248, 30–31.
  * [4] Hawking, S.W. Particle creation by black holes. Commun. Math. Phys. 1975, 43, 199–220; Erratum: ibid 1976, 46, 206.
  * [5] Page, D.N., 2005. Hawking radiation and black hole thermodynamics. New J. Phys. 2005, 7, 203.
  * [6] Unruh, W.G. and Schützhold, R. Universality of the Hawking effect. Phys. Rev. D 2005, 71, 024028.
  * [7] Volovik, G.E. Black hole and Hawking radiation by type-II Weyl fermions. JETP Lett. 2016, 104, 645–648.
  * [8] Volovik, G.E. and Zhang, K. Lifshitz transitions, type-II Dirac and Weyl fermions, event horizon and all that. J. Low Temp. Phys. 2017, 189, 276–299.
  * [9] Morice, C., Moghaddam, A.G., Chernyavsky, D., van Wezel, J. and van den Brink, J. Synthetic gravitational horizons in low-dimensional quantum matter. Phys. Rev. Res. 2021, 3, L022022.
  * [10] Sims, C. Topologically Protected Wormholes in Type-III Weyl Semimetal Co3In2X2 (X= S, Se). Condens. Matter 2021, 6, 18.
  * [11] Leahy, D.A. and Unruh, W.G., Effects of a $\lambda\phi^{4}$ interaction on black-hole evaporation in two dimensions. Phys. Rev. D 1983, 28, 694–702.
  * [12] Frasca, M. Hawking radiation and interacting fields. Eur. Phys. J. Plus 2017, 132, 467.
  * [13] Parikh, M.K. and Wilczek, F. Hawking radiation as tunneling. Phys. Rev. Lett. 2000, 85, 5042–5045.
  * [14] Painlevé, P. La mécanique classique et la théorie de la relativité. C. R. Acad. Sci. 1921, 173, 677–680.
  * [15] Gullstrand, A. llgemeine lösung des statischen einkörperproblems in der Einsteinschen gravitationstheorie. Ark. Mat., Astron. Fys. 1922, 16, 1.
  * [16] Kanai, Y., Siino, M. and Hosoya, A. Gravitational collapse in Painlevé-Gullstrand coordinates. Prog. Theor. Phys. 2011, 125, 1053–1065.
  * [17] Srinivasan, K., and Padmanabhan, T. Particle production and complex path analysis. Phys.Rev. D 1999 60, 024007
  * [18] Shankaranarayanan, S., Padmanabhan, T. and Srinivasan, K. Hawking radiation in different coordinate settings: complex paths approach. Class. Quant. Grav 2002, 19, 2671-2688.
  * [19] Akhmedov, E. T., Akhmedova, V. and Singleton, D. Hawking temperature in the tunneling picture. Phys. Lett. B 2006, 642, 124-128.
  * [20] Akhmedov, E. T., Akhmedova, V., Singleton, D. and Pilling, T. Thermal radiation of various gravitational backgrounds. Int. J. Mod. Phys. A 2007, 22, 1705-1715.
  * [21] Okugawa, R. and Murakami, S. Dispersion of Fermi arcs in Weyl semimetals and their evolutions to Dirac cones. Phys. Rev. B 2014, 89, 235315.
  * [22] Armitage, N.P., Mele, E.J. and Vishwanath, A. Weyl and Dirac semimetals in three-dimensional solids. Rev. Mod. Phys. 2018, 90, 015001.
  * [23] Wu, H. and An, J.H. Non-Hermitian Weyl semimetal and its Floquet engineering. Phys. Rev. B 2022, 105, L121113.
  * [24] Yang, X., Cao, Y. and Zhai, Y. Non-Hermitian Weyl semimetals: Non-Hermitian skin effect and non-Bloch bulk–boundary correspondence. Chin. Phys. B 2022, 31, 010308.
  * [25] Yang, Z., Chiu, C.K., Fang, C. and Hu, J. Jones polynomial and knot transitions in Hermitian and non-Hermitian topological semimetals. Phys. Rev. Lett. 2020, 124, 186402.
  * [26] De Beule, C., Groenendijk, S., Meng, T. and Schmidt, T.L. Artificial event horizons in Weyl semimetal heterostructures and their non-equilibrium signatures. SciPost Phys. 2021, 11, 095.
  * [27] Sabsovich, D., Wunderlich, P., Fleurov, V., Pikulin, D.I., Ilan, R. and Meng, T. Hawking fragmentation and Hawking attenuation in Weyl semimetals. Phys. Rev. Res. 2022, 4, 013055.
  * [28] Xu, Y., Zhang, F. and Zhang, C. Structured Weyl points in spin-orbit coupled fermionic superfluids. Phys. Rev. Lett. 2015, 115, 265304.
  * [29] Soluyanov, A.A., Gresch, D., Wang, Z., Wu, Q., Troyer, M., Dai, X. and Bernevig, B.A. Type-II weyl semimetals. Nature 2015, 527, 495–498.
  * [30] Huhtala, P. and Volovik, G.E. Fermionic microstates within the Painlevé-Gullstrand black hole. J. Exp. Theor. Phys. 2002, 94, 853–861.
  * [31] Jana, S. and Sirota, L. A classical-mechanical analogue of Hawking black hole radiation. In 2022 Sixteenth International Congress on Artificial Materials for Novel Wave Phenomena (Metamaterials) (pp. 205–207) IEEE, Sienna, Italy, 2022.
  * [32] Bagchi, B. and Sen, S. Artificial Hawking radiation, weak pseudo-Hermiticity, and Weyl semimetal blackhole analogy. J. Math. Phys. 2022 63, 122102.
  * [33] Stålhammar, M., Larana-Aragon, J., Rødland, L. and Kunst, F.K. $\mathcal{PT}$ symmetry-protected exceptional cones and analogue Hawking radiation. New J. Phys. 2023, 25, 043012.
  * [34] Kato, T., 2013. Perturbation theory for linear operators, 2nd ed.; Springer: Berlin, Germany, 1995.
  * [35] Heiss, W.D. Exceptional points of non-Hermitian operators. J. Phys. A: Math. and Gen. 2004, 37, 2455.
  * [36] Heiss, W.D. The physics of exceptional points. J. Phys. A: Math. and Theor. 2012, 45, 444016.
  * [37] Znojil, M. Exceptional points and domains of unitarity for a class of strongly non-Hermitian real-matrix Hamiltonians. J. Math. Phys. 2021, 62, 052103.
  * [38] Correa, F. and Plyushchay, M.S. Spectral singularities in PT-symmetric periodic finite-gap systems. Phys. Rev. D 2012, 86, 085028.
  * [39] Cerjan, A., Xiao, M., Yuan, L. and Fan, S. Effects of non-Hermitian perturbations on Weyl Hamiltonians with arbitrary topological charges. Phys. Rev. B 2018, 97, 075128.
  * [40] Liang, S.D. and Huang, G.Y. Topological invariance and global Berry phase in non-Hermitian systems. Phys. Rev. A 2013, 87, 012118.
  * [41] Longhi, S. and Feng, L. Complex Berry phase and imperfect non-Hermitian phase transitions. Phys. Rev. B 2023, 107, 085122.
  * [42] Bender, C.M. and Boettcher, S. Real spectra in non-Hermitian Hamiltonians having PT symmetry. Phys. Rev. Lett. 1998, 80, 5243–5246.
  * [43] Bender, C.M., Dorey, P.E., Dunning, C., Fring, A., Hook, D.W., Jones, H.F., Kuzhel, S., Levai, G. and Tateo, R. $\mathcal{PT}$-symmetry: in quantum and classical physics, World Scientific: Singapore, 2018.
  * [44] Noble, J.H., Lubasch, M., Stevens, J. and Jentschura, U.D. Diagonalization of complex symmetric matrices: Generalized Householder reflections, iterative deflation and implicit shifts. Comp. Phys. Comm. 2017, 221, 304–316.
  * [45] Weimann, S., Kremer, M., Plotnik, Y., Lumer, Y., Nolte, S., Makris, K.G., Segev, M., Rechtsman, M.C. and Szameit, A. Topologically protected bound states in photonic parity–time-symmetric crystals. Nature materials 2017, 16, 433–438.
  * [46] Yuce, C. and Oztas, Z. PT symmetry protected non-Hermitian topological systems. Scientific Reports 2018, 8, 17416.
  * [47] Goerbig, M.O. Electronic properties of graphene in a strong magnetic field. Rev. Mod. Phys. 2011, 83, 1193–1243.
  * [48] Berry, M.V. Physics of nonhermitian degeneracies. Czech. J. Phys. 2004, 54, 1039–1047.
  * [49] Bender, C.M., Brody, D.C., Jones, H.F. and Meister, B.K. Faster than Hermitian quantum mechanics. Phys. Rev. Lett. 2007, 98, 040403.
  * [50] Kim, D.Y., Lasenby, A.N. and Hobson, M.P. Spherically-symmetric solutions in general relativity using a tetrad-based approach. Gen. Relativ. Gravit. 2018, 50, 1–37.
  * [51] Bagchi, B. and Sen, S. Tunneling of hawking radiation for BTZ black hole revisited. Int. J. Mod. Phys. A 2022, 37, 2150252.
  * [52] Akhmedova, V., Pilling, T., de Gill, A. and Singleton, D. Temporal contribution to gravitational WKB-like calculation. Phys. Lett. B 2008, 666, 269-271.
  * [53] Akhmedov, E. T., Pilling, T., Singleton, D. Subtleties in the quasi-classical calculation of Hawking radiation. Int. J. Mod. Phys. A 2008, D 17, 2453-2458.
  * [54] Castelvecchi, D. Black-hole mimic triumphs. Nature 2016, 536, 258–259.
  * [55] Steinhauer, J. Observation of self-amplifying Hawking radiation in an analogue black-hole laser. Nature Phys. 2014, 10, 864–869.
  * [56] Drori, J., Rosenberg, Y., Bermudez, D., Silberberg, Y. and Leonhardt, U. Observation of stimulated Hawking radiation in an optical analogue. Phys. Rev. Lett. 2019, 122, 010404.
  * [57] Dente, A. D., Bustos-Marún, R. A., and Pastawski, H. M. Dynamical regimes of a quantum SWAP gate beyond the Fermi golden rule. Phys. Rev. A 2008, 78, 062116.
  * [58] Bustos-Marún, R. A., Coronado, E. A., and Pastawski, H. M. Buffering plasmons in nanoparticle waveguides at the virtual-localized transition. Phys. Rev. B 2010, 82, 035434.
  * [59] Rotter, I. A non-Hermitian Hamilton operator and the physics of open quantum systems. J. Phys. A: Math. and Theor. 2009, 42, 153001.
  * [60] Kim, H. Hawking radiation as tunneling from charged black holes in 0A string theory. Phys. Lett. B 2011, 703, 94-99.
  * [61] Umetsu, K. Hawking Radiation from Kerr-Newman Black Hole and Tunneling Mechanism. Int. J. Mod. Phys. A 2010, 25, 4123-4140.


# Constrained Reinforcement Learning for Adaptive Controller Synchronization
in Distributed SDN

Ioannis Panitsas, Akrit Mudvari, Leandros Tassiulas  Yale University, New
Haven, CT
Emails: {ioannis.panitsas, akrit.mudvari<EMAIL_ADDRESS>
###### Abstract

In software-defined networking (SDN), the implementation of distributed SDN
controllers, with each controller responsible for managing a specific sub-
network or domain, plays a critical role in achieving a balance between
centralized control, scalability, reliability, and network efficiency. These
controllers must be synchronized to maintain a logically centralized view of
the entire network. While there are various approaches for synchronizing
distributed SDN controllers, most tend to prioritize goals such as
optimization of communication latency or load balancing, often neglecting to
address both the aspects simultaneously. This limitation becomes particularly
significant when considering applications like Augmented and Virtual Reality
(AR/VR), which demand constrained network latencies and substantial
computational resources. Additionally, many existing studies in this field
predominantly rely on value-based reinforcement learning (RL) methods,
overlooking the potential advantages offered by state-of-the-art policy-based
RL algorithms. To bridge this gap, our work focuses on examining deep
reinforcement learning (DRL) techniques, encompassing both value-based and
policy-based methods, to guarantee an upper latency threshold for AR/VR task
offloading within SDN environments, while selecting the most cost-effective
servers for AR/VR task offloading. Our evaluation results indicate that while
value-based methods excel in optimizing individual network metrics such as
latency or load balancing, policy-based approaches exhibit greater robustness
in adapting to sudden network changes or reconfiguration.

###### Index Terms:

Software Defined Networking, SDN, Controller Synchronization, DRL, DQN, DDQN,
PPO, Wireless Networks, Dynamic Networks, Augmented Reality

##  I Introduction

###  I-A Motivation

Software-Defined Networking (SDN) [1], [2] introduces a groundbreaking shift
in network architecture, fundamentally differentiating the decision-making
entity, known as the control plane, from the physical elements that handle
traffic routing and forwarding, called the data plane. This separation
enhances network performance through programmable management and
reconfiguration, offering network operators increased flexibility and
adaptability. SDN represents a major advancement in network design and
management, providing enhanced efficiency and customization. In this
architecture, a central SDN controller, which possesses a comprehensive
understanding of the network’s state, is responsible for distributing flow
tables to the data plane devices for data routing. However, this centralized
approach introduces challenges in scalability, availability, and security,
making the SDN controller a focal point for potential attacks [3] and a single
point of failure. Due to its centralized nature, the controller struggles to
efficiently manage network resources in large-scale environments, inherently
limiting its scalability. Moreover, the centralization intensifies security
vulnerabilities, as it presents a target for attackers, especially through
DDoS attacks [4], which can significantly disrupt network availability and
reliability. To address these challenges, SDN employs a distributed model [5]
with multiple independent controllers spread across the network. Although
physically separated, these controllers operate in a ’logically-centralized’
manner. This distributed SDN model, proposed to balance centralized and
distributed control systems, involves each controller managing a specific set
of subnetworks or domains. These controllers synchronize to maintain a
cohesive, logically centralized network view. In large-scale networks,
maintaining complete synchronization among controllers can be cost-
prohibitive. Consequently, many distributed SDN networks adopt partial inter-
controller synchronization, accepting temporary inconsistencies in the
controllers’ network views [6]. This is known as the ’eventual consistency’
model, which effectively manages costs and complexities.

Augmented Reality (AR) and Virtual Reality (VR) technologies are swiftly
becoming popular due to their exceptional capacity to seamlessly integrate
computer-generated graphics with real-life environments. These applications,
while offering immersive experiences, are both computationally intensive and
acutely delay-sensitive [7], [8]. This makes their operation on mobile or
local devices often impractical, particularly with the concern of battery
life. A solution is to offload the most resource-intensive computational tasks
of AR/VR applications to proximate edge servers [9]. However, in mobile edge
computing environments, this offloading process encounters substantial
challenges in adhering to the stringent delay requirements crucial for an
uninterrupted AR/VR experience [10]. Incorporating these applications into SDN
adds another layer of complexity. SDN’s ability to dynamically manage network
resources and optimize data flow can be pivotal in meeting the low latency
demands of AR/VR applications. Maintaining stringent latency requirements in
SDN-controlled networks poses challenges, particularly given the distributed
topology of SDN controllers. Furthermore, in dynamic wireless networks,
network parameters like link latency and edge server computational capacity
are in constant flux. Controllers must dynamically synchronized, ensuring
efficient path selection to servers while adhering to latency constraints.
This demands a proactive synchronization protocol, capable of real-time
adjustments to the network’s evolving topology and performance metrics,
ensuring consistent quality of service and latency adherence.

Inspired by the remarkable achievements of RL across various fields, such as
mastering the game of Go [11], optimizing energy efficiency in data centers,
researchers have applied various RL techniques to tackle synchronization
problems in Software-Defined Networking (SDN). By formulating the
synchronization problem as a Markov Decision Process (MDP), RL algorithms can
learn the optimal policy for synchronizing SDN controllers, ensuring that the
latency and QoS demands of AR/VR applications are met.

###  I-B Methodology and Contributions

In this paper, we developed a policy for synchronizing the distributed SDN
controllers, adhering to the eventual consistency model. Inspired by recent
successes in DRL, we formulated our problem as a MDP and we employed DRL
algorithms for approximating the optimal synchronization policy. We examined
both value-based and policy-based DRL methods to assist controllers in
learning close to optimal synchronization policies and we evaluated them in
various distributed SDN environments, where we varied multiple factors such as
network topology, number of connected devices in each domain, latency
constraints, and operational costs associated with edge servers. Our primary
objective was to reduce network operation costs by offloading tasks to edge
servers while adhering to specific latency constraints. A secondary objective
of our study was to expand the DRL framework to encompass existing SDN
applications, including shortest path routing. This extension aimed to
leverage and test the capabilities of our DRL framework in optimizing these
traditional SDN functionalities. Finally, our last objective in this paper was
to examine whether value based or policy based DRL algorithms are more
effective in dynamic SDN settings, particularly in developing a
synchronization policy in dynamic environments.

The contributions of this work can be summarized as follows:

  1. 1.

We tackle the challenge of developing a synchronization policy in SDN
environments, aiming to simultaneously ensure latency constraints and minimize
network operator costs by strategically offloading tasks to edge servers.

  2. 2.

We show that policy-based DRL methods are superior in learning and converging
faster to optimal synchronization policies when undergoing abrupt and
significant changes or network reconfigurations, compared to value-based
methods. We are the first to implement policy-based RL algorithms for
synchronizing distributed SDN controllers in dynamic networks.

  3. 3.

We evaluate our DRL framework in additional SDN applications, including
shortest path routing. Our results demonstrate that value-based methods
surpass policy-based approaches in effectively determining the optimal number
of network paths and server allocations.

  4. 4.

We evaluate the robustness and adaptability of our DRL framework in a range of
network conditions. This included altering the number of data and control
plane devices, edge servers, as well as varying background traffic and server
operational costs.

##  II Related Works

Recent breakthroughs [11], [12] have sparked considerable interest in applying
RL techniques to complex decision-making tasks. In the context of SDN, the
authors in [13] propose a novel approach by combining RL techniques with deep
neural networks. They develop the ”Deep-Q (DQ) Scheduler”, an advanced
controller synchronization policy. This scheduler significantly outperformed
the anti-entropy algorithm used in the ONOS controller, particularly for
inter-domain routing tasks. In this work [14], the researchers applied a deep
reinforcement and transfer learning-based method. This method equipped
controllers with an efficient policy for synchronization and maintaining a
logically centralized view in network systems. The effectiveness of this
approach has been demonstrated across various applications, including shortest
path routing and load balancing. The authors in [15] identified the optimal
frequency for synchronizing network views between controller pairs. Their
approach, focusing on two objectives - maximizing synchronized controller
pairs and enhancing application performance affected by synchronization rates
- showed considerable advantages over traditional methods that synchronize all
controller pairs at a uniform rate. In this study [16], the authors applied RL
techniques combined with deep neural networks (DNNs) to develop a nuanced,
scalable controller synchronization policy named ”Multi-Armed Cooperative
Synchronization” (MACS). The goal of MACS is to maximize the performance
improvements gained through controller synchronizations, showcasing an
innovative strategy in SDN management. Finally, in [17] the authors applied a
multi-objective DRL approach for controller synchronization and placement.

##  III System Description

In this section, we begin by providing an overview of the distributed SDN
environment and introducing the controller synchronization problem. Following
this introduction, we delve into the applications of interest that are
pertinent to this context. Subsequently, we explore the formulation of the
synchronization problem as a MDP, discussing how RL algorithms can be utilized
to approximate the optimal synchronization policy. In our approach, we employ
both value-based and policy-based methods to address the problem effectively.

###  III-A Distributed SDN environment

The network architecture consists of distributed controllers, denoted as $c$,
and data plane devices, denoted as $d$. A network domain contains a variety of
data plane devices, such as hubs, switches, and routers, as well as computing
resources, including edge servers for task offloading. Edge servers are
utilized to increase the computational capacity of the system, primarily to
offload tasks from various network applications. Each edge server has an
assigned cost variable, indicating the expense involved in offloading and
completing a task on that server. Neighbouring domains are interconnected
through several gateway nodes. These gateways play a critical role in
directing data flow and facilitating communication both across different
domains and within the same domain. Data plane devices forward network traffic
to the appropriate destination based on their flow table, which is controlled
and updated by a control plane device, such as a distributed SDN controller.
Control plane devices manage all critical network and security functions,
including updating and distributing flow table information to data plane
devices and filtering malicious traffic, among others. Link capacities and
server costs are highly dynamic and are evolving due to the background traffic
generated by users and network applications. Figure 1 illustrates a
distributed SDN environment, including both control and data plane devices.

Figure 1: A distributed SDN environment.

###  III-B Controller Synchronization

In this distributed SDN environment, our objective in this paper is to
implement an efficient and effective controller synchronization algorithm
under the eventual consistency model. Instead of synchronizing all the
distributed controllers, we aim to synchronize only a subset of the available
controllers while optimizing crucial metrics for different network
applications. We define the subset of distributed controllers to synchronize
in each time step as synchronization rate denoted as $SR$. In each
synchronization time step, after deciding which specific controllers to
synchronize, network domain state information will be exchanged only with the
synchronized controllers. This information can include link capacities,
network domain load etc. By synchronizing only a subset of the total
controllers, communication and signaling overheads will be minimized. The
selection of controllers for synchronization needs to be carefully aligned
with the goals of the network application. For instance, in the shortest path
routing application, the main objective is to synchronize the most critical
controllers that capture the dynamic nature of specific network domains,
thereby maximizing the accurate number of the shortest paths between all
source and destination nodes. This presents a significant challenge,
especially when the synchronization rate is limited.

###  III-C Applications of Interest

In this paper, our primary network application is offloading AR/VR tasks, that
are generated by users in the network edge, to servers under pre-defined
latency requirements. More specifically, recognizing the high computational
demand and the low latency transmission requirements of AR/VR tasks [18], we
considered an application where the generated tasks need to be transmitted
under pre-defined latency constraints/requirements and concurrently the
offloading procedure needs to be carefully designed to optimize network
operator objectives. In our case, the objective was to minimize servers
operation cost by selecting the optimal server for task offloading while
adhering to pre-defined latency constraints. Latency requirements are divided
into three categories: low latency task offloading, mid latency task
offloading and high latency task offloading. In the context of controller
synchronization, the goal is to implement an algorithm or policy that
strategically synchronizes a certain group of controllers based on the network
application constraints and optimizing the network application objectives.
This synchronization aims to achieve two main objectives: firstly, to ensure
the maximum number of network paths meet the latency requirements for the
tasks at hand, and secondly, to minimize network operator costs by efficiently
allocating tasks to the most appropriate servers. Note that, the implemented
policy can be easily modified and seamlessly integrated in every network
application that follows the approach of optimizing network metrics under
specific constraints. Figure 2, illustrates various scenarios encountered in
the task offloading process for AR/VR applications. In the optimal scenario,
tasks are offloaded to a server that both meets the latency requirements and
minimizes computational costs. However, the figure also highlights suboptimal
conditions, including cases where the latency constraints are not satisfied,
the selected server does not provide the most cost-effective solution, or
scenarios where neither latency nor cost criteria are adequately met.

Additionally, we evaluated our policy’s effectiveness in a different network
application, shortest path routing (SPR). In SPR, given the distributed
architecture of controllers, each must determine global paths based on the
network’s logical view. However, due to the eventual consistency model, if a
controller lacks accurate routing information, it may lead to incorrect path
calculations [19].

###  III-D MDP Formulation

For approximating the optimal policy for our described network application, we
formulated our problem as a MDP [20]. MDP provides a mathematical framework
ideal for modeling sequential decision-making scenarios. We use an MDP
framework to model the decision-making process involved in synchronizing
distributed network controllers.

  * •

Each state $s$ in state space $S$ is represented as a vector
$s=[s_{1},s_{2},\ldots,s_{n}]$. Each element $s_{i}$ in the vector corresponds
to the synchronization status of the $i^{th}$ SDN controller in a distributed
environment comprising $n$ controllers. The value of $s_{i}$ indicates the
number of time periods elapsed since the last synchronization of the $i^{th}$
controller. For instance, a state $s=[0,2,0]$ in a system with three
controllers ($n=3$) signifies that the first and third controllers were
synchronized in the current time step (indicated by 0), while the second
controller was last synchronized two time periods prior (indicated by 2). This
representation enables us to monitor the frequency of updates from the
controllers, providing insights into the synchronization policy and the
network’s dynamics.

  * •

Each action $a$ in action space $A$ is represented as a one-dimensional binary
vector $a=[a_{1},a_{2},\ldots,a_{n}]$, where each component $a_{i}$
corresponds to the decision made for the $i^{th}$ controller. Specifically,
$a_{i}=1$ indicates that the $i^{th}$ controller is to be synchronized during
the current time step, whereas $a_{i}=0$ signifies that the $i^{th}$
controller will not be synchronized. Given a predefined synchronization rate
$SR$, the number of elements in the action vector $a$ that are set to 1 must
be exactly equal to $SR$ following the eventual consistency model.

  * •

The reward function is an application-specific reward that depends on the
primary objective of the application, as well as on whether or not the chosen
policy violates certain constraint that the application may present. For
instance, in the primary AR/VR application that we discuss, reward function is
defined to impose penalties for synchronizing controllers that do not maximize
the total number of correct network paths satisfying the latency requirements
and penalizes decisions that do not minimize network operator costs
constrained on the latency requirements. More specifically, the reward
function is defined as follows:

$R(s,a)=\begin{cases}-r_{1},&\begin{subarray}{c}\text{if }s_{\text{sel}}\in
S\text{ and}\\\ s_{\text{sel}}\notin S_{L}\end{subarray}\\\
-K(C(s_{\text{sel}})-C(s_{\text{opt}})),&\begin{subarray}{c}\text{if
}s_{\text{sel}}\in S_{L}\text{ and}\\\
C(s_{\text{sel}})>C(s_{\text{opt}})\end{subarray}\\\ 0,&\text{if
}s_{\text{sel}}=s_{\text{opt}}\\\ -r_{2},&\text{otherwise}\end{cases}$ (1)

Where:

    * –

$s_{\text{sel}}$ is the edge server selected for task offloading.

    * –

$S$ is the set of all available edge servers.

    * –

$S_{L}$ is the subset of edge servers that meet the latency constraint $L$.

    * –

$r_{1}$ and $r_{2}$ are negative rewards (penalties) for selecting a edge
server that does not meet the latency or other criteria.

    * –

$K$ is a scaling factor for the cost-based penalty.

    * –

$C(s)$ is the cost associated with using server $s$.

    * –

$s_{\text{opt}}$ is the optimal edge server that meets both latency and cost
criteria.

For the non-constrained example, shortest path routing application, the reward
function is defined as:

$R(s,a)=\sum_{t}k.\sum_{l}SP(l,t)$ (2)

where, $SP(l,t)$ is a binary variable in $\\{0,1\\}$ with value $1$ if the
shortest path at time-step $t$ and link $l$ is accurately recognized, and $k$
is a scaling factor.

More details about the selected values of the parameters can be seen in
section IV-C.

Figure 2: Illustrating SDN controller synchronization with consistent and
eventual consistency models. Top left: A consistent model where all
controllers are synchronized, ideal for offloading an AR/VR task from domain 2
to an cost-effective edge server with maximum latency requirement equals to 4.
Remaining images depict an eventual consistency model with a synchronization
rate of 2. Top right: Meets latency requirements but not cost-effective.
Bottom left: Optimal edge server selected for task offloading, but fails to
meet latency constraint. Bottom right: Latency constraint and cost objective
are unmet

###  III-E Reinforcement learning formulation

In RL, an agent takes sequential decisions by interacting with an environment,
executing actions, and receiving feedback as rewards or penalties. The agent’s
goal is to develop a policy, a method for choosing actions based on its
current state, to maximize the cumulative reward over time. This learning
process is dynamic and adaptive, with the agent continually refining its
policy based on ongoing interactions and feedback from the environment [21].
The primary aim of employing RL to our problem is to find the optimal policy
denoted as $\pi^{*}$ for synchronizing the distributed controllers given the
synchronization rate constraint $SR$ for minimizing network operator costs,
while adhering to one additional constraint: limiting latency to not exceed a
pre-defined maximum value for an offloading task.

In the next sub-section, we will present value-based and policy-based RL
methods, that applied to address the challenges in our synchronization
problem. In this paper, we focused on a selection of value-based algorithms,
specifically Deep Q-Networks and Double Deep Q-Networks. Additionally, we
explored various policy-based methods, including the REINFORCE algorithm and
Proximal Policy Optimization algorithm. While we also considered A2C-A3C, a
hybrid approach combining aspects of both value and policy-based methods, it
was excluded from our in-depth analysis due to its performance.

###  III-F Value Based Methods

In value-based RL approaches, the primary objective is to estimate the value
function, from which the policy is then inferred. These algorithms are
categorized as off-policy, since the policy is determined based on the
Q-function. Value-based methods typically fall under the umbrella of model-
free techniques. This implies that they operate without a predetermined model
of the environment, learning to optimize the policy through direct
interactions and experiences within the environment itself.

####  III-F1 Deep Q-Learning

Deep Q-Networks (DQNs) [22], developed by DeepMind, were a major breakthrough
in RL, combining deep neural networks with Q-learning to handle complex, high-
dimensional problems. The core concept employed a function approximator to
estimate the Q-function using the Bellman equation for iterative updates.
Inspired by this work, for a given synchronization state $S$ and rate $SR$,
our implemented Deep Q-network outputs a vector of action values
$Q(s,\cdot;\theta)$ to select which subset of controllers need to be
synchronized at each time-step. The neural network’s input comprises a state
vector that indicates the number of time steps since each controller was last
synchronized. The output of this neural network corresponds to the total
number of possible actions, specifically detailing various synchronization
methods for the controllers. We trained the Deep Q-network by minimizing a
sequence of loss functions $L_{i}(\theta_{i})$ that change at each iteration
$i$. The loss function was defined as:

$L_{i}(\theta_{i})=\mathbb{E}_{s,a\sim\rho(\cdot)}\left[(y_{i}-Q(s,a;\theta_{i}))^{2}\right],$
(3)

where $y_{i}=\mathbb{E}_{s^{\prime}\sim
E}\left[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i-1})|s,a\right]$
is the target for iteration $i$, and $\rho(s,a)$ is a probability distribution
over sequences $s$ and actions $a$. The parameters from the previous
iteration, $\theta_{i-1}$, were held fixed when optimizing the loss function
$L_{i}(\theta_{i})$. The derivative of the loss function was calculated as:

$\displaystyle\nabla_{\theta_{i}}L_{i}(\theta_{i})=\mathbb{E}_{s,a\sim\rho(\cdot);s^{\prime}\sim
E}$
$\displaystyle\left[\left(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i-1})\right.\right.$
$\displaystyle\left.\left.-Q(s,a;\theta_{i})\right)\nabla_{\theta_{i}}Q(s,a;\theta_{i})\right],$
(4)

Instead of computing the full expectations in this gradient, we utilized a
replay memory, a data structure storing the agent’s transitions, to
approximate the expectation. By randomly sampling mini-batches from the replay
memory, we were able to optimise the loss function through stochastic gradient
descent

$\theta_{i+1}\leftarrow\theta_{i}-\eta\nabla_{\theta_{i}}L_{i}(\theta_{i}),$
(5)

where $\eta$ is the learning rate.

During the training phase, an epsilon-greedy strategy was employed. This
method allowed the agent to explore and learn effective decision-making
strategies by balancing between choosing random distributed controllers for
synchronization (exploration) and the best-known controller synchronization
action (exploitation). Once training was completed, a greedy approach was
adopted for selecting the optimal subset of controllers for synchronizing.
This approach involved choosing the optimal controllers for synchronization
with the highest Q-value from the deep Q-network’s output layer, thereby
identifying the optimal action for any given state.

####  III-F2 Double Deep Q-Learning

An issue encountered in the DQN algorithm was that the gradient descent
updates of $\theta$ differed from classical supervised learning. This
distinction arose because the training target

$y_{i}=R(s,a)+\gamma\max_{a^{\prime}}Q_{i}(s^{\prime},a^{\prime};\theta_{i})$
(6)

was generated by the same Q-function $Q_{i}(s,a;\theta_{i})$ that was trained.
This made it more likely to select overestimated values, resulting in
overoptimistic value estimates. Double Q-learning (DDQN) [23], an updated
version of DQN, prevented this issue by decoupling the selection from the
evaluation. For that reason, we used two value functions that were learned by
assigning each experience randomly to update one of the two value functions.
This resulted in two sets of weights, $\theta$ and $\theta^{\prime}$. For each
update, one set of weights was used to determine the greedy action (optimal
action), and the other set was used to determine its value. The secondary
network learned at a slower pace compared to the main network, which helped to
prevent overfitting. The updated training target value can be calculated as
follows:

$y_{i}=R(s,a)+\gamma\
Q_{\theta^{\prime}}(s^{\prime},\text{argmax}_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime}))$
(7)

Additionally, we incorporated a replay memory mechanism in our training
process to store training samples, allowing the agent to revisit previous
experiences. Once training was completed, a greedy approach was adopted for
controller selection. This approach involved choosing the optimal controllers
for synchronization with the highest Q-value from the neural network’s output
layer. Once training was completed, a greedy approach was adopted for
selecting the optimal subset of controllers for synchronizing similar to DQN
approach.

###  III-G Policy Based Methods

In policy-based RL approaches, the strategy is to approximate a stochastic
policy $\pi$ directly instead of deriving a deterministic policy from an
estimated value function. A stochastic policy is a function
$\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$ that maps each state $s$
to a probability distribution over actions assigning a probability $\pi(a|s)$
for each action $a$ in that state. A stochastic policy is often estimated with
a function approximator having trainable parameters $\theta$. The policy is
parameterized as $p_{\theta}(a|s)$, where the primary objective involves
computing the gradient of the expected rewards with respect to the policy
parameters $\theta$. Subsequently, gradient ascent is applied to iteratively
refine $\theta$, thereby enhancing the maximization of cumulative long-term
rewards.

####  III-G1 REINFORCE

REINFORCE [24] is a policy gradient algorithm, that employs a neural network
to approximate the stochastic policy, characterized by parameters $\theta$. We
implemented a neural network that takes as input the distributed controllers
state and produces a probability distribution over possible pairs of
controllers to synchronize. We directly computed the gradient of the long term
expected return denoted as $\nabla_{\theta}J(\theta)$, with respect to the
policy parameters $\theta$ and we used gradient ascent for updating the
trainable parameters.

The gradient was calculated as follows:

$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{T}\gamma^{t}R(s,a)\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\right]$
(8)

To optimize $J(\theta)$ and improve the policy, the parameters $\theta$ are
updated using gradient ascent:

$\theta\leftarrow\theta+\eta\nabla_{\theta}J(\theta)$ (9)

where, $\eta$ is the learning rate. This process iteratively adjusts the
policy towards higher expected returns by following the direction of the
approximated gradient.

####  III-G2 Proximal Policy Optimization

The fundamental challenge with certain methodologies in policy optimization
arises when implementing a large step size. Such an approach can lead to
substantial alterations in the policy at each iteration. This significant
change can result in a considerable gap between the parameters of the
consecutive policies — the new and the old — which, in turn, risks leading to
a collapse of the model. To address this issue, Proximal Policy Optimization
(PPO) [25] emerges as an advanced strategy, refining the traditional policy
gradient methods. PPO’s core objective is to facilitate substantial updates in
the policy while concurrently imposing a constraint. This constraint, known as
the trust region constraint, ensures that the new and old policies remains
within a predefined limit. The objective of PPO is encapsulated in the
following equation, which defines its core operational mechanism:

$\displaystyle L^{CLIP}(\theta)=\hat{E}_{t}\bigg{[}\min\bigg{(}$
$\displaystyle\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\hat{A}_{t},$
(10)
$\displaystyle\text{clip}\left(r_{t}(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_{t}\bigg{)}\bigg{]}$

where $r(\theta)$ denotes the probability ratio:

$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}.$
(11)

The first component of the objective function represents the core objective of
the TRPO algorithm. However, this part of the function does not incorporate a
constraint. As a result, maximizing this objective without any restrictions
could potentially lead to unduly large updates in the policy, a scenario that
might be undesirable in maintaining the stability and effectiveness of the
algorithm. By introducing the second term,
$\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}$, PPO alters the
surrogate objective by implementing a clipping mechanism on the probability
ratio. This modification effectively constrains the expansion of $r_{t}$
beyond the interval [1 - $\epsilon$, 1 + $\epsilon$]. Consequently, the
objective is finalized by choosing the lesser of the clipped or unclipped
objectives, thus establishing the ultimate objective as a conservative
estimate, or a ’guarded bound’, on the unclipped objective. This approach
ensures that alterations in the probability ratio are taken into account only
when they adversely affect the objective, fostering a more balanced policy
update process.

In our PPO implementation, we utilized a neural network to directly
approximate the synchronization policy. The input to this neural network is
the controllers state information while the output is a probability
distribution across all possible actions, where each action corresponds to a
different method of synchronizing the controllers. This setup allows our agent
to select the most optimal actions based on the generated probability
distribution, thereby eliminating the need for an epsilon-greedy strategy due
to the stochastic nature of the output layer. During training, the focus was
on refining the weights of the neural network in order to select optimal
controllers that adhere to latency constraints and efficiently allocate edge
servers, while minimizing the objective function $L^{CLIP}(\theta)$ using
stochastic gradient descent.

More details about the selected hyper-parameters of the above algorithms can
be seen in section IV-C.

##  IV Evaluation

This section begins by introducing the performance benchmarks in Section IV-A.
Then, the distributed network settings, reinforcement learning settings, and
experimental settings for evaluation scenarios are described in Sections IV-B,
IV-C, and IV-D, respectively. Finally, the evaluation results and analysis are
presented in Sections IV-E, IV-F, and IV-G.

Figure 3: Value-based algorithms outperform both policy-based and baseline
strategies in terms of cost minimization. Figure 4: Value-based algorithms
outperform both policy-based and baseline strategies in terms of finding
correct allocations. Figure 5: Value-based algorithms outperform both policy-
based and baseline strategies in finding correct path that satisfy latency
constraints. Figure 6: Costs, correct server allocations and network paths of
value-based and policy-based algorithms as well as random and round-robin
baselines for low latency task-offloading. Figure 7: Costs, correct server
allocations and network paths of value-based and policy-based algorithms as
well as random and round-robin baselines for mid latency task-offloading.
Figure 8: Costs, correct server allocations and network paths of value-based
and policy-based algorithms as well as random and round-robin baselines for
high latency task-offloading.

###  IV-A Performance Benchmarks

For bench-marking purposes, we specifically selected round robin (RRobin) and
random synchronization methods for their straightforwardness and intuitive
nature. In round robin synchronization, domain controllers are sequentially
synchronized, starting with a set of $SR$ controllers in a cyclical pattern.
Conversely, random synchronization selects $SR$ controllers randomly at each
time step for a varied sequence.

###  IV-B Distributed SDN Environment Settings

Our distributed network environment is composed of various domains and
distributed controllers. Experiments were conducted with a variety of network
configurations, altering the numbers of data plane devices, controllers, and
edge servers. Each domain in our network is assigned a distributed controller,
responsible for maintaining routing information and monitoring link statuses.
Within each domain, there are multiple servers available for offloading tasks
from AR/VR applications, with each server allocated a cost reflective of real-
world scenarios. For this scenario, we generated server costs as uniformly
random values between 20 and 100, denoted as $C$. It’s important to note that
these values can be interpreted with a different minimization objective;
rather than minimizing network costs, tasks are offloaded to edge servers that
consume the least energy for the task. Additionally, we created traffic
patterns that closely mimic reality, based on selected network metrics. The
network was tested across 5 to 12 domains, with each domain comprising a
random number of data plane devices ranging from 2 to 15. While we established
a synchronization rate ranging from 2 to 4 for each network topology
implemented, in section IV-E, we specifically present results using 6 to 8
network domains and synchronization rates ranging from 2 to 3.

###  IV-C Reinforcement Learning Settings

Our DRL framework was developed using the PyTorch library in Python. For the
DQN and DDQN models, we designed a neural network architecture with a single
hidden layer comprising 50 neurons and utilized ReLU as the activation
function. The training parameters included a batch size of 256 and an epsilon
decay factor of 10 to adjust the likelihood of random actions. The replay
memory was set at a capacity of 40000. Throughout the training, we employed
the Mean Squared Error (MSE) as the loss function and used the Adam optimizer
with a learning rate of 0.01. Additionally, we set a discount factor of 0.1 in
these models to better assess future rewards. Regarding policy gradient
methods, specifically REINFORCE and PPO, we followed a similar neural network
structure with a single hidden layer of 50 neurons using ReLU for activation.
The output layer, however, utilized a Softmax layer for generating probability
distributions. These models were trained with a batch size of 256, using the
Adam optimizer at a learning rate of 0.001, and each training cycle included 5
iterations. For effective learning, we opted for a clip epsilon of 0.1 and a
gradient clipping norm of 7. Like the DQN and DDQN models, these policy
gradient methods also incorporated a discount factor of 0.1.

For the AR/VR application, the parameters of the reward function were
initiated with these values $r_{1}=-10000$, $r_{2}=-8000$, and $K=80$. For the
SPR application, the scalar variable $k$ was set $k=100$.

###  IV-D Experimental Settings

Our experiments were carried out on a high-performance computer equipped with
an Intel(R) Core(TM) i9-10920X CPU running at 3.50GHz, paired with an NVIDIA
GPU, specifically the GeForce RTX 3090.

###  IV-E Evaluation Results

Analysis of Figures 3, 4, and 5 shows that value-based algorithms perform
better than policy-based algorithms, as well as the random and round robin
synchronization algorithms, in terms of minimizing the network operator’s
costs, maximizing the number of optimal server allocations, and maximizing the
number of correct network paths, respectively. PPO does seem to perform quite
well as opposed to other non-value based methods, with results much closer to
value based approaches. Specifically, Figure 3 highlights the efficiency of
the DQN algorithm in reducing network operator costs. It shows that DQN
outperforms other methods, being 0.45% more effective than DDQN, 7.32% more
than PPO, 17.77% more than Round Robin, 18.37% more than REINFORCE, and 22.95%
more effective compared to the Random algorithm. Moreover, Figure 4
illustrates that the DDQN algorithm excels in identifying optimal server
allocations. Specifically, it outperforms the DQN by 0.28%, PPO by 5.70%,
Round Robin by 15.15%, REINFORCE by 16.41% and Random by 20.61%. In Figure 5,
the results indicate that DDQN once again emerges as the optimal algorithm for
maximizing the correct number of network paths within the given latency
constraints. DDQN shows a 0.19% advantage over DQN, 4.26% over PPO, 10.45%
over Round Robin, 11.53% over REINFORCE, and 15.18% over Random.

Additionally, we evaluated our DRL framework across various network
topologies, focusing on controllers ranging from 6 to 8 and synchronization
rates between 2 and 3, as well as, different latency requirements. As depicted
in Figure 6 (low latency, 8 ms), Figure 7 (mid latency, 10 ms), and Figure 8
(high latency, 12ms), we observed that value-based algorithms outperform both
policy-based methods and traditional synchronization approaches like random
and round-robin in terms of reducing network operator costs while also
maximizing the efficiency in server allocations and identifying correct
network paths. Specifically, for low latency task offloading scenarios (shown
in Figure 6), we found that both DQN and DDQN exhibit similar levels of high
performance, making them preferred choices for this application. Among the
policy-based methods, PPO demonstrated notable effectiveness in reducing
network costs and successfully finding correct servers and paths. In scenarios
with mid latency requirements, as depicted in Figure 7, the DQN and DDQN
algorithms show remarkably similar effectiveness, marking them as optimal
choices for these conditions. The PPO algorithm also delivers a robust
performance, although not quite on par with DQN and DDQN. In contrast, the
REINFORCE algorithm, along with random and round robin approaches, demonstrate
a lower level of performance, indicating their limited suitability for these
specific latency demands. For high latency requirements as shown in Figure 8,
DQN and DDQN again stand out as the most effective algorithms, with PPO also
showing strong performance. However, REINFORCE, random, and round robin
methods lag behind in effectiveness for these requirements.

###  IV-F Additional Application: Shortest Path Routing

To further validate our DRL framework beyond varying network environments and
latency requirements for AR/VR task offloading, we extended our evaluation to
a distinct SDN application: shortest path routing. Here, we examined only the
best-performing algorithm from each category - value-based and policy-based.
As illustrated in Figure 9, DDQN emerged as the most accurate in detecting
correct network paths with an accuracy of 86.76%, closely followed by PPO at
85.98%. Comparatively, the random and round robin methods achieved lower
accuracies of 84.6% and 80.7%, respectively.

Figure 9: Percentage accurate detection of shortest paths under different
reinforcement learning algorithms

###  IV-G Robustness to significant network changes or reconfiguration

Figure 10: Percentage accurate detection of shortest paths with network
reconfiguration at random intervals

In this section, our goal will be to observe the effects of random and abrupt
network reconfiguration (or significant changes to the network), which is a
scenario where the subdomain (nodes/switches and links) assigned to each
neighboring controller is changed. On top of the previous assumptions of a
dynamic network, i.e., where network structure is changing relatively slowly,
in this new paradigm the network will undergo significant reconfiguration
(links and nodes are randomly recreated). As shown in Figure 10, the network
undergoes such reconfiguration at episodes 30 and 58 (denoted by horizontal
dotted line.). Like in previous cases, we see that the learning occurs better
in RL methods, with value-based DDQN approaching the most efficient policy
faster than PPO. But when reconfiguration happens, PPO is shown to be more
robust to the significant network change occurring after episodes 30 and 58;
it is shown to be capable of reaching higher efficiency in the new environment
very fast after each. PPO outperforms with 89% accuracy in path detection,
closely followed by DDQN at 87.23%, random at 86.5%, and round robin at 82.8%
This implies that, at least in absence of further modifications to value based
methods, which could be change in exploitation vs. exploration trade off among
other improvements, PPO is a better alternative for extremely vocative network
configurations.

##  V Conclusion

In this paper, we address the synchronization problem in distributed SDN
controllers under constrained synchronization rates and latency requirements.
Our goal was to create a DRL-based synchronization policy for efficient AR/VR
application offloading, targeting both an upper limit on network delay and
cost reduction for network operators. In addition to exploring value-based
methods such as DQN and DDQN, our research was extended to include policy-
based RL techniques, and their application in other network tasks such as
shortest path routing. Initially, we formulated the synchronization problem as
a MDP and subsequently defined our SDN environment. Besides approximating the
optimal synchronization policy with value-based RL algorithms, we also
employed policy-based algorithms and explored their benefits. Furthermore,
this research was not limited to developing strategies for optimizing an
application objective under a strict constraint, but also included an
evaluation of our RL framework across multiple network applications. Our
evaluation results indicate that while value-based methods outperform in
optimizing single network metrics such as latency followed closely by PPO,
policy-based approaches are more robust in sudden network changes or re-
configurations and can achieve higher performance in fast evolving dynamic
networks.

## References

  * [1] Kreutz, D., Ramos, F. M. V., Veríssimo, P. E., Rothenberg, C. E., Azodolmolky, S., and Uhlig, S., “Software-defined networking: A comprehensive survey,” _Proceedings of the IEEE_ , vol. 103, no. 1, pp. 14–76, 2015.
  * [2] Nunes, B. A. A., Mendonca, M., Nguyen, X.-N., Obraczka, K., and Turletti, T., “A survey of software-defined networking: Past, present, and future of programmable networks,” _IEEE Communications Surveys & Tutorials_, vol. 16, no. 3, pp. 1617–1634, 2014.
  * [3] Scott-Hayward, S., Natarajan, S., and Sezer, S., “A survey of security in software defined networks,” _IEEE Communications Surveys & Tutorials_, vol. 18, no. 1, pp. 623–654, 2016.
  * [4] Dharma, N. I. G., Muthohar, M. F., Prayuda, J. D. A., Priagung, K., and Choi, D., “Time-based ddos detection and mitigation for sdn controller,” in _2015 17th Asia-Pacific Network Operations and Management Symposium (APNOMS)_ , 2015, pp. 550–553.
  * [5] Bannour, F., Souihi, S., and Mellouk, A., “Distributed sdn control: Survey, taxonomy, and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 1, pp. 333–354, 2018.
  * [6] Sakic, E. and Kellerer, W., “Impact of adaptive consistency on distributed sdn applications: An empirical study,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 12, pp. 2702–2715, 2018.
  * [7] Chen, X. and Liu, G., “Joint optimization of task offloading and resource allocation via deep reinforcement learning for augmented reality in mobile edge network,” in _2020 IEEE International Conference on Edge Computing (EDGE)_ , 2020, pp. 76–82.
  * [8] Nyamtiga, B. W., Hermawan, A. A., Luckyarno, Y. F., Kim, T.-W., Jung, D.-Y., Kwak, J. S., and Yun, J.-H., “Edge-computing-assisted virtual reality computation offloading: An empirical study,” _IEEE Access_ , vol. 10, pp. 95 892–95 907, 2022.
  * [9] Erol-Kantarci, M. and Sukhmani, S., “Caching and computing at the edge for mobile augmented reality and virtual reality (ar/vr) in 5g,” in _Ad Hoc Networks_ , Zhou, Y. and Kunz, T., Eds. Cham: Springer International Publishing, 2018, pp. 169–177.
  * [10] Dab, B., Aitsaadi, N., and Langar, R., “Joint optimization of offloading and resource allocation scheme for mobile edge computing,” in _2019 IEEE Wireless Communications and Networking Conference (WCNC)_ , 2019, pp. 1–7.
  * [11] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M. _et al._ , “Mastering the game of go with deep neural networks and tree search,” _Nature_ , vol. 529, no. 7587, p. 484, 2016.
  * [12] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [13] Zhang, Z., Ma, L., Poularakis, K., Leung, K. K., and Wu, L., “Dq scheduler: Deep reinforcement learning based controller synchronization in distributed sdn,” in _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_ , 2019, pp. 1–7.
  * [14] Mudvari, A., Poularakis, K., and Tassiulas, L., “Robust sdn synchronization in mobile networks using deep reinforcement and transfer learning,” in _ICC 2023 - IEEE International Conference on Communications_ , 2023, pp. 1080–1085.
  * [15] Poularakis, K., Qin, Q., Ma, L., Kompella, S., Leung, K. K., and Tassiulas, L., “Learning the optimal synchronization rates in distributed sdn control architectures,” in _IEEE INFOCOM 2019 - IEEE Conference on Computer Communications_ , 2019, pp. 1099–1107.
  * [16] Zhang, Z., Ma, L., Poularakis, K., Leung, K. K., Tucker, J., and Swami, A., “Macs: Deep reinforcement learning based sdn controller synchronization policy design,” in _2019 IEEE 27th International Conference on Network Protocols (ICNP)_ , 2019, pp. 1–11.
  * [17] Mudvari, A. and Tassiulas, L., “Joint sdn synchronization and controller placement in wireless networks using deep reinforcement learning,” Year of Publication. [Online]. Available: URL
  * [18] Yang, X., Chen, Z., Li, K., Sun, Y., Liu, N., Xie, W., and Zhao, Y., “Communication-constrained mobile edge computing systems for wireless virtual reality: Scheduling and tradeoff,” _IEEE Access_ , vol. 6, pp. 16 665–16 677, 2018.
  * [19] Kuzniar, M., Perešíni, P., and Kostic, D., “What you need to know about sdn flow tables,” in _International Conference on Passive and Active Network Measurement_. Springer, 2015, pp. 347–359.
  * [20] White, C., _Markov decision processes_. Springer, 2001.
  * [21] Sutton, R. S. and Barto, A. G., _Reinforcement Learning: An Introduction_ , 2nd ed. MIT Press, 2018.
  * [22] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M., “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [23] van Hasselt, H., Guez, A., and Silver, D., “Deep reinforcement learning with double q-learning,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [24] Williams, R. J., “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 229–256, 1992.
  * [25] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O., “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017.