# Leveraging Opinions and Vaccination to Eradicate Networked Epidemics

Humphrey C. H. Leung, Zhuocong Li, Baike She, and Philip E. Paré Humphrey C.
H. Leung, Zhuocong Li, and Philip E. Paré are with the School of Electrical
and Computer Engineering at Purdue University. Email: {leung61, li3975,
<EMAIL_ADDRESS>Baike She is affiliated with the Department of
Mechanical and Aerospace Engineering at University of Florida. Email:
<EMAIL_ADDRESS>This work is partially supported by the National Science
Foundation, grant NSF-ECCS #2032258.

###### Abstract

We introduce a multi-layer networked compartmental $SIRS-V_{o}$ model that
captures opinion dynamics, disease spread, risk perception, and self-interest
vaccine-uptake behavior in an epidemic process. We characterize the target
vaccination criterion of the proposed model and conditions that guarantee the
criterion is obtainable by influencing opinions on disease prevalence. We
leverage this result to design an eradication strategy that leverages opinions
and vaccination. Through numerical simulations, we show that the proposed
eradication strategy is able to stabilize the epidemic process around a
healthy state equilibrium, and the outbreak rebounds after the control signal
is relaxed.

##  I Introduction

There has been a long history of studying epidemic mitigation and eradication
problems [1]. One popular approach for analyzing, predicting, and controlling
epidemic spreading processes is to leverage networked epidemic spreading
models. However, it is also critical to integrate other factors such as human
behavior [2], opinions [3], and vaccination [4] into the spreading models.
Hence, in order to study the impact of vaccination on epidemic mitigation, as
determined by opinions on the outbreak’s severity and perceived risk towards
the infection, we propose a networked compartmental model by coupling a
classic networked susceptible-infected-recovered-susceptible model with
opinion dynamics. The proposed model also includes vaccination and the loss of
immunity, i.e., the vaccinated population can become susceptible again.

Network-structured compartmental models were implemented to study disease
spreading over heterogeneous mixed population [5, 6, 7]. In order to better
capture complex spreading behaviors that are affected by various factors,
researchers proposed modern epidemic models by implementing human awareness
(opinions) [8, 3] and vaccination [9, 4] to epidemic modeling and control.

In order to address the critical impact of human awareness on vaccination over
epidemic spreading networks, mathematical models from social science were
imported to capture these opinions in epidemic models. An opinion dynamics
model was proposed in [10] which was coupled with epidemic spreading models in
[3]. In addition, multiplex and multi-layer epidemic spreading networks that
incorporate social dynamics and multi-transmission pathways were studied by
[11, 12].

There have been numerous works on game theoretic modeling to study the impact
of volunteer vaccination on the spread of infectious disease over networks
[13]. The basic assumption of these game theoretic models is that people act
according to their self interest, maximizing their personal payoff. This
assumption has been challenged in more recent works. Specifically, simulated
data has been used to compare with historical events to show that imitation
dynamics can reproduce salient features of vaccination level over time [14];
and self-interest is a major factor in vaccine-uptake decisions [15].

In this work, we incorporate opinion dynamics and imitation dynamics over
networks to construct a novel multi-layer networked epidemic model that
captures the aforementioned characteristics of epidemic spreading processes.
The main results and contributions of this work are:

  1. 1.

we propose a novel multi-layer network model to connect the opinions towards
the disease prevalence, risk perception on the infection, and self-interest
vaccination behavior on networked epidemic processes;

  2. 2.

we show the existence of the healthy state equilibrium and that it is locally
stable;

  3. 3.

we characterize conditions on vaccination levels which ensure disease
eradication; and

  4. 4.

we propose a control strategy for disease eradication through opinions.

While speech censorship, propaganda, and opinion control are characteristics
of malfunctioning dystopian states, we hope that through this work we can open
up discussions on the role and limitations that opinions play in epidemic
control from an analytical and computational perspective.

We organize the paper as follows: in Section II, we introduce our networked
$SIRS-V_{o}$ model with opinion and imitation dynamics and formulate the
problem of interest. In Section III, we present the stability analysis of our
model and develop a control strategy. Section IV demonstrates the numerical
results of our work via simulation. Section V concludes the paper and
discusses potential future research directions.

### Notation

Let [$n$] denote the index set {1, 2, …, $n$}, $\forall$
$n\in\mathbb{Z}_{>0}$. We view vectors as column vectors and write $x^{\top}$
to denote the transpose of a column vector $x$. We use $x_{i}$ to denote the
$i$th entry of a vector $x$. For any matrix $M\in\mathbb{R}^{n\times n}$, we
use $[M]_{i,:}$, $[M]_{:,j}$, and $[M]_{ij}$ to denote its $i$th row, $j$th
column, and $ij$th entry, respectively. Similarly, we denote
$[a_{ij}]_{i,j\in[n]}$ as the $n\times n$ matrix for any $n\times n$
2-dimensional array $a_{ij}$. We use $\widetilde{M}$ =
$diag\\{m_{1},...,m_{n}\\}$ to represent a diagonal matrix
$\widetilde{M}\in\mathbb{R}^{n\times n}$ with
$[M]_{ii}=m_{i},\leavevmode\nobreak\ \forall i\in[n]$. We use $0_{n}$ and
$1_{n}$ to denote the vectors whose entries all equal 0 and 1, respectively,
and $I_{n}$ to denote the $n\times n$ identity matrix. For a real square
matrix $M$, we use $\alpha(M)$, $\rho(M)$, and $\sigma(M)$ to denote the
spectral abscissa (the largest real part among its eigenvalues), spectral
radius, and singular value of $M$, respectively. For any two vectors
$v,w\in\mathbb{R}^{n}$, we write $v\geq w$ if $v_{i}\geq w_{i}$, $v>w$ if
$v\geq w$ and $v\neq w$, $v\gg w$ when $v_{i}>w_{i}$, $\forall i,j\in[n]$. The
comparison notations between vectors are used for matrices as well, for
instance, for $A,B\in\mathbb{R}^{n\times n}$, $A>B$ indicates that
$A_{ij}>B_{ij}$, $\forall i,j\in[n]$. Consider a directed graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, with the node set
$\mathcal{V}=\\{v_{1},...,v_{n}\\}$ and the edge set
$\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$. Let matrix
$A\in\mathbb{R}^{n\times n},\leavevmode\nobreak\ [A]_{ij}=a_{ij}$, denote the
adjacency matrix of $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$a_{ij}\in\mathbb{R}^{+}$ if $(v_{j},v_{i})\in\mathcal{E}$ and $a_{ij}=0$
otherwise, $\forall i\in[n]$. Let $k_{i}[A]=\sum_{j\in\mathcal{N}_{i}}a_{ij}$,
where $\mathcal{N}_{i}=\\{v_{j}:(v_{i},v_{j})\in\mathcal{E}\\}$ denotes the
neighbor set of $v_{i}$. The graph Laplacian of $\mathcal{G}$, whose adjacency
matrix is $A$, is defined as $L[A]\triangleq\widetilde{K}[A]-A$, where
$\widetilde{K}[A]\triangleq diag\\{k_{1}[A],...,k_{n}[A]\\}$.

##  II Model And Problem Formulation

In this section, we first introduce a networked ($SIRS-V_{o}$) model, which
integrates the classic networked susceptible-infected-recovered-susceptible
($SIRS$) model with vaccinations ($V$) and opinion ($o$). The goal is to
incorporate three interacting networks in our compartmental model to capture
the potential impact of media to a disease spreading process through human
decision making process. The three interacting networks include: a disease
transition network that captures the disease spreading through physical
interactions; an opinion spreading network that captures agents’ perception of
the current state of the prevalence of the disease; and a strategy
substitution network that describes the dynamics of agents switching their
vaccination strategies to substitute their neighbors’ decision of not taking
vaccine.

We first consider an epidemic spreading over a network and use a weighted
directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ to capture the physical
transmission network. $\mathcal{V}=\\{v_{1},...,v_{n}\\}$ represents the set
of $n$ nodes and the edge set
$\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ represents the disease
transmission channels over $\mathcal{V}$. Each edge is weighted by
$\beta_{ij}\in\mathbb{R}_{\geq 0}$, which denotes the infection rate from host
$j$ to $i$. Let $x_{i}(t)\in[0,1]$, $\forall i\in[n]$, $t\geq t_{0}$, denote
the probability that node $i$ is infected at time $t$.

We define the network of the opinions of the agents toward the prevalence of
the epidemic spread as a directed graph
$\mathcal{\bar{G}}=(\mathcal{V},\mathcal{\bar{E}})$, where
$\mathcal{\bar{E}}\subseteq\mathcal{V}\times\mathcal{V}$ represents the
opinion exchange over $\mathcal{V}$. Each edge is weighted by
$a_{ij}\in\mathbb{R^{+}}$, denoting the impact of node $j$ on node $i$ in
terms of opinions. Let $o_{i}(t)\in[0,1]$, $\forall i\in[n]$ at time $t\geq
t_{0}$, denote node $i$’s belief in the seriousness of the epidemic. A higher
$o_{i}(t)$ means that node $i$ considers the epidemic to be more serious, and
vice versa. Further, we assume, for node $i$, that a higher infection level in
their neighborhood ($x_{j}(t)$) will raise their belief in the seriousness of
the epidemic.

Lastly, we denote the strategy imitation network as a weighted directed graph
$\mathcal{\hat{G}}=(\mathcal{V},\mathcal{\hat{E}})$, where the edge set
$\mathcal{\hat{E}}\subseteq\mathcal{V}\times\mathcal{V}$ represents the set of
pairs of agents $(i,j)$ where agent $i$ imitates agent $j$’s vaccine-uptake
strategy with a non-zero probability. Furthermore, we assume in this model
that each agent $i$ randomly samples another agent $j$ according to some
distribution $p_{ij}$; if the strategy of the sampled agent provides a higher
payoff, then the agent will switch to the new strategy with a probability
proportional to the expected payoff gain [16, 17].

The weight of each directed edge $(v_{j},v_{i})$ is $\eta_{ij}(t)$, and it is
defined in the following way. Let $u^{v}_{i}$ be the probability of a serious
vaccine side effects, such as allergic reactions, after vaccine-uptake;
$u^{x}_{i}$ be the probability of significant morbidity after an infection of
agent $i$; and $p_{ij}$ be the probability of agent $i$’s decision being
influenced by the decision of agent $j$ and $[p_{ij}]_{i,j\in[n]}$ is a row
stochastic matrix. Let $m^{v}_{i}$ and $m^{x}_{i}$ represent the severity of a
serious vaccine side effects and significant morbidity after infection,
respectively, and $c$ be the government compensation for serious vaccine side
effects such that $c>m^{v}_{i}u^{v}_{i}$ for all $i\in[n]$. Then the payoff of
getting vaccinated is $c-m^{v}_{i}u^{v}_{i}$, and the perceived payoff of $i$
not getting vaccinated is $-o_{i}(t)(m^{x}_{i}u^{x}_{i})$, where $o_{i}(t)$ is
the perceived disease prevalence of node $i$ at time $t$. Note that $c$ can be
interpreted as the sum of the expected payoff of vaccine-uptake and expected
cost of remaining unprotected that is not captured by $m^{x}_{i}u^{x}_{i}$ and
$m^{v}_{i}u^{v}_{i}$, such as the inconvenience of mask wearing. The above
interpretation only suggest one scenario to interpret these parameters. The
above non-vaccine-uptake payoff formulation assumes that each agent determines
its payoff evaluation on the current perceived disease prevalence. We further
assume that the vaccine has perfect efficacy; an agent takes the vaccine once
it switches its strategy to vaccine-uptake; and the agent cannot switch back
except through the loss of immunity. Therefore, the perceived payoff gain for
agent $i$ at time $t$ is
$g_{i}(t)=o_{i}(t)(m^{x}_{i}u^{x}_{i})+(c-m^{v}_{i}u^{v}_{i})$.

Finally, let $\mu_{i}$ be the sampling rate of agent $i$ and $q$ be an
arbitrary constant. Then, we define the strategy transition matrix
$H(o(t))\in\mathbb{R}^{n\times n}$ such that
$[H(o(t))]_{ij}=\eta_{ij}(t)=q\mu_{i}g_{i}(t)p_{ij}(t)$. We denote
$\eta^{min}_{ij}=q\mu_{i}(c-m_{i}^{v}u_{i}^{v})p_{ij}$,
$\Delta\eta_{ij}=q\mu_{i}(m^{x}_{i}u^{x}_{i})p_{ij}$, and
$\eta_{ij}(t)=o_{i}(t)\Delta\eta_{ij}+\eta^{min}_{ij}$. We now formally
introduce the networked ($SIRS-V_{o}$) model:

$\displaystyle\dot{o}_{i}(t)$
$\displaystyle=\sum_{j\in\mathcal{N}_{i}}a_{ij}(o_{j}(t)-o_{i}(t))+a_{ij}(x_{j}(t)-o_{i}(t)),$
(1a) $\displaystyle\dot{s}_{i}(t)$
$\displaystyle=\delta_{i}v_{i}(t)-s_{i}(t)\sum_{j\in\mathcal{N}_{i}}[\beta_{ij}x_{j}(t)+\eta_{ij}(t)v_{j}(t)]+\omega_{i}r_{i}(t),$
(1b) $\displaystyle\dot{x}_{i}(t)$
$\displaystyle=s_{i}(t)\sum_{j\in\mathcal{N}_{i}}\beta_{ij}x_{j}(t)-\gamma_{i}x_{i}(t),$
(1c) $\displaystyle\dot{r}_{i}(t)$
$\displaystyle=\gamma_{i}x_{i}(t)-\omega_{i}r_{i}(t)-r_{i}(t)\sum_{j\in\mathcal{N}_{i}}\eta_{ij}(t)v_{j}(t),$
(1d) $\displaystyle\dot{v}_{i}(t)$
$\displaystyle=(s_{i}(t)+r_{i}(t))\sum_{j\in\mathcal{N}_{i}}\eta_{ij}(t)v_{j}(t)-\delta_{i}v_{i}(t),$
(1e)

where $s_{i}$, $x_{i}$, $r_{i}$, and $v_{i}$ represent the percentage of
susceptible, infected, removed, and vaccinated populations at node $i$. Note
that $\eta_{ij}(t)=((1-o_{i}(t))\eta_{ij}+o_{i}(t)\eta^{\min}_{ij})$ is a
convex combination of $\eta_{ij}$ and $\eta^{\min}_{ij}$. To further simplify
the model, we express (1) in a compact form:

$\displaystyle\dot{o}(t)$ $\displaystyle=A(x(t)-o(t))-2L[A]o(t),$ (2a)
$\displaystyle\dot{s}(t)$
$\displaystyle=\widetilde{D}v(t)-\widetilde{S}(t)(Bx(t)+H(o(t))v(t)+\widetilde{W}r(t),$
(2b) $\displaystyle\dot{x}(t)$
$\displaystyle=\widetilde{S}(t)Bx(t)-\widetilde{G}x(t),$ (2c)
$\displaystyle\dot{r}(t)$
$\displaystyle=\widetilde{G}x(t)-\widetilde{W}r(t)-\widetilde{R}(t)H(o(t))v(t),$
(2d) $\displaystyle\dot{v}(t)$
$\displaystyle=(\widetilde{S}(t)+\widetilde{R}(t))H(o(t))v(t)-\widetilde{D}v(t),$
(2e)

where $\widetilde{S}(t)=diag(s(t))$, $\widetilde{R}(t)=diag(r(t))$, and $L[A]$
is the Laplacian matrix of the opinion spreading graph $\mathcal{\bar{G}}$.
$\widetilde{G}$ and $\widetilde{D}$ are diagonal matrices, with
$[\widetilde{G}]_{ii}=\gamma_{i}$ and $[\widetilde{D}]_{ii}=\delta_{i}$,
$\forall i\in[n]$. We define $H(o(t))$ as a function of $o(t)$, where
$H(o(t))=\widetilde{O}(t)\Delta H+H_{\min}$, and
$[H_{\min}]_{ij}=\eta^{\min}_{ij}$ if $\eta_{ij}>0$, otherwise,
$[H_{\min}]_{ij}=0$.

Recall that our goal is to investigate the potential impact of media on
epidemic spreading through human decision processes, we introduce the media
actuator $m(t)=[m_{i}(t)]_{i=1\ldots n}$ and augment (2a) by including the set
of media nodes:

$\dot{o}(t)=A(x(t)-o(t))-2L[A]o(t)+\widetilde{M}(t)(1-o(t)),$ (3)

where $\widetilde{M}(t)=diag(M(t))$. The problems that we will answer in this
paper include:

  1. 1.

Show the existence and uniqueness of a healthy state equilibrium of system (2)
and the local stability condition of its infection subsystem.

  2. 2.

Construct an eradication algorithm stabilizing 2c around $x=0_{n}$ through the
control of a media actuator $g(t)$, and provide the assumptions and conditions
that guarantee the validity of the algorithm.

Solutions to these problems will be provided in Section III-A and Section
III-B, respectively.

##  III Main Results

This section explores the stability conditions, the properties of the opinion
dynamics, and the strategies to mitigate the networked epidemic process in
(2). The results are developed under the following assumptions.

###### Assumption 1

All parameters are real and non-negative.

###### Assumption 2

The initial conditions at $t_{0}$ should obey
$(o_{i}(t_{0}),s_{i}(t_{0}),x_{i}(t_{0}),r_{i}(t_{0}),v_{i}(t_{0}))\in[0,1]$
and
$s_{i}(t_{0})+x_{i}(t_{0})+r_{i}(t_{0})+v_{i}(t_{0})=1,\leavevmode\nobreak\
\forall i\in[n]$.

###### Assumption 3

We assume that the opinion spreading graph $\bar{\mathcal{G}}$, disease
transmission graph ${\mathcal{G}}$, and strategy imitation graph
$\mathcal{\hat{G}}$ are strongly connected.

First, one can show that the model in (2) is well defined by checking the
derivative of a state being always pointing inside the set $[0,1]$, when the
state hits the boundary of $[0,1]$. Thus, we omit the proof by giving the
following statements.

###### Lemma 1

For all $t>t_{0}$ and $i\in[n]$, $s_{i}(t)+x_{i}(t)+r_{i}(t)+v_{i}(t)=1$ and
($s_{i}(t)$, $x_{i}(t)$, $r_{i}(t)$, $v_{i}(t)$) $\in[0,1]$.

###### Lemma 2

If $o(t_{0})\in[0,1]^{n}$, $o_{i}(t)$ varies in the range of [0,1], $\forall
i\in[n]$ and $t>t_{0}$.

###### Proposition 1

Without the loss of vaccine immunity, the vaccinated population monotonically
increases.

Let $(o^{*},s^{*},x^{*},r^{*},v^{*})$ denote the equilibria of the $SIRS-
V_{o}$ model. We start to analyze the equilibria of the $SIRS-V_{o}$ model
through the following definition.

###### Definition 1

(Healthy State Equilibrium). _A healthy state equilibrium_ is an equilibrium
with the steady-state vector of infected populations $x^{*}=0_{n}$ , where
$\dot{s}^{*}=\dot{x}^{*}=\dot{r}^{*}=\dot{v}^{*}=\dot{o}^{*}=0_{n}$.

###  III-A Healthy State Equilibrium

In this subsection, we prove that the healthy state equilibrium of the coupled
networked system in (2) exists and derive the stability condition of the
infection subsystem around any healthy state equilibrium.

###### Lemma 3

At any healthy state equilibrium, where $x^{*}=0_{n}$, the recovered
population $r^{*}=0_{n}$. Furthermore, $s^{*}+v^{*}=1_{n}$.

###### Proof:

According to (2d), when $x^{*}=0_{n}$,

$\widetilde{W}r^{*}+\widetilde{R}^{*}H_{min}v^{*}=0_{n},$

where $[\widetilde{W}]_{ii}>0$, $v^{*}\geq 0_{n}$, and $H_{min}\geq 0_{n\times
n}$, $\forall i\in[n]$. Therefore, $r^{*}=0_{n}$, and $s^{*}+v^{*}=1_{n}$ via
Lemma 1. ∎

###### Lemma 4

At any healthy state equilibrium, where $x^{*}=0_{n}$, the opinions will reach
consensus at $o_{i}^{*}=0$, $\forall i\in[n]$.

###### Proof:

At an equilibrium where $x^{*}=0_{n}$, the opinions in (2a) satisfy

$\displaystyle 0_{n}$ $\displaystyle=A(0_{n}-o^{*})-2L[A]o^{*}$
$\displaystyle=-(L[A]+\widetilde{K}[A])o^{*}.$

Based on the Gershgorin circle theorem, we have that the spectrum of the
matrix lies within at least one of the Gershgorin discs, where the center of
the $i$th disc is given by
$c_{ii}=[-(L[A]+\widetilde{K}[A])]_{ii}=-2k_{i}[A]$, and the corresponding
radius of the disc is $[L[A]]_{ii}=k_{i}[A]$, $\forall i\in[n]$. Hence, all
discs are strictly located on the left-hand side of the complex plane, which
indicates that the matrix $-(L[A]+\widetilde{K}[A])$ is negative definite,
i.e., the null space of the matrix contains only $0_{n}$. Further, at an
equilibirium where $x^{*}=0_{n}$, the only solution for (2a) is $o^{*}=0_{n}$,
which completes the proof. ∎

The above lemmas outline some important characteristics about the healthy
state equilibrium when it exists, which will be useful when proving the
following proposition.

###### Proposition 2

If $\alpha(-D+H_{min})\leq 0$, then $(0_{n},1_{n},0_{n},0_{n},0_{n})$ is the
unique healthy state equilibrium of system (2). If $\alpha(-D+H_{min})>0$,
then system (2) has two healthy state equilibria,
$(0_{n},1_{n},0_{n},0_{n},0_{n})$ and $(0_{n},1_{n}-v^{*},0_{n},0_{n},v^{*})$,
where $v^{*}\gg 0_{n}$.

###### Proof:

By Lemma 1, 3, and 4, if there exists a healthy state equilibrium, then it
will have the form $(0_{n},1_{n}-v^{*},0_{n},0_{n},v^{*})$. Furthermore,
system (2) can be simplified to:

$\dot{v}(t)=(I-\widetilde{V}(t))H_{\min}v(t)-\widetilde{D}v(t),$ (4)

where $H(o(t))=H_{\min}$ since $o(t)=0_{n}$. Therefore, to show that a healthy
state equilibrium exists, we limit our search of the healthy state equilibrium
within the domain of
$\\{0_{n}\\}\times[0,1]^{n}\times\\{0_{n}\\}\times\\{0_{n}\\}\times[0,1]^{n}$
through Brouwer’s fixed-point theorem [18, pg. 20]. After the above
simplifications, the problem of showing the existence and uniqueness of the
healthy state equilibrium of system (2) is reduced to the equilibrium analysis
of the single viral networked SIS model, by replacing $v$ with $x$ and
$H_{\min}$ with $B$. The desired result then follows directly from [19, Prop.
2] since $H_{\min}$ is irreducible by Assumption 3. ∎

Proposition 2 implies that the networked model in (2) has a unique healthy
equilibrium when $\alpha(-D+H_{\min})<0$, where the susceptible and vaccinated
population within the community will reach a balance at a unique ratio, which
is determined by the model parameters.

###### Lemma 5

The infection subsystem (2c) is locally asymptotically stable around
$x^{*}=0$, if $\rho(\widetilde{G}^{-1}\widetilde{S}^{*}B)<1$.

###### Proof:

Considering the Jacobian matrix of (2c), we have that if the spectral abscissa
of $\widetilde{S}(t)B-\widetilde{G}$ is less than $0$, then (2c) is locally
asymptotically stable around $x^{*}=0$. Note that $\widetilde{S}^{*}$ is a
positive diagonal matrix and $B$ is irreducible by Assumption 3. The result
then follows as a direct application of [19, Prop. 1] to (2c). ∎

Since $S^{*}\leq 1_{n}$, $\rho(\widetilde{G}^{-1}B)<1$ implies
$\rho(\widetilde{G}^{-1}\widetilde{S}^{*}B)<1$ and in turn implies local
asymptotic stability of the infection subsystem (2c). We define
$\mathcal{R}_{0}=\rho(\widetilde{G}^{-1}B)$ as the basic reproduction number
and $\mathcal{R}_{t}=\rho(\widetilde{G}^{-1}\widetilde{S}(t)B)$ as the
effective reproduction number of the subsystem (2c).

###  III-B Control Strategy

In this section, we consider a fixed opinion media actuator, which connects to
each node $i$ in the opinion network with strength $m_{i}(t)\in[0,\infty)$.
Equation (1a) can be rewritten as:

$\dot{o}_{i}(t)=m_{i}(t)(1-o_{i}(t))+\sum_{j\in\mathcal{N}_{i}}a_{ij}(o_{j}(t)-o_{i}(t))+a_{ij}(x_{j}(t)-o_{i}(t))$

to include the influence of the actuator, and the corresponding matrix form of
the system with the actuator input is:

$\dot{o}(t)=A(x(t)-o(t))-2L[A]o(t)+\widetilde{M}(t)(1_{n}-o(t)),$ (5)

where $\widetilde{M}(t)=diag(m(t))$.

Intuitively, if $m_{i}(t)$ are maintained on a sufficiently high level, then
the epidemic will eventually die out because individuals will be motivated to
take the vaccination due to their perception of the high disease prevalence.
The main goal of this section is to derive a lower bound on $m(t)$ that is
sufficient to guarantee the local stability of the infection subsystem (2c)
around the healthy state equilibrium. To achieve this goal, we will introduce
two concepts which we will formally characterize for system (2) with the media
actuator in (5) later in the section. They are the target vaccination
criterion and the target opinion criterion:

######  Definition 2 (Target Vaccination Criterion)

We call $v_{c}$ a target vaccination criterion, if for every $v(t)$ that
satisfies the target vaccination criterion $v_{c}$, that is, $v_{i}(t)>v_{c}$
for all $i\in[n]$ and $t\geq T$ for some $T\in\mathbb{R}_{\geq 0}$, the
healthy state equilibrium is locally asymptotically stable for the
corresponding infection subsystem (2c).

In other words, if the vaccination levels of all $n$ nodes stay above the
target vaccination criterion, $v_{c}$, when time $t$ is sufficiently large,
then the epidemic will die out. Furthermore, since the eradication solution is
trivial if $v_{c}\leq 0$ and infeasible if $v_{c}=1$ , we only consider the
case where $v_{c}\in(0,1)$.

######  Definition 3 (Target Opinion Criterion)

We define $\underline{o}_{i}^{v_{c}}$ as a target opinion criterion with
respect to $v_{c}$ if $v_{i}(t)>v_{c}$ when
$o_{i}(t)\geq\underline{o}_{i}^{v_{c}}$ for all $i\in[n]$ and $t\geq T$, where
$T\geq 0$.

The target opinion criterion is a lower bound on the opinion that guarantees
the vaccination level at node $i$ to remain above the target vaccination
criterion. The control criterion $\underline{m}^{v_{c}}$ will be expressed as
a lower bound of $m(t)$ in terms of the target opinion criterion
$\underline{o}_{i}^{v_{c}}$ such that the vaccination level at each node
$v_{i}(t)$ remains above $v_{c}$ despite loss of immunity. Before introducing
the key results, we make the following assumption for the control strategy.

###### Assumption 4

The infection transmission matrix $B$ is symmetric, i.e. $B=B^{\top}$.

The following theorem characterizes the target vaccination criterion of (2).

###### Theorem 1

Under Assumptions 3 and 4,

$v_{c}=1-\frac{1}{\rho(\widetilde{G}^{-1})\rho(B)}$ (6)

is a target vaccination criterion.

###### Proof:

Recall that under Assumption 3, $B$ is strongly connected. Since $B$ is non-
negative and irreducible, $\rho(B)>0$, by the Perron–Frobenius theorem [20].
Therefore, the inverse of $\rho(B)\in\mathbb{R}_{>0}$ exists and
$1-\frac{1}{\rho(\widetilde{G}^{-1})\rho(B)}$ is well-defined.

After showing that $v_{c}$ is well-defined, we will show that for any
arbitrary $v(t)$ satisfying the target vaccination criterion, the healthy
state equilibrium is locally asymptotically stable for the corresponding
infection subsystem (2c), consistent with Definition 2. Consider an arbitrary
$v(t)$ such that for all $i\in[n]$:

$\displaystyle v_{i}(t)$
$\displaystyle>1-\frac{1}{\rho(\widetilde{G}^{-1})\rho(B)}.$

Therefore, for all $i\in[n]$,

$\displaystyle\left(\rho(\widetilde{G}^{-1})\rho(B)\right)^{-1}$
$\displaystyle>1-v_{i}(t).$ (7)

Since (7) holds for all $i\in[n]$,

$\displaystyle\left(\rho(\widetilde{G}^{-1})\rho(B)\right)^{-1}$
$\displaystyle>\max\\{1-v_{i}(t):i\in[n]\\}$
$\displaystyle\Rightarrow\left(\rho(\widetilde{G}^{-1})\rho(B)\right)^{-1}$
$\displaystyle>\rho(I_{n}-\widetilde{V}(t)).$

By Lemma 1, $\widetilde{S}(t)\leq I_{n}-\widetilde{V}(t)$. Therefore,

$1>\rho(\widetilde{G}^{-1})\rho(\widetilde{S}(t))\rho(B).$

By the log-majorization theorem [21, Thm 5.12], we have
$\sigma(A)\sigma(B)\geq\sigma(AB)$, where $\sigma(M)$ is the singular value of
$M$. Also note that $\rho(A)\leq\sigma(A)$ and equality holds if $A$ is
symmetric. Therefore, since $\widetilde{G}^{-1}$ and $\widetilde{S}^{*}$ are
diagonal and hence symmetric,
$\rho(\widetilde{G}^{-1})\rho(\widetilde{S}(t))=\sigma(\widetilde{G}^{-1})\sigma(\widetilde{S}(t))\geq\sigma(\widetilde{G}^{-1}\widetilde{S}(t))=\rho(\widetilde{G}^{-1}\widetilde{S}(t))$.
Similarly, since $B$ is symmetric under Assumption 4,

$\displaystyle\rho(\widetilde{G}^{-1}S(t))\rho(B)$
$\displaystyle=\sigma(\widetilde{G}^{-1}S(t))\sigma(B)$
$\displaystyle\geq\sigma(\widetilde{G}^{-1}\widetilde{S}(t)B)$
$\displaystyle\geq\rho(\widetilde{G}^{-1}\widetilde{S}(t)B),$

leading to the following conclusion:

$1>\rho(\widetilde{G}^{-1}\widetilde{S}(t)B),$ (8)

which is the effective reproduction number of the infection subsystem for all
time $t\geq T$. It follows from Lemma 5 that the sub-system (2c) is locally
asymptotically stable around $x^{*}=0_{n}$ if (8) holds for all $t\geq T$ for
some $T\in\mathbb{R}_{\geq 0}$. Therefore, $v_{c}$ is a target vaccination
criterion. ∎

Note that in the single-node case ($n=1$),
$v_{c}=1-\frac{1}{\mathcal{R}_{0}}$, which is consistent with the literature
[22].

In the next lemma, we introduce the target opinion criterion
$\underline{o}^{v^{*}}$, corresponding to some target vaccination goal $v^{*}$
and its corresponding control signal $\underline{m}^{v^{*}}$. Notice that the
target vaccination goal $v^{*}$ can be any constant such that $v_{c}\leq
v^{*}<1$.

###### Lemma 6

If

$\underline{m}^{v^{*}}=(I-\widetilde{\underline{O}}^{v^{*}})^{-1}(A+2L[A])\underline{o}^{v^{*}},$
(9)

where:

$\underline{o}^{v^{*}}=k^{-1}[\Delta H](\frac{\delta}{1-v^{*}}-k[H_{\min}]),$
(10)

then $(\underline{o}^{v^{*}},(1-v^{*})1_{n},0_{n},0_{n},v^{*}1_{n})$ is the
unique healthy state equilibrium of system (2) with the media actuator in (5).

###### Proof:

The existence of the healthy state equilibrium can be shown by setting the
state values in system (2) with the media actuator in (5) equal to
$(\underline{o}^{v^{*}},(1-v^{*})1_{n},0_{n},0_{n},v^{*}1_{n})$,
$m(t)=\underline{m}^{v^{*}}$ and observing that
$\dot{o}=\dot{s}=\dot{x}=\dot{r}=\dot{v}=0_{n}$. The uniqueness of the healthy
state equilibrium can be shown by assuming, by way of contradiction, that
there exists a $p^{*}>\underline{o}^{v^{*}}$ such that:

$0_{n}=A(0_{n}-p^{*})-2L[A]p^{*}+\widetilde{\underline{M}}^{v^{*}}.$ (11)

Moving terms around in (11) and using the definition of
$\widetilde{\underline{M}}^{v^{*}}$ in (9), we obtain the following:

$(A+2L[A])p^{*}=(I-\widetilde{P}^{*})(I-\widetilde{\underline{O}}^{v^{*}})^{-1}(A+2L[A])\underline{o}^{v^{*}}.$

Since $p^{*}>\underline{o}^{v^{*}}$, which implies
$(I-\widetilde{P}^{*})(I-\widetilde{\underline{O}}^{v^{*}})^{-1}<I$, we obtain
$(A+2L[A])p^{*}<(A+2L[A])\underline{o}^{v^{*}}$ by replacing
$(I-\widetilde{P}^{*})(I-\widetilde{\underline{O}}^{v^{*}})^{-1}$ with $I$.
Furthermore, since $A+2L[A]$ is non-singular by the Gershgorin circle theorem,
we multiply both sides by $(A+2L[A])^{-1}$ and conclude that
$p^{*}<\underline{o}^{v^{*}}$, which is a contradiction. Therefore,
$p^{*}>\underline{o}^{v^{*}}$ does not hold. Without loss of generality, the
opposite direction where $p^{*}<\underline{o}^{v^{*}}$ can also be rejected
with a similar argument, leading to the conclusion that
$p^{*}=\underline{o}^{v^{*}}$. Therefore, since $\underline{o}^{v^{*}}$ is the
unique solution of (5) when $\dot{o}=0_{n}$ and
$\widetilde{M}(t)=\widetilde{\underline{M}}^{v^{*}}$, and $H(o)$ is
irreducible by definition in Section II and Assumption 3 for all
$o\in[0,1]^{n}$, we can replace $H_{\min}$ with $H(\underline{o}^{v^{*}})$ in
(4), and then it follows from the same argument in the proof of Proposition 2
that $(\underline{o}^{v^{*}},(1-v^{*})1_{n},0_{n},0_{n},v^{*}1_{n})$ is the
unique healthy state equilibrium of system (2). ∎

Note that Lemma 6 does not specify that the value of $v^{*}$ needs to be
greater than $v_{c}$. Instead, the lemma introduces a tracking scheme for any
vaccination goal $v^{*}\in[0,1]$. A natural consequence of this lemma is that
the system will return to its original equilibrium when the control signal is
relaxed. Furthermore, we will see that not all vaccination goals are feasible
via opinion control. The feasibility of the control signal depends on the
upper and lower bounds of the opinions. that is,
$\underline{o}_{i}^{v^{*}}\in[0,1]$ $\forall i\in[n]$. The next lemma
characterizes the constraints in terms of the system parameters and the
vaccination goal $v^{*}$.

###### Lemma 7

We have $0_{n}\leq\underline{o}_{i}^{v^{*}}\leq 1_{n}$ if and only if
$(1-v^{*})k_{i}(H_{\min})\leq\delta_{i}\leq(1-v^{*})k_{i}(H)$.

###### Proof:

The following inequalities are equivalent by (10):

$\displaystyle 0$ $\displaystyle\leq\underline{o}_{i}^{v^{*}}\leq 1$
$\displaystyle 0$ $\displaystyle\leq k_{i}^{-1}[\Delta
H]\left(\frac{\delta_{i}}{1-v^{*}}-k_{i}[H_{\min}]\right)\leq 1$
$\displaystyle 0$
$\displaystyle\leq\frac{\delta_{i}}{1-v^{*}}-k_{i}[H_{\min}]\leq k_{i}[\Delta
H]$ $\displaystyle(1-v^{*})k_{i}[H_{\min}]$
$\displaystyle\leq\delta_{i}\leq(1-v^{*})k_{i}[H],$

which concludes the proof. ∎

Lemma 7 shows that only a subset of systems of the form in (2) can be
controlled via opinions, that is, there exist parameterizations of (2) such
that the vaccination goal $v^{*}\geq v_{c}$ can be achieved only if
$\underline{o}_{i}^{v^{*}}>1$ or $\underline{o}_{i}^{v^{*}}<0$. While there is
flexibility in choosing $v^{*}$, the fundamental constraint lies in the
relationship between the rate of loss of immunity from vaccination
$\delta_{i}$, the target vaccination criterion $v_{c}$, and the maximum rate
of vaccine-uptake $k_{i}[H]$. If the ratio between the rate of loss of
immunity from vaccination and the maximum rate of vaccine-uptake is greater
than $1-v_{c}$, then no control signals of the form in (5) are guaranteed to
eradicate the disease, unless we have a more relaxed $v_{c}$ that guarantees
stability around the healthy state equilibrium of the system.

The next theorem shows that the suggested control law can provide local
stability around the healthy state equilibrium of system (2) with a media
actuator (5).

###### Theorem 2

If $0_{n}\ll\underline{o}^{v^{*}}\ll 1_{n}$ and $m(t)=\underline{m}^{v^{*}}$
as defined in (10) and (9), respectively, then for all $v^{*}\geq v_{c}$ (2)
with media actuator (5) is locally asymptotically stable around the healthy
state equilibrium
$(\underline{o}^{v^{*}},(1-v^{*})1_{n},0_{n},0_{n},v^{*}1_{n})$.

###### Proof:

Since $s(t)=1_{n}-x(t)-r(t)-v(t)$ by Lemma 1, we can linearize (2) around the
healthy state equilibrium as follows, without including the dynamics of
$s(t)$:

$\begin{bmatrix}\dot{x}\\\ \dot{o}\\\ \dot{r}\\\
\dot{v}\end{bmatrix}=\begin{bmatrix}J_{x}&0&0&0\\\ A&J_{o}&0&0\\\
\widetilde{G}&0&J_{r}&0\\\
0&v^{*}(1-v^{*})I&v^{*}\widetilde{K}[H(\underline{o})]&J_{v}\end{bmatrix}\begin{bmatrix}x\\\
o\\\ r\\\ v\end{bmatrix}$

where $J_{o}=-(A+2L[A]+\widetilde{\underline{M}}^{v^{*}})$,
$J_{x}=(1-v_{c})B-\widetilde{G}$,
$J_{r}=-(\widetilde{W}+v^{*}K[H(\underline{o}^{v^{*}})])$, and
$J_{v}=(1-v^{*})H(\underline{o}^{v^{*}})-\widetilde{D}$.

The linearized system is stable if and only if each diagonal block is stable
because the Jacobian is lower triangular. By Theorem 1, the infection
subsystem $J_{x}$ is locally asymptotically stable around $x^{*}=0_{n}$ for
all $v^{*}\geq v_{c}$. Moreover, we notice that each Gershgorin disc of
$J_{o}$ has a center at $-(\underline{m}_{i}^{v^{*}}+2k_{i}[A])$ with radius
$k_{i}[A]$, and $\underline{m}_{i}^{v^{*}}$ is non-negative. Therefore, the
spectral abscissa of $J_{o}$ lies on the open left half plane by the
Gershgorin circle theorem. Furthermore, $J_{r}$ is locally asymptotically
stable by observing that $-(\widetilde{W}+v^{*}K[H(\underline{o}^{v^{*}})])$
is a negative diagonal matrix.

Lastly, an upper bound of the spectral abscissa of $J_{v}$ can be obtained by
replacing $\widetilde{D}$ with $(1-v^{*})k_{i}[H_{\min}]$, because Lemma 7
states that $\widetilde{D}\gg(1-v^{*})\widetilde{K}(H_{\min})$ if
$\underline{o}^{v^{*}}\gg 0_{n}$. By assumption, $\underline{o}^{v^{*}}\gg
0_{n}$ holds. Then, by shifting the center of each Gershgorin disc to the
left, we obtain:

$\displaystyle\alpha(J_{v})$
$\displaystyle=\alpha((1-v^{*})H(\underline{o}^{v^{*}})-\widetilde{D})$
$\displaystyle<\alpha((1-v^{*})H(\underline{o}^{v^{*}})-(1-v^{*})\widetilde{K}[H_{\min}])$
$\displaystyle=\alpha\left((1-v^{*})\left[H(\underline{o}^{v^{*}})-\widetilde{K}[H_{\min}]\right]\right).$
(12)

By replacing each term with its corresponding definition:
$H(\underline{o}^{v^{*}}):=\widetilde{\underline{O}}^{v^{*}}\Delta
H+(H_{\min}-K[H_{\min}])$ and
$\widetilde{\underline{O}}^{v^{*}}:=\widetilde{K}^{-1}[\Delta
H](\frac{1}{1-v^{*}}\widetilde{D}-\widetilde{K}[H_{\min}])$, we have the
following:

$H(\underline{o}^{v^{*}})=\left(\frac{1}{1-v^{*}}\widetilde{D}-\widetilde{K}[H_{\min}]\right)\widetilde{K}^{-1}[\Delta
H]+H_{\min}.$ (13)

Notice that by replacing $\widetilde{D}$ with
$(1-v^{*})\widetilde{K}(H_{\min})$ in (13), terms cancel out and we are left
with $H(\underline{o}^{v^{*}})=H_{\min}$. Therefore, if we substitute
$H(\underline{o}^{v^{*}})$ from (13) into (III-B) and apply the inequality
$\widetilde{D}\gg(1-v^{*})$, we have:

$\displaystyle\alpha(J_{v})$
$\displaystyle<\alpha\left((1-v^{*})\left[H(\underline{o}^{v^{*}})-\widetilde{K}[H_{\min}]\right]\right)$
$\displaystyle<\alpha\left((1-v^{*})\left[H_{\min}-\widetilde{K}[H_{\min}]\right]\right).$

By the Gershgorin circle theorem,
$\alpha\left((1-v^{*})\left[H_{\min}-\widetilde{K}[H_{\min}]\right]\right)\leq
0$, which completes the proof. ∎

##  IV Simulation

In this section, we illustrate and visualize the results in Section III. There
are 4 nodes in our simulated system, where each node represents an individual
or a community. The key difference between a network of communities and a
network of individuals in a networked epidemic model is that whether we allow
self-loops in the disease transmission network. When modelling interaction
between individuals, it is most natural to assume that each individual does
not induce sickness to themselves, while the opposite is true when modelling
interaction between communities.

The infectious network, the strategy immitation network, and the opinion
network are characterized by the following matrices:

$\displaystyle B$ $\displaystyle=\left[\begin{matrix}0&0.4&0.35&0.3\\\
0.4&0&0.2&0.25\\\ 0.35&0.2&0&0.2\\\ 0.3&0.25&0.2&0\\\
\end{matrix}\right],\leavevmode\nobreak\
H=\left[\begin{matrix}0&0.4&&0.2&0.5\\\ 0.7&0&&0.8&1\\\ 1&0.5&&0&0.7\\\
1&0.8&&0.5&0\\\ \end{matrix}\right].$

Other parameters are defined as follows:

  * •

$\gamma=[0.4,1.6,0.8,0.3]$;

  * •

$\omega=[0.4,0.5,0.6,0.7]$;

  * •

$\delta=[0.1,0.2,0.2,0.1]$;

  * •

$\eta^{\min}_{ij}=0.1$, $\forall i\neq j\in[n]$;

  * •

$\alpha_{ij}=0.1$, $\forall i,j\in[n]$.

The initial conditions of the system:

$(s(t_{0}),x(t_{0}),r(t_{0}),v(t_{0}),o(t_{0})),$

are $((0,0.2,0.3,0.2)$, $(0.2,0.4,0.2,0.3)$, $(0.6,0.3,0.5,0.5)$,
$(0.2,0.1,0,0)$, $(0,0.1,0.9,1))$, respectively, where $t_{0}=0$ in the
following simulations.

Figure 1 illustrates the existence of the healthy state equilibrium with a
non-zero vaccination equilibrium which is formally presented in Proposition 2.
We change the initial conditions (Figure 1 (b)) to:
$s(t_{0})=(0.9,0.2,0.5,0.4)$, $x(t_{0})=(0,0.4,0.2,0.1)$,
$r(t_{0})=(0.1,0,0,0)$, $v(t_{0})=(0,0.4,0.3,0.5)$, and $o(t_{0})$ $=$
$(0,0.1,$ $0.9,1)$. Figure 1 shows that $v^{*}\gg 0$ is invariant to a single
perturbation in the initial states. According to Lemma 3, at a healthy
equilibrium, if $v^{*}$ is unique, $s^{*}$, $r^{*}$, and $x^{*}$ are also
unique.

(a) $v(t)$ with the original
initial condition.

(b) $v(t)$ with the second
initial condition.

Figure 1: The existence and uniqueness of the healthy state equilibrium,
illustrating Proposition 2. Only the vaccination levels over time are shown.

To show the impact of the proposed controller, we first construct a system
with an endemic state equilibrium, which is as an equilibrium where
$x^{*}>0_{n}$, by setting $\gamma=[0.2,0.8,0.4,0.15]$. Other parameters are
the same as in the previous setting. Note $B$ is symmetric in the original
setup. Figure 2 shows the dynamics of the system without the control strategy.

Figure 2: The system with an endemic state equilibrium.

Then, we apply the control strategy based on Theorem 2. If there exists a
$t_{1}^{i}$ such that $o_{i}(t_{1}^{i})<\underline{o}_{i}^{v_{c}}$ defined in
(10), then $m_{i}(t)=\underline{m}_{i}^{v_{c}}$ defined in (9), $\forall
t>t_{1}^{i}$, and $m_{i}(t)=0$, $\forall t<t_{1}^{i}$, where $t_{1}^{i}$
represents when the control applied on Node $i$ starts. In our case, as shown
by Figure 3, $t_{1}^{[1-3]}$ are $0$, $0$, $2.29$, respectively. Node 4 is
never connected to the actuator, since
$o_{4}(t)\geq\underline{o}_{i}^{v_{c}}$, $\forall t>t_{0}$. In Figure 3, we
can see that the system with the applied control strategy results in a healthy
state equilibrium. Note, we need to maintain the control input in order to
ensure that the healthy state equilibrium remains stable. Figure 4 shows that
the healthy state equilibrium is no longer stable if we remove the media
actuator at $t_{2}=74$, and the system returns to its natural endemic
equilibrium. As indicated by the red dashes in Figure 4, the system no longer
maintains the healthy state equilibrium $\forall t>t_{2}$.

Figure 3: The system with the control strategy applied.

Figure 4: The healthy state equilibrium is unstable after ending the control.
The vertical red dotted line indicates the time when the control signal is
lifted.

##  V Conclusion

In this work, we propose a networked $SIRS-V_{o}$ model to study the
interaction between opinion dynamics, self-interest decision-making, vaccine
uptake, and disease spreading process. We provide in this paper a proof on the
existence of the healthy state equilibrium of the networked $SIRS-V_{o}$
model, and show that the problem can be reduced to the equilibrium analysis of
a single viral networked SIS model. We then characterize the local stability
condition of the infection subsystem of the networked multi-layer epidemic
model around the healthy state equilibria. Building on this model, we develop
target vaccination and opinion criteria which guarantee disease eradication.
These criteria provide sufficient conditions that leverage opinions and
vaccination to eradicate the spread of an epidemic. We then show analytically
that the family of control signals which satisfies the target vaccination and
opinion criteria stabilizes the networked $SIRS-V_{o}$ model locally around
the healthy state equilibrium under some mild assumptions. Among these mild
assumptions, the requirement that the target vaccination criterion is feasible
at node $i$ only when the ratio between the rate of loss of immunity
$\delta_{i}$ and the maximum rate of vaccine-uptake $k_{i}[H]$ is greater than
$1-v_{c}$ gives us insight into the limitation of epidemic control through the
influence of a centralized media actuator. We also note that the infection
subsystem returns to its natural equilibrium after the control algorithm is
relaxed, both analytically and in simulations. In the future, we will extend
the control analysis to study robust government-subsidized vaccination
programs for disease control under adversarial attacks on opinions through
fake news or other mediums.

## References

  * [1] W. O. Kermack and A. G. McKendrick, “A contribution to the mathematical theory of epidemics,” _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 115, pp. 700–721, 08 1927.
  * [2] V. Capasso and G. Serio, “A generalization of the Kermack-McKendrick deterministic epidemic model,” _Mathematical Biosciences_ , vol. 42, no. 1-2, pp. 43–61, 1978.
  * [3] B. She, H. C. Leung, S. Sundaram, and P. E. Paré, “Peak infection time for a networked SIR epidemic with opinion dynamics,” in _Proceedings of the 60th IEEE Conference on Decision and Control (CDC)_. IEEE, 2021, pp. 2104–2109.
  * [4] C. Leung, M. E. Gibbs, and P. E. Paré, “The impact of vaccine hesitancy on epidemic spreading,” in _Proceedings of the American Control Conference (ACC)_. IEEE, 2022, pp. 3656–3661.
  * [5] P. Van Mieghem, J. Omic, and R. Kooij, “Virus spread in networks,” _IEEE/ACM Transactions on Networking_ , vol. 17, pp. 1–14, 02 2009.
  * [6] W. Mei, S. Mohagheghi, S. Zampieri, and F. Bullo, “On the dynamics of deterministic epidemic propagation over networks,” _Annual Reviews in Control_ , vol. 44, pp. 116–128, 2017.
  * [7] P. E. Paré, C. L. Beck, and T. Başar, “Modeling, estimation, and analysis of epidemics over networks: An overview,” _Annual Reviews in Control_ , vol. 50, pp. 345–360, 2020.
  * [8] J. S. Weitz, S. W. Park, C. Eksin, and J. Dushoff, “Awareness-driven behavior changes can shift the shape of epidemics away from peaks and toward plateaus, shoulders, and oscillations,” _Proceedings of the National Academy of Sciences_ , vol. 117, p. 32764–32771, 12 2020.
  * [9] M. J. Keeling and P. Rohani, _Modeling Infectious Diseases in Humans and Animals_. Princeton University Press, 2008\.
  * [10] M. Taylor, “Towards a mathematical theory of influence and attitude change,” _Human Relations_ , vol. 21, no. 2, pp. 121–139, 1968.
  * [11] S. Boccaletti, G. Bianconi, R. Criado, C. I. Del Genio, J. Gómez-Gardenes, M. Romance, I. Sendina-Nadal, Z. Wang, and M. Zanin, “The structure and dynamics of multilayer networks,” _Physics Reports_ , vol. 544, no. 1, pp. 1–122, 2014.
  * [12] M. Salehi, R. Sharma, M. Marzolla, M. Magnani, P. Siyari, and D. Montesi, “Spreading processes in multilayer networks,” _IEEE Transactions on Network Science and Engineering_ , vol. 2, no. 2, pp. 65–83, 2015.
  * [13] Z. Wang, C. T. Bauch, S. Bhattacharyya, A. d’Onofrio, P. Manfredi, M. Perc, N. Perra, M. Salathé, and D. Zhao, “Statistical physics of vaccination,” _Physics Reports_ , vol. 664, pp. 1–113, 2016.
  * [14] F. Fu, D. I. Rosenbloom, L. Wang, and M. A. Nowak, “Imitation dynamics of vaccination behaviour on social networks,” _Proceedings of the Royal Society B: Biological Sciences_ , vol. 278, no. 1702, pp. 42–49, 2011.
  * [15] E. Shim, G. B. Chapman, J. P. Townsend, and A. P. Galvani, “The influence of altruism on influenza vaccination decisions,” _Journal of The Royal Society Interface_ , vol. 9, no. 74, pp. 2234–2243, 2012.
  * [16] C. T. Bauch, “Imitation dynamics predict vaccinating behaviour,” _Proceedings of the Royal Society B: Biological Sciences_ , vol. 272, no. 1573, pp. 1669–1675, 2005.
  * [17] J. Hofbauer, K. Sigmund, _et al._ , _Evolutionary Games and Population Dynamics_. Cambridge University Press, 1998\.
  * [18] R. F. Brown, _A Topological Introduction to Nonlinear Analysis_. Springer, 1993.
  * [19] J. Liu, P. E. Paré, A. Nedić, C. Y. Tang, C. L. Beck, and T. Başar, “Analysis and control of a continuous-time bi-virus model,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 12, pp. 4891–4906, 2019.
  * [20] R. Bapat, “A max version of the Perron-Frobenius theorem,” _Linear Algebra and its Applications_ , vol. 275-276, pp. 3–18, 05 1998.
  * [21] T. Ando, “Majorizations and inequalities in matrix theory,” _Linear Algebra and its Applications_ , vol. 199, pp. 17–67, 1994.
  * [22] P. R. Matthew James Keeling, _Modeling Infectious Diseases in Humans and Animals_. Princeton University Press, 2008\.


# Remote Electrical Tilt Optimization via Safe Reinforcement Learning

Filippo Vannella12§*  |, Grigorios Iakovidis12§*  |, Ezeddin Al Hakim2*  |, Erik Aumayr3*  |, Saman Feghhi3*  |
1KTH Royal Institute of Technology, Stockholm, Sweden
2Ericsson Research, Stockholm, Sweden
3Network Management Research Lab, LM Ericsson, Athlone, Ireland
Email: {vannella<EMAIL_ADDRESS>{ezeddin.al.hakim, erik.aumayr,
<EMAIL_ADDRESS>
###### Abstract

Remote Electrical Tilt (RET) optimization is an efficient method for adjusting
the vertical tilt angle of Base Stations (BSs) antennas in order to optimize
Key Performance Indicators (KPIs) of the network. Reinforcement Learning (RL)
provides a powerful framework for RET optimization because of its self-
learning capabilities and adaptivity to environmental changes. However, an RL
agent may execute unsafe actions during the course of its interaction, i.e.,
actions resulting in undesired network performance degradation. Since the
reliability of services is critical for Mobile Network Operators (MNOs), the
prospect of performance degradation has prohibited the real-world deployment
of RL methods for RET optimization. In this work, we model the RET
optimization problem in the Safe Reinforcement Learning (SRL) framework with
the goal of learning a tilt control strategy providing performance improvement
guarantees with respect to a safe baseline. We leverage a recent SRL method,
namely Safe Policy Improvement through Baseline Bootstrapping (SPIBB), to
learn an improved policy from an offline dataset of interactions collected by
the safe baseline. Our experiments show that the proposed approach is able to
learn a safe and improved tilt update policy, providing a higher degree of
reliability and potential for real-world network deployment.

This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible. §§footnotetext: Equal contribution.

##  I Introduction

Network performance optimization represents one of the major challenges for
Mobile Network Operators (MNOs). Due to the growing complexity of modern
networks, MNOs increasingly face the need to satisfy consumer demand that is
highly variable in both the spatial and temporal domains. In order to provide
a high level of Quality of Service (QoS) efficiently to each User Equipment
(UE), networks must adjust their configuration in an automatic and timely
manner. The antenna downtilt, defined as the inclination angle of the
antenna’s radiating beam with respect to the horizontal plane, plays a crucial
role in the network performance optimization. Remote Electrical Tilt (RET)
optimization is a powerful technique to remotely control the downtilt of Base
Stations (BSs) antennas in order to optimize Key Performance Indicators (KPIs)
while efficiently utilizing network resources.

In this work, we focus on RET optimization techniques using Reinforcement
Learning (RL) methods [1]. RL is a decision-making paradigm used to solve
complex problems by direct interaction with an environment. RL offers an
effective framework for RET optimization due to its self-learning capabilities
and adaptivity to environmental changes. Because of these potentialities,
during the past decade, there has been a significant amount of research into
the field of RET optimization using RL methods [2, 3, 4, 5]. Such RL methods
have shown great promise by outperforming traditional control-based [6] and
Rule-Based (RB) [7] approaches.

However, RL methods for RET optimization ignore their inherent unsafe nature.
During the RL interaction, the agent is faced with a fundamental dilemma: it
must decide whether to prioritize further gathering of information
(exploration) or to execute the best action given the current knowledge
(exploitation). The primary source of the unsafety of RL methods can be
imputed to the agent’s exploration, resulting in undesired network performance
degradation. Since the reliability of services is a critical feature for MNOs,
the prospect of performance degradation has prohibited the real-world
deployment of RL methods for RET optimization, limiting their use to simulated
network environments. For this reason, safety has become an essential feature
in RET optimization approaches using RL techniques, since it represents the
primary obstacle for the deployment of such techniques, and precludes the
possibility to achieve improved network performance.

Safe Reinforcement Learning (SRL) aims at solving RL problems in which a
minimum performance level must be guaranteed during learning and deployment.
Recently, there have been many attempts in the field of SRL to design safe RL
algorithms. We refer the interested reader to [8] for a complete survey on SRL
[8]. We focus on SRL methods falling in the domain of Offline RL [9, 10, 11],
in which the goal is to devise an optimal policy from offline data, without
further interaction with the real environment. The dataset is collected by a
baseline policy, considered to be safe and sub-optimal.

In this work, we formulate the RET optimization problem in the SRL framework
and apply an SRL method to devise a safe downtilt update policy. In
particular, we leverage a recent Offline RL method, namely Safe Policy
Improvement with Baseline Bootstrapping (SPIBB) [11], in order to learn a
policy having performance improvement guarantees over a baseline policy. We
empirically evaluate the SRL approach to RET optimization for different types
of baseline policies and over different worst-case scenarios. The outcomes of
the experiments show that the proposed approach is able to learn a safe and
improved downtilt update policy, providing a higher degree of reliability and
potential for real-world network deployment.

The remainder of the paper is organized as follows: section II reviews related
works for RET optimization, highlighting the lack of safety of RL methods and
the shortcomings of SRL methods. Section III introduces the system model for
RET optimization, and the background on RL and SRL. Section IV introduce the
RET optimization problem formulation as an SRL problem and illustrates the use
of SPIBB for RET optimization. Section V presents the experimental setup, the
evaluation methodology, reports the results of the experiments. Finally,
Section VI contains our concluding thoughts.

##  II Related work

A wide range of RL approaches for the RET optimization problem have been
explored in the literature. For example, the authors of [5] tackle RET
optimization using a $Q$-learning method with linear function approximation,
and the authors of [3] employ an RL technique based on Boltzmann exploration
to maximize the throughput fairness while increasing energy efficiency.
Another class of methods combines RL and Fuzzy Logic in the Fuzzy RL (FRL)
framework. Razavi et al. [2] propose an FRL method that utilizes fuzzy logic
to model the intrinsic uncertainty of the states in cellular networks. Fan et
al. [4] apply FRL, introducing cooperative learning between different cells in
a network, resulting in faster and more efficient training. Despite having
good performance, both RL and FRL methods utilize traditional exploration-
exploitation strategies that do not preclude the possibility of selecting
unsafe actions, failing to provide any performance guarantee.

The authors of [12] address the RET optimization problem in the off-policy
Contextual Multi-Armed Bandit (CMAB) formulation. The goal is to learn a RET
policy completely offline from real-world network data. However, the CMAB
formulation imposes severe approximations to the RET optimization model, and
the off-policy learning performance is highly susceptible to the data quality.
In [13], the authors consider an SRL method based on the use of an actor-
critic model executing only actions for which the probability is
$\varepsilon$-close in Euclidean distance to the actions of a safe baseline
policy. The constant $\varepsilon$ acts as a safety threshold and is
progressively adjusted during training. However, this method may be
excessively conservative if the safety threshold $\varepsilon$ is not
correctly tuned, and it does not have convergence guarantees.

##  III Background

In this section, we introduce the network environment model considered for RET
optimization, the background on the RL and SRL framework, and algorithms in
both frameworks utilized in the remainder of the paper.

Figure 1: Representation of the downtilt $\theta_{t,c}$ at time $t$ and for
cell $c$.

###  III-A Network environment model

We consider a geographical area covered by a sectorized mobile network
environment containing $\mathcal{B}$ Base Stations (BSs) and $\mathcal{C}$
cells. The network environment is populated by $\mathcal{U}$ UEs, uniformly
distributed in the considered area. Time is indexed by $t=1,2,\dots$, and
$\theta_{t,c}$, represented in Fig. 1, denotes the downtilt of the antenna at
time $t$ and for cell $c$.

The RET problem formulation used in this paper follows previous works in the
literature based on Coverage-Capacity Optimization (CCO) [7, 12]. The goal in
CCO is to maximize both network coverage and capacity, while minimizing inter-
cell interference.

The coverage KPI measures to which extent the area of interest is covered with
a minimum level of Reference Signal Received Power (RSRP). The capacity KPI
measures the amount of UEs that the cell can handle simultaneously (e.g., in
terms of calls) with a reference bit rate. We consider the network
measurements for capacity and coverage KPIs to be collected at time $t$ and
for cell $c$, and we denote them by $\textsc{Cov}_{t,c}$ and
$\textsc{Cap}_{t,c}$ respectively. The downtilt determines a trade-off between
coverage and capacity: low values of downtilt result in a larger covered area
and smaller capacity due to lower signal power received. In contrast, high
values of downtilt result in a smaller covered area and the possibility of
creating coverage holes in the cells, but leading to increased capacity. The
joint optimization of these mutually opposed KPIs ensures that the targeted
area remains covered while maximizing the capacity. It is critical to note
that the cell’s performance is strongly influenced by the downtilt of
neighboring cells, that introduce coupled interference between cells. In our
formulation, the effect of negative interference from neighboring cells is
modeled by the quality KPI, denoted as $\textsc{Qual}_{t,c}$ at time $t$ and
for cell $c$.

In this paper, we consider the class of RET control strategies whose input
consists of the downtilt, coverage, capacity, and quality KPIs for cell $c$
and time step $t$, and whose output consists of one between the actions
increase the downtilt, decrease the downtilt, or keep the same downtilt. The
RET control strategy manages the antenna tilt on a cell-by-cell basis and is
global, i.e., the same control strategy is executed independently for all
cells $c=1,\dots,\mathcal{C}$.

###  III-B The RL framework

RL [1] is a powerful algorithmic framework in which an agent learns an optimal
control strategy by direct interaction with an unknown environment. The RL
agent iteratively observes a representation of the state of the environment,
executes an action, receives a scalar reward, and transitions to a new state.
The goal of the agent is to learn a policy maximizing the expected cumulative
reward.

Formally, an RL problem is defined by a Markov Decision Process (MDP),
described by a tuple $\langle\mathcal{S},\mathcal{A},R,P,\gamma,p_{0}\rangle$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$R(\cdot|s,a)$ is the reward probability distribution, $P(\cdot|s,a)$ is the
transition probability distribution, $\gamma$ is the discount factor, and
$p_{0}$ is the initial state distribution. We assume the state space is
continuous $\mathcal{S}\subseteq\mathbb{R}^{d}$ and the action space
$\mathcal{A}$ is discrete.

The policy of the agent $\pi(\cdot|s):\mathcal{S}\to\Delta_{\mathcal{A}}$ is a
mapping from states to probability distributions over actions※※※This
formulation includes the case of a deterministic policy
$\pi:\mathcal{S}\to\mathcal{A}$., defining the agent’s control strategy. The
RL interaction proceeds as follows: at each time instant $t=1,2,\dots$, the
agent receives a representation of the environment state
$s_{t}\in\mathcal{S}$, selects an action
$a_{t}\sim\pi(\cdot|s_{t})\in\mathcal{A}$, receives a stochastic reward
${r_{t}\sim R\left(\cdot|s_{t},a_{t}\right)}$, and transitions to a new state
${s_{t+1}\sim P\left(\cdot\middle|s_{t},a_{t}\right)}$. The goal of the agent
is to devise an optimal policy, i.e., a policy maximizing the expected
cumulative reward over a predefined period of time.

###  III-C $Q$-learning and Deep $Q$-Network

$Q$-learning [14] is one of the most popular RL algorithms. It aims at
estimating the state-action value function (a.k.a. $Q$-function) when
following policy $\pi$, defined as

$Q^{\pi}\left(s,a\right)=\mathbb{E}_{\pi}\left[\sum_{\tau=0}^{\infty}{\gamma^{\tau}r_{t+\tau+1}\Big{|}s_{t}=s,a_{t}=a}\right],$

where $\mathbb{E}_{\pi}[\cdot]$ denotes the expected value operator with
respect to actions sampled by policy $\pi$. In its tabular form the
$Q$-function is stored as matrix whose entries
$(s,a)\in\mathcal{S}\times\mathcal{A}$ correspond to the $Q$-value of a state-
action pair. The learning is performed by direct interaction with the
environment: at each time step, an action $a_{t}$ is executed in a state
$s_{t}$, and the state-action value is updated as

$Q\left(s_{t},a_{t}\right)\leftarrow
r_{t+1}+\gamma\max_{a^{\prime}}Q\left(s_{t+1},a^{\prime}\right).$

In the case of continuous state space, the $Q$-function is parametrized
through a function approximator $Q_{w}(s,a)$, where $w\in\mathbb{R}^{\Omega}$
is the parameter vector, allowing to generalize the $Q$-function
representation to unseen states. When an Artificial Neural Network (ANN)
parametrization is chosen, the $Q$-learning algorithm is also known as Deep
$Q$-Network (DQN) [15]. DQN usually makes use of experience replay, a
biologically inspired learning technique that consists in storing trajectories
of experience in a dataset and reusing them in future training steps in order
to remove the correlation between the samples. DQN uses optimization methods
(e.g. Stochastic Gradient Descent (SGD)) to minimize the Mean Squared Error
(MSE) between the target
${y_{t}=r_{t}+\gamma\max_{a^{\prime}}Q_{w}(s^{\prime}_{t},a^{\prime})}$ and
the parametrized $Q$-function $Q_{w}(s,a)$. The weight update step is executed
as

$w\leftarrow
w-\eta\cdot\nabla_{w}\left(y_{t}-Q_{w}\left(s_{t},a_{t}\right)\right)^{2},$

where $\nabla_{w}$ denotes the gradient operator with respect to the weight
vector $w$ and $\eta$ denotes the learning rate.

###  III-D SPIBB-DQN

In the context of RL, an optimal policy is usually learned in a trial-and-
error fashion by direct interaction with the environment. In the course of
such interaction, the agent will explore sub-optimal regions of the state-
action space. In some domains, sub-optimal exploration may result in
unacceptable risky circumstances or performance degradation. DQN-based
algorithms have shown superior performance in many applications. However, due
to the inherent exploratory nature of $Q$-learning policies, DQN fails to
provide safety guarantees.

One of the most effective ways to achieve safety in RL is to rely on a safe
baseline policy $\pi_{b}$ and on the baseline dataset
$\mathcal{D}_{\pi_{b}}=\\{(s_{n},a_{n},r_{n},s^{\prime}_{n})\\}_{n=1}^{N}$ of
previous interactions of $\pi_{b}$ with the environment. The goal is to learn
a policy $\pi$ having better performance (e.g. in terms of Q-function) with
respect to the baseline policy $\pi_{b}$ in an offline manner from
$\mathcal{D}_{\pi_{b}}$, and without further interaction with the environment.

We make use of the SPIBB method, as presented in [11]. The high-level idea of
SPIBB is to rely on the guidance provided by a safe baseline policy $\pi_{b}$
for state-action pairs $(s,a)$ that are rarely observed in the baseline
dataset $\mathcal{D}_{\pi_{b}}$. Specifically, the method first constructs a
bootstrapped set containing a set of state action pairs
${(s,a)\in\mathcal{S}\times\mathcal{A}}$ that have been observed with a low
frequency in $\mathcal{D}_{\pi_{b}}$. Formally, let
$N_{\mathcal{D}_{\pi_{b}}}(s,a)=\sum_{n=1}^{N}\mathbbm{1}\\{s_{n}=s,a_{n}=a\\}$
be the state-action count of dataset $\mathcal{D}_{\pi_{b}}$, and define the
bootstrapped set as

$\mathfrak{B}_{\pi_{b}}=\\{(s,a)\in\mathcal{S}\times\mathcal{A}:N_{\mathcal{D}_{\pi_{b}}}(s,a)\leq
N_{\wedge}\\},$ (1)

where $N_{\wedge}$ is a hyperparameter controlling the level of safety. In
particular we focus on the SPIBB-DQN algorithm, which can be considered as a
safe variation of the DQN through baseline bootstrapping. The main variation
with respect to DQN is in the target of the $Q$-function update, executed on a
batch of samples
$\\{(s_{\beta},a_{\beta},r_{\beta},s^{\prime}_{\beta})\\}_{\beta=1}^{N_{B}}$
and for update index $j$ as

$\displaystyle y^{(j)}_{\beta}$
$\displaystyle=r_{\beta}+\gamma\sum_{a^{\prime}\mid\left(s_{\beta}^{\prime},a^{\prime}\right)\in\mathfrak{B}_{\pi_{b}}}\pi_{b}\left(a^{\prime}\mid
s_{\beta}^{\prime}\right)Q^{(j)}\left(s_{\beta}^{\prime},a^{\prime}\right)$
(2)
$\displaystyle+\gamma\sum_{a^{\prime}\mid\left(s_{\beta}^{\prime},a^{\prime}\right)\notin\mathfrak{B}_{\pi_{b}}}\pi_{b}\left(a^{\prime}\mid
s_{\beta}^{\prime}\right)\max_{a^{\prime}\mid\left(s_{\beta}^{\prime},a^{\prime}\right)\notin\mathfrak{B}_{\pi_{b}}}Q^{(j)}\left(s_{\beta}^{\prime},a^{\prime}\right).$

This update consists of three elements: $(i)$ the immediate return observed in
the dataset’s recorded transition, $(ii)$ the estimated return of the
bootstrapped actions (where the agent must follow the policy of the baseline),
and $(iii)$ the maximum over the return estimates of the non-bootstrapped
actions (where the agent is free to exploit the optimal strategy).

It is easy to observe that for $N_{\wedge}\to\infty$, the bootstrapped set
contains the whole state-action space, and the training policy converges to
the baseline policy. Meanwhile, for $N_{\wedge}\to 0$, the bootstrapped set is
empty, and the SPIBB-DQN degenerates into a vanilla DQN with no safety-
preserving influence from the baseline. Thus, the safety parameter
$N_{\wedge}$ must be chosen large enough to ensure safety, and small enough to
ensure improvement.

##  IV RET Optimization via SRL

In this section, we formulate the RET optimization problem as an SRL problem,
and we illustrate how to apply the SPIBB-DQN algorithm to RET optimization.

Figure 2: Simulated mobile network in urban environment.

###  IV-A SRL formulation for RET optimization

We formulate the RET optimization as an SRL problem by specifying its
constituent elements:

  * •

Environment $\mathcal{E}$: consists of a simulated mobile network in an urban,
environment as described in Section III-A, and represented in Fig. 2.

  * •

State space $\mathcal{S}\subseteq[0,1]^{4}$: comprises normalized values of
downtilt, coverage, capacity and quality for a given cell $c$., i.e.
$s_{t,c}=[\bar{\theta}_{t,c},\textsc{Cov}_{t,c},\textsc{Cap}_{t,c},\textsc{Qual}_{t,c}]$.

  * •

Action space $\mathcal{A}=\\{-\alpha,0,\alpha\\}$: consists of discrete
variations of the current downtilt of magnitude $\alpha$.

  * •

Transition probability distribution $P(\cdot|s_{t,c},a_{t,c})$: describes the
state evolution given the current state and the executed action. It is
composed by a deterministic part and a stochastic part: the downtilt varies
deterministically as $\theta_{t+1,c}=\theta_{t,c}+a_{t,c}$, while the
transition probabilities for
$\textsc{Cov}_{t,c},\textsc{Cap}_{t,c},\textsc{Qual}_{t,c}$ are assumed to be
stochastic and unknown.

  * •

Reward
$r_{t,c}=\log(1+\textsc{Cov}_{t,c}^{2}+\textsc{Cap}_{t,c}^{2}+\textsc{Qual}_{t,c}^{2})$:
consists of the squared log-sum of the coverage, capacity, and quality KPIs.

  * •

Baseline dataset
$\mathcal{D}_{\pi_{b}}=\\{(s_{n},a_{n},r_{n},s^{\prime}_{n})\\}_{n=1}^{N}$:
consists of a collection of state, action, reward, next state tuples,
collected by a baseline policy $\pi_{b}$. More information and statistics on
$\pi_{b}$ and $\mathcal{D}_{\pi_{b}}$ are presented in Section V.

  * •

Agent policy $\pi_{w}:\mathcal{S}\to\mathcal{A}$: consists of a deterministic
policy parametrized by an ANN with weight vector $w\in\mathbb{R}^{\Omega}$. We
assume that the RL agent is trained independently across data coming from each
cell $c=1,\dots,\mathcal{C}$.

###  IV-B SPIBB-DQN for RET optimization

This section details the application of the SPIBB-DQN algorithm to the RET use
case. The pseudocode of SPIBB-DQN for RET optimization is presented in
Algorithm 1.

Algorithm 1 SPIBB-DQN for RET optimization

1:

2:Phase 1 - Data Collection

3:

4:Input: Baseline policy $\pi_{b}$, time budget $T$.

5:Initialize: Initial state $s_{0}\sim p_{0}$.

6:for $t=0,\dots,T$ do

7: for $c=1,\dots,\mathcal{C}$ do

8: Select $a_{t,c}\sim\pi_{b}(\cdot|s_{t,c})$,

9: Receive $r_{t,c}\sim R(\cdot|s_{t,c},a_{t,c})$,

10: Transition to $s_{t+1,c}\sim P(\cdot|s_{t,c},a_{t,c})$,

11: Append
$\mathcal{D}_{\pi_{b}}\leftarrow\mathcal{D}_{\pi_{b}}\cup(s_{t,c},a_{t,c},r_{t,c},s_{t+1,c})$.

12:Return $\mathcal{D}_{\pi_{b}}$.

13:

14:Phase 2 - SPIBB-DQN Learning

15:

16:Input: Baseline dataset $\mathcal{D}_{\pi_{b}}$, and parameters
$N_{\wedge}$, $N_{B}$, $N_{\text{epoch}}$.

17:Initialize: $Q$-function weights $w$ randomly.

18:Compute pseudo-counts $\widetilde{N}_{\mathcal{D}_{\pi_{b}}}(s,a)$ as in
(3).

19:Construct bootstrapped set $\mathfrak{B}_{\pi_{b}}$ as in (1).

20:for $m=1,\dots,N_{\text{epoch}}$ do

21: for $j=1,\dots,\left\lfloor{N/N_{B}}\right\rfloor$ do

22: Sample batch
$\\{(s_{\beta},a_{\beta},r_{\beta},s^{\prime}_{\beta})\\}_{\beta=1}^{N_{B}}$
from $\mathcal{D}_{\pi_{b}},$

23: Set the target $y_{\beta}^{(j)}$ as in (2) for $\beta=1,\dots,N_{B}$,

24: Update $w\leftarrow
w-\eta\sum_{\beta}^{N_{B}}\nabla_{w}\left(y_{\beta}^{(j)}-Q_{w}\left(s_{\beta},a_{\beta}\right)\right)^{2}$

25:Return $\pi_{w}^{\star}(s)=\operatorname*{arg\,max}_{a}Q_{w}(s,a)$.

The algorithm can be divided into two phases:

  1. 1.

Data collection: Collect $\mathcal{D}_{\pi_{b}}$ by letting the safe baseline
policy $\pi_{b}$ interact with the environment $\mathcal{E}$. At each time
step $t=1,\dots,T$, and for each cell $c=1,\dots,\mathcal{C}$, the tuple
$(s_{t,c},a_{t,c},r_{t,c},s_{t+1,c})$ is generated according to the RL
interaction cycle and stored. This process is repeated for a total of
$N=T\cdot\mathcal{C}$ dataset entries.

  2. 2.

SPIBB-DQN learning: Learn a safe policy $\pi_{w}$ from the collected dataset
$\mathcal{D}_{\pi_{b}}$. First, the state-action counts are computed. Since we
deal with a continuous state space, we utilize the pseudo-count
$\widetilde{N}_{\mathcal{D}}(s,a)$ for counting the state-action pairs in a
dataset $\mathcal{D}$, defined as

$\widetilde{N}_{\mathcal{D}}(s,a)=\sum_{(s_{i},a_{i})\in\mathcal{D}}\lVert
s-s_{i}\rVert_{2}\mathbbm{1}\\{a=a_{i}\\},$ (3)

where $\left\|\cdot\right\|_{2}$ denotes the Euclidean norm. Based on the
pseudo-count computation, the bootstrapped set $\mathfrak{B}_{\pi_{b}}$ is
constructed. Subsequently, the learning proceeds in batches of length $N_{B}$
sampled from the dataset $\mathcal{D}_{\pi_{b}}$ for $N_{\text{epochs}}$. For
each batch, the bootstrapped update on the $Q$-function is executed. At the
end of the process, the learned policy $\pi_{w}^{\star}$ is returned.

##  V Experimental setting and results

This section describes the experimental setup, the evaluation metrics, and our
experimental results.

###  V-A Experimental setup

We execute our experiments in a simulated network environment. The network
environment map is shown in Fig. 2, and the simulation parameters are reported
in Table LABEL:tab:simulator_setup.

TABLE I: Simulator parameters. Simulator parameter | Symbol | Value
---|---|---
Number of BSs | $\mathcal{B}$ | $7$
Number of cells | $\mathcal{C}$ | $21$
Number of UEs | $\mathcal{U}$ | $2000$
Frequency | $f$ |  $2$ GHz
Traffic volume average | $\mu_{\tau}$ |  $20$ Mbps
Traffic volume variance | $\sigma^{2}_{\tau}$ |  $4$ Mbps
Antenna height | $h$ |  $32$ m
Minimum downtilt angle | $\theta_{\text{min}}$ | $1^{\circ}$
Maximum downtilt angle | $\theta_{\text{max}}$ | $16^{\circ}$

Two different baselines are used in our experiments: a RB baseline
$\pi_{{}_{\text{RB}}}$, and a  DQN baseline $\pi_{{}_{\text{DQN}}}$. The RB
baseline consists of a legacy policy currently deployed in a real-world
network and is thus considered safe. The DQN baseline consists of a sub-
optimal RL policy trained in the simulation environment for $500$ steps and
stopped before convergence. In contrast to the RB baseline, the DQN baseline
is stochastic and explores a larger portion of the state-action space. For
both baseline policies, we conduct a series of experiments by applying
Algorithm 1. We collect the datasets with size
$|\mathcal{D}_{\pi_{\text{RB}}}|=|\mathcal{D}_{\pi_{\text{DQN}}}|=500$, as in
Phase $1$, and subsequently train the SPIBB-DQN model by constructing the
bootstrapped sets $\mathfrak{B}_{\pi_{\text{RB}}}$,
$\mathfrak{B}_{\pi_{\text{DQN}}}$, and running the bootstrapped $Q$-function
updates, as in Phase $2$. We execute $K=20$ independent runs for each
experiment, reporting mean and standard deviation of the performance. We
execute experiments for different values of
$N_{\wedge}\in\\{5,10,20,30,40,50,100,150,200,300\\}$ and offline dataset size
$N\in\\{25,50,100,200,300,400,500\\}$. The experiments are executed according
to the following methodology: for each episode, we reset the downtilt
uniformly at random, i.e. at $t=k\cdot T_{\textrm{episode}}$,
$k=0,\dots,N_{\textrm{episode}}-1$, we set
${\theta_{t,c}\sim\textsc{U}\left(\theta_{\textrm{min}},\theta_{\textrm{max}}\right)}$,
where $\textsc{U}(\cdot)$ denotes the discrete uniform distribution. Finally,
the training and testing hyperparameters are reported in Table
LABEL:tab:hyperparams.

TABLE II: Training $\&$ Testing Hyperparameters. Hyperparameter | Symbol | Value
---|---|---
Discount factor | $\gamma$ | $9\times 10^{-1}$
Learning rate | $\eta$ | $1\times 10^{-3}$
Batch size | $N_{B}$ | $50$
Test episode length | $T_{\text{episode}}$ | $20$
Number of evaluation episodes | $N_{\text{episode}}$ | $25$
Test epoch length | $T_{\text{epoch}}$ | $500$
Number of training epochs | $N_{\text{epoch}}$ | $20$

Fig. 3 illustrates the ANN architecture used for the SPIBB policies. The ANN
model takes as input a state-action pair $(s,a)$ and outputs the $Q$-value for
$(s,a)$. It is composed of three hidden fully-connected layers, whose number
of units is reported in parenthesis in Fig. 3. The activation function for the
hidden layers is the Rectified Linear Unit (ReLU), while for the output layer,
we use a linear activation.

$Q_{w}(s,a)$$(s,a)$ReLuFully Connected Layer$(100)$ReLuLinearFully Connected
Layer$(50)$Fully Connected Layer$(20)$

Figure 3: SPIBB network architecture.

###  V-B Evaluation metrics

(a) Average Network Reward vs $t$

(b)

(c) Minimum cell reward

(d)

(e)

(f)

(g)

(h)

(i)

Figure 4: Experimental results.

For evaluation, we let the policies interact with the simulated network
environment on unseen test environment configurations. The policies considered
for the evaluations are: RB and DQN baseline policies, the SPIBB-RB and SPIBB-
DQN policies, trained on the RB and DQN dataset respectively, the uniform
random policy, and the (empirical) optimal policy. We utilize the following
evaluation metrics:

  1. 1.

Average network reward: $\bar{r}_{t}$, a general performance indicator
considering the average over the cells’ performance of the controlled network
area, defined as:

$\bar{r}_{t}\triangleq\frac{1}{\mathcal{C}}\sum_{c=1}^{\mathcal{C}}r_{t,c}.$

With a slight abuse of notation we denote by
${\bar{r}=\frac{1}{T_{\textrm{epoch}}}\sum_{t=1}^{T_{\textrm{epoch}}}\bar{r}_{t}}$
the time average of $\bar{r}_{t}$.

  2. 2.

Conditional Value-at-Risk: $\textrm{CVaR}_{a}(r)$, the average reward obtained
from the $a\%$ worst runs, defined as

$\textrm{CVaR}_{a}(r)\triangleq\mathbb{E}[r\mid r\leq\textrm{VaR}_{a}(r)],$

where VaR denotes the Value at Risk, defined as
$\textrm{VaR}_{a}(r)\triangleq\min\\{\rho\in\mathbb{R}:\mathbb{P}[r\leq\rho]\geq
a\\}$.

  3. 3.

Minimum cell reward: $r^{\text{min}}_{t}$, considering the reward performance
of the worst cell, defined as

$r^{\text{min}}_{t}\triangleq\min_{c\in\mathcal{C}}r_{t,c}.$

Finally, we denote by
$r^{\text{min}}=\frac{1}{T_{\textrm{epoch}}}\sum_{t=1}^{T_{\textrm{epoch}}}r^{\text{min}}_{t}$
the time average of $r_{t,c}$.

###  V-C Results and discussion

The results of our experiments are shown in Fig. 4. The columns contain the
result for the average network reward, the $5\%$ CvaR of the average network
performance, and the minimum cell reward. The first row shows an example of
performance evaluation for fixed data size $N=300$ and safety parameter
$N_{\wedge}=100$. The second row shows the performance of the considered
policies when varying the data size $N$, and for fixed safety parameter
$N_{\wedge}=100$. The third row shows the performance of the considered
policies when varying the safety parameter $N_{\wedge}$ and for fixed data
size $N=100$. Finally, in Table LABEL:tab:results we report the experimental
results for the average and standard deviation of the reward across the $K=20$
runs for $N=N_{\wedge}=100$.

TABLE III: Experimental Results for Reward ($N=N_{\wedge}=100$) Policy | $\bar{r}$ | $\textrm{CVaR}_{5}(\bar{r})$ | $\bar{r}_{\textrm{min}}$
---|---|---|---
Random | $-0.562\pm 0.049$ | $-0.649$ | $-0.681$
DQN | $-0.327\pm 0.061$ | $-0.448$ | $-0.527$
SPIBB-DQN | $-0.278\pm 0.059$ | $-0.399$ | $-0.486$
Rule-Based | $-0.219\pm 0.046$ | $-0.309$ | $-0.626$
SPIBB-RB | $-0.151\pm 0.036$ | $-0.224$ | $-0.295$
Optimal | $-0.141\pm 0.033$ | $-0.211$ | $-0.295$

In the following points, we summarize and discuss the key conclusions from the
experimental results:

  1. 1.

The SPIBB policies outperform the respective baselines: from Figs. 4a, 4b, 4c,
we may observe that safe improvement over the baselines is achieved at any
time step of the evaluation. Also, Figs. 4d, 4e, 4f show that the SPIBB
policies outperform the respective baseline in terms of average network
reward, $5\%$ CVaR of the average network reward and minimum cell reward. The
first metric represents the overall network performance, while the second and
third metrics embody worst-case scenarios. Thus, the result validates the
effectiveness of the SPIBB, providing safe policy improvement over the
respective baselines for average performance and different worst-case
performance.

  2. 2.

The performance of the SPIBB policies increases with data size: a larger
amount of data allows learning better policies during the offline learning
process. This is confirmed for both the RB policy, which is closer to a
deterministic policy, and the DQN policy, that exhibits a higher degree of
randomization. Amazingly, we observe that even small data sizes ($N\geq 50$)
allow the SPIBB policies to outperform the respective baselines. This
highlights the high sample efficiency of the SPIBB method and its robustness
to different types of baselines.

  3. 3.

The SPIBB-RB outperforms the SPIBB-DQN: this is a direct consequence of three
facts: first, the RB baseline policy outperforms the DQN baseline policy. As
mentioned in Section V-A, the DQN baseline is stopped before convergence and
thus has worse performance than the RB baseline. This is confirmed by the
almost totality of results in Fig. 4. Second, the RB policy collects high-
performance trajectories that are stored in the RB baseline dataset and used
in the offline training of the SPIBB-RB policy. Third, the RB action
probabilities, used to bootstrap the state-action pairs on the RB bootstrapped
set, are closer to the ones of the optimal policy when compared to the DQN
policy.

  4. 4.

The SPIBB policy is highly sensitive to the safety parameter $N_{\wedge}$:
from Figs. 4g, 4h, 4i, we can observe how different values of the safety
parameter $N_{\wedge}$ significantly affect the SPIBB performance. Generally,
we expect that higher values of the safety parameter make the policy safer but
more prone to sub-optimal performance. On the one hand, we observe how the
SPIBB-DQN policy exhibits a more stable behavior to variations of this
parameter: it outperforms the DQN baseline for all experimented $N_{\wedge}$.
This may be due to the high degree of randomization of the DQN baseline and
the consequent increased diversity of the dataset. On the other hand, the
SPIBB-RB baseline shows a very unstable behavior, especially for the minimum
cell reward performance. This may be due to the fact that the RB baseline is
mostly deterministic and a very biased composition of the Bootstrapped set
makes the performance highly sensitive to $N_{\wedge}$. Note that the values
of $N_{\wedge}$ are chosen from a wide range of values, and more concentrated
values may produce more consistent curves.

##  VI Conclusion

In this paper, we devise and evaluate a safe downtilt update policy using
SPIBB, a recent and effective SRL method based on the use of a safe baseline
policy and batches of data containing trajectories collected by the baseline
policy. We demonstrate the validity of the approach for different baseline
policies and dataset sizes through extensive experimental evaluation on a
mobile network simulator. We show that this method is able to produce a policy
that outperforms different baseline policies in a safe manner. As opposed to
RL techniques using standard exploration paradigms, the proposed method is
able to achieve safety at any instant, providing a higher degree of
reliability and potential for real-world network deployment. Possible future
research directions include exploring the use of SRL methodologies for the
optimization of multi-antenna systems (e.g., 5G MIMO systems) and the
combination of multiple and diverse baselines in SPIBB to achieve an overall
increased performance.

## Acknowledgement

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

## References

  * [1] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [2] R. Razavi, S. Klein, and H. Claussen, “A fuzzy reinforcement learning approach for self-optimization of coverage in LTE networks,” _Bell Labs Technical Journal_ , 2010.
  * [3] W. Guo, S. Wang _et al._ , “Spectral-and energy-efficient antenna tilting in a HetNet using reinforcement learning,” _IEEE Wireless Communications and Networking Conference, WCNC_ , 2013.
  * [4] S. Fan, H. Tian, and C. Sengul, “Self-optimization of coverage and capacity based on a fuzzy neural network with cooperative reinforcement learning,” _Eurasip Journal on Wireless Communications and Networking_ , 2014.
  * [5] E. Balevi and J. G. Andrews, “Online Antenna Tuning in Heterogeneous Cellular Networks with Deep Reinforcement Learning,” _IEEE Transactions on Cognitive Communications and Networking_ , 2019.
  * [6] A. Tall, Z. Altman, and E. Altman, “Self-optimizing Strategies for Dynamic Vertical Sectorization in LTE Networks,” in _IEEE Wireless Communications and Networking Conference, WCNC_ , 2015.
  * [7] V. Buenestado, M. Toril _et al._ , “Self-tuning of remote electrical tilts based on call traces for coverage and capacity optimization in LTE,” _IEEE Transactions on Vehicular Technology_ , 2017.
  * [8] J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” _Journal of Machine Learning Research_ , 2015.
  * [9] S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” in _Adaptation, Learning, and Optimization_. Springer Verlag, 2012.
  * [10] P. S. Thomas, G. Theocharous, and M. Ghavamzadeh, “High confidence policy improvement,” _32nd International Conference on Machine Learning, ICML_ , 2015.
  * [11] R. Laroche, P. Trichelair, and R. T. Des Combes, “Safe policy improvement with baseline bootstrapping,” _36th International Conference on Machine Learning, ICML_ , 2019.
  * [12] F. Vannella, J. Jeong, and A. Proutiere, “Off-policy Learning for Remote Electrical Tilt Optimization,” _IEEE 92nd Vehicular Technology Conference, VTC Fall_ , 2020.
  * [13] V. Raghunath, “Antenna tilt optimization using Reinforcement Learning with doubly-safe exploration,” MSc Thesis, KTH Royal Institute of Technology, 2020\.
  * [14] C. J. C. H. Watkins, “Learning from delayed rewards,” Ph.D. dissertation, King’s College, Cambridge, UK, May 1989.
  * [15] V. Mnih, K. Kavukcuoglu _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , 2015.