# A Barrier Certificate-based Simplex Architecture with Application to
Microgrids††thanks: This work was supported in part by NSF grants OIA-2134840,
OIA-2040599, CCF-1954837, CCF-1918225, and CPS-1446832.

Amol Damare  Shouvik Roy  Scott A. Smolka  Scott D. Stoller

(Stony Brook University, Stony Brook NY, USA
)

###### Abstract

We present _Barrier Certificate-based Simplex_ (_BC-Simplex_), a new, provably
correct design for runtime assurance of continuous dynamical systems. _BC-
Simplex_ is centered around the Simplex Control Architecture, which consists
of a high-performance _advanced controller_ which is not guaranteed to
maintain safety of the plant, a verified-safe _baseline controller_ , and a
_decision module_ that switches control of the plant between the two
controllers to ensure safety without sacrificing performance. In _BC-Simplex_
, _Barrier certificates_ are used to prove that the baseline controller
ensures safety. Furthermore, _BC-Simplex_ features a new automated method for
deriving, from the barrier certificate, the conditions for switching between
the controllers. Our method is based on the Taylor expansion of the barrier
certificate and yields computationally inexpensive switching conditions.

We consider a significant application of _BC-Simplex_ to a microgrid featuring
an advanced controller in the form of a neural network trained using
reinforcement learning. The microgrid is modeled in RTDS, an industry-standard
high-fidelity, real-time power systems simulator. Our results demonstrate that
_BC-Simplex_ can automatically derive switching conditions for complex
systems, the switching conditions are not overly conservative, and _BC-
Simplex_ ensures safety even in the presence of adversarial attacks on the
neural controller.

##  1 Introduction

_Barrier certificates_ (BaCs) [28, 27] are a powerful method for verifying the
safety of continuous dynamical systems without explicitly computing the set of
reachable states. A BaC is a function of the state satisfying a set of
inequalities on the value of the function and value of its time derivative
along the dynamic flows of the system. Intuitively, the zero-level-set of BaC
forms a “barrier” between the reachable states and unsafe states. Existence of
BaC assures that starting from a state where the BaC is positive, safety is
forever maintained [7, 27, 28]. Moreover, there are automated methods to
synthesize BaCs, e.g., [14, 35, 39, 32].

Proving safety of plants with complex controllers is difficult with any formal
verification technique, including barrier certificates. However, as we now
show, BaCs can play a crucial role in applying the well-established Simplex
Control Architecture [30, 31] to provide provably correct runtime safety
assurance for systems with complex controllers.

We present _Barrier Certificate-based Simplex_ (_BC-Simplex_), a new, provably
correct design for runtime assurance of continuous dynamical systems. _BC-
Simplex_ is centered around the Simplex Control Architecture, which consists
of a high-performance _advanced controller_ (AC) that is not guaranteed to
maintain safety of the plant, a verified-safe _baseline controller_ (BC), and
a _decision module_ that switches control of the plant between the two
controllers to ensure safety without sacrificing performance. In _BC-Simplex_
, _Barrier certificates_ are used to prove that the baseline controller
ensures safety. Furthermore, _BC-Simplex_ features a new scalable (relative to
existing methods that require reachability analysis, e.g., [4, 6, 5, 11]) and
automated method for deriving, from the BaC, the conditions for switching
between the controllers. Our method is based on the Taylor expansion of the
BaC and yields computationally inexpensive switching conditions.

We consider a significant application of _BC-Simplex_ , namely _microgrid
control_. A _microgrid_ is an integrated energy system comprising distributed
energy resources and multiple energy loads operating as a single controllable
entity in parallel to, or islanded from, the existing power grid [34]. The
microgrid we consider features an advanced controller (for voltage control) in
the form of a neural network trained using reinforcement learning. For this
purpose, we use _BC-Simplex_ in conjunction with the _Neural Simplex
Architecture_ (NSA) [25], where the AC is an AI-based _neural controller_
(NC). NSA also includes an _adaptation module_ (AM) for online retraining of
the NC while the BC is in control.

The microgrid we consider is modeled in RTDS, an industry-standard high-
fidelity, real-time power systems simulator. Our results demonstrate that _BC-
Simplex_ can automatically derive switching conditions for complex systems,
the switching conditions are not overly conservative, and _BC-Simplex_ ensures
safety even in the presence of adversarial attacks on the neural controller.

#### Architectural overview of _BC-Simplex_.

Figure 1 shows the overall architecture of the combined Barrier Certificate-
based Neural Simplex Architecture. The green part of the figure depicts our
design methodology; the blue part illustrates NSA. Given the BC, the required
safety properties, and a dynamic model of the plant, our methodology generates
a BaC and then derives the switching condition from it. The reinforcement
learning module learns a high-performance NC, based on the performance
objectives encoded in the reward function.

The structure of the rest of the paper is the following. Section 2 provides
background material on barrier certificates. Section 3 features our new
approach for deriving switching conditions from barrier certificates. Section
4 introduces our Microgrid case study and the associated controllers used for
microgrid control. Section 5 presents the results of our microgrid case study.
Section 6 discusses the related work. Section 7 offers our concluding remarks.

Figure 1: Overview of the Barrier Certificate-based Neural Simplex
Architecture

##  2 Preliminaries

We use Barrier Certificates (BaCs) to prove that the BC ensures safety. We
implemented two automated methods for BaC synthesis from the literature. As
discussed next, one of the methods is based on sum-of-squares optimization
(SOS) and the other uses deep learning. Our design methodology for computing
switching conditions (see Section 3) requires a BaC, but is independent of how
the BaC is obtained.

#### BaC Synthesis using SOS Optimization.

This method first derives a Lyapunov function $V$ for the system using the
expanding interior-point algorithm in [3]. It then uses the SOS-based
algorithm in [35] to obtain a BaC from $V$. Note that the largest super-level
set of a Lyapunov function within a safety region is a BaC. The algorithm in
[14, 35] computes a larger BaC by starting with that sub-level set and then
expanding it, by allowing it to take shapes other than that of a sub-level set
of the Lyapunov function. This method involves a search of Lyapunov functions
and BaCs of various degrees by choosing different candidate polynomials and
parameters of the SOS problem. It is limited to systems with polynomial
dynamics. In some cases, non-polynomial dynamics can be recast as polynomial
using, e.g., the techniques in [3].

#### BaC Synthesis using Deep Learning.

We also implemented _SyntheBC_ [40], which uses deep learning to synthesize a
BaC. First, training samples obtained by sampling different areas of the state
space are used to train a feedforward ReLU neural network with 2 hidden layers
as a candidate BaC. Second, the validity of this candidate BaC must be
verified. The NN’s structure allows the problem of checking whether the NN
satisfies the defining conditions of a BaC to be transformed into mixed-
integer linear programming (MILP) and mixed-integer quadratically-constrained
programming (MIQCP) problems, which we solve using the Gurobi optimizer. If
the verification fails, the Gurobi optimizer provides evidence that can be
used to focus continued training of the NN. In this way, the training and
verification steps can be iterated as needed.

##  3 Deriving the Switching Condition

We employ our novel methodology to derive the switching logic from the BaC.
The Decision Module (DM) implements this switching logic for both forward and
reverse switching. When the forward-switching condition (FSC) is true, control
is switched from the NC to the BC; likewise, when the reverse-switching
condition (RSC) is true, control is switched from the BC to the NC. The
success of our approach rests on solving the complex problems discussed in
this section to derive an FSC. Consider a continuous dynamical system of the
form:

$\dot{x}=f(x,u)$ (1)

where $x\in\mathbb{R}^{k}$ is the state of the plant at time $t$ and
$u\in\Omega$ is the control input provided to the plant at time $t$. The set
of all valid control actions is denoted by $\Omega$. The set of _unsafe
states_ is denoted by $\mathcal{U}$. Let $x_{lb},x_{ub}\in\mathbb{R}^{k}$ be
_operational bounds_ on the ranges of state variables, reflecting physical
limits and simple safety requirements.

The set $\mathcal{A}$ of _admissible states_ is given by:
$\mathcal{A}=\\{x:x_{lb}\leq x\leq x_{ub}\\}$. A state of the plant is
_recoverable_ if the BC can take over in that state and keep the plant
invariably safe. For a given BC, we denote the _recoverable region_ by
$\mathcal{R}$. Note that $\mathcal{U}$ and $\mathcal{R}$ are disjoint. The
safety of such a system can be verified using a BaC
$h(x):\mathbb{R}^{k}\rightarrow\mathbb{R}$ of the following form [28, 27, 35,
14]:

$\displaystyle h(x)\geq 0,$ $\displaystyle\forall
x\in\mathbb{R}^{k}\mathbin{\backslash}\mathcal{U}$ (2) $\displaystyle h(x)<0,$
$\displaystyle\forall x\in\mathcal{U}$
$\displaystyle(\nabla_{x}h)^{T}f(x,u)+\sigma(h(x))\geq 0,$
$\displaystyle\forall x\in\mathbb{R}^{k}$

where $\sigma(.)$ is an extended class-$\mathcal{K}$ function. The BaC is
negative over the unsafe region and non-negative otherwise. $\nabla_{x}h$ is
the gradient of $h$ w.r.t $x$ and the expression $(\nabla_{x}h)^{T}f(x,u)$ is
the time derivative of $h$. The zero-super-level set of a BaC $h$ is
$\mathcal{Z}(h)=\\{x:h(x)>0\\}$. In [35], the invariance of this set is used
to show $\mathcal{Z}(h)\subseteq\mathcal{R}$.

Let $\eta$ denote the control period a.k.a. time step. Let
$\hat{h}(x,u,\delta)$ denote the $n^{\rm th}$-degree Taylor approximation of
BaC $h$’s value after time $\delta$, if control action $u$ is taken in state
$x$. The approximation is computed at the current time to predict $h$’s value
$\delta$ time units later and is given by:

$\hat{h}(x,u,\delta)=h(x)+\sum_{i=1}^{n}\frac{h^{i}(x,u)}{i!}\delta^{i}$ (3)

where $h^{i}(x,u)$ denotes the $i^{\rm th}$ time derivative of $h$ evaluated
in state $x$ if control action $u$ is taken. The control action is needed to
calculate the time derivatives of $h$ from the definition of $h$ and Eq. 1 by
applying the chain rule. Since we are usually interested in predicting the
value one time step in the future, we use $\hat{h}(x,u)$ as shorthand for
$\hat{h}(x,u,\eta)$. By Taylor’s theorem with the Lagrange form of the
remainder, the remainder error of the approximation $\hat{h}(x,u)$ is:

$\frac{h^{n+1}(x,u,\delta)}{(n+1)!}\eta^{n+1}\ \mbox{for some}\
\delta\in(0,\eta)$ (4)

An upper bound on the remainder error, if the state remains in the admissible
region during the time interval, is:

$\lambda(u)=\sup\left\\{\frac{|h^{n+1}(x,u)|}{(n+1)!}\eta^{n+1}:x\in\mathcal{A}\right\\}$
(5)

The FSC is based on checking recoverability during the next time step. For
this purpose, the set $\mathcal{A}$ of admissible states is shrunk by margins
of $\mu_{\rm dec}$ and $\mu_{\rm inc}$, a vector of upper bounds on the amount
by which each state variable can decrease and increase, respectively, in one
time step, maximized over all admissible states. Formally,

$\displaystyle\mu_{\rm dec}(u)=|\min(0,\eta\dot{x}_{min}(u))|$ (6)
$\displaystyle\mu_{\rm inc}(u)=|\max(0,\eta\dot{x}_{max}(u))|$

where $\dot{x}_{min}$ and $\dot{x}_{max}$ are vectors of solutions to the
optimization problems:

$\displaystyle\dot{x}_{i}^{min}(u)=\inf\\{\dot{x}_{i}(x,u):x\in\mathcal{A}\\}$
(7)
$\displaystyle\dot{x}_{i}^{max}(u)=\sup\\{\dot{x}_{i}(x,u):x\in\mathcal{A}\\}$

The difficulty of finding these extremal values depends on the complexity of
the functions $\dot{x}_{i}(x,u)$. For example, it is relatively easy if they
are convex. In our case study of a realistic microgrid model, they are
multivariate polynomials with degree 1, and hence convex. The set
$\mathcal{A}_{r}$ of _restricted admissible states_ is given by:

$\mathcal{A}_{r}(u)=\\{x:x_{lb}+\mu_{\rm dec}(u)<x<x_{ub}-\mu_{\rm inc}(u)\\}$
(8)

Let $\textit{Reach}_{=\eta}(x,u)$ denote the set of states reachable from
state $x$ after exactly time $\eta$ if control action $u$ is taken in state
$x$. Let $\textit{Reach}_{\leq\eta}(x,u)$ denote the set of states reachable
from $x$ within time $\eta$ if control action $u$ is taken in state $x$.

######  Lemma 1.

For all $x\in\mathcal{A}_{r}(u)$ and all control actions $u$,
$\textit{Reach}_{\leq\eta}(x,u)\subseteq\mathcal{A}$.

###### Proof.

The derivative of $x$ is bounded by $\dot{x}_{min}(u)$ and $\dot{x}_{max}(u)$
for all states in $\mathcal{A}$. This implies that $\mu_{\rm dec}$ and
$\mu_{\rm inc}$ are the largest amounts by which the state $x$ can decrease
and increase, respectively, during time $\eta$, as long as $x$ remains within
$\mathcal{A}$ during the time step. Since $\mathcal{A}_{r}(u)$ is obtained by
shrinking $\mathcal{A}$ by $\mu_{\rm dec}$ and $\mu_{\rm inc}$ (i.e., by
moving the lower and upper bounds, respectively, of each variable inwards by
those amounts), the state cannot move outside of $\mathcal{A}$ during time
$\eta$. ∎

###  3.1 Forward Switching Condition

To ensure safety, a forward-switching condition (FSC) should switch control
from the NC to the BC if using the control action $u$ proposed by NC causes
any unsafe states to be reachable from the current state $x$ during the next
control period, or causes any unrecoverable states to be reachable at the end
of the next control period. These two conditions are captured in the following
definition:

######  Definition 1 (Forward Switching Condition).

A condition $FSC(x,u)$ is a forward switching condition if for every
recoverable state $x$, every control action $u$, and control period $\eta$,
$\textit{Reach}_{\leq\eta}(x,u)\cap\mathcal{U}\neq\emptyset\lor\textit{Reach}_{=\eta}(x,u)\not\subset\mathcal{R}$
implies $FSC(x,u)$ is true.

######  Theorem 1.

A Simplex architecture whose forward switching condition satisfies Definition
1 keeps the system invariably safe provided the system starts in a recoverable
state.

###### Proof.

Our definition of an FSC is based directly on the switching logic in Algorithm
1 of [37]. The proof of Theorem 1 in [37] shows that an FSC that is exactly
the disjunction of the two conditions in our definition invariantly ensures
system safety. It is easy to see that any weaker FSC also ensures safety. ∎

We now propose a new and general procedure for constructing a switching
condition from a BaC and prove its correctness.

######  Theorem 2.

Given a barrier certificate $h$, the following condition is a forward
switching condition: $FSC(x,u)=\alpha\lor\beta$ where
$\alpha\equiv\hat{h}(x,u)-\lambda(u)\leq 0$ and $\beta\equiv
x\notin\mathcal{A}_{r}(u)$

###### Proof.

Intuitively, $\alpha\lor\beta$ is an FSC because (1) if condition $\alpha$ is
false, then control action $u$ does not lead to an unsafe or unrecoverable
state during the next control period, provided the state remains admissible
during that period; and (2) if condition $\beta$ is false, then the state will
remain admissible during that period. Thus, if $\alpha$ and $\beta$ are both
false, then nothing bad can happen during the control period, and there is no
need to switch to the BC.

Formally, suppose $x$ is a recoverable state, $u$ is a control action, and
$\textit{Reach}_{\leq\eta}(x,u)\cap\mathcal{U}\neq\emptyset\;\lor\;\textit{Reach}_{=\eta}(x,u)\not\subset\mathcal{R}$,
i.e., there is an unsafe state in $\textit{Reach}_{\leq\eta}(x,u)$ or an
unrecoverable state in $\textit{Reach}_{=\eta}(x,u)$. Let $x^{\prime}$ denote
that unsafe or unrecoverable state. Recall that
$\mathcal{Z}(h)\subseteq\mathcal{R}$, and
$\mathcal{R}\cap\mathcal{U}=\emptyset$. Therefore, $h(x^{\prime},u)\leq 0$. We
need to show that $\alpha\lor\beta$ holds. We do a case analysis based on
whether $x$ is in $\mathcal{A}_{r}(u)$.

Case 1: $x\in\mathcal{A}_{r}(u)$. In this case, we use a lower bound on the
value of the BaC $h$ to show that states reachable in the next control period
are safe and recoverable. Using Lemma 1, we have
$\textit{Reach}_{\leq\eta}(x,u)\subseteq\mathcal{A}$. This implies that
$\lambda(u)$, whose definition maximizes over $x\in\mathcal{A}$, is an upper
bound on the error in the Taylor approximation $\hat{h}(x,u,\delta)$ for
$\delta\leq\eta$. This implies that $\hat{h}(x,u)-\lambda(u)$ is a lower bound
on value of BaC for all states in $\textit{Reach}_{\leq\eta}(x,u)$. As shown
above, there is a state $x^{\prime}$ in $\textit{Reach}_{\leq\eta}(x,u)$ with
$h(x^{\prime},u)\leq 0$. $\hat{h}(x,u)-\lambda(u)$ is lower bound on
$h(x^{\prime},u)$ and hence must also be less than or equal to 0. Thus,
$\alpha$ holds.

Case 2: $x\notin\mathcal{A}_{r}(u)$. In this case, $\beta$ holds. Note that in
this case, the truth value of $\alpha$ is not significant (and not relevant,
since $FSC(x,u)$ holds regardless), because the state might not remain
admissible during the next control period. Hence, the error bound obtained
using Eq. 5 is not applicable. ∎

###  3.2 Reverse Switching Condition

The RSC is designed with a heuristic approach, since it does not affect safety
of the system. To prevent frequent switching between the NC and BC, we design
the RSC to hold if the FSC is likely to remain false for at least $m$ time
steps, with $m>1$. The RSC, like the FSC, is the disjunction of two
conditions. The first condition is $h(x)\geq m\eta|\dot{h}(x)|$, since $h$ is
likely to remain non-negative for at least $m$ time steps if its current value
is at least that duration times its rate of change. The second condition
ensures that the state will remain admissible for $m$ time steps. In
particular, we take:

$RSC(x)=h(x)\geq m\eta|\dot{h}(x)|\,\land\,x\in\mathcal{A}_{r,m},$ (9)

where the $m$-times-restricted admissible region is:

$\mathcal{A}_{r,m}=\\{x:x_{lb}+m\mu_{\rm dec}<x<x_{ub}-m\mu_{\rm inc}\\},$
(10)

where vectors $\mu_{\rm dec}$ and $\mu_{\rm inc}$ are defined in the same way
as $\mu_{\rm dec}(u)$ and $\mu_{\rm inc}(u)$ in Eqs. 6 and 7 except with
optimization over all control actions $u$. An RSC that guarantees absence of
forward switches for at least $m$ time steps can be designed by using the
maximum of $\dot{h}(x)$ over the admissible region; however, this conservative
approach might leave the BC in control longer than desired.

###  3.3 Decision Logic

The DM’s switching logic has three inputs: the current state $x$, the control
action $u$ currently proposed by the NC, and the name $c$ of the controller
currently in control (as a special case, we take $c=NC$ in the first time
step). The switching logic is defined by cases as follows: $DM(x,u,c)$ returns
$BC$ if $c=NC$ $\land$ $FSC(x,u)$, returns $NC$ if $c=BC$ $\land$ $RSC(x)$,
and returns $c$ otherwise.

##  4 Application to Microgrids

A _microgrid_ (MG) is an integrated energy system comprising distributed
energy resources (DERs) and multiple energy loads. DERs tend to be _renewable_
energy resources and include solar panels, wind turbines, batteries, and
emergency diesel generators. By satisfying energy needs from local renewable
energy resources, MGs can reduce energy costs and improve energy supply
reliability for energy consumers. Some of the major control requirements for
an MG are power control, load sharing, and frequency and voltage regulation.

An MG can operate in two modes: grid-connected and islanded. When operated in
grid-connected mode, DERs act as constant source of power which can be
injected into the network on demand. In contrast, in islanded or autonomous
mode, the DERs form a grid of their own, meaning not only do they supply power
to the local loads, but they also maintain the MG’s voltage and frequency
within the specified limits [26]. For our case study, we focus on voltage
regulation in both grid-connected and islanded modes. Specifically, we apply
_BC-Simplex_ to the controller for the inverter for a Photovoltaic (PV) DER.

Applying _BC-Simplex_ to other DERs which have inverter interfaces such as
battery is straightforward. Of the three controllers necessary for diesel
generator DER, our methodology can be applied to voltage and frequency
controllers straightforwardly. The exciter system controls the magnetic flux
flowing through the rotor generator, and its dynamics are coupled with that of
the diesel engine. We plan to explore using the approach presented in [13] to
handle these coupled dynamics and apply _BC-Simplex_ to the exciter system.

###  4.1 Baseline Controller

Figure 2: Lyapunov-function level sets (black-dotted ellipses). Innermost
ellipse also indicates initial BaC, which is optimized iteratively (green
ellipses). Red lines are voltage safety limits.

For our experiments, we used the SOS-based methodology described in Section 2
to derive a Barrier Certificate (as a proof of safety) for the baseline
controller. We use a droop controller as the BC. A droop controller is a type
of proportional controller, traditionally used in power systems for control
objectives such as voltage regulation, power regulation, and current sharing
[10, 15, 41]. The droop controller tries to balance the electrical power with
voltage and frequency. Variations in the active and reactive powers result in
frequency and voltage magnitude deviations, respectively [21].

Consider the following model of an MG’s droop-controlled inverters:

$\dot{\theta_{i}}=\omega_{i}$ (11a)
$\dot{\omega_{i}}=\omega_{i}^{0}-\omega_{i}+\lambda^{p}_{i}(\mathbf{P_{i}}-P_{i})$
(11b) $\dot{v_{i}}=v_{i}^{0}-v_{i}+\lambda^{q}_{i}(\mathbf{Q_{i}}-Q_{i})$
(11c)

where $\theta_{i}$, $\omega_{i}$, and $v_{i}$ are the phase angle, frequency,
and voltage of the $i^{th}$ inverter, respectively. $\mathbf{P_{i}}$ and
$\mathbf{Q_{i}}$ are the inverter’s active and reactive power set-points, and
$\lambda^{p}$ and $\lambda^{q}$ are the droop controller’s coefficients. The
values of set-points $\mathbf{P_{i}}$ and $\mathbf{Q_{i}}$ of an inverter
depend upon local loads and power needed by the rest of the MG. The loads are
not explicitly modeled here. In our case studies, we vary these power set-
points to simulate changing loads. Let $\mathcal{M}$ be the set of all
inverter indices. The active power $P_{i}$ and reactive power $Q_{i}$ are
given by:

$\displaystyle P_{i}$
$\displaystyle=v_{i}\sum_{j\in\mathcal{N}_{i}}{v_{k}(G_{i,j}\cos{\theta_{i,j}}+B_{i,j}\sin{\theta_{i,j}})}$
(12) $\displaystyle Q_{i}$
$\displaystyle=v_{i}\sum_{j\in\mathcal{N}_{i}}{v_{k}(G_{i,j}\sin{\theta_{i,j}}-B_{i,j}\cos{\theta_{i,j}})}$

where $\theta_{i,j}=\theta_{i}-\theta_{j}$, and
$\mathcal{N}_{i}\subseteq\mathcal{M}$ is the set of neighbors of inverter $i$.
$G_{i,j}$ and $B_{i,j}$ are respectively the conductance and susceptance
values of the transmission line connecting inverters $i$ and $j$. As shown in
[3], the stability of such a system can be verified using Lyapunov theory.
Detailed dynamic models for an MG with multiple inverters connected by
transmission lines and with droop controllers for frequency and voltage are
given in [3, 14].

Fig. 2 shows this process of incrementally expanding the Lyapunov function to
obtain the BaC. SOS-based algorithms apply only to polynomial dynamics so we
first recast our droop controller dynamics to be polynomial using a DQ0
transformation [23] to AC waveforms. This transformation is exact; i.e., it
does not introduce any approximation error. In our experimental evaluation
(Section 5), we obtain the BaCs for BCs in the form of droop controllers for
voltage regulation, in the context of MGs containing up to three DERs of
different types. Note that battery DERs operate in two distinct modes,
charging and discharging, resulting in a hybrid system model with different
dynamics in different modes. For now, we consider only runs in which the
battery remains in the same mode for the duration of the run. Extending our
framework to hybrid systems is future work.

###  4.2 Neural Controller

To help address the control challenges related to microgrids, the application
of _neural networks for microgrid control_ is on the rise [17]. Increasingly,
Reinforcement learning (RL) is being used to train powerful Deep Neural
Networks (DNNs) to produce high-performance MG controllers.

We present our approach for learning neural controllers (NCs) in the form of
DNNs representing deterministic control policies. Such a DNN maps system
states (or raw sensor readings) to control inputs. We use RL in form of Deep
Deterministic Policy Gradient (DDPG) algorithm, with the safe learning
strategy of penalizing unrecoverable actions [25]. DDPG was chosen because it
works with deterministic policies and is compatible with continuous action
spaces.

#### Deep Deterministic Policy Gradient Algorithm.

The DDPG algorithm is a model-free, off-policy Reinforcement Learning method.
_Model-free_ means that the algorithm does not have access to a model of the
environment (in our case, the microgrid dynamics). While model-free methods
forego the potential gains in sample efficiency from using a model, they tend
to be easier to implement and tune. An _off-policy_ learner learns the value
of the optimal policy independently of the current learned policy. A major
challenge of learning in continuous action spaces is exploration. An advantage
of off-policy algorithms such as DDPG is that the problem of exploration can
be treated independently from the learning algorithm [16]. Off-policy learning
is advantageous in our setting because it enables the NC to be (re-)trained
using actions taken by the BC rather than the NC or the learning algorithm.
The benefits of off-policy retraining are further considered in Section 4.3.

We consider a standard Reinforcement Learning setup consisting of an agent
interacting with an environment in discrete time. At each time step $t$, the
agent receives a (microgrid) state $x_{t}$ as input, takes an action $a_{t}$,
and receives a scalar reward $r_{t}$. An agent’s behavior is defined by a
policy that maps states to a probability distribution over the actions. The
goal of Reinforcement Learning is to learn a policy that maximizes the reward
function $r$ from the starting state distribution $J$. Reward function $r$ is
an incentive mechanism that tells the agent what actions it should take (in
terms of performance and safety) and, conversely, which ones it should avoid,
using rewards and penalties.

The DDPG algorithm employs an _actor-critic framework_. The actor generates a
control action and the critic evaluates its quality. The Actor network
representing the actor is a DNN which in our case takes the vector state of
the DER voltages and currents as its input, and outputs a continuous action
$a_{t}=\mu(x_{t}|\theta^{\mu})$, where $\theta^{\mu}$ is the weight of the
actor network. The Critic network representing the critic is a DNN that
receives a state $x_{t}$ and an action $\mu(x_{t}|\theta^{\mu})$ as input, and
produces a scalar $\mathcal{Q}$-value. In order to learn from prior knowledge,
DDPG uses a replay buffer $R$ to store training samples of the form
$(x_{t},a_{t},r_{t},x_{t+1})$, where $x_{t}$ is the state at time $t$, $a_{t}$
is the action taken at time $t$, $r_{t}$ is the reward associated with the
current state and action, and $x_{t+1}$ is the next state.

At every training iteration, a set $\mathcal{S}$ of samples is randomly chosen
from the replay buffer. As such, $\mathcal{S}$ is not necessarily generated
using the current policy, but rather by the policies the DNN learned at
different stages of training. Hence, DDPG is an off-policy algorithm. The
critic agent $\mathcal{Q}(x,a|\theta^{\mathcal{Q}})$ for each state $x$, where
$\theta^{\mathcal{Q}}$ is the weight of the critic agent, is updated using the
Bellman equation. The actor policy is updated iteratively by the following
policy gradient:

$\nabla_{\theta^{\mu}}J\approx\frac{1}{\mathcal{S}}\sum_{t}\nabla_{a}\mathcal{Q}(x,a|\theta^{\mathcal{Q}})|_{x=x_{t},a=\mu(x_{t})}\,\nabla_{\theta^{\mu}}\mu(x|\theta^{\mu})|_{x_{t}}$
(13)

The critic network evaluates the action of the actor network based on its
current reward. For further details regarding the implementation of the DDPG
algorithm, please refer to Algorithm 1 [16].

To learn an NC for DER voltage control, we designed the following reward
function, which guides the actor network to learn the desired control
objective.

$r(x_{t},a_{t})=\begin{cases}-1000&\text{if FSC}(x_{t},a_{t})\\\
\;\;\;\;\;100&\text{if }v_{od}\in[v_{ref}-\epsilon,v_{ref}+\epsilon]\\\
-w\cdot\left(v_{od}-v_{ref}\right)^{2}&\text{otherwise}\\\ \end{cases}$ (14)

where $w$ is a weight ($w=100$ in our experiments), $v_{od}$ is the
$d$-component of the output voltage of the DER whose controller is being
learned, $v_{ref}$ is the reference or nominal voltage, and $\epsilon$ is the
tolerance threshold. We assign a high negative reward for triggering the FSC,
and a high positive reward for reaching the tolerance region, i.e.,
$v_{ref}\pm\epsilon$. The third clause rewards actions that lead to a state in
which the DER voltage is close to its reference value.

#### Adversarial Inputs.

Controllers obtained via deep RL algorithms are vulnerable to _adversarial
inputs_ (AIs): those that lead to a state in which the NC produces an
unrecoverable action, even though the NC behaves safely on very similar
inputs. NSA provides a defense against these kinds of attacks. If the NC
proposes a potentially unsafe action, the BC takes over in a timely manner,
thereby guaranteeing the safety of the system. To demonstrate NSA’s resilience
to AIs, we use a gradient-based attack algorithm [24] to construct such
inputs, and show that the DM switches control to the BC in time to ensure
safety.

The gradient-based algorithm takes as input the critic network, actor network,
adversarial attack constant $c$, parameters $a$,$b$ of beta distribution
$\beta(a,b)$, and the number of times $n$ noise is sampled. For a given
(microgrid) state $x$, the critic network is used to ascertain its
$\mathcal{Q}$-value and the actor network determines its optimal action. Once
the gradient of the critic network’s loss function is computed using the
$\mathcal{Q}$-value and the action, the $l_{2}$-constrained norm of the
gradient $(grad\\_dir)$ is obtained. An initial (microgrid) state $x_{0}$, to
be provided as input to the actor network, is then perturbed to obtain a
potential adversarial state $x_{adv}$, determined by the sampled noise in the
direction of the gradient: $x_{adv}=x_{0}-c\cdot\beta(a,b)\cdot grad\\_dir$.

We can now compute the $\mathcal{Q}$-value of $x_{adv}$ and its (potentially
adversarial) action $a_{adv}$. If this value is less tha
$\mathcal{Q}(x_{0},a_{0})$, then $x_{adv}$ leads to a sub-optimal action. A
sub-optimal action, however, does not necessarily guarantee that the FSC will
be triggered. Thus, we iterate the procedure $n$ times in an attempt to find
an adversarial state that produces an action that triggers the FSC.

Note that the gradient-based attack algorithm does not guarantee the
successful generation of AIs every time it is executed, as this largely
depends on the quality of the training (e.g., the training error) of the NC:
the higher the quality of training, the lower the success rate of generating
AIs. In our experiments (see Section 5.4), the highest rate of AI generation
we observed is $0.008\%$.

###  4.3 Adaptation Module

The Adaptation Module (AM) retrains the NC in an online manner when the NC
produces an unrecoverable action that causes the DM to failover to the BC.
With retraining, the NC is less likely to repeat the same or similar mistakes
in the future, allowing it to remain in control of the system more often,
thereby improving performance. We use Reinforcement Learning with the reward
function defined in Eq. 14 for online retraining.

As in initial training, we use the DDPG algorithm (with the same settings) for
online retraining. When the NC outputs an unrecoverable action, the DM
switches control to the BC, and the AM computes the (negative) reward for this
action and adds it to a pool of training samples. As in [25], we found that
reusing the pool of training samples (DDPG’s experience replay buffer) from
initial training of the NC evolves the policy in a more stable fashion, as
retraining samples gradually replace initial training samples in the pool.
Another benefit of reusing the initial training pool is that retraining of the
NC can start almost immediately, without having to wait for enough samples to
be collected online.

There are two methods to retrain the NC:

  1. 1.

Off-policy retraining: At every time step while the BC is active, the BC’s
action is used in the training sample. The reward for the BC’s action is based
on the observed next state of the system.

  2. 2.

Shadow-mode retraining: At every time step while the BC is active, the AM
takes a sample by running the NC in shadow mode to compute its proposed
action, and then simulates the behavior of the system for one time step to
compute a reward for it.

In our experiments, both methods produce comparable benefits. Off-policy
retraining is therefore preferable because it does not require simulation (or
a dynamic model of the system) and hence is less costly.

##  5 Experimental Evaluation

Figure 3: RTDS Microgrid Model [22] Figure 4: Integration of External NC with
RTDS

We apply our _BC-Simplex_ methodology to a model of a microgrid [22] with
three DERs: a battery, photovoltaic (PV, a.k.a. solar panels), and diesel
generator. The three DERs are connected to the main grid via bus lines. As
depicted in Fig. 3, the three DERs are connected to the main grid via bus
lines.We are primarily interested in the PV control, since we apply _BC-
Simplex_ to PV voltage regulation. The PV control includes multiple
components, such as “three-phase to DQ0 voltage and current” transformer,
average voltage and current control, power and voltage measurements, inner-
loop $dq$ current control, and outer-loop Maximum Power Point Tracking (MPPT)
control. Our experimental evaluation of _BC-Simplex_ was carried out on RTDS,
a high-fidelity power systems simulator.

We ran experiments for three configurations of the microgrid: Configuration 1:
grid-connected mode with only the PV DER connected within the MG;
Configuration 2: islanded mode with PV and diesel generator DERs connected
within the MG; Configuration 3: islanded mode with PV, diesel generator, and
battery (in discharging mode) DERs connected within the MG. All configurations
also include a load. These configurations demonstrate _BC-Simplex_ ’s ability
to handle a wide variety of MG configurations involving various types of DERs.
We did not perform experiments with the battery in charging mode, because in
this mode, the battery is simply another load, and the configuration is
equivalent to Configuration 1 or Configuration 2 with a larger load.

We use _BC-Simplex_ to ensure the safety property that the $d$-component of
the output voltage of the inverter for the PV DER is within $\pm 3\%$ of the
reference voltage $v_{ref}=0.48$ kV. We adopted a 3% tolerance based on the
discussion in [22]. _BC-Simplex_ could similarly be used to ensure additional
desired safety properties. All experiments use runs of length 10 seconds, with
the control period, RTDS time step, and simulation time step in MATLAB all
equal to 3.2 milliseconds (msec), the largest time step allowed by RTDS.

###  5.1 Integration of _BC-Simplex_ in RTDS

The BC is the original droop controller described in [22], implemented in RTDS
using components in the RTDS standard libraries. The DM is implemented as an
RTDS _custom component_ written in C. For an MG configuration, expressions for
the BaC, $\lambda$ and $\mu$ (see Section 3) are derived in MATLAB, converted
to C data structures, and then included in a header file of the custom
component. The BaCs are polynomials comprising 41, 67, and 92 monomials,
respectively, for configurations 1, 2, and 3.

The NC is trained and implemented using Keras [9], a high-level neural network
API written in Python, running on top of TensorFlow [1]. For training, we
customized an existing skeleton implementation of DDPG in Keras, which we then
used with the Adam optimizer [12]. Hyperparameters used during training
involved a learning rate $lr=0.0001$, discounting factor $\gamma=0.99$, and
target network update weight $\tau=0.001$.

RTDS imposes limitations on custom components that make it difficult to
implement complex NNs within RTDS. Existing NN libraries for RTDS, such as
[18, 19], severely limit the NN’s size and the types of activation functions.
Therefore, we implemented the NC external to RTDS, following the _software-
defined microgrid control_ approach in [36]. Fig. 4 shows our setup. We used
RTDS’s GTNET-SKT communication protocol to establish a TCP connection between
the NC running on a PC and an “NC-to-DM” relay component in the RTDS MG model.
This relay component repeatedly sends the plant state to the NC, which
computes its control action and sends it to the relay component, which in turn
sends it to the DM.

Running the NC outside RTDS introduces control latency. We measured the round-
trip time between RTDS and NC (including the running time of NC on the given
state) to be 4.34 msec. Since the control period is 3.2 msec, each control
action is delayed by one control period. The latency is mostly from network
communication, since the PC running the NC was off-campus. We plan to reduce
the latency by moving the NC to a PC connected to the same LAN as RTDS.

###  5.2 Consistency of RTDS and MATLAB Models

Our methodology requires an analytical model of the microgrid dynamics to
derive a BaC for the BC and a switching condition for the DM. We therefore
developed an analytical model in MATLAB based on the RTDS model and the
description given in [22]. To verify consistency of MATLAB and RTDS models, we
compared trajectories obtained from them under various operating conditions.

Table 1: Voltage deviation (VD) and current deviation (CD) between output of
PV DER in RTDS and MATLAB models

(a) Configuration 1 | VD (kV) | VD (%) | CD (Amp) | CD (%)
---|---|---|---|---
Avg | 0.000214 | 0.04 | 0.000129 | 0.028
Min | 0.000187 | 0.03 | 0.000124 | 0.015
Max | 0.000378 | 0.08 | 0.000181 | 0.036

(b) Configuration 2 | VD (kV) | VD (%) | CD (Amp) | CD (%)
---|---|---|---|---
Avg | 0.000348 | 0.07 | 0.000126 | 0.032
Min | 0.000103 | 0.02 | 0.000104 | 0.019
Max | 0.000493 | 0.10 | 0.000187 | 0.052

(c) Configuration 3 | VD (kV) | VD (%) | CD (Amp) | CD (%)
---|---|---|---|---
Avg | 0.001041 | 0.12 | 0.000238 | 0.047
Min | 0.000119 | 0.02 | 0.000133 | 0.019
Max | 0.001403 | 0.21 | 0.000187 | 0.102

Table 1 reports deviations in output voltage and current trajectories of the
PV DER between the two models under the control of the BC. The results are
based on 100 trajectories starting from random initial states.

As expected, the two models are in close agreement. The small deviations are
due to a few factors: (1) the RTDS model uses realistic dynamic models of
transmission lines including their noise, whereas the MATLAB model ignores
transmission line dynamics; and (2) the RTDS model uses average-value modeling
to more efficiently simulate the dynamics in real-time [22], whereas in
MATLAB, trajectories are calculated by solving ordinary differential equations
of the dynamics at each simulation time-step.

###  5.3 Evaluation of Forward Switching Condition

We derive a BaC using the SOS-based methodology presented in Section 2, and
then derive a switching condition from the BaC, as described in Section 3.1.
To find values of $\lambda$ and $\mu$, we use MATLAB’s fmincon function to
solve the constrained optimization problems given in Eqs. 6 and 7.

An ideal FSC triggers a switch to BC only if an unrecoverable state is
reachable in one time step. For systems with complex dynamics, switching
conditions derived in practice are conservative, i.e., may switch sooner. To
show that our FSC is not overly conservative, we performed experiments using
an AC that continuously increases the voltage and hence soon violates safety.
The PV voltage controller has two outputs, $m_{d}$ and $m_{q}$, for the $d$
and $q$ components of the voltage, respectively. The dummy AC simply uses
constant values for its outputs, with $m_{d}=0.5$ and $m_{q}=1e-6$.

These experiments were performed with PV DER in grid connected mode, with
reference voltage and voltage safety threshold of 0.48 kV and 0.4944 kV,
respectively, and a FSC derived using a $4^{\rm{th}}$-order Taylor
approximation of the BaC. We averaged over 100 runs from initial states with
initial voltage selected uniformly at random from the range $0.48$ kV
$\pm\;1\%$. The mean voltage at switching is 0.4921 kV (with standard
deviation 0.0002314 kV), which is only 0.46% below the safety threshold. The
mean numbers of time steps before switching, and before a safety violation if
_BC-Simplex_ is not used, are 127.4 and 130.2, respectively. Thus, our FSC
triggered a switch about three time steps, on average, before a safety
violation would have occurred.

###  5.4 Evaluation of Neural Controller

The NC for a microgrid configuration is a DNN with four fully-connected hidden
layers of 128 neurons each and one output layer. The hidden layers and output
layer use the ReLU and tanh activation function, respectively. The input state
to the NC (DNN) is the same as the inputs to the BC (droop controller) i.e.,
$[i_{ld}\ i_{lq}]$, where $i_{ld}$ and $i_{lq}$ are the $d$\- and
$q$-components of the input current to the droop controller. Thus the NC has
same inputs and outputs as the BC. The NC is trained on 1 million samples
(one-step transitions) from MATLAB simulations, processed in batches of 200.
Transitions start from random states, with initial values uniformly sampled
from $[0.646,0.714]$ for $i_{ld}$ and $[-0.001,0.001]$ for $i_{lq}$ [22].
Training takes approximately 2 hours. The number of trainable parameters in
the actor and critic networks are 198,672 and 149,111, respectively.

We created an infrastructure for training the NC using samples from RTDS. The
main challenge is setting the RTDS state to a starting state selected by the
training algorithm. RTDS does not provide a native facility for this, and we
needed to use different techniques and some custom components to set the
states of different types of microgrid components. Training with samples from
RTDS would yield a slightly higher-performing controller but would be
significantly slower, due to the overhead of sending states back and forth
between RTDS and the training algorithm running on a PC.

Table 2: Performance comparison of NC and BC

(a) Experimental Results for Configuration 1 Controller | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
NC | 100 | 67.5 | 5.8 | $1.1\mathrm{e}{-4}$ | $1.0\mathrm{e}{-5}$
BC | 100 | 102.3 | 8.2 | $4.2\mathrm{e}{-4}$ | $3.7\mathrm{e}{-5}$

(b) Experimental Results for Configuration 2 Controller | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
NC | 100 | 76.8 | 6.1 | $1.3\mathrm{e}{-4}$ | $1.2\mathrm{e}{-5}$
BC | 100 | 108.8 | 8.3 | $5.1\mathrm{e}{-4}$ | $3.8\mathrm{e}{-5}$

(c) Experimental Results for Configuration 3 Controller | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
NC | 100 | 81.1 | 7.7 | $1.5\mathrm{e}{-4}$ | $1.3\mathrm{e}{-5}$
BC | 100 | 115.7 | 9.8 | $5.8\mathrm{e}{-4}$ | $3.8\mathrm{e}{-5}$

#### Performance

We evaluate a controller’s performance based on three metrics: convergence
rate $(CR)$, the percentage of trajectories in which the DER voltage converges
to the tolerance region $v_{ref}\pm\epsilon$; average convergence time $(CT)$,
the average time required for convergence of the DER voltage to the tolerance
region; and mean deviation $(\delta)$, the average deviation of the DER
voltage from $v_{ref}$ after the voltage enters the tolerance region. We
always report CR as a percentage, CT in milliseconds, and $\delta$ in kV.

We show that the NC outperforms the BC. For this experiment, we used RTDS to
run the BC and NC starting from the same 100 initial states. Table 2 compares
their performance, averaged over 100 runs, with $\epsilon=0.001$. We observe
that for all three configurations, the NC outperforms the BC both in terms of
average convergence time and mean deviation.We also report the standard
deviations $(\sigma)$ for these metrics and note that they are small compared
to the average values. The FSC was not triggered even once during these runs,
showing that the NC is well-trained.

Table 3: Generalization performance of NC

(a) Experimental Results for Configuration 1 | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
Gen 1 | 100 | 108.7 | 9.8 | $1.7\mathrm{e}{-4}$ | $1.5\mathrm{e}{-5}$
Gen 2 | 100 | 77.1 | 6.9 | $1.3\mathrm{e}{-4}$ | $1.1\mathrm{e}{-5}$

(b) Experimental Results for Configuration 2 | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
Gen 1 | 100 | 118.2 | 10.1 | $2.1\mathrm{e}{-4}$ | $1.9\mathrm{e}{-5}$
Gen 2 | 100 | 81.2 | 6.2 | $1.5\mathrm{e}{-4}$ | $1.4\mathrm{e}{-5}$

(c) Experimental Results for Configuration 3 | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
Gen 1 | 100 | 120.4 | 10.8 | $2.2\mathrm{e}{-4}$ | $1.9\mathrm{e}{-5}$
Gen 2 | 100 | 88.5 | 7.3 | $1.6\mathrm{e}{-4}$ | $1.4\mathrm{e}{-5}$

#### Generalization

Generalization refers to the NC’s ability to perform well in contexts beyond
the ones in which it was trained. First, we consider two kinds of
generalization with respect to the microgrid state:

  * •

Gen 1: the initial states of the DERs are randomly chosen from a range outside
of the range used during training.

  * •

Gen 2: the power set-point $P^{\star}$ is randomly chosen from the range
$[0.2,1]$, whereas all training was done with $P^{\star}=1$.

Table 3 presents the NC’s performance in these two cases, based on 100 runs
for each case. We see that the NC performs well in both cases.

Second, we consider generalization with respect to the microgrid
configuration. Here we evaluate how the NC handles dynamic changes to the
microgrid configuration during runtime. For the first experiment, we start
with all the 3 DERs connected, but the diesel generator DER is disconnected
after the voltage has converged. For the second experiment, we again start
with all the 3 DERs connected, but both the diesel generator and battery DER
are disconnected after the voltage has converged. For both instances, the NC
succeeded in continuously keeping the voltage in the tolerance region
($v_{ref}\pm\epsilon$) after the disconnection. The disconnection caused a
slight drop in the subsequent steady-state voltage, a drop of $0.114\%$ and
$0.132\%$, averaged over 100 runs for each case.

Finally, we consider generalization with respect to the microgrid
configuration. We perform two sets of experiment for this. Let NC-$i$ denote
the NC trained for Configuration $i$. In the first set of experiments, we test
the performance of NC-1 for Configuration 2 and NC-2 for Configuration 1 on
100 runs from random initial states. In both cases, the CR was 100%. However,
the mean deviation for NC-1 was 4.7 times larger than when it was used with
Configuration 1. The mean deviation for NC-2 was 2.4 times larger than when it
was used with Configuration 2. We conclude that an NC trained on a more
complex microgrid generalizes better than one trained on a simpler microgrid.

In the second set of experiments, we evaluate how NC-1 and NC-2 handle dynamic
changes to the microgrid configuration, even though no changes occurred during
training. Each run starts with the PV and diesel generator DERs both
connected, and the diesel generator DER disconnected after the voltage has
converged. Both NCs succeed in continuously keeping the voltage in the
tolerance region ($v_{ref}\pm\epsilon$) after the disconnection. The
disconnection causes a slight drop in the subsequent steady-state voltage, a
drop of $0.195\%$ for NC-1 and $0.182\%$ for NC-2.

(a)

(b)

Figure 5: NC with adversarial inputs (left: without NSA, right: with NSA)

#### Adversarial input attacks

We demonstrate that RL-based neural controllers are vulnerable to adversarial
input attacks. We use the gradient-based attack algorithm described in Section
4.2 to generate adversarial inputs for our NCs. We use an adversarial attack
constant $c=0.05$ and the parameters for the beta distributions are $\alpha=2$
and $\beta=4$. From $100,000$ unique initial states, we obtain 8, 6, and 5
adversarial states for Configurations 1, 2, and 3, respectively. In these
experiments, we perturb all state variables simultaneously. In a real-life
attack scenario, an attacker might have the capability to modify only a subset
of them. Nevertheless, our experiments illustrate the fragility of RL-based
neural controllers and the benefits of protecting them with NSA.

We confirmed with simulations that all generated adversarial states lead to
safety violations when the NC alone is used, and that safety is maintained
when _BC-Simplex_ is used. Fig. 5 (left) shows one such case, where the NC
commits a voltage safety violation. The red horizontal line shows the
reference voltage $v_{ref}=0.48$ kV. The black dashed horizontal line shows
the lower boundary of the safety region, 3% below $v_{ref}$. Fig. 5 (right)
shows how _BC-Simplex_ prevents the safety violation. The pink dotted vertical
line marks the switch from NC to BC.

We also confirmed that for all generated adversarial states, the forward
switch is followed by a reverse switch. The time between forward switch and
reverse switch depends on the choice of $m$ (see Section 3.2). In the run
shown in Fig. 5 (right), they are 5 time steps (0.016 sec) apart; the time of
the reverse switch is not depicted explicitly, because the line for it would
mostly overlap the line marking the forward switch. For $m=2,3,4$ with
Configuration 1, the average number of time steps between them are 7 (0.0244
sec), 11 (0.0352 sec), and 16 (0.0512 sec), respectively. For $m=2,3,4$ with
Configuration 2, the average time steps between them are 7 (0.0244 sec), 13
(0.0416 sec), and 17 (0.0544 sec), respectively. For $m=2,3,4$ with
Configuration 3, the average time steps between them are 8 (0.0256 sec), 14
(0.0448 sec), and 19 (0.0608 sec), respectively.

###  5.5 Evaluation of Adaptation Module

To measure the benefits of online retraining, we used the adversarial inputs
described above to trigger switches to BC. For each microgrid configurations,
we ran the original NC from the first adversarial state for that
configuration, performed online retraining while the BC is in control, and
repeated this procedure for the remaining adversarial states for that
configuration except starting with the updated NC from the previous step. As
such, the retraining is cumulative for each configuration. We performed this
entire procedure separately for different RSCs corresponding to different
values of $m$. After the cumulative retraining, we ran the retrained
controller from all of the adversarial states, to check whether the retrained
NC was still vulnerable (i.e., whether those states caused violations).

For Configuration 1, the BC was in control for a total of 56, 88, and 128 time
steps for $m=2,3,4$, respectively. For Configuration 2, the BC was in control
for a total of 42, 78, and 102 time steps for $m=2,3,4$, respectively. For
Configuration 3, the BC was in control for a total of 40, 70, and 95 time
steps for $m=2,3,4$, respectively. For $m=2$, the retrained controllers were
still vulnerable to some adversarial statesfor each configuration. For
$m=3,4$, the retrained controllers were not vulnerable to any of the
adversarial states, and voltage always converged to the tolerance region.

Table 4 compares the performance of the original and retrained NCs for each
configuration, averaged over 100 runs starting from random (non-adversarial)
states. The retraining shows a slight improvement in the performance of the
NC; thus, retraining improves both safety and performance.

A potential concern is whether with online retraining can be done in real-
time; i.e., whether a new retraining sample can be processed within one
control period, so the retrained NC is available as soon as the RSC holds. In
the above experiments, run on a laptop with an Intel i5-6287U CPU, retraining
is done nearly in real-time: on average, the retraining finishes 0.285
milliseconds (less than one-tenth of a control period) after the RSC holds.

Table 4: Performance comparison of original NC and NC retrained by AM

(a) Experimental Results for Configuration 1 NC | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
retrained | 100 | 60.4 | 5.6 | $1.0\mathrm{e}{-4}$ | $1.0\mathrm{e}{-5}$
original | 100 | 67.5 | 5.8 | $1.1\mathrm{e}{-4}$ | $1.0\mathrm{e}{-5}$

(b) Experimental Results for Configuration 2 NC | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
retrained | 100 | 69.4 | 5.3 | $1.1\mathrm{e}{-4}$ | $1.0\mathrm{e}{-5}$
original | 100 | 76.8 | 6.1 | $1.3\mathrm{e}{-4}$ | $1.2\mathrm{e}{-5}$

(c) Experimental Results for Configuration 3 NC | CR | CT | $\sigma(CT)$ | $\delta$ | $\sigma(\delta)$
---|---|---|---|---|---
retrained | 100 | 70.2 | 5.7 | $1.4\mathrm{e}{-4}$ | $1.3\mathrm{e}{-5}$
original | 100 | 81.1 | 7.7 | $1.5\mathrm{e}{-4}$ | $1.3\mathrm{e}{-5}$

##  6 Related Work

The use of BaCs in the Simplex architecture originated in [37]. There are,
however, significant differences between their method for obtaining the
switching condition and ours. Their switching logic involves computing, at
each decision period, the set of states reachable from the current state
within one control period, and then checking whether that set of states is a
subset of the zero-level set of the BaC. Our approach avoids the need for
reachability calculations by using a Taylor approximation of the BaC, and
bounds on the BaC’s derivatives, to bound the possible values of the BaC
during the next control period and thereby determine recoverability of states
reachable during that time. Our approach is computationally much cheaper: a
reachability computation is expensive compared to evaluating a polynomial.
Their framework can handle hybrid systems. Extending our method to hybrid
systems is a direction for future work.

Mehmood et al. [20] propose a distributed Simplex architecture with BCs
synthesized using control barrier functions (CBFs) and with switching
conditions derived from the CBFs, which are BaCs satisfying additional
constraints. A derivation of switching conditions based on Taylor
approximation of CBFs is briefly described but does not consider the remainder
error, admissible states, or restricted admissible states, and does not
include a proof of correctness (which requires an analysis of the remainder
error).

Kundu et al. [14] and Wang et al. [35] use BaCs for safety of microgrids, and
Prajna et al. [29] propose an approach for stochastic safety verification of
continuous and hybrid systems using BaCs. These approaches are based on the
use of verified-safe controllers; they do not allow the use of unverified
high-performance controllers, do not consider switching conditions, etc.

The application of neural networks for microgrid control is gaining in
popularity [17]. Amoateng et al. [2] use adaptive neural networks and
cooperative control theory to develop microgrid controllers for inverter-based
DERs. Using Lyapunov analysis, they prove that their error-function values and
weight-estimation errors are uniformly ultimately bounded. Tan et al. [33] use
Recurrent Probabilistic Wavelet Fuzzy Neural Networks (RPWFNNs) for microgrid
control, since they work well under uncertainty and generalize well. We used
more traditional DNNs, since they are already high performing, and our focus
is on safety assurance. Our _BC-Simplex_ framework, however, allows any kind
of neural network to be used as the AC and can provide the safety guarantees
lacking in their work. Unlike our approach, none of these works provide safety
guarantees.

##  7 Conclusion

We have presented _BC-Simplex_ , a new, provably correct design for runtime
assurance of continuous dynamical systems. _BC-Simplex_ features a new
scalable automated method for deriving, from the barrier certificate,
computationally inexpensive conditions for switching between advanced and
baseline controllers.

We combined _BC-Simplex_ with the Neural Simplex Architecture and applied the
combined framework to micgrogrid control. We conducted an extensive
experimental evaluation of the framework on a realistic model of a microgrid
with multiple types of energy sources. The experiments demonstrate that the
framework can be used to develop high-performance, generalizable neural
controllers (NCs) while assuring specified safety properties, even in the
presence of adversarial input attacks on the NC. Our experiments also
demonstrate that the derived forward switching conditions are not too
conservative, i.e., that they switch control from the NC to the BC only a
short time before a safety violation becomes unavoidable, and that online
retraining of the NC is effective in preventing subsequent safety violations
by the NC.

In future work, we plan to extend our framework to systems with noise or other
sources of uncertainty in the dynamics. We plan to eliminate the need for
complete manually developed analytical dynamic models by learning neural ODEs
[8, 42] that capture unknown parts of the dynamics, and deriving BaCs and
switching conditions from the resulting dynamics. We also plan to apply our
approach to networked microgrids [38].

## References

  * [1] Abadi, M., et al.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015), https://www.tensorflow.org/
  * [2] Amoateng, D.O., Al Hosani, M., Elmoursi, M.S., Turitsyn, K., Kirtley, J.L.: Adaptive voltage and frequency control of islanded multi-microgrids. IEEE Transactions on Power Systems 33(4), 4454–4465 (2018)
  * [3] Anghel, M., Milano, F., Papachristodoulou, A.: Algorithmic construction of Lyapunov functions for power system stability analysis. IEEE Transactions on Circuits and Systems I: Regular Papers 60(9), 2533–2546 (2013)
  * [4] Bak, S., Greer, A., Mitra, S.: Hybrid cyberphysical system verification with simplex using discrete abstractions. In: 16th IEEE Real-Time and Embedded Technology and Applications Symposium. pp. 143–152 (2010)
  * [5] Bak, S., Johnson, T.T., Caccamo, M., Sha, L.: Real-time reachability for verified Simplex design. ACM Transactions on Embedded Computing Systems 2, 138–148 (Jan 2015)
  * [6] Bak, S., Manamcheri, K., Mitra, S., Caccamo, M.: Sandboxing controllers for cyber-physical systems. In: Proceedings of the IEEE/ACM International Conference on Cyber-Physical Systems (ICCPS 2011). pp. 3 –12 (Apr 2011)
  * [7] Borrmann, U., Wang, L., Ames, A.D., Egerstedt, M.: Control barrier certificates for safe swarm behavior. In: Egerstedt, M., Wardi, Y. (eds.) ADHS. IFAC-PapersOnLine, vol. 48, pp. 68–73. Elsevier (2015)
  * [8] Chen, T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS 2018). pp. 6572–6583 (2018)
  * [9] Chollet, F., et al.: Keras (2015), https://github.com/keras-team/keras.git
  * [10] Guerrero, J.M., Vasquez, J.C., Matas, J., de Vicuna, L.G., Castilla, M.: Hierarchical control of droop-controlled AC and DC microgrids — A general approach toward standardization. IEEE Transactions on Industrial Electronics 58(1), 158–172 (2011)
  * [11] Johnson, T.T., Bak, S., Caccamo, M., Sha, L.: Real-time reachability for verified Simplex design. ACM Trans. Embedded Comput. Syst. 15(2), 26:1–26:27 (2016)
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [13] Krishnamurthy, S., Jahns, T.M., Lasseter, R.H.: The operation of diesel gensets in a CERTS microgrid. In: 2008 IEEE Power and Energy Society General Meeting - Conversion and Delivery of Electrical Energy in the 21st Century. pp. 1–8 (2008)
  * [14] Kundu, S., Geng, S., Nandanoori, S.P., Hiskens, I.A., Kalsi, K.: Distributed barrier certificates for safe operation of inverter-based microgrids. In: 2019 American Control Conference (ACC). pp. 1042–1047 (2019)
  * [15] Lasseter, R., Paigi, P.: Microgrid: A conceptual solution. In: 2004 IEEE 35th Annual Power Electronics Specialists Conference (IEEE Cat. No.04CH37551). vol. 6, pp. 4285–4290 (2004)
  * [16] Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. In: ICLR (2016)
  * [17] Lopez-Garcia, T.B., Coronado-Mendoza, A., Domínguez-Navarro, J.A.: Artificial neural networks in microgrids: A review. Engineering Applications of Artificial Intelligence 95, 103894 (2020)
  * [18] Luitel, B., Venayagamoorthy, G.K.: Neural networks in RSCAD for intelligent real-time power system applications. In: 2013 IEEE Power Energy Society General Meeting. pp. 1–5 (2013)
  * [19] Luitel, B., Venayagamoorthy, G.K., Oliveira, G.: Developing neural networks library in RSCAD for real-time power system simulation. In: 2013 IEEE Computational Intelligence Applications in Smart Grid (CIASG). pp. 130–137 (2013)
  * [20] Mehmood, U., Stoller, S.D., Grosu, R., Roy, S., Damare, A., Smolka, S.A.: A distributed Simplex architecture for multi-agent systems. In: Proceedings of the Symposium on Dependable Software Engineering: Theories, Tools and Applications (SETTA 2021). Lecture Notes in Computer Science, vol. 13071. Springer (2021)
  * [21] Mehrizi-Sani, A.: Distributed control techniques in microgrids. In: Mahmoud, M.S. (ed.) Microgrid: Advanced Control Methods and Renewable Energy System Integration, pp. 43–62. Butterworth-Heinemann (2017)
  * [22] Nzimako, O., Rajapakse, A.: Real time simulation of a microgrid with multiple distributed energy resources. In: 2016 International Conference on Cogeneration, Small Power Plants and District Energy (ICUE). pp. 1–6 (2016)
  * [23] O’Rourke, C.J., Qasim, M.M., Overlin, M.R., Kirtley, J.L.: A geometric interpretation of reference frames and transformations: dq0, Clarke, and Park. IEEE Transactions on Energy Conversion 34(4), 2070–2083 (2019)
  * [24] Pattanaik, A., Tang, Z., Liu, S., Bommannan, G., Chowdhary, G.: Robust deep reinforcement learning with adversarial attacks (2017)
  * [25] Phan, D., Grosu, R., Jansen, N., Paoletti, N., Smolka, S.A., Stoller, S.D.: Neural Simplex architecture. In: NASA Formal Methods Symposium (2020)
  * [26] Pogaku, N., Prodanovic, M., Green, T.C.: Modeling, analysis and testing of autonomous operation of an inverter-based microgrid. IEEE Transactions on Power Electronics 22(2), 613–625 (2007)
  * [27] Prajna, S.: Barrier certificates for nonlinear model validation. Automatica 42(1), 117–126 (2006)
  * [28] Prajna, S., Jadbabaie, A.: Safety verification of hybrid systems using barrier certificates. In: Alur, R., Pappas, G.J. (eds.) Proceedings of the 7th International Workshop on Hybrid Systems: Computation and Control (HSCC 2004). Lecture Notes in Computer Science, vol. 2993, pp. 477–492. Springer (2004)
  * [29] Prajna, S., Jadbabaie, A., Pappas, G.J.: A framework for worst-case and stochastic safety verification using barrier certificates. IEEE Transactions on Automatic Control 52(8), 1415–1428 (2007)
  * [30] Seto, D., Krogh, B., Sha, L., Chutinan, A.: The Simplex architecture for safe online control system upgrades. In: Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No. 98CH36207). vol. 6, pp. 3504–3508 (1998)
  * [31] Sha, L.: Using simplicity to control complexity. IEEE Software 18(4), 20–28 (2001)
  * [32] Sha, M., Chen, X., Ji, Y., Zhao, Q., Yang, Z., Lin, W., Tang, E., Chen, Q., Li, X.: Synthesizing barrier certificates of neural network controlled continuous systems via approximations. In: 2021 58th ACM/IEEE Design Automation Conference (DAC). pp. 631–636 (2021)
  * [33] Tan, K.H., Lin, F.J., Shih, C.M., Kuo, C.N.: Intelligent control of microgrid with virtual inertia using recurrent probabilistic wavelet fuzzy neural network. IEEE Transactions on Power Electronics 35(7), 7451–7464 (2020)
  * [34] Ton, D.T., Smith, M.A.: The U.S. department of energy’s microgrid initiative. The Electricity Journal 25(8), 84–94 (2012)
  * [35] Wang, L., Han, D., Egerstedt, M.: Permissive barrier certificates for safe stabilization using sum-of-squares. In: 2018 Annual American Control Conference (ACC). pp. 585–590 (2018)
  * [36] Wang, L., Qin, Y., Tang, Z., Zhang, P.: Software-defined microgrid control: The genesis of decoupled cyber-physical microgrids. IEEE Open Access Journal of Power and Energy 7, 173–182 (2020)
  * [37] Yang, J., Islam, M.A., Murthy, A., Smolka, S.A., Stoller, S.D.: A Simplex architecture for hybrid systems using barrier certificates. In: Proceedings of the 36th International Conference on Computer Safety, Reliability, and Security (SAFECOMP 2017). Lecture Notes in Computer Science, vol. 10488. Springer (2017)
  * [38] Zhang, P.: Networked Microgrids. Cambridge University Press (2021)
  * [39] Zhao, H., Zeng, X., Chen, T., Liu, Z.: Synthesizing barrier certificates using neural networks. In: Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control (HSCC ’20) (2020)
  * [40] Zhao, Q., Chen, X., Zhang, Y., Sha, M., Yang, Z., Lin, W., Tang, E., Chen, Q., Li, X.: Synthesizing ReLU neural networks with two hidden layers as barrier certificates for hybrid systems. In: Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control (HSCC ’21). New York, NY, USA (2021)
  * [41] Zhou, Y., Ngai-Man Ho, C.: A review on microgrid architectures and control methods. In: 2016 IEEE 8th International Power Electronics and Motion Control Conference (IPEMC-ECCE Asia). pp. 3149–3156 (2016)
  * [42] Zhou, Y., Zhang, P.: Neuro-reachability of networked microgrids. IEEE Transactions on Power Systems (2021)