# Reinforcement Learning based Distributed Control of Dissipative Networked
Systems

K. C. Kosaraju, S. Sivaranjani, W. Suttle, V. Gupta, and J. Liu K.C. Kosaraju,
V. Gupta are with the Department of Electrical Engineering, University of
Notre Dame, Notre Dame, IN 46556, USA (email: {kkosaraj, vgupta2}@nd.edu).S.
Sivaranjani is with the Department of Electrical and Computer Engineering,
Texas A&M University, College Station, TX 77843, USA (email:
sivaranjani@tamu.edu).W. Suttle and J. Liu are with the Department of Applied
Mathematics and Statistics and the Department of Electrical and Computer
Engineering at Stony Brook University, Stony Brook, NY 11794, USA (email:
{wesley.suttle, ji.liu}@stonybrook.edu).

###### Abstract

We consider the problem of designing distributed controllers to stabilize a
class of networked systems, where each subsystem is dissipative and designs a
reinforcement learning based local controller to maximize an individual
cumulative reward function. We develop an approach that enforces dissipativity
conditions on these local controllers at each subsystem to guarantee stability
of the entire networked system. The proposed approach is illustrated on a DC
microgrid example, where the objective is maintain voltage stability of the
network using local distributed controllers at each generation unit.

##  I Introduction

Distributed control of large scale networked systems is a classical research
topic, with practical applications in a variety of fields such as
transportation, chemical reaction, and hydraulic networks, multi-body
mechanical systems, and microgrids [1, 2, 3, 4, 5]. The problem provides many
challenges such as non-classical information patterns, computational
complexity due to the large state-space, scalability of control design
methods, complex system dynamics that may be imperfectly known, and so on.
Despite many important advances, the field continues to be a focus of intense
research.

An interesting direction in recent times has been the utilization of
reinforcement learning for distributed and multi-agent control. Reinforcement
Learning (RL) is especially powerful for the control of systems where the
dynamics and/or the environment are unknown [6]. In a typical RL-based design,
the aim is to learn a controller that maximizes its cumulative reward while
exploring the unknown environment. A wide variety of model-based and model-
free algorithms are now available (see, e.g., [7] for a survey). While
initially developed for single agent settings, the scope of RL based
techniques has also been expanded to multi-agent networked systems (see [8, 9,
10] for surveys). Further, while the typical focus of RL-based techniques for
controller design has been through simulations and demonstrations, a growing
line of research now considers obtaining guarantees about concerns traditional
to control theory, e.g., stability, safety, and robustness, through
controllers obtained using RL [11].

In this paper, we consider the problem of guaranteeing stability when RL is
used for distributed control of networked dynamical systems. Specifically,
consider a large scale system consisting of many subsystems that are coupled
through their inputs and outputs, such as a network of microgrids. Each
subsystem designs a local controller based on information about the subsystem
state, inputs, and outputs. In particular, we assume that the controller is
implemented using an RL algorithm since the dynamics of the subsystems may be
unknown. Of note, however, different controllers may potentially use different
RL algorithms. How do we design the controllers that guarantee that the entire
system is still stable? There are at least two challenges here. First, we
would like the control strategy to be distributed. While there exists a wide
literature on RL techniques for multi-agent systems, distributed control
strategies using RL that provide guarantees like stability, safety, and
robustness [12] are still scant. Works that consider the problem of
guaranteeing stability and robustness with RL controllers have largely been
limited to contexts such as model-based RL and LQR designs for single-agent
systems [13, 14, 15, 16]. Second, most available literature on multi-agent RL
considers the case when all subsystems implement the same RL algorithm and
further share information such as a global state or rewards with other
subsystems. Development of RL-based controllers at the subsystems that ensure
stability and robustness for the entire networked system, especially when
different agents may not use the same RL algorithm, largely remains an open
problem.

As a first step towards addressing this problem, we focus on a class of
networked systems where each subsystem is dissipative [17] in open loop.
Dissipativity is an input-output concept that can be used to guarantee a broad
range of useful properties such as $\mathcal{L}_{2}$ stability, robustness
with respect to disturbances, and stability under time-delays [18, 19, 20] and
has been widely used in traditional control theory for distributed controller
synthesis [21, 22, 23, 24, 25, 26, 27, 28, 29]. In the context of RL,
dissipativity has been used to enhance the convergence/performance of various
learning schemes [30] and has been enforced as a system property for specific
systems like Port-Hamiltonian systems [31, 32]. However, there has been
limited literature on enforcing it using model-free RL techniques or on
exploring its potential to permit distributed controller design that
guarantees properties such as stability at the system level. The challenge in
our formulation is that an RL controller aiming to optimize the local
performance metric at a subsystem can easily disrupt the dissipativity of the
subsystem with respect to the variables that it exchanges with the other
subsystems.

In this paper, we develop a reinforcement learning based distributed control
design approach that exploits the dissipativity property of individual
subsystems to guarantee stability of the entire networked system. Our proposed
approach can be summarized as follows. We first use a control barrier function
to characterize the set of controllers that enforce a dissipativity condition
at each subsystem (Propositions 2 and 3). We impose a minimal energy
perturbation on the control input learned by the RL algorithm to project it to
an input in this set (Theorem 3). Together, these results guarantee the
stability of the entire networked system even when the subsystems utilize
potentially heterogeneous RL algorithms to design their local controllers
(Theorem 4).

Our approach of utilizing a control barrier function (CBF) to impose the
constraint that the controller designed for each subsystem using RL preserves
the dissipativity of the subsystem in the closed loop parallels the use of
CBFs to enforce safety in RL algorithms [11]. CBFs guarantee the existence of
control inputs under which a super-level set of a function (typically
representing specifications like safety) is forward invariant under a given
dynamics [33, 34, 35]. However, their use to impose input-output properties
such as dissipativity is less studied. Here, we utilize CBFs to characterize
the set of dissipativity ensuring controllers, and then learn a dissipativity
ensuring controller for each subsystem from this set.

The main contribution of this work is a distributed approach to ensure
stability of a networked system with dissipative subsystems when the
individual subsystems utilize RL to design their own controllers. Beyond the
specific stabilization problem that we focus on, integrating dissipativity
(and other input-output) specifications into RL-based control is useful since
it allows a wide landscape of tools from classical dissipativity theory to be
integrated into RL-based control design. The proposed algorithm guarantees
stability irrespective of the choice of the RL algorithm used at each
subsystem. In particular, the results also hold for heterogeneous RL
algorithms being used at each subsystem. We also note that as opposed to most
existing literature on multi-agent RL, the proposed approach requires only the
output from neighboring subsystems to learn the control policy at each
subsystem. In other words, to guarantee stability, no information about the
states, rewards, or policies of other subsystems is required.

The paper is organized as follows. In Section II, we present the model of the
networked system, state the necessary assumptions, and provide the problem
formulation. In Section III-A, we utilize CBFs to characterize the set of
controllers that guarantees dissipativity of each subsystem. In Section III-B,
we present an RL algorithm to compute a control input that preserves the
dissipativity of each subsystem, and show that it stabilizes the networked
system. In Section IV, we numerically illustrate our approach on a Direct-
Current microgrid application. Finally, in Section V, we provide some
directions for future work. Proofs of all the results in the paper, and the
definitions of dissipativity, are provided in the Appendix.

Notation: $\mathbb{R}^{m}$ denotes the space of $m$-dimensional real vectors,
$\mathbb{R}$ denotes the space of real numbers, and ${\mathbb{R}}_{+}$ denotes
the set of all positive real numbers. $\otimes$ denotes the Kronecker product.
$z^{\top}$ denotes the transpose of a vector or a matrix $z$ and $\|z\|_{2}$
(or simply $\|z\|$) denotes its 2-norm. For a symmetric matrix $M$ and a
vector $z$ of compatible dimensions, $\|z\|^{2}_{M}$ is defined to be equal to
$z^{\top}Mz.$ Given square matrices $M_{1}$, $M_{2}$, $\cdots$, $M_{n}$,
define the matrix $\textrm{diag}(M_{i})$ as the block diagonal matrix whose
main-diagonal blocks are matrices $M_{1}$, $M_{2}$, $\cdots$, $M_{n}$, and all
off-diagonal blocks are zero matrices. For a symmetric matrix $M$,
$\lambda_{min}(M)$ denotes its smallest eigenvalue. $I$ denotes the identity
matrix with dimensions clear from the context. A directed graph
$\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$ is defined by a finite set
of nodes (or vertices) $\mathcal{V}$ and a set of directed edges (or arcs)
$\mathcal{E},$ together with a mapping from $\mathcal{E}$ to the set of pairs
of $\mathcal{V}$. By convention, we disregard self-loops. Thus, to any arc
$e\in{\mathcal{E}}$, there corresponds an ordered pair
$(u,v)\in{\mathcal{V}}\times{\mathcal{V}},$ with $u\neq v,$ representing the
head vertex $u$ and the tail vertex $v$. Given this, a shorthand notation is
to simply say $(u,v)\in{\mathcal{E}}.$ A graph is undirected if whenever
$(u,v)\in{\mathcal{E}}$ then $(v,u)\in{\mathcal{E}}$. The in-neighbor set
$\mathcal{N}_{i}$ of node $i$ is the set of all vertices $j$ such that
$(j,i)\in{\mathcal{E}}.$ Let $\mathcal{D}\subset{\mathbb{R}}^{n}$. A function
$f:\mathcal{D}\rightarrow{\mathbb{R}}^{n}$ is Lipschitz if there exists a
constant $L$ satisfying $\|f(b)-f(a)\|_{2}\leq L\|b-a\|_{2}$ for all
$a,\leavevmode\nobreak\ b\in\mathcal{D}$, and class $C^{1}$ if it is
continuously differentiable. We denote a value obtained by sampling the
probability distribution function $f_{X}(x)$ for a random variable $X$ as
$y\sim f_{X}(x).$ When the random variable is clear from the context, we
denote the distribution function simply by $f(x).$

Figure 1: Schematic of the system configuration

##  II Problem Formulation

We adapt the general framework described in [23] and shown in Figure 1.

Node dynamics: Consider a networked system described by a directed graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where each node $i\in V$ is a
subsystem $\Sigma^{i}_{n}$, given by

$\displaystyle\begin{split}\Sigma_{\mathrm{n}}^{i}:\leavevmode\nobreak\
\left\\{\begin{matrix}x_{t+1}^{i}\\\ y^{i}_{u,t}\\\
y^{i}_{\nu,t}\end{matrix}\begin{matrix}=\\\ =\\\
=\end{matrix}\begin{matrix}f^{i}(x^{i}_{t},u^{i}_{t},\nu^{i}_{t})\\\
g^{i}(x^{i}_{t},u^{i}_{t})\\\
h^{i}(x^{i}_{t},\nu^{i}_{t})\end{matrix}\right.\end{split}$ (1)

where at time $t$, $x^{i}_{t}\in\mathbb{R}^{n_{i}}$ denotes the state of the
$i$-th subsystem, $u^{i}_{t}\in\mathbb{R}^{m_{i}}$ denotes the control input
applied by the subsystem controller that needs to be designed, and
$\nu^{i}_{t}\in\mathbb{R}^{p_{i}}$ is the input to the $i$-th subsystem that
depends on the output of the other subsystems in the in-neighbor set of node
$i$. The subsystem has two outputs:
$y_{u,t}^{i}\in\mathbb{R}^{\overline{o}_{i}}$ which is the output that is used
to design the control input $u_{t}^{i}$, and
$y_{\nu,t}^{i}\in\mathbb{R}^{\hat{o}_{i}}$ which is the output that is used to
compute the inputs $\nu^{j}_{t}$ for other subsystems $j$ for whom $i$ is an
in-neighbor. We will define the exact relation between $\nu^{i}_{t}$ and
$y_{t}^{j}$, $j\in\mathcal{N}_{i}$, later. Given that each subsystem
corresponds to a unique node in the graph, we use the terms subsystem dynamics
and node dynamics interchangeably. We assume that the state transition
function $f^{i}$ and the output functions $g^{i},\leavevmode\nobreak\ h^{i}$
are of Class $C^{1}$. Without loss of generality we assume that
$(x^{i}=0,u^{i}=0,\nu^{i}=0)$ is an equilibrium point of the subsystem
$\Sigma_{\mathrm{n}}^{i}$.

For future reference, define
$x^{\top}\triangleq[x^{1\top},\ldots,x^{N\top}]\in{\mathbb{R}}^{n}$,
$u^{\top}\triangleq[u^{1\top},\ldots,u^{N\top}]\in{\mathbb{R}}^{m}$,
$y^{\top}_{u}\triangleq[y^{1\top}_{u},\ldots,y^{N\top}_{u}]\in{\mathbb{R}}^{\overline{o}}$,
$y^{\top}_{\nu}\triangleq[y^{1\top}_{\nu},\ldots,y^{N\top}_{\nu}]\in{\mathbb{R}}^{\hat{o}}$,
$y^{i}\triangleq[y^{i\top}_{u},y^{i\top}_{\nu}]\in{\mathbb{R}}^{o}$,
$y^{\top}\triangleq[y^{1\top},\ldots,y^{N\top}]\in{\mathbb{R}}^{o}$, and
$\nu^{\top}\triangleq[\nu^{1\top},\ldots,\nu^{N\top}]\in{\mathbb{R}}^{p}$.

As stated earlier, definitions of dissipativity are provided in Appendix -A
for the sake of completeness. We make the following assumption throughout the
paper.

######  Assumption 1 (Dissipative node dynamics).

Each subsystem $\Sigma_{n}^{i}$ with dynamics defined in (1) is dissipative,
in the set $\mathcal{S}^{i}_{n}$, with respect to the supply function

$\displaystyle\begin{split}w^{i}_{n}(u^{i},\nu^{i},y_{u}^{i},y_{\nu}^{i})=&\underbrace{u^{i\top}S^{i\top}_{u}y_{u}^{i}-\|u^{i}\|^{2}_{R_{u}^{i}}-\|y_{u}^{i}\|^{2}_{Q_{u}^{i}}}_{\triangleq
w^{i}_{u}(u^{i},y^{i}_{u})}\\\
&+\underbrace{\nu^{i\top}S^{i\top}_{\nu}y_{\nu}^{i}-\|\nu^{i}\|^{2}_{R^{i}_{\nu}}-\|y_{\nu}^{i}\|^{2}_{Q^{i}_{\nu}}}_{\triangleq
w^{i}_{\nu}(\nu^{i},y^{i}_{\nu})},\end{split}$ (2)

where $S^{i}_{u},\leavevmode\nobreak\
R_{u}^{i}=\left(R_{u}^{i}\right)^{\top},\leavevmode\nobreak\
Q_{u}^{i}=\left(Q_{u}^{i}\right)^{\top},$ $S^{i}_{\nu},\leavevmode\nobreak\
R^{i}_{\nu}=\left(R_{\nu}^{i}\right)^{\top},$ and
$Q^{i}_{\nu}=\left(Q_{\nu}^{i}\right)^{\top}$ are matrices of appropriate
dimensions.

For future reference, define $S_{u}\triangleq\textrm{diag}(S_{u}^{i})$,
$R_{u}\triangleq\textrm{diag}(R_{u}^{i})$,
$Q_{u}\triangleq\textrm{diag}(Q_{u}^{i})$,
$S_{\nu}\triangleq\textrm{diag}(S_{\nu}^{i})$,
$R_{\nu}\triangleq\textrm{diag}(R_{\nu}^{i})$, and
$Q_{\nu}\triangleq\textrm{diag}(Q_{\nu}^{i})$. Further, denote
$\epsilon_{\nu}=\lambda_{min}\left(Q_{\nu}\right)$,
$\delta_{\nu}=\lambda_{min}\left(R_{\nu}\right)$,
$\epsilon_{u}=\lambda_{min}\left(Q_{u}\right)$,
$\delta_{u}=\lambda_{min}\left(R_{u}\right)$,
$\epsilon_{e}=\lambda_{min}\left(Q_{e}\right)$ and
$\delta_{e}=\lambda_{min}\left(R_{e}\right)$.

######  Remark 1.

Even though Assumption 1 states that the subsystem is dissipative, it is an
assumption in the ‘open loop’. Note that the design of the controller that
determines the inputs $u^{i}$ has not been specified. The dissipativity
property required for system stability concerns the inputs $\mu^{i}$ and the
outputs $y_{\mu}^{i}$ and this may easily be disrupted by the additional
dynamics, say of the form $u^{i}=\zeta^{i}(x^{i})$, introduced through the
design of the controller. For a simple illustration of this fact, note that
from [36, Corollary 4.1.5], Assumption 1 holds if and only if the condition

$\displaystyle\sum_{t=t_{0}}^{t-1}\sum_{i=1}^{N}\left(w^{i}_{u}(u^{i}_{t},y^{i}_{u_{t}})+w^{i}_{\nu}(\nu^{i},y^{i}_{\nu})\right)\geq
0,$ (3)

holds for all $0\leq t_{0}\leq t$. Consider subsystem (1) in closed-loop with
a Lipschitz controller $u^{i}=\zeta^{i}(x^{i})\in{\mathbb{R}}^{m_{i}}$. Then,
we notice that

$\displaystyle\sum_{t=t_{0}}^{t-1}\sum_{i=1}^{N}w^{i}_{\nu}(\nu^{i}_{t},y^{i}_{\nu,t})\geq-\sum_{t=t_{0}}^{t-1}\sum_{i=1}^{N}w^{i}_{u}(\zeta^{i}(x^{i}_{t}),y^{i}_{u,t})\ngeq
0,$ (4)

which implies that unless the controller has been designed to ensure that
$w^{i}_{u}(\zeta^{i}(x^{i}),y^{i}_{u})\leq 0$, dissipativity of the subsystem
in the closed loop with the controller may not be preserved.

Edge dynamics: While the simplest form of coupling among the subsystems would
be to equate the inputs $\nu^{i}_{t}$ for the subsystem $i$ with the output
$y_{t}^{j}$ of subsystem $j$ if $(j,i)\in\mathcal{E}$, inspired by [23], we
consider a more general model that allows the edges in the graph $\mathcal{G}$
to be described a dynamic system as well. Specifically for edge
$k\in{\mathcal{E}}$, the dynamics are given by

$\displaystyle\begin{split}\Sigma_{\mathrm{e}}^{k}:\leavevmode\nobreak\
\left\\{\begin{matrix}z_{t+1}^{k}\\\
\omega^{k}_{t}\end{matrix}\begin{matrix}=\\\
=\end{matrix}\begin{matrix}g^{k}(z_{t}^{k},\mu_{t}^{k})\\\
j^{k}(z_{t}^{k},\mu_{t}^{k})\end{matrix}\right.\end{split}$ (5)

where $z^{k}_{t}\in{\mathbb{R}}^{q_{i}}$ denotes the edge subsystem state at
time $t$, $\mu^{k}_{t}\in{\mathbb{R}}^{r_{i}}$ denotes the input at time $t$,
and $\omega^{k}_{t}\in{\mathbb{R}}^{s_{i}}$ denotes the output at time $t$. We
assume that the state transition function $g^{k}$ and the output function
$j^{k}$ are of Class $C^{1}$. Once again, without loss of generality we assume
that $(z^{k}=0,\mu^{k}=0)$ is an equilibrium point of the subsystem
$\Sigma_{\mathrm{e}}^{k}$. For future reference, define
$z^{\top}\triangleq[z^{1\top},\ldots,z^{M\top}]\in{\mathbb{R}}^{q}$,
$\omega^{\top}\triangleq[\omega^{1\top},\ldots,\omega^{M\top}]\in{\mathbb{R}}^{s}$,
and $\mu^{\top}\triangleq[\mu^{1\top},\ldots,\mu^{M\top}]\in{\mathbb{R}}^{r}$,
where $M$ denotes the cardinality of the set ${\mathcal{E}}$.

######  Assumption 2 (Dissipative edge dynamics).

Each subsystem $\Sigma_{e}^{k}$ with its dynamics defined in (5) is
dissipative in the set $\mathcal{S}_{e}^{k}$ with supply-function

$\displaystyle w^{k}_{e}(\mu^{k},\omega^{k})=$
$\displaystyle\mu^{k\top}S^{k\top}_{e}\omega^{k}-\|\mu^{k}\|^{2}_{R^{k}_{e}}-\|\omega^{k}\|^{2}_{Q^{k}_{e}},$
(6)

where $S^{k}_{e},\leavevmode\nobreak\
R^{k}_{e}=\left(R^{k}_{e}\right)^{\top},\leavevmode\nobreak\
Q^{k}_{e}=\left(Q^{k}_{e}\right)^{\top}$ are matrices of appropriate
dimensions.

For future reference, define $S_{e}\triangleq\textrm{diag}(S_{e}^{k}$),
$R_{e}\triangleq\textrm{diag}(R_{e}^{k}$), and
$Q_{e}\triangleq\textrm{diag}(Q_{e}^{k})$.

Interconnection among subsystems: The entire networked system is defined
through the interconnection of the subsystems defined by the nodes and edges
by relating the inputs $\nu$ and outputs $y_{\nu}$ of the node subsystems with
the inputs $\mu$ and outputs $\omega$ of the edge subsystems as specified
below. Define $s^{\top}\triangleq[x^{\top},z^{\top}]$ as the state variable of
the overall network. Further, define

$\displaystyle\begin{split}w_{u}(u,y_{u})&\triangleq\left(u^{\top}S^{\top}_{u}y_{u}-\|u\|^{2}_{R_{u}}-\|y_{u}\|_{Q_{u}}^{2}\right),\\\
w_{\nu}(\nu,y_{\nu})&\triangleq\left(\nu^{\top}S^{\top}_{\nu}y_{\nu}-\|\nu\|_{R_{\nu}}^{2}-\|y_{\nu}\|_{Q_{\nu}}^{2}\right),\\\
w_{e}(\mu,\omega)&\triangleq\left(\mu^{\top}S^{\top}_{e}\omega-\|\mu\|_{R_{e}}^{2}-\|\omega\|_{Q_{e}}^{2}\right).\end{split}$
(7)

Following [23], we model the interconnection among the subsystems through the
equation

$\displaystyle\Sigma_{\mathrm{i}}:\leavevmode\nobreak\ \begin{bmatrix}\nu\\\
\mu\end{bmatrix}=\begin{bmatrix}0&\mathcal{B}\\\
-\mathcal{B}^{\top}&0\end{bmatrix}\begin{bmatrix}y_{\nu}\\\
\omega\end{bmatrix}$ (8)

for a suitably defined matrix $\mathcal{B}$. Further, we make the following
assumption.

######  Assumption 3.

Matrices $S_{\nu}$ and $S_{e}$ in (7) satisfy

$\displaystyle\mathcal{B}^{\top}S_{\nu}^{\top}-S_{e}\mathcal{B}^{\top}=0$ (9)

An interpretation of (8) and Assumption 3 is that the edges of the system do
not generate any energy. Although equation (8) appears intricate, most
interconnected physical systems can be written in this form (see [23] for
examples from various domains; an example of interconnected distributed
generation units is discussed in detail below). Similarly, several relevant
subclasses of dissipative systems including, but not limited to,
$\mathcal{L}_{2}$ gain systems and passive systems satisfy Assumption 3, see
[22] for other examples. For future reference, denote

$\displaystyle\mathcal{B}_{\delta}(x)$
$\displaystyle\triangleq\epsilon_{e}I+x\mathcal{B}^{\top}\mathcal{B},$ (10)
$\displaystyle\mathcal{B}_{\epsilon}(y)$ $\displaystyle\triangleq
yI+\delta_{e}\mathcal{B}\mathcal{B}^{\top}.$ (11)

$R^{i}$$I^{i}_{t}$$L^{i}$$-$$+$$u^{i}_{t}V_{s}^{i}$$V_{t}^{i}$$G^{i}$$C^{i}$$I_{t}^{k}$$R^{k}_{l}$$L^{k}_{l}$DGU
$i$ (node $i$)Line $k$ (edge $k$) Figure 2: Electrical scheme of DGU $i$ and
transmission line $k$ as considered in Example 1.
$1$$2$$4$$3$$I_{l,t}^{1}$$I_{l,t}^{2}$$I_{l,t}^{3}$$I_{l,t}^{4}$ Figure 3: The
topology of network considered in Example 1.

######  Example 1.

Consider the electrical schematic of a microgrid, containing four Distributed
Generating Units (DGUs) and interconnected through four transmission lines, as
shown in Figures 2 and 3. The DGUs correspond to the nodes and the
transmission lines correspond to the edges of the graph describing this
networked system. Let the DGUs and the transmission lines be numbered as shown
in Fig 3. Each DGU contains a DC-DC buck converter that is operating on a
constant impedance load. The controller to be designed sets
$u_{t}^{i}\in(0,1)$ for the $i$-th DGU. Denote by $I_{t}^{k}$ the current
through the $k$-th transmission line at time $t$ and by $V_{t}^{i}$ the
voltage across the $i$-th DGU at time $t$. Define the state of the subsystem
at the $i$-th node (corresponding to the $i$-th DGU) by
$x_{t}^{i}\triangleq\left[\begin{array}[]{lr}I_{t}^{i}&V_{t}^{i}\end{array}\right]^{\top}.$
The dynamics of the $DGU$ at node $i\in\mathcal{V}:=\left\\{1\ldots
4\right\\}$, which forms the $i$-th subsystem, can be written as

$\displaystyle\begin{split}I_{t+1}^{i}&=I_{t}^{i}-(T_{s}/L^{i})(R^{i}I^{i}_{t}+V^{i}_{t}-u^{i}_{t}V_{s})\\\
V_{t+1}^{i}&=V_{t}^{i}+(T_{s}/C^{i})(I^{i}_{t}-G^{i}V^{i}_{t}+\nu^{i}_{t}),\end{split}$
(12)

where $T_{s},L^{i},\leavevmode\nobreak\ C^{i},\leavevmode\nobreak\
R^{i},\leavevmode\nobreak\ G^{i},\leavevmode\nobreak\
V_{s}^{i}\in{\mathbb{R}}_{>0}$ are constants,
$u^{i}_{t}\in(0,\leavevmode\nobreak\ 1)$ is the local control input to be
designed, and $\nu_{t}^{i}\in{\mathbb{R}}$ is the input to the $i$-th
subsystem that depends on the output of the other subsystems in its in-
neighbor set through the relations

$\left[\begin{array}[]{c}\nu_{t}^{1}\\\ \nu_{t}^{2}\\\ \nu_{t}^{3}\\\
\nu_{t}^{4}\end{array}\right]=\left[\begin{array}[]{c}I_{l,t}^{4}-I_{l,t}^{1}\\\
I_{l,t}^{1}-I_{l,t}^{2}\\\ I_{l,t}^{2}-I_{l,t}^{3}\\\
I_{l,t}^{3}-I_{l,t}^{4}\end{array}\right],$ (13)

where $I_{l,t}^{k}$ denotes the current through the edge $k$. We denote the
outputs $y_{\nu,t}^{i}\triangleq V_{t}^{i}.$

The edges correspond to the transmission lines connected to each DGU. The
dynamics of the transmission line at edge $k\in\mathcal{E}:=\left\\{1\ldots
4\right\\}$ are given by

$\displaystyle\begin{split}I_{l,t+1}^{k}&=I_{l,t}^{k}-(T_{s}/L_{l}^{k})(R_{l}^{k}I_{l,t}^{k}+\mu^{k}_{t})\\\
\omega_{t}^{k}&=I_{l,t}^{k}\end{split}$ (14)

where $L_{l}^{k},\leavevmode\nobreak\ R_{l}^{k}\in{\mathbb{R}}_{>0}$ are
constants, $I_{l,t}^{k}\in{\mathbb{R}}$ denotes the state variable, and
$\mu^{k}_{t}\in{\mathbb{R}}$ denotes the input from the nodes connected to the
edge $k$ defined as

$\left[\begin{array}[]{c}\mu_{t}^{1}\\\ \mu_{t}^{2}\\\ \mu_{t}^{3}\\\
\mu_{t}^{4}\end{array}\right]=\left[\begin{array}[]{c}V_{t}^{2}-V_{t}^{1}\\\
V_{t}^{3}-V_{t}^{2}\\\ V_{t}^{4}-V_{t}^{3}\\\
V_{t}^{1}-V_{t}^{4}\end{array}\right].$ (15)

Define the incidence matrix $\mathcal{B}\in{\mathbb{R}}^{4\times 4}$ to model
the network topology. Specifically, if the ends of each edge $k$ are
arbitrarily labeled with a $+$ and a $-$, then the entries of ${\mathcal{B}}$
are given by

${\mathcal{B}}_{ik}=\begin{cases}+1\quad&\text{if $i$ is the positive end of
$k$}\\\ -1\quad&\text{if $i$ is the negative end of $k$}\\\
0\quad&\text{otherwise}.\end{cases}$

The interconnection between the nodes and edges can then be expressed as

$\displaystyle\begin{bmatrix}\nu_{t}\\\
\mu_{t}\end{bmatrix}=\begin{bmatrix}0&\mathcal{B}\\\
-\mathcal{B}^{\top}&0\end{bmatrix}\begin{bmatrix}y_{\nu,t}\\\
\omega_{t}\end{bmatrix}=\begin{bmatrix}\mathcal{B}I_{l,t}\\\
\mathcal{B}^{\top}V_{t}\end{bmatrix}$ (16)

Controller design: We assume that each subsystem $i$ wishes to design its
controller to maximize the expected discounted cumulative reward,

$\displaystyle
J^{i}=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}(x_{t}^{i},u_{t}^{i})\right],$
(17)

where $\gamma\in(0,1)$ is the discount factor,
$r^{i}_{t}(x_{t}^{i},u_{t}^{i})$ is the per step reward function evaluated at
time $t$, and the expectation is over any stochasticity that may arise due to
the control policy itself. We assume that each agent utilizes a RL algorithm
to design its controller. For a given control policy $\pi^{i}$, we define the
value function $V_{\pi}^{i}$, and the state-action value function
$Q_{\pi}^{i}$ below:

$\displaystyle V_{\pi}^{i}(x^{i})$
$\displaystyle=\mathbb{E}_{\pi^{i}}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}(x^{i}_{t},u^{i}_{t})\
|\ x^{i}_{0}=x^{i}\right],$ (18) $\displaystyle Q^{i}_{\pi}(x^{i},u^{i})$
$\displaystyle=\mathbb{E}_{\pi^{i}}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}(x^{i}_{t},u^{i}_{t})\
|\ x^{i}_{0}=x^{i},u^{i}_{0}=u^{i}\right],$ (19) $\displaystyle
A_{\pi}^{i}(x^{i},u^{i})$
$\displaystyle=Q^{i}_{\pi}(x^{i},u^{i})-V_{\pi}^{i}(x^{i}).$ (20)

Note that we do not assume that each subsystem utilizes the same RL algorithm.
However, we assume that the RL algorithms converge.

Problem statement: Equations (1), (5) and (8) jointly define the networked
system $\Sigma$ under consideration, with state defined as
$s_{t}^{\top}\triangleq\left[x_{t}^{\top},z_{t}^{\top}\right].$ From
Assumption 1, we know that the each subsystem $i$ is dissipative with the
supply-function $w^{i}_{u}(u^{i},y^{i}_{u})+w^{i}_{\nu}(\nu^{i},y^{i}_{\nu})$.
However, since the subsystems use RL to design their local controllers, the
closed loop subsystems may not remain dissipative (see Remark 1). Further, the
control actions of all the subsystems may end up destabilizing the entire
networked system. We are interested in the problem of how to design the RL
algorithm at each subsystem to guarantee the stability of the networked
system. Specifically, consider a networked system on a directed graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, described by (1), (5), and (8), and
satisfying Assumptions 1, 2 and 3. Assume that the controller at each
subsystem $i$ is designed using an RL algorithm to maximize the discounted
cumulative reward $J^{i}$ in (17). How should the updates in the RL algorithms
be done so that the control policies at convergence guarantee Lyapunov
stability of the overall networked system?

##  III Dissipativity ensuring Reinforcement Learning

In this section, we present the main results of the paper through a new
distributed RL algorithm that guarantees the stability of the entire networked
system. The proposed approach is as follows.

  * (a)

Control barrier functions for dissipativity: As stated in Remark 1, even
though each subsystem $i$ is dissipative with supply-function
$w^{i}_{u}(u^{i},y^{i}_{u})+w^{i}_{\nu}(\nu^{i},y^{i}_{\nu}),$ with the
controller for the input $u^{i}$ the subsystem may no longer remain
dissipative with the input-output pair $w^{i}_{\nu}(\nu^{i},y^{i}_{\nu})$. Our
first step is to utilize control barrier functions to characterize the set of
all controllers that ensure that the closed loop subsystem $i$ is dissipative
with respect to the input $\nu^{i}$ and output $y^{i}$ (c.f. Fig 1) with the
supply function

$\displaystyle w_{d}^{i}(\nu^{i},y^{i})$
$\displaystyle=\nu^{i\top}S_{\nu}^{i\top}y_{\nu}^{i}-\delta^{i}_{d}\|\nu^{i}\|^{2}_{2}-\epsilon^{i}_{d}\|y^{i}\|^{2}_{2},$
(21)

where $\delta^{i}_{d}\in{\mathbb{R}}$ and $\epsilon^{i}_{d}\in{\mathbb{R}}$
are tuning parameters set by the designer.

  * (b)

Projection-based RL algorithm for dissipativity: In the second step, at each
subsystem $i$, we consider the control input generated by an RL algorithm that
seeks to maximize the discounted cumulative reward given by (17) and use a
quadratic program (QP) to project this control input onto the set of control
inputs that ensure that the closed loop subsystem remains dissipative with
supply-function $w^{i}_{d}(\nu^{i},y^{i}_{\nu})$. Note that the RL algorithms
used at different nodes can be different.

  * (c)

Networked system stability: We finally show that if each subsystem designs the
controller to ensure that it is dissipative, the entire networked system is
also stable.

We now develop these steps one by one. We will make the following assumption
in the sequel.

######  Assumption 4.

Denote $\alpha=\min(\delta_{d}^{1},\ldots,\delta_{d}^{N})$, and
$\beta=\min(\epsilon_{d}^{1},\ldots,\epsilon_{d}^{N})$. The conditions

$\displaystyle\begin{split}\mathcal{B}_{\delta}(\alpha)&\geq 0,\\\
\mathcal{B}_{\epsilon}(\beta)&\geq 0,\end{split}$ (22)

hold, where $\mathcal{B}_{\delta},$ and $\mathcal{B}_{\epsilon}$ have been
defined in (10).

###  III-A Control barrier functions for dissipativity

Control barrier functions (CBFs) are now a popular tool for enforcing safety
constraints in nonlinear control systems. The following definition follows the
development in [37, 38, 39].

######  Definition 1 (Time-varying Zeroing Control Barrier Functions).

Consider a function
$b:\mathbb{R}_{+}\times\mathbb{R}^{n+q}\rightarrow\mathbb{R}$ that is
continuously differentiable in both arguments. Define a closed set
$\mathcal{C}$ as the super-level set of this function as follows:

$\displaystyle\mathcal{C}\triangleq\left\\{s\in{\mathbb{R}}^{n+q}\;|\;b(t,s)\geq
0\right\\}.$ (23)

The function $b(t,s_{t})$ is a time-varying zeroing control barrier function,
for the networked system $\Sigma$ described by (1), (5) and (8) and with state
$s_{t}$, if there exists an $\eta\in[0,\leavevmode\nobreak\ 1]$ such that for
all $s_{t}\in\mathcal{C}$, $t\in\mathbb{R}_{+}$,

$\displaystyle\sup_{u_{t}\in{\mathbb{R}}^{m}}\left[b(t+1,s_{t+1})+(\eta-1)b(t,s_{t})\right]\geq
0.$ (24)

Control barrier functions can be used to derive sufficient conditions under
which a super-level set of a function of the state of the networked system
$\Sigma$ is forward invariant. These conditions also characterize the set of
control inputs achieving such forward invariance through the relation

$\displaystyle\mathrm{B}(t,s_{t})\triangleq\left\\{u_{t}\in{\mathbb{R}}^{m}|b(t+1,s_{t+1})+(\eta-1)b(t,s_{t})\geq
0\right\\}.$ (25)

The following result, given for completeness for a discrete time setting such
as ours, shows that the set $\mathcal{C}$ defined in (23) is forward invariant
for every $u_{t}\in\mathrm{B}(t,s_{t})$.

######  Proposition 1 (Discrete-time time-varying Control Barrier Functions).

Consider a time-varying zeroing control barrier function $b(t,s_{t})$ and its
super level set $\mathcal{C}$ defined in (23). Then any Lipschitz input
$u_{t}\in\mathrm{B}(t,s_{t})$, where $\mathrm{B}(t,s_{t})$ is given in (25),
will render the set $\mathcal{C}$ forward invariant.

Although dissipativity is a property defined by the input, and the output, we
can utilize control barrier functions to characterize the set of controllers
that ensures dissipativity in the closed loop of the subsystems, which in turn
guarantee the stability of the overall networked system [40] . Following
Proposition 1, we define a control barrier function for each subsystem $i$ as
follows. Denote

$\displaystyle\tilde{w}^{i}(u^{i},\nu^{i},y_{u}^{i},y_{\nu}^{i})\triangleq
w_{n}^{i}(u^{i},\nu^{i},y_{u}^{i},y_{\nu}^{i})-w_{d}^{i}(\nu^{i},y^{i}).$ (26)

Then, define the control barrier function

$b^{i}(t,x^{i}_{t})\triangleq-\sum_{\tau=t_{0}}^{t-1}\tilde{w}^{i}(u^{i}_{\tau},\nu^{i}_{\tau},y_{u_{,}\tau}^{i},y_{\nu,\tau}^{i}),$
(27)

whose super-level set is given by

$\displaystyle\mathcal{C}^{i}=\left\\{x^{i}_{t}\in{\mathbb{R}}^{n_{i}}\;|\;b^{i}(t,x^{i}_{t})\geq
0\right\\}.$ (28)

To use the control barrier function $b^{i}(t,x^{i}_{t})$ to enforce
dissipativity of the closed loop subsystem, we proceed as follows. Denote

$\mathrm{D}^{i}(x^{i}_{t},\nu_{t}^{i})\triangleq\\{u^{i}\in{\mathbb{R}}^{m_{i}}|-\tilde{w}^{i}(u^{i}_{t},\nu^{i}_{t},y_{u,t}^{i},y_{\nu,t}^{i})\\\
+\eta^{i}b^{i}(t,x^{i}_{t})\geq 0\\},$ (29)

where $\eta^{i}\in[0,\leavevmode\nobreak\ 1]$ is a designer specified
parameter. We can then state the following result.

######  Proposition 2 (Control barrier function for dissipativity).

Consider the problem formulation in Section II. If
$u^{i}\in\mathrm{D}^{i}(x^{i},\nu^{i})$ at all time steps, then the subsystem
(1) is dissipative with respect to input $\nu^{i}$ and output $y^{i}$ with
supply function $w^{i}_{d}(\nu^{i},y^{i}_{\nu})$.

From Proposition 2, if the set $\mathrm{D}^{i}(x^{i}_{t},\nu_{t}^{i})$ is non-
empty, then any control input
$u_{t}^{i}\in\mathrm{D}^{i}(x^{i}_{t},\nu_{t}^{i})$ renders (1) dissipative
with respective to the supply function
$w^{i}_{d}(\nu_{t}^{i},y_{\nu_{t}}^{i})$. We can choose a particular control
input in this set from other considerations, such as minimizing the control
cost. We can also use this set to ensure that the control input from an RL
algorithm ensures that the subsystem is dissipative as shown next.

###  III-B Dissipativity ensuring RL policies

We now consider the case when an RL algorithm is used for designing the
control inputs $u^{i}$ and show how the input can be chosen to one that
preserves the dissipativity of the closed-loop subsystem $\Sigma_{n}^{i}$ with
respective to the supply function $w^{i}_{d}(\nu^{i},y_{\nu}^{i})$. The key
idea is similar to shielded RL techniques [11, 41, 42] and uses the control
barrier function based characterization of the set of dissipativity ensuring
controllers obtained above to both project the control policy and to guide the
future exploration of the RL algorithm.

We assume that the RL algorithm proceeds in an episodic fashion. Let
$\pi^{\mathrm{RL}_{i}}_{k}$ denote the policy at the $k$-th policy iteration
of the RL algorithm. This policy will in general be stochastic and may be
parameterized by some parameters $\theta_{k}^{i}$ that may correspond to,
e.g., the neural network being used to learn the policy. The paramterization
is not relevant to our arguments and to minimize notational complexity, we
suppress it in the sequel. Let
$u^{\mathrm{RL_{i}}}_{k}(x_{t}^{i})\sim\pi^{\mathrm{RL}_{i}}_{k}(\cdot|x_{t}^{i})$.
Our algorithm proceeds by projecting this input on the set of dissipativity
ensuring controllers. Specifically, we propose that the overall dissipativity
ensuring control input in the $k$-th episode takes the following structure:

$\displaystyle u_{k}^{\mathrm{DEC}_{i}}(x_{t}^{i})$
$\displaystyle=u_{k}^{\mathrm{FF}_{i}}(x^{i}_{t})+u_{k}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{k}^{\mathrm{FF}_{i}}),$
(30)

where $u_{k}^{\mathrm{FF}_{i}}(x^{i})$ represents the feedforward
compensation, given by

$\displaystyle u_{k}^{\mathrm{FF}_{i}}(x^{i}_{t})$
$\displaystyle=u^{\mathrm{RL_{i}}}_{k}(x^{i}_{t})+\sum_{j=0}^{k-1}u_{j}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{j}^{\mathrm{FF}_{i}}(x^{i}_{t})),$
(31)

and $u_{k}^{\mathrm{CBF}_{i}}$ is computed using the optimization problem:

$\displaystyle u_{k}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{k}^{\mathrm{FF}_{i}})=$
$\displaystyle\arg\min_{a_{t}^{i}\in{\mathbb{R}}^{m_{i}}}\|a_{t}^{i}\|$ (32)
s.t.
$\displaystyle-\tilde{w}(u_{t}^{i},\nu_{t}^{i},y_{u_{t}}^{i},y_{\nu}^{i})+\eta^{i}b^{i}(t,x^{i}_{t})\geq
0,$ $\displaystyle a_{t}^{i}+u_{k}^{\mathrm{FF}_{i}}(x^{i}_{t})=u_{t}^{i}.$

As in the usual control barrier function based works, the formulation in the
relation (30) seeks to minimize the energy of the perturbation needed to
project the control input in the set of dissipativity ensuring controllers
[11, 37]. The feedforward compensation in (31) is split into two parts:
$u_{k}^{\mathrm{RL}_{i}}(x^{i})$ represents the control input obtained from
the RL policy. However, this might not ensure dissipativity of the closed loop
subsystem. The second term in (31) represents our best guess to rectify the
input to ensure dissipativity. Furthermore, the term $u^{\mathrm{CBF}_{i}}$ in
(30) may be interpreted as the feedback part of the controller. The complete
algorithm description is given in Algorithm 1.

for  _$i=1,\ldots,N$_ do

Initialize RL input $\pi_{0}^{\mathrm{RL}_{i}}$, and arrays $\hat{D}^{i}$ and
$\hat{A}^{i}$.

end for

for  _$t=0,\ldots,T$_ do

for  _$i=1,\ldots,N$_ do

Sample $u^{\mathrm{RL}_{i}}_{0}(x_{t}^{i})\sim\pi^{\mathrm{RL}_{i}}_{0}$ and
compute $u_{0}^{\mathrm{CBF}_{i}}(x_{t}^{i},u_{0}^{FF_{i}})$ using (32).

Deploy
$u_{0}^{i}(x^{i}_{t})=u^{\mathrm{RL}_{i}}_{0}(x^{i}_{t})+u_{0}^{\mathrm{CBF}_{i}}(x_{t}^{i},u_{0}^{FF_{i}})$

Store state-action pairs
$(x_{t}^{i},u_{0}^{\mathrm{CBF}_{i}}(x_{t}^{i},u_{0}^{FF_{i}}))$ in
$\hat{A}^{i}$

end for

for  _$i=1,\ldots,N$_ do

Observe $x^{i}_{t},u_{0}^{i}(x^{i}_{t}),x_{t+1}^{i},r_{t}^{i}$ and store in
$\hat{D}^{i}$ for use in the RL algorithm

end for

end for

for  _$i=1,\ldots,N$_ do

Collect Episode Reward $\sum_{t=1}^{T}r_{t}^{i}$

end for

Set $k=1$ (representing the $k$-th episode or input iteration step)

while  _$k <$ Max_Episodes_ do

for  _$i=1,\ldots,N$_ do

Do input iteration using RL algorithm based on previously observed episode to
obtain $\pi^{\mathrm{RL}_{i}}_{k}$

end for

Initialize state $s_{0}$ from an initial state distribution

for  _t = 0,…, T_ do

for  _$i=1,\ldots,N$_ do

Compute the feed-forward term
$u_{k}^{\mathrm{FF}_{i}}(x_{t}^{i})=u^{\mathrm{RL_{i}}}_{k}(x_{t}^{i})+\sum_{j=0}^{k-1}u_{j}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{j}^{FF_{i}}(x_{t}^{i}))$

Use (32) solve for $u_{k}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{k}^{FF_{i}})$

Deploy controller
$u_{k}^{i}(x^{i}_{t})=u_{k}^{\mathrm{FF}_{i}}(x_{t}^{i})+u_{k}^{\mathrm{CBF}_{i}}(x^{i}_{t},u_{k}^{\mathrm{FF}_{i}}(x_{t}^{i}))$

Store state-action pairs $(x_{t}^{i},u_{k}^{i}(x^{i}_{t}))$

end for

for  _$i=1,\ldots,N$_ do

Observe $x^{i}_{t},u_{k}^{i}(x^{i}_{t}),x_{t+1}^{i},r_{t}^{i}$ and store in
$\hat{D}^{i}$ for use in the RL algorithm

end for

end for

$k=k+1$

end while

Algorithm 1 RL-DEC algorithm.

We assume that the parameter $Max\\_Episodes$ has been chosen to be large
enough that the algorithm converges. Upon convergence, denote
$u^{\textrm{DEC}_{i}}(x_{t}^{i})$ to be the final deployed controller
$u_{k}^{i}(x_{t}^{i})$ for $k=Max\\_Episodes$. The following result shows that
Algorithm 1 renders the closed loop subsystem dissipative. For brevity, we
skip the proof as it is a direct consequence of Proposition 2 and Definition
2.

######  Proposition 3.

Consider the problem formulation in Section II. Let the controller
$u^{\textrm{DEC}_{i}}(x_{t}^{i})$ designed with Algorithm 1 be used as the
input $u_{t}^{i}$ for the subsystem (1). If there exists a solution to the
optimization problem (32) for all $(x^{i},\nu^{i})$, then the closed-loop
subsystem (30) is dissipative with supply function
$w^{i}_{d}(\nu^{i},y_{\nu}^{i})$.

######  Remark 2.

Computing $u_{k}^{\mathrm{FF}_{i}}(x)$ requires the solution of the
optimization problem $k$ times; further, the knowledge of all
$u^{\mathrm{RL_{i}}}_{0},\ldots,u^{\mathrm{RL}_{i}}_{{k-1}}$ is required.
Consequently, for large $k$, the proposed algorithm can become memory
intensive and computationally expensive. However, we need not compute
$u_{k}^{\mathrm{FF}_{i}}(x)$ very accurately because of the presence of the
feedback term $u_{k}^{CBF_{i}}$. This raises the possibility of approximating
$u_{k}^{\mathrm{FF}_{i}}(x)$ by using a feed-forward neural network
$u^{\mathrm{bar}}_{\phi_{k}}$ to learn the term
$\sum_{j=0}^{k-1}u_{j}^{\mathrm{CBF}_{i}}$. In this case, (31) should be
replaced by

$\displaystyle u_{k}^{\mathrm{FF}_{i}}(x)$
$\displaystyle=u^{\mathrm{RL_{i}}}_{k}(x)+u^{\mathrm{bar}}_{\phi_{k}^{i}}(x),$
(33)

where $\phi_{k}$ parameterizes the neural network, which is updated using the
data from previously collected samples.

The following is the main result of the paper, which shows that the controller
calculated using Algorithm 1 stabilizes the networked system.

######  Theorem 4 (Stability of networked system in closed-loop).

Consider the problem formulation in Section II with Assumption 4. If
$u_{t}^{i}$ is chosen to be equal to $u^{\mathrm{DEC}_{i}}(x_{t}^{i})$ at all
time steps and for all subsystems $i$, then the networked system defined by
(1), (5) and (8) is Lyapunov stable with respect to the origin. Further,
suppose that $\mathcal{B}_{\delta}(\alpha)>0$,
$\mathcal{B}_{\epsilon}(\beta)>0$, and
$R_{u}\triangleq\textrm{diag}(R_{u}^{i})>0$. If the systems (1), and (5) are
zero state detectable, then the networked system defined by (1), (5), and (8)
is also asymptotically stable with respect to the origin.

The definition of zero-state detectability is provided in Definition 3 of
Appendix A.

######  Remark 3 (Decentralized and Distributed).

In (32), each agent needs to evaluate $\tilde{w}$ which requires the
information of $\nu_{t}$. From (8), computing $\nu_{t}$ requires information
from its neighbours. Then, the proposed RL algorithm is distributed. However,
in the event when the desired supply-function $w_{d}$ is equal to $w_{\nu}$,
then $\tilde{w}=w_{u}$. Consequently, the RL algorithm takes a decentralized
form.

##  IV Case study: DC Microgrid

We now evaluate the proposed control barrier function based RL Algorithm 1 in
simulation. We consider the DC microgrid in Example 1 with 4 DGU’s,
interconnected through resistive and inductive lines as shown in Figure 3. The
control objective is to regulate the voltage $V^{i}$ across the load of each
DGU’s to its desired value $\overline{V}^{i}\in{\mathbb{R}}$. Thus, we define
the set of all feasible forced equiliria of the node subsystems (12) and the
edge subsystems (14) as

$\displaystyle\mathcal{C}_{i}^{n}=\left\\{(\overline{I}^{i},\overline{V}^{i},\overline{u}^{i},\overline{\nu}^{i})\in\mathbb{R}^{4}|\right.$
$\displaystyle\left.R^{i}\overline{I}^{i}+\overline{V}^{i}-\overline{u}^{i}V_{s}^{i}=0,\right.$
(34)
$\displaystyle\left.\overline{I}^{i}-G\overline{V}^{i}+\overline{\nu}^{i}=0\right\\},$

and

$\displaystyle\mathcal{C}_{k}^{e}=\left\\{(\overline{I}_{l}^{i},\overline{\mu}^{i})\in\mathbb{R}^{2}|R_{l}^{i}\overline{I}_{l}^{i}+\overline{\mu}^{i}=0\right\\},$
(35)

respectively. In the development above, we have assumed that $(s=0)$ is the
desired equilibrium. However, the results are agnostic to the choice of the
equilibrium. Since the objective in this case study is to stabilize the system
at a non-trivial operating point
$(\overline{I}^{i},\overline{V}^{i},\overline{u}^{i},\overline{\nu}^{i},\overline{I}_{l}^{i},\overline{\mu}^{i})\in\mathcal{C}_{i}^{n}\times\mathcal{C}_{k}^{e}$,
we shift the equilibrium of the networked system to the trivial equilibrium
via a simple change of variables. In what follows, for a given variable $\nu$,
denote the error between $\tilde{\nu}=\nu-\overline{\nu}$.

In [43], the authors show that the subsystems at the node (12) and the edge
(14) are dissipative with the supply-functions

$\displaystyle\begin{split}w^{i}_{n}(\tilde{u}^{i},\tilde{\nu}^{i},\tilde{y}_{u}^{i},\tilde{y}_{\nu}^{i})=&\underbrace{\tilde{u}^{i\top}\tilde{y}_{u}^{i}-R^{i}\|\tilde{y}_{u}^{i}\|^{2}_{2}}_{w^{i}_{u}(\tilde{u}^{i},\tilde{y}^{i}_{u})}+\underbrace{\tilde{\nu}^{i\top}\tilde{y}_{\nu}^{i}-G^{i}\|\tilde{y}_{\nu}^{i}\|^{2}_{2}}_{w^{i}_{\nu}(\tilde{\nu}^{i},\tilde{y}^{i}_{\nu})}\end{split}$
(36)

and

$\displaystyle\begin{split}w^{k}_{e}(\tilde{\mu}^{k},\tilde{\omega}^{k})=&\tilde{\mu}^{k\top}\tilde{\omega}^{k}-R^{k}_{l}\|\tilde{\omega}^{k}\|^{2}_{2},\end{split}$
(37)

respectively. As a next step, we define the desired supply function
corresponding to (21) as

$\displaystyle
w_{d}^{i}(\tilde{\nu}^{i},\tilde{y}^{i})=w^{i}_{\nu}(\tilde{\nu}^{i},\tilde{y}^{i}_{\nu})-R^{i}\|\tilde{y}_{u}^{i}\|^{2}_{2}$

where we chose $\delta_{d}^{i}=0,\leavevmode\nobreak\ \epsilon_{d}^{i}=R^{i}$,
which satisfies equation (22) in Assumption 4. Consequently, using (26) we
compute the resulting control barrier function as

$\displaystyle
b^{i}(t,x^{i}_{t})=-\sum_{\tau=t_{0}}^{t-1}\left(\tilde{u}^{i\top}\tilde{y}_{u}^{i}-G^{i}\|\tilde{y}_{\nu}^{i}\|^{2}_{2}\right),\;t\geq
t_{0}\geq 0,$ (38)

and its super-level is defined as in (28).

Figure 4: (Top) time evolution of control barrier function, (bottom) voltage
across the load of each DGU, considering a load variation of $5\%$ at time
$t=0.05$ seconds.

Finally, we define the instantaneous reward function at each node as

$\displaystyle r^{i}(V^{i}):=-k^{i}\left(\tilde{V^{i}}\right)^{2}$ (39)

where $k^{i}\in{\mathbb{R}}_{>0}$. For numerical simulation, the parameters of
the microgrids are taken from [43, Tables 3, and 4].

Though the general framework described in the preceding can be used with
almost any RL algorithm, we chose to use Deep Deterministic Policy Gradient
(DDPG) [44] to showcase the performance of Algorithm 1. Figure 5 compares the
accumulated rewards of vanilla DDPG and the proposed dissipativity-ensuring
Algorithm 1 using DDPG during training. As the plot shows, Algorithm 1 coupled
with DDPG converges faster that the vanilla DDPG algorithm; however, this may
not be a general observation.

Next, we validate the performance of the controllers designed using the
proposed approach. The voltage across the load and the value of the control
barrier function at each node are plotted in Figure 4. At $t=0$ seconds, we
start by initializing the microgrid near the desired operating point. We
observe that the voltage signals stabilize to their desired values. However,
in the DC microgrid, the value of load $G^{i}$ is unknown and subject to
change over time. To verify the robustness of the controller with respect to
this uncertainty, the load at each DGU was increased by $5\%$ of its original
value at $t=0.05$ seconds. In Figure 4 we see that, after a minor
perturbation, the voltage signals again stabilized to their desired values.
Furthermore, the control barrier function is positive, thus validating the
dissipativity-ensuring nature of the proposed approach.

Figure 5: Comparison of accumulated rewards from nodes of DC microgrid for
each episode during training using DDPG and the propose Dissipative CBF
approach.

##  V Conclusions

In this paper, we considered the problem of designing distributed controllers
to stabilize a class of networked systems, where each subsystem is
dissipative. We assumed that each subsystem designs a local controller using
reinforceent learning to optimize its own reward function. We develop an
approach that enforces dissipativity conditions on the local controller design
to guarantee stability of the entire networked system. The proposed approach
was illustrated on a microgrid example.

###  -A Dissipativity

Consider the following discrete time nonlinear system with state
$x\in{\mathbb{R}}^{n}$ and inputs $a\in{\mathbb{R}}^{m}$

$\displaystyle\begin{split}\left\\{\begin{matrix}x_{t+1}\\\
y_{t}\end{matrix}\leavevmode\nobreak\ \begin{matrix}=\\\
=\end{matrix}\leavevmode\nobreak\ \begin{matrix}f(x_{t},a_{t}),\\\
h(x_{t},a_{t}).\end{matrix}\right.\end{split}$ (40)

where the functions $f,\leavevmode\nobreak\ h$ as assumed to be sufficiently
smooth. Consider the mapping
$w:{\mathbb{R}}^{m},{\mathbb{R}}^{m}\rightarrow{\mathbb{R}}$. Then,
dissipativity of system $\Sigma$ with $w(a_{t},y_{t})$ as supply-function is
defined as follows:

######  Definition 2 (Dissipativity [45]).

System (40) is said to be dissipative with respect to the supply function
$w(a_{t},y_{t})$, if there exist a non-negative function
$S:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}_{+}$, called as storage function,
satisfying $S(0)=0$ such that for all $s_{t_{0}}\in X$, all $t>t_{0}\geq 0$
and all $a_{t}\in A$,

$\displaystyle
S(x_{t})-S(s_{t_{0}})\leq-\sum_{i=t_{0}}^{t-1}\mathcal{D}(x_{t})+\sum_{i=t_{0}}^{t-1}w(a_{t},y_{t}),$
(41)

or equivalently [46],

$\displaystyle\sum_{i=t_{0}}^{t-1}w(a_{t},\leavevmode\nobreak\
y_{t})\geq\sum_{i=t_{0}}^{t-1}\mathcal{D}(x_{t})\geq 0$ (42)

where $\mathcal{D}(x_{t})\in\mathbb{R}_{+}$ is a non-negative function, and
$s_{t}$ is the state at time $t$, resulting from state $s_{t-1}$ with input
$u_{t-1}$. Furthermore, we call the system $QSR$ dissipative if the inequality
(42) holds with

$\displaystyle\begin{split}w(a_{t},y_{t})&=-\|y_{t}\|_{Q}^{2}+a_{t}^{\top}Sy_{t}-\|a_{t}\|_{R}^{2}\end{split}$
(43)

where $Q=Q^{\top}$, $S$, and $R=R^{\top}$ are matrices of appropriate
dimensions.

######  Definition 3 (zero-state detectability).

Consider (40) with $f(0,0)=0$, and $h(0,0)=0$. Then system (40) is called
zero-state detectable if

$\displaystyle a_{t}=0\leavevmode\nobreak\ \textit{and}\leavevmode\nobreak\
y_{t}=0\leavevmode\nobreak\ \implies\lim_{t\rightarrow\infty}x_{t}\rightarrow
0.$

###  -B Proofs

#### Proof of Proposition 1

Without loss of generality, we assume the initial state as $s_{0}\in\rho_{0}$
at time $t=0$ and $b(0,s_{0})\geq 0$. It suffices to show that $b(t,s_{t})\geq
0$, for all $a_{t}\in\mathrm{DEC}(t,s_{t})$. From (24) and (25), for all
$a_{t}\in\mathrm{DEC}(t,s_{t})$, we have

$\displaystyle b(t+1,s_{t+1})\geq(1-\eta)b(t,s_{t}).$ (44)

Now, consider the following boundary value problem:

$\displaystyle X_{t+1}$ $\displaystyle=(1-\eta)X_{t}$ (45)

with initial condition $X_{0}=b(0,s_{0})\geq 0$. Then, the solution to (45) is
$X_{t}=(1-\eta)^{t}X_{0}\geq 0$, $\forall k\in\mathbb{Z}^{+}$, $0<\eta\leq 1$.
From (44) and (45),

$\displaystyle b(t,s_{t})\geq X_{t}.$ (46)

Thus $\mathcal{C}$ is forward invariant.

#### Proof of Proposition 2

Consider the barrier function $b^{i}(t,x^{i}_{t})$ defined in (28). From
Proposition 1, for all $u_{t}\in\mathrm{D}^{i}(x^{i}_{t},\nu_{t})$, it implies
that $\mathcal{C}^{i}$ is forward invariant. Consequently, we have
$b^{i}(t,x^{i}_{t})=-\sum_{\tau=t_{0}}^{t-1}\tilde{w}\geq 0$

$\displaystyle\implies$ $\displaystyle\sum_{\tau=t_{0}}^{t-1}\tilde{w}\leq 0$
(47) $\displaystyle\implies$
$\displaystyle\sum_{\tau=t_{0}}^{t-1}(w_{n}-w_{d})\leq 0$ (48)
$\displaystyle\implies$
$\displaystyle\sum_{\tau=t_{0}}^{t-1}w_{d}\geq\sum_{\tau=t_{0}}^{t-1}w_{n}.$
(49)

From Assumption 1 the subsystem (1) is dissipative, which further implies

$\displaystyle\implies$
$\displaystyle\sum_{\tau=t_{0}}^{t-1}w_{d}\geq\sum_{\tau=t_{0}}^{t-1}w_{n}\geq
0.$ (50)

From Definition 2, we conclude the proof.

#### Proof of Theorem 4

As a consequence of Assumption 1, Proposition 3 implies that node dynamics in
closed-loop with control input (30) are dissipative with supply function (21)
$w_{d}^{i}(\nu^{i},y^{i})$. Consequently, for all $i\in{\mathcal{V}}$ there
exist a storage function
$S_{d}^{i}:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}_{+}$, satisfying

$\displaystyle S_{d}^{i}(x^{i}_{t})\leq
S_{d}^{i}(x^{i}_{t_{0}})+\sum_{t=t_{0}}^{t-1}w_{d}^{i}(\nu^{i},y^{i}).$ (51)

From Assumption 2, the edge dynamics are dissipative with supply-function
$w_{e}^{k}(\mu^{k},\omega^{k})$. Consequently, for all
$k\in\left\\{1,\dots,M\right\\}$, there exist a storage function
$S_{e}^{i}:{\mathbb{R}}^{m}\rightarrow{\mathbb{R}}_{+}$, satisfying

$\displaystyle S_{e}^{k}(z^{k}_{t})\leq
S_{e}^{i}(z^{k}_{t_{0}})+\sum_{t=t_{0}}^{t-1}w^{k}_{e}(\mu_{t}^{k},\omega_{t}^{k}).$
(52)

Consider
$S(s_{t})=\sum_{i=1}^{N}S_{d}^{i}(x^{i}_{t})+\sum_{k=1}^{M}S_{e}^{k}(z^{k}_{t})$,
consequently

$\displaystyle S(s_{t})-S(s_{t_{0}})$ (53a) $\displaystyle\leq$
$\displaystyle\sum_{i=1}^{N}\sum_{t=t_{0}}^{t-1}w_{d}^{i}(\nu^{i},y^{i})+\sum_{k=1}^{M}\sum_{t=t_{0}}^{t-1}w^{k}_{e}(\mu_{t}^{k},\omega_{t}^{k})$
$\displaystyle=$
$\displaystyle\sum_{t=t_{0}}^{t-1}\sum_{i=1}^{N}w_{d}^{i}(\nu^{i},y^{i})+\sum_{t=t_{0}}^{t-1}\sum_{k=1}^{M}w^{k}_{e}(\mu_{t}^{k},\omega_{t}^{k})$
$\displaystyle\leq$
$\displaystyle\sum_{t=t_{0}}^{t-1}\left(\nu^{\top}S_{\nu}^{\top}y_{\nu}-\alpha\|\nu\|^{2}_{2}-\beta\|y\|^{2}_{2}+\mu^{\top}S_{e}^{\top}\omega-\delta_{e}\|\mu\|_{2}^{2}\right.$
$\displaystyle\left.-\epsilon_{e}\|\omega\|_{2}^{2}\right)$ (53b)
$\displaystyle\leq$
$\displaystyle\sum_{t=t_{0}}^{t-1}\left(\omega^{\top}\mathcal{B}^{\top}S_{\nu}^{\top}y_{\nu}-\alpha\|\mathcal{B}\omega\|^{2}_{2}-\beta\|y_{\nu}\|^{2}_{2}-\beta\|y_{u}\|^{2}_{2}\right.$
$\displaystyle\left.-y_{\nu}^{\top}\mathcal{B}^{\top}S_{e}^{\top}\omega-\delta_{e}\|\mathcal{B}y_{\nu}\|_{2}^{2}-\epsilon_{e}\|\omega\|_{2}^{2}\right)$
(53c) $\displaystyle=$
$\displaystyle-\sum_{t=t_{0}}^{t-1}\left(\|\omega\|^{2}_{\mathcal{B}_{\delta}(\alpha)}+\|y_{\nu}\|^{2}_{\mathcal{B}_{\epsilon}(\beta)}+\beta\|y_{u}\|^{2}_{2}\right)$
(53d)

In (53a) we use (51) and (52). In (53b) we use the interconnection laws from
(8). In (53c), we use Assumption 3. This implies that the overall networked
system is stable.

Furthermore, consider $\mathcal{B}_{\delta}(\alpha)>0$, and
$\mathcal{B}_{\epsilon}(\beta)>0$. Then from (53d) there exists a forward
invariant set $\Pi$ and by LaSalle’s invariance principle, the solutions that
start in $\Pi$ converge to the largest invariant set contained in

$\displaystyle\Pi\cap\left\\{s\in{\mathbb{R}}^{n+p}|\leavevmode\nobreak\
\omega=0,\leavevmode\nobreak\ y=0\right\\}.$ (54)

Moreover, from (8) this implies $\mu=0,\leavevmode\nobreak\ \nu=0$. From
Assumption 1 and $R_{u}>0$ this further implies that $u=0$. Finally on this
set, we have $(y=0,u=0,\nu=0)$ and $(\omega=0,\mu=0)$. Given that that
subsystems (1) and (5) are zero-state detectable, the trajectories in $\Pi$
converges asymptotically to the largest invariant set contained in

$\displaystyle\Pi\cap\left\\{s=0\right\\},$ (55)

following [36, Corollary 4.2.2].

## References

  * [1] M. Egerstedt and X. Hu, “Formation constrained multi-agent control,” _IEEE transactions on robotics and automation_ , vol. 17, no. 6, pp. 947–951, 2001.
  * [2] S. Sivaranjani, S. Sadraddini, V. Gupta, and C. Belta, “Distributed control policies for localization of large disturbances in urban traffic networks,” in _2017 American Control Conference (ACC)_. IEEE, 2017, pp. 3542–3547.
  * [3] T. Dragičević, X. Lu, J. C. Vasquez, and J. M. Guerrero, “Dc microgrids—part i: A review of control strategies and stabilization techniques,” _IEEE Transactions on power electronics_ , vol. 31, no. 7, pp. 4876–4891, 2015.
  * [4] F. Horn and R. Jackson, “General mass action kinetics,” _Archive for rational mechanics and analysis_ , vol. 47, no. 2, pp. 81–116, 1972.
  * [5] R. H. Lasseter and P. Paigi, “Microgrid: A conceptual solution,” in _2004 IEEE 35th Annual Power Electronics Specialists Conference (IEEE Cat. No. 04CH37551)_ , vol. 6. IEEE, 2004, pp. 4285–4290.
  * [6] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [7] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” _Journal of artificial intelligence research_ , vol. 4, pp. 237–285, 1996.
  * [8] L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 38, no. 2, pp. 156–172, 2008.
  * [9] K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _arXiv preprint arXiv:1911.10635_ , 2019.
  * [10] ——, “Decentralized multi-agent reinforcement learning with networked agents: Recent advances,” _arXiv preprint arXiv:1912.03821_ , 2019.
  * [11] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick, “End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3387–3395.
  * [12] L. Buşoniu, T. de Bruin, D. Tolić, J. Kober, and I. Palunko, “Reinforcement learning for control: Performance, stability, and deep approximators,” _Annual Reviews in Control_ , vol. 46, pp. 8–28, 2018.
  * [13] F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, “Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers,” _IEEE Control Systems Magazine_ , vol. 32, no. 6, pp. 76–105, 2012.
  * [14] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” in _Advances in neural information processing systems_ , 2017, pp. 908–918.
  * [15] M. Fazel, R. Ge, S. M. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” _arXiv preprint arXiv:1801.05039_ , 2018.
  * [16] K. Zhang, B. Hu, and T. Başar, “Policy optimization for $\mathcal{H}_{2}$ linear control with $\mathcal{H}_{\infty}$ robustness guarantee: Implicit regularization and global convergence,” _arXiv preprint arXiv:1910.09496_ , 2019.
  * [17] J. C. Willems, “Dissipative dynamical systems part ii: Linear systems with quadratic supply rates,” _Archive for rational mechanics and analysis_ , vol. 45, no. 5, pp. 352–393, 1972.
  * [18] c. J. Van der Schaft, _$L_{2}$ -Gain and Passivity Techniques in Nonlinear Control_. Springer, 2000, vol. 2.
  * [19] C. A. Desoer and M. Vidyasagar, _Feedback systems: input-output properties_. SIAM, 2009.
  * [20] G. Niemeyer and J. E. Slotine, “Stable adaptive teleoperation,” _IEEE Journal of Oceanic Engineering_ , vol. 16, no. 1, pp. 152–162, 1991\.
  * [21] N. Chopra and M. W. Spong, “Passivity-based control of multi-agent systems,” in _Advances in robot control_. Springer, 2006, pp. 107–134.
  * [22] M. Arcak, C. Meissen, and A. Packard, _Networks of dissipative systems: compositional certification of stability, performance, and safety_. Springer, 2016.
  * [23] A. Van der Schaft and B. Maschke, “Port-hamiltonian systems on graphs,” _SIAM Journal on Control and Optimization_ , vol. 51, no. 2, pp. 906–937, 2013.
  * [24] E. Agarwal, _Compositional Control of Large-Scale Cyber-Physical Systems Using Hybrid Models and Dissipativity Theory_. University of Notre Dame, 2019.
  * [25] E. Agarwal, S. Sivaranjani, V. Gupta, and P. J. Antsaklis, “Distributed synthesis of local controllers for networked systems with arbitrary interconnection topologies,” _IEEE Transactions on Automatic Control_ , 2020.
  * [26] K. C. Kosaraju, M. Cucuzzella, J. M. A. Scherpen, and R. Pasumarthy, “Differentiation and passivity for control of brayton-moser systems,” _IEEE Transactions on Automatic Control_ , 2020.
  * [27] M. J. Tippett and J. Bao, “Dissipativity based distributed control synthesis,” _Journal of Process Control_ , vol. 23, no. 5, pp. 755–766, 2013\.
  * [28] S. Sivaranjani, E. Agarwal, L. Xie, V. Gupta, and P. Antsaklis, “Mixed voltage angle and frequency droop control for transient stability of interconnected microgrids with loss of pmu measurements,” in _2020 American Control Conference (ACC)_ , 2020, pp. 2382–2387.
  * [29] E. Agarwal, S. Sivaranjani, V. Gupta, and P. J. Antsaklis, “Sequential synthesis of distributed controllers for cascade interconnected systems,” in _2019 American Control Conference (ACC)_. IEEE, 2019, pp. 5816–5821.
  * [30] B. Gao and L. Pavel, “On passivity, reinforcement learning and higher-order learning in multi-agent finite games,” _IEEE Transactions on Automatic Control_ , 2020.
  * [31] S. P. Nageshrao, G. A. Lopes, D. Jeltsema, and R. Babuška, “Passivity-based reinforcement learning control of a 2-dof manipulator arm,” _Mechatronics_ , vol. 24, no. 8, pp. 1001–1007, 2014.
  * [32] O. Sprangers, R. Babuška, S. P. Nageshrao, and G. A. Lopes, “Reinforcement learning for port-hamiltonian systems,” _IEEE transactions on cybernetics_ , vol. 45, no. 5, pp. 1017–1027, 2014.
  * [33] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs for safety critical systems,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 8, pp. 3861–3876, 2016.
  * [34] M. Z. Romdlony and B. Jayawardhana, “Uniting control lyapunov and control barrier functions,” in _53rd IEEE Conference on Decision and Control_. IEEE, 2014, pp. 2293–2298.
  * [35] P. Wieland and F. Allgöwer, “Constructive safety using control barrier functions,” _IFAC Proceedings Volumes_ , vol. 40, no. 12, pp. 462–467, 2007\.
  * [36] A. J. van der Schaft, _$L_{2}$ -gain and passivity techniques in nonlinear control_. Springer, London, 2000.
  * [37] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in _2019 18th European Control Conference (ECC)_. IEEE, 2019, pp. 3420–3431.
  * [38] X. Xu, P. Tabuada, J. W. Grizzle, and A. D. Ames, “Robustness of control barrier functions for safety critical control,” _IFAC-PapersOnLine_ , vol. 48, no. 27, pp. 54–61, 2015.
  * [39] G. Notomista and M. Egerstedt, “Persistification of robotic tasks,” _IEEE Transactions on Control Systems Technology_ , 2020.
  * [40] G. Notomista, X. Cai, J. Yamauchi, and M. Egerstedt, “Passivity-based decentralized control of multi-robot systems with delays using control barrier functions,” in _2019 International Symposium on Multi-Robot and Multi-Agent Systems (MRS)_. IEEE, 2019, pp. 231–237.
  * [41] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [42] J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin, “A general safety framework for learning-based control in uncertain robotic systems,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 7, pp. 2737–2752, 2018.
  * [43] M. Cucuzzella, K. C. Kosaraju, and J. Scherpen, “Voltage control of dc networks: robustness for unknown zip-loads,” _arXiv preprint arXiv:1907.09973_ , 2019.
  * [44] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [45] E. Navarro-López, D. Cortés, and E. Fossas-Colet, “Implications of dissipativity and passivity in the discrete-time setting,” _IFAC Proceedings Volumes_ , vol. 35, no. 1, pp. 55–60, 2002.
  * [46] M. Xia, P. J. Antsaklis, V. Gupta, and M. J. McCourt, “Determining passivity using linearization for systems with feedthrough terms,” _IEEE Transactions on Automatic Control_ , vol. 60, no. 9, pp. 2536–2541, 2014.


# Comments on:
FANO VARIETIES WITH TORSION IN $H^{3}(X,{\mathbb{Z}})$
by J. C. OTTEM and J. V. RENNEMO

János Kollár<EMAIL_ADDRESS>
###### Abstract.

Ottem and Rennemo constructed a Fano 4-fold with torsion in
$H_{2}(X,{\mathbb{Z}})$. We simplify the computation of
$H_{2}(X,{\mathbb{Z}})$ and also exhibit 2 lines
$L^{\prime},L^{\prime\prime}\subset X$ such that $L^{\prime}-L^{\prime\prime}$
generates the torsion.

[OR23] constructs a smooth Fano 4-fold $X$ such that
$H_{2}(X,{\mathbb{Z}})\cong{\mathbb{Z}}+{\mathbb{Z}}/2$. Below is a shorter
computation of $H_{2}(X,{\mathbb{Z}})$, see Claims 2 and 4.

We also add two new results. In Claim 6 we exhibit 2 lines
$L^{\prime},L^{\prime\prime}\subset X$ such that $L^{\prime}-L^{\prime\prime}$
generates the torsion in
$H_{2}(X,{\mathbb{Z}})\cong{\mathbb{Z}}+{\mathbb{Z}}/2$.

Then in Paragraph 14 we show that $X$ is birational to a double cover of
${\mathbb{P}}^{4}$ ramified along a degree 18 hypersurface $R$, which is
obtained as the 5-secants of a degree 15, smooth, determinantal surface
$S=\bigl{(}\operatorname{rank}N\leq 4\bigr{)}\subset{\mathbb{P}}^{4},$ where
$N$ is a $6\times 5$ matrix whose entries are linear forms. Although $S$ is
smooth, it is not a general determinantal surface, since the latter have only
1-parameter families of 5-secants.

The higher dimensional examples constructed in [OR23] can also be treated with
minor changes.

We refer to [Roo38, Chaps.VIII-IX] for symmetric determinantal varieties and
to [Kro1890] for the classification of lines on them.

######  1Basic set-up.

Following [OR23], let $Z_{i}\subset{\mathbb{P}}^{14}$ be the space of rank
$\leq i$ quadrics in ${\mathbb{P}}^{4}$. Our main interest is
$Z=Z_{4}\subset{\mathbb{P}}^{14}$, the space of rank $\leq 4$ quadrics in
${\mathbb{P}}^{4}$. It is a quintic hypersurface.

The universal deformation of a rank 3 quadric is given by

$None$
$x_{0}^{2}-x_{1}^{2}-x_{2}^{2}+z_{1}x_{3}^{2}+z_{2}x_{3}x_{4}+z_{3}x_{4}^{2}=0.$

This has rank $\leq 4$ iff $z_{2}^{2}-4z_{1}z_{3}=0$. So $Z$ is singular along
$Z_{3}$, with transversal singularity type $A_{1}$.

Let $Z^{\circ}:=Z_{4}\setminus Z_{2}$ be the space of quadrics of rank $3$ or
$4$. Note that $Z_{2}\subset Z_{4}$ has codimension 5.

Let $U$ be the space of pairs $(L^{2}\subset Q)$ where
$L^{2}\subset{\mathbb{P}}^{4}$ is a 2-plane and $Q\subset{\mathbb{P}}^{4}$ a
quadric. Note that this forces $\operatorname{rank}Q\leq 4$, and $U$ is a
${\mathbb{P}}^{8}$-bundle over $G:=\operatorname{Grass}(2,4)$. We have
projections

$G\stackrel{{\scriptstyle p_{1}}}{{\longleftarrow}}U\stackrel{{\scriptstyle
p_{2}}}{{\longrightarrow}}Z,$

and $\operatorname{Pic}(U)\cong{\mathbb{Z}}^{2}$ is generated by
$p_{1}^{*}{\mathcal{O}}_{G}(1)$ and $p_{2}^{*}{\mathcal{O}}_{Z}(1)$.

Let $U_{2}\subset U$ be the subset where the quadric has rank $\leq 2$. These
quadrics split into 2 hyperplanes, one of which must contain $L^{2}$. So
$U_{2}$ is a ${\mathbb{P}}^{1}\times{\mathbb{P}}^{4}$-bundle over
$\operatorname{Grass}(2,4)$. Set $U^{\circ}:=U\setminus U_{2}$.

Since $U_{2}\subset U$ has codimension 3,
$H^{i}(U^{\circ},{\mathbb{Z}})=H^{i}(U,{\mathbb{Z}})$ for $i\leq 4=2(3-1)$. In
particular,

$H^{0}(U^{\circ},{\mathbb{Z}})\cong{\mathbb{Z}},\
H^{1}(U^{\circ},{\mathbb{Z}})\cong 0,\
H^{2}(U^{\circ},{\mathbb{Z}})\cong{\mathbb{Z}}^{2},\
H^{3}(U^{\circ},{\mathbb{Z}})\cong 0.$

Now look at the morphism $p_{2}:U\to Z$ with Stein factorization

$U\stackrel{{\scriptstyle\pi}}{{\longrightarrow}}W\stackrel{{\scriptstyle\tau}}{{\longrightarrow}}Z.$

In the coordinates (1.1),
$Z\cong(z_{2}^{2}-4z_{1}z_{3}=0)\times{\mathbb{A}}^{11}$. Set
$w_{1}:=\sqrt{z_{1}}$ and $w_{3}:=\sqrt{z_{3}}$. Since $w_{1}w_{3}=z_{2}/2$,
adjoining both $w_{1},w_{2}$ is a degree 2 covering and

$z_{1}x_{3}^{2}+z_{2}x_{3}x_{4}+z_{3}x_{4}^{2}=(w_{1}x_{3}+w_{3}x_{4})^{2}.$

Thus, locally analytically over the points of $Z_{3}\setminus Z_{2}$, we have
$W\cong{\mathbb{A}}^{2}_{w_{1},w_{2}}\times{\mathbb{A}}^{11}$, and the family
of quadrics becomes

$None$
$(x_{0}^{2}-x_{1}^{2}-x_{2}^{2}+(w_{1}x_{3}+w_{2}x_{4})^{2}=0)\times{\mathbb{A}}^{11}.$

Thus $U$ is the family of 2-planes in the same family as

$\bigl{(}x_{0}-x_{1}=x_{2}-(w_{1}x_{3}+w_{2}x_{4})=0\bigr{)}.$

Each of these has a unique intersection point with $(x_{3}=x_{4}=0)$. Thus $U$
is locally analytically isomorphic to the trivial family

$W\times(x_{0}^{2}-x_{1}^{2}-x_{2}^{2}=0)\subset W\times{\mathbb{P}}^{2}.$

Therefore, restricting to $U^{\circ}$ we get

$U^{\circ}\stackrel{{\scriptstyle\pi^{\circ}}}{{\longrightarrow}}W^{\circ}\stackrel{{\scriptstyle\tau^{\circ}}}{{\longrightarrow}}Z^{\circ},$

and $\pi^{\circ}:U^{\circ}\to W^{\circ}$ is a smooth morphism with conics as
fibers.

######  Claim 2.

$H^{3}(W^{\circ},{\mathbb{Z}})\cong{\mathbb{Z}}/2$ and $\pi$ has no rational
sections.

Proof. Let $C\subset U^{\circ}$ be a fiber of $\pi^{\circ}$ over a rank 3
conic. We can think of $C$ as (cones over) lines on a quadric cone, or after
further degeneration, as (cones over) 2 pencils of lines on 2 planes. So $C$
is 2-times the class of (cones over) a pencil of lines in a plane. Thus the
image of $H^{2}\bigl{(}U^{\circ},{\mathbb{Z}}\bigr{)}\to
H^{2}(C,{\mathbb{Z}})\cong{\mathbb{Z}}$ is twice the generator. (Note that
this splitting of $C$ into 2 components happens in the fibers over $Z_{2}$,
thus outside $Z^{\circ}$.)

In the Leray spectral sequence for $\pi^{\circ}$, the only interesting map is
on the $E_{3}$ page:

${\mathbb{Z}}\cong
H^{0}\bigl{(}W^{\circ},R^{2}\pi^{\circ}_{*}{\mathbb{Z}}\bigr{)}\to
H^{3}\bigl{(}W^{\circ},{\mathbb{Z}}\bigr{)}.$

As we noted, the kernel is $2{\mathbb{Z}}$ and
$H^{3}\bigl{(}U^{\circ},{\mathbb{Z}}\bigr{)}=0$. Thus
$H^{3}\bigl{(}W^{\circ},{\mathbb{Z}}\bigr{)}\cong{\mathbb{Z}}/2$ and
$H^{2}\bigl{(}W^{\circ},{\mathbb{Z}}\bigr{)}\cong{\mathbb{Z}}$. ∎

######  3Construction of $X$.

Since $Z\subset{\mathbb{P}}^{14}$ is a quintic hypersurface, $K_{Z}\sim-10H$.
Since $\tau:W\to Z$ is étale over $Z\setminus Z_{3}$, $K_{W}\sim-10\tau^{*}H$.
As we noted, $Z_{2}\subset Z$ has codimension $5$, so its preimage
$W_{2}\subset W$ has codimension $5$.

As in [OR23], let $X_{Z}\subset Z$ be the complete intersection of $9$ general
hyperplanes and $X:=X_{W}\subset W$ its preimage. Then $X\subset W^{\circ}$
and $K_{X}\sim-\tau^{*}H$.

######  Claim 4.

$H_{2}(X,{\mathbb{Z}})\cong{\mathbb{Z}}+{\mathbb{Z}}/2$.

Proof. The isomorphism $H_{2}(X,{\mathbb{Z}})\cong
H_{2}(W^{\circ},{\mathbb{Z}})$ follows from the Lefschetz hyperplane theorem.
A slight problem is that $W$ is singular and $W^{\circ}$ is not compact. There
are several ways to take care of these.

One way is to use the Lefschetz theorem in intersection homology. A more
elementary argument is given in [OR23, 4.9]. ∎

Note that by [Voi06], $H_{2}(X,{\mathbb{Z}})$ is generated by algebraic
curves. Next we write down a difference of 2 smooth, degree 1 rational curves
that generates the ${\mathbb{Z}}/2$-summand of $H_{2}(X,{\mathbb{Z}})$.

######  Claim 5.

Let $L\subset Z\setminus Z_{3}\subset{\mathbb{P}}^{14}$ be a line. Its
preimage in $W$ is a pair of lines $L^{\prime}\cup L^{\prime\prime}$ such that

  1. (1)

$L^{\prime}$ and $L^{\prime\prime}$ are numerically equivalent,

  2. (2)

$U\times_{W}L^{\prime}$ is the ruled surface ${\mathbb{F}}_{1}\cong
B_{p}{\mathbb{P}}^{2}$,

  3. (3)

$U\times_{W}L^{\prime\prime}$ is the ruled surface
${\mathbb{F}}_{0}\cong{\mathbb{P}}^{1}\times{\mathbb{P}}^{1}$, and

  4. (4)

$L^{\prime}-L^{\prime\prime}$ is a generator of the ${\mathbb{Z}}/2$-summand
of $H_{2}(W^{\circ},{\mathbb{Z}})\cong{\mathbb{Z}}+{\mathbb{Z}}/2$.

By Paragraph 12, $X_{Z}$ contains a 2-parameter family of lines, and Claim 5
applies to them. Thus we obtain the following.

######  Claim 6.

Let $L\subset X_{Z}\setminus Z_{3}$ be a line. Its preimage in $X$ is a pair
of lines $L^{\prime}\cup L^{\prime\prime}$, and $L^{\prime}-L^{\prime\prime}$
is a generator of the ${\mathbb{Z}}/2$-summand of
$H_{2}(X,{\mathbb{Z}})\cong{\mathbb{Z}}+{\mathbb{Z}}/2$. ∎

######  7Beginning of the proof of Claim 5.

By Claim 4 $H_{2}(X,{\mathbb{Z}})/(\mbox{torsion})$ is generated by
$K_{X}\sim-\tau^{*}H$, so $L^{\prime}$ and $L^{\prime\prime}$ are numerically
equivalent.

By Paragraph 11, in suitable coordinates we can write $L$ as a family of
quadrics

$Q(\lambda{:}\mu):=\bigl{(}x_{0}(\lambda x_{2}-\mu x_{3})=x_{1}(\mu
x_{4}-\lambda x_{3})\bigr{)}.$

All of these contain the 2-plane $(x_{0}=x_{1}=0)$, defining a section
$s_{0}:L\to U$.

The preimage of $L$ in $W$ is a disjoint union of 2 lines $L^{\prime}\cup
L^{\prime\prime}$. We choose $L^{\prime}$ to be $\pi\circ s_{0}:L\to U\to W$.

For any nonzero linear form $\ell=a\lambda+b\mu$, a section $C^{\prime}(\ell)$
of $\pi:U\to W$ over $L^{\prime}$ is given by

$\ell x_{0}=\mu x_{4}-\lambda x_{3}\quad\mbox{and}\quad\lambda x_{2}-\mu
x_{3}=\ell x_{1}.$

For $\ell_{1}\neq\ell_{2}$ the 2 sections
$C^{\prime}(\ell_{1}),C^{\prime}(\ell_{2})$ meet at the point where
$\ell_{1}=\ell_{2}$. Thus $U\times_{W}L^{\prime}$ is the ruled surface
${\mathbb{F}}_{1}$.

In the other family of 2-planes, we have sections $C^{\prime\prime}(\ell)$
given by

$cx_{0}=x_{1}\quad\mbox{and}\quad\mu x_{4}-\lambda x_{3}=c(\lambda x_{2}-\mu
x_{3}).$

These are disjoint for $c_{1}\neq c_{2}$. Thus $U\times_{W}L^{\prime\prime}$
is the the trivial ${\mathbb{P}}^{1}$-bundle.

These show Claim 5.2–3. ∎

Claim 5.4 is a formal consequence of (5.1–3). To see this, we need to discuss
how to detect 2-torsion in $H_{2}$ using ${\mathbb{P}}^{1}$-bundles.
(Similarly, $n$-torsion can be detected using ${\mathbb{P}}^{n-1}$-bundles.)

######  8Comments on ${\mathbb{P}}^{1}$-bundles.

Let $X$ be a normal, proper variety and $\pi:Y\to X$ a
${\mathbb{P}}^{1}$-bundle (étale locally trivial). For a smooth curve $C\to
X$, let $C_{Y}\to Y_{C}:=C\times_{X}Y$ be a lifting. Set

$None$ $\sigma_{Y}(C):=(C_{Y}\cdot K_{Y_{C}/C})\mod 2.$

This is well defined as a function on $A_{1}(X)$, the group of 1-cycles modulo
algebraic equivalence.

If $X$ is smooth and $Y\to X$ has a rational section $S\subset Y$, then
$K_{Y/X}\sim-2S+\pi^{*}D$ for some Cartier divisor $D$ on $X$. In this case

$\sigma_{Y}(C)\equiv(C\cdot D)\mod 2.$

Conversely, assume that we are over ${\mathbb{C}}$, $H_{2}(X,{\mathbb{Z}})$ is
generated by algebraic curves, and $\sigma_{Y}(C)\equiv(C\cdot D)\mod 2$ for
every $C$. Then $K_{Y/X}-\pi^{*}D$ is divisible by 2, giving a rational
section. In any case, we get the following.

######  Claim 9.

If there is a numerically trivial 1-cycle $C$ such that $\sigma_{Y}(C)\equiv
1\mod 2$, then $[C]$ is a nontrivial torsion element in
$H_{2}(X,{\mathbb{Z}})$, and $Y\to X$ has no rational sections. ∎

######  10End of the proof of Claim 5.

Since $U\times_{W}L^{\prime}\cong{\mathbb{F}}_{1}$, (8.1) shows that
$\sigma_{U}(L^{\prime})\equiv 1\mod 2$. Similarly,
$U\times_{W}L^{\prime\prime}\cong{\mathbb{F}}_{0}$ implies that
$\sigma_{U}(L^{\prime\prime})\equiv 0\mod 2$. Thus
$C:=L^{\prime}-L^{\prime\prime}$ is numerically trivial and
$\sigma_{U}(C)\equiv 1\mod 2$. We can now apply Claim 9. ∎

In both cases we could have used the isomorphism

$\omega_{U/W}\cong p_{1}^{*}{\mathcal{O}}_{G}(-1)\otimes
p_{2}^{*}{\mathcal{O}}_{Z}(1)$

to compute $\sigma_{U}(L^{\prime})$ and $\sigma_{U}(L^{\prime\prime})$. ∎

######  11Lines on $Z$.

By [Kro1890], the lines on $Z\setminus Z_{2}$ form 3 families of dimension 20
each. These are the following.

(11.1) $\langle Q_{1},Q_{2}\rangle$ where the $Q_{i}$ contain a common
2-plane. The general such line $L$ is disjoint from $Z_{3}$, and its preimage
in $W$ is a pair of disjoint lines $L^{\prime}\cup L^{\prime\prime}$. After
coordinate change, these can be written as

$x_{0}(\lambda x_{2}-\mu x_{3})=x_{1}(\mu x_{4}-\lambda x_{3}).$

(11.2) $\langle Q_{1},Q_{2}\rangle$ where the $Q_{i}$ have a common singular
point. After coordinate change, these can be written as

$\lambda q_{1}(x_{1},\dots,x_{4})+\mu q_{2}(x_{1},\dots,x_{4})=0,$

where the $q_{i}$ are quadratic forms. The general such line $L$ intersects
$Z_{3}$ at 4 points, and its preimage in $W$ is a smooth, elliptic curve of
degree 2.

(11.3) $\langle Q_{1},Q_{2}\rangle$ where the $Q_{i}$ have rank 2 and
$\operatorname{Sing}Q_{i}$ is tangent to $Q_{3-i}$. The general such line $L$
intersects $Z_{3}$ at 2 points, and its preimage in $W$ is a smooth, rational
curve of degree 2. After coordinate change, these can be written as

$\lambda(x_{0}^{2}+x_{1}x_{2})+\mu(x_{2}x_{3}+x_{4}^{2})=0.$

######  12Lines on $X_{Z}$.

The space of lines in $Z\setminus Z_{3}$ has dimension 20, and with each
hyperplane section the dimension drops by 2. So the lines on $X_{Z}$ form 3
families of dimension 2 each. Only (11.1) contains lines that are disjoint
from $\operatorname{Sing}X_{Z}$.

Since there are no lines on $Z_{3}\setminus Z_{2}$, the only common liness to
any 2 of these families are the finitely many double tangents of
$\operatorname{Sing}X_{Z}$.

######  13Another representation of $X_{Z}$.

Let $P_{5}$ be a general 5-dimensional linear system of quadrics on
$P_{4}:={\mathbb{P}}^{4}$. For $i=4,5$, we have the projections
$\pi_{i}:P_{4}\times P_{5}\to P_{i}$. For brevity let us write
$(a,b):=a\pi_{4}^{*}H_{4}+b\pi_{5}^{*}H_{5}$, where $H_{i}$ is the hyperplane
class on $P_{i}$. Set

$Y:=\bigl{\\{}(p,Q):p\in P_{4},Q\in
P_{5},p\in\operatorname{Sing}Q\bigr{\\}}\subset P_{4}\times P_{5}.$

The condition $p\in\operatorname{Sing}Q$ is equivalent to the partial
derivatives of the equation of $Q$ vanishing at $p$. Thus $Y\subset
P_{4}\times P_{5}$ is the complete intersection of 5 divisors of bidegree
$(1,1)$. Write these as

$None$
$\textstyle{\sum}_{i=0}^{4}\textstyle{\sum}_{j=0}^{5}a^{\ell}_{ij}y_{i}x_{j}\quad\mbox{for}\quad\ell=1,\dots,5.$

Over $P_{5}$, (13.1) is equivalent to a $5\times 5$ symmetric matrix whose
entries are the linear forms
$m_{i}^{\ell}=\textstyle{\sum}_{j=0}^{5}a^{\ell}_{ij}x_{j}$. The condition
$\det(m_{i}^{\ell})=0$ defines $X_{Z}$ as in Paragraph 3.

Over $P_{4}$, (13.1) is equivalent to a $6\times 5$ matrix whose entries are
the linear forms $n_{j}^{\ell}=\textstyle{\sum}_{i=0}^{4}a^{\ell}_{ij}y_{i}$.
Note that $\pi_{4}:Y\to P_{4}$ is birational. Its inverse is the blow-up of a
surface

$S\subset P_{4},\quad\mbox{defined by
}\quad\operatorname{rank}(n_{j}^{\ell})\leq 4.$

Let $E_{4}\subset Y$ denote the exceptional divisor. $Y$ defines a rational
map $P_{4}\dasharrow X_{Z}\subset P_{5}$, which is given by the $5\times 5$
subdeterminants of $(n_{j}^{\ell})$. Thus $E_{4}\sim(5,-1)|_{Y}$.

The inverse rational map $X_{Z}\dasharrow P_{4}$ is a bit harder to see. It is
given by a linear system of divisors as follows. Let $H\in|H_{4}|$ be a
hyperplane and set

$D_{H}:=\bigl{\\{}Q\in
X_{Z}:H\cap\operatorname{Sing}Q\neq\emptyset\bigr{\\}}\subset X_{Z}.$

Note that the condition $H\cap\operatorname{Sing}Q$ is equivalent to $Q|_{H}$
being singular. (Here we need that $Q$ itself is singular.) This gives us the
equation $\det(Q|_{H})=0$ for $D_{H}$. It has degree 4.

We claim that the intersection of $\left(\det(Q|_{H})=0\right)$ with $X_{Z}$
has multiplicity 2. To see this, choose coordinates such that $H=(x_{0}=0)$
and $Q=(x_{0}^{2}+x_{2}^{2}+x_{3}^{2}+x_{4}^{2}=0)$. For its deformations we
can make linear coordinate changes to the $x_{2},x_{3},x_{4}$, but $x_{0}$ can
only be multiplied by a constant. Thus we get a miniversal deformation family

$\bigl{(}x_{0}^{2}+t_{1}x_{0}x_{1}+t_{2}x_{1}^{2}+x_{2}^{2}+x_{3}^{2}+x_{4}^{2}=0\bigr{)}\subset
P_{5}\times{\mathbb{A}}^{2}_{t_{1},t_{2}}.$

For a given $t_{1},t_{2}$, the quadric has rank $4$ iff $t_{1}^{2}-4t_{2}=0$,
and the singular point is on $(x_{0}=0)$ iff $t_{2}=0$. Their intersection is
the length 2 scheme $(t_{1}^{2}=0)$.

Thus the $D_{H}\subset X_{Z}$ have degree $10=\frac{1}{2}(4\cdot 5)$ and
$2D_{H}\sim 4H_{5}|_{Y}$. In particular, the divisor class $D_{H}-2H_{5}|_{Y}$
is 2-torsion in the class group $\operatorname{Cl}(X_{Z})$. The corresponding
double cover is our $X$, constructed in Paragraph 3.

Let $E_{5}\subset Y$ denote the exceptional divisor of $\pi_{5}:Y\to X_{Z}$.
The previous computations suggest that it should be linearly equivalent to
$(-1,2)$. However, $X_{Z}$ has multiplicity 2 along the base locus of
$|D_{H}|$, so the correct bidegree is $(-2,4)$.

On $P_{4}$, the 3 families of lines (11.1–3) correspond to

(13.1) conics that are 9-secants of $S$,

(13.2) fibers of $E_{4}\to S$, and

(13.3) lines that are 4-secants of $S$.

######  14$X$ as a double ${\mathbb{P}}^{4}$.

By the previous description, $X$ is birational to a double cover of
${\mathbb{P}}^{4}$ ramified along the hypersurface $R:=\pi_{4}(E_{5})\subset
P_{4}$.

The degree of $R$ is given by $(1,0)^{3}[E_{5}](1,1)^{5}$, which works out to
be 18. The degree of the surface $S$ is $(1,0)^{2}[E_{4}](0,1)(1,1)^{5}=15$.
Note that $S$ is a $6\times 5$ determinantal surface. However, it is not
general since we have a symmetry condition on the $P_{5}$ side, so results
about general determinantal surfaces do not apply to $S$.

The multiplicity of $Y$ along $S$ is 4. This follows from the computation

$(1,0)^{2}[E_{4}][E_{5}](1,1)^{5}=60=4\cdot\deg S.$

Thus $R$ is in the 4th symbolic power of the homogeneous ideal of $S$, but not
in its 4th power. For general determinantal surfaces these are equal by
[dCEP80].

Another interesting property of $S$ is that the fibers of
$E_{5}\to\operatorname{Sing}X_{Z}$ give 5-secants of $S$. Thus $S$ has a
2-parameter family of 5-secants. Note that by (13.3), the family of 4-secants
has dimension 2 as well.

Most surfaces in ${\mathbb{P}}^{4}$, including general $6\times 5$
determinantal surfaces, have only 1-parameter families of 5-secants.

######  Acknowledgments.

I thank J. Ottem, C. Raicu and B. Totaro for many useful comments. Partial
financial support was provided by the NSF under grant number DMS-1901855.

## References

  * [dCEP80] C. de Concini, D. Eisenbud, and C. Procesi, _Young diagrams and determinantal varieties_ , Invent. Math. 56 (1980), no. 2, 129–165. MR 558865
  * [Kro1890] L. Kronecker, _Algebraische Reduction der Scharen bilinearer Formen_ , Berl. Ber. 1890 (1890), 1225–1237.
  * [OR23] J. C. Ottem and J. V. Rennemo, _Fano varieties with torsion in the third cohomology group_ , 2023, https://arxiv.org/abs/2309.10793.
  * [Roo38] T. G. Room, _The geometry of determinantal loci_ , Cambridge University Press, London, 1938.
  * [Voi06] Claire Voisin, _On integral Hodge classes on uniruled or Calabi-Yau threefolds_ , Moduli spaces and arithmetic geometry, Adv. Stud. Pure Math., vol. 45, Math. Soc. Japan, Tokyo, 2006, pp. 43–73. MR 2306166

Princeton University, Princeton NJ 08544-1000,