# Learning a Better Control Barrier Function
Under Uncertain Dynamics

Bolun Dai<EMAIL_ADDRESS>Prashanth Krishnamurthy
<EMAIL_ADDRESS>Farshad Khorrami<EMAIL_ADDRESS>Control/Robotics Research Laboratory (CRRL), Dept. of Electrical and Computer
Engineering,
New York University Tandon School of Engineering, Brooklyn, NY 11201

###### Abstract

Using control barrier functions (CBFs) as safety filters provides a
computationally inexpensive yet effective method for constructing controllers
in safety-critical applications. However, using CBFs requires the construction
of a valid CBF, which is well known to be a challenging task, and accurate
system dynamics, which are often unavailable. This paper presents a learning-
based approach to learn a valid CBF and the system dynamics starting from a
conservative handcrafted CBF (HCBF) and the nominal system dynamics. We devise
new loss functions that better suit the CBF refinement pipeline and are able
to produce well-behaved CBFs with the usage of distance functions. By adopting
an episodic learning approach, our proposed method is able to learn the system
dynamics while not requiring additional interactions with the environment.
Additionally, we provide a theoretical analysis of the quality of the learned
system dynamics. We show that our proposed learning approach can effectively
learn a valid CBF and an estimation of the actual system dynamics. The
effectiveness of our proposed method is empirically demonstrated through
simulation studies on three systems, a double integrator, a unicycle, and a
two-link arm.

###### keywords:

Control Barrier Function, Uncertain Dynamics, Learning

††thanks: An earlier version [12] of a portion of this paper was presented at
the 61st IEEE Conference on Decision and Control, Cancun, Mexico, December
2022.

, ,

##  1 Introduction

Ensuring safety is crucial when designing controllers for real-world
applications [10][13][9][11]. With the increasing usage of automated systems,
e.g., self-driving cars [6], the ability to guarantee the safety of such
systems becomes increasingly important. In optimal control, safety is often
ensured by casting the safety requirements as constraints [18]. However, when
the optimization problem gets larger [19], the solution time increases
rapidly, limiting its usage for guaranteeing safety in complex environments,
which usually requires the control system to react quickly. Recently,
Hamilton-Jacobi reachability analysis has been used to generate safe controls
[4][24]. When solved offline, it provides a way to generate safe controls
quickly online. However, its usage is greatly limited by the curse of
dimensionality [4]. With the rise in popularity of learning-based methods in
control synthesis, learning-based methods have also been used to synthesize
controllers for safety-critical tasks [14][33][5]. However, most learning-
based methods require a significant amount of unsafe interactions to learn a
safe controller [28], which might be costly or impossible to obtain.

Another popular method to synthesize safe control is utilizing control barrier
functions (CBFs) [1]. CBFs can be used with control Lyapunov functions (CLFs)
or as a safety filter for an unsafe performance controller [1]. In both cases,
the control can be obtained by solving a quadratic program (QP) [3] which can
be done at a very high frequency using modern optimization solvers. Given its
many advantages, CBFs have been used on many safety-critical tasks, e.g.,
biped and quadrupedal locomotion on stepping stones [26] [16], adaptive cruise
control [2], and multi-agent aerial maneuver [29].

Although CBF provides a promising direction in safe controller synthesis,
there are two significant assumptions when applying CBF-based controllers:
having access to a valid CBF and having accurate system dynamics. A common
approach to finding a valid CBF is to start with a description of the safe
set, usually in the form of state constraints, and find a function that is
positive only within the safe set and has the appropriate relative degree with
respect to the system dynamics. This method is plausible for simple
constraints. However, finding a valid CBF that recovers the entire safe set
becomes increasingly challenging [7] as the constraints become nonlinear or
nonconvex. To mitigate this issue, work has been done in learning the CBF. In
[31], human demonstrations have been used to map the boundaries of the safe
set, and a CBF is then learned. This method may not scale to constraints in
higher dimensions. Instead of having information on safe set boundaries, work
has been done on utilizing expert demonstrations of safe and unsafe
trajectories [30]. Additionally, work has been done in learning CBFs using
data collected online. In [22], a CBF is synthesized using only onboard
sensors.

The aforementioned learning-based methods assume no knowledge of the CBF and
learn it from scratch. This is an overly restricting assumption because
handcrafting a conservative CBF is usually possible in many cases. Recently,
work has been done in learning a CBF starting from an initial conservative
CBF. In [34], an HCBF is used to warm start a dynamic program that refines the
HCBF to enlarge the recovered safe set. In [12], a learning-based approach is
used to learn the difference between a conservative HCBF and a CBF that
recovers a more significant portion of the safe set.

Another assumption made in many CBF-related works is having access to the
system dynamics, which is usually not the case in real-world applications [8].
Work has been done in learning the CBF in a model-free fashion [28]. However,
like learning CBFs from scratch, having no knowledge of the system dynamics is
also overly restricting since an approximate nominal model of the system
dynamics is often known in many real-world applications. Recently, work has
been done in learning the system dynamics for CBF-based controllers [32][35]
while assuming access to a ground truth CBF. In this paper, we build on our
earlier work in learning-based CBF refinement [12] and further develop and
evaluate the methodology under uncertain system dynamics.

In this paper, we propose an algorithmic approach to learn both the CBF and
the system dynamics starting from an HCBF and a nominal model of system
dynamics. The main contribution of this paper is threefold: (1) starting from
an HCBF, we develop a method to learn a well-behaved CBF that recovers a more
significant portion of the safe set (also known as CBF refinement [34]) using
a CBF prior (i.e., distance function); (2) we extend the CBF refinement
problem to include problems with uncertain dynamics; (3) we show the
effectiveness of our proposed approach using extensive simulation studies on
three systems: double integrator, unicycle, and a two-link arm. The remainder
of this paper is structured as follows. In Section II, the foundations of CBF
are briefly summarized. In Section III, the problem formulation is given. In
Section IV, the proposed method is presented. In Section V, the results of the
simulation studies on a double-integrator, a unicycle, and a two-link arm are
presented. Section VI concludes the paper with a summary and discussion of
future works.

##  2 Preliminaries

In this section, we review the concept of CBF and how it is utilized in
safety-critical applications. Consider a control affine system

$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u},$
(1)

where the state is represented as $\mathbf{x}\in\mathbb{R}^{n}$ and the
control as $\mathbf{u}\in\mathcal{U}\subset\mathbb{R}^{m}$, with $\mathcal{U}$
being the admissible set of controls. The locally Lipschitz continuous
functions $\mathbf{f}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and
$\mathbf{g}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times m}$ represent the
drift and the control influence matrix, respectively. We assume access to a
feedback controller

$\mathbf{u}=\pi(\mathbf{x}),$ (2)

with $\pi:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ also being a locally
Lipschitz continuous function. Substituting (2) into (1), the closed-loop
dynamics are given by:

$\dot{\mathbf{x}}=\mathbf{f}_{\mathrm{cl}}(\mathbf{x})=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\pi(\mathbf{x}).$
(3)

For any initial state $\mathbf{x}_{0}\in\mathbb{R}^{n}$, there exists a
maximal time interval of existence

$\mathbf{I}(\mathbf{x}_{0})=[t_{0},t_{\mathrm{max}}),$ (4)

where $\mathbf{x}(t)$ is a unique solution to (3) on
$\mathbf{I}(\mathbf{x}_{0})$; when $t_{\mathrm{max}}=\infty$, the system
defined in (3) is considered forward complete [20].

The notion of safety is defined for this work as forward invariance with
respect to the safe set $\mathcal{C}\subset\mathbb{R}^{n}$:

######  Definition 1 (Forward Invariance & Safety).

The system defined in (3) is forward invariant with respect to $\mathcal{C}$
if for every $\mathbf{x}_{0}\in\mathcal{C}$, we have
$\mathbf{x}(t)\in\mathcal{C}$ for all $t\in\mathbf{I}(\mathbf{x}_{0})$. A
system that is forward invariant with respect to $\mathcal{C}$ is said to be
safe with respect to $\mathcal{C}$. A controller that makes a closed-loop
system safe with respect to $\mathcal{C}$ is said to be safe with respect to
$\mathcal{C}$.

We consider $\mathcal{C}$ to be the 0-superlevel set of a continuously
differentiable function $h:\mathbb{R}^{n}\rightarrow\mathbb{R}$, yielding

$\displaystyle\mathcal{C}$ $\displaystyle=\\{\mathbf{x}\in\mathbb{R}^{n}\mid
h(\mathbf{x})\geq 0\\},$ (5a) $\displaystyle\partial\mathcal{C}$
$\displaystyle=\\{\mathbf{x}\in\mathbb{R}^{n}\mid h(\mathbf{x})=0\\},$ (5b)
$\displaystyle\mathrm{Int}(\mathcal{C})$
$\displaystyle=\\{\mathbf{x}\in\mathbb{R}^{n}\mid h(\mathbf{x})>0\\},$ (5c)

where $\partial\mathcal{C}$ represents the boundary of $\mathcal{C}$ and
$\mathrm{Int}(\mathcal{C})$ represents the interior of $\mathcal{C}$.
Additionally, we assume that $\mathrm{Int}(\mathcal{C})$ is not an empty set,
i.e., $\mathrm{Int}(\mathcal{C})\neq\emptyset$, and that $\mathcal{C}$ does
not contain any isolated points. Before defining CBFs, we first define
extended class $\mathcal{K}$ functions:

######  Definition 2 (Extended class $\mathcal{K}$ function).

A continuous function $\alpha:(-b,a)\rightarrow\mathbb{R}$ is called an
extended class $\mathcal{K}$ function when $\alpha(0)=0$ and $\alpha$ is
strictly monotonically increasing. When $a=\infty$, $b=\infty$, and

$\lim_{r\rightarrow\infty}\alpha(r)=\infty\ \ \ \ \mathrm{\&}\ \ \ \
\lim_{r\rightarrow-\infty}\alpha(r)=-\infty,$

$\alpha$ is called an extended class $\mathcal{K}_{\infty}$ function.

With the aforementioned concepts, the CBF is defined:

######  Definition 3 (Control Barrier Function [1]).

Let $\mathcal{C}\subset\mathcal{D}\subset\mathbb{R}^{n}$ be the 0-superlevel
set of a continuously differentiable function
$h:\mathcal{D}\rightarrow\mathbb{R}$, then $h$ is a control barrier function
(CBF) on $\mathcal{C}$ if there exists an extended class
$\mathcal{K}_{\infty}$ function $\alpha(\cdot)$ such that for all
$\mathbf{x}\in\mathcal{D}$, the system defined in (1) satisfies

$\sup_{\mathbf{u}\in\mathcal{U}}\Big{[}\frac{\partial
h(\mathbf{x})}{\partial\mathbf{x}}\Big{(}\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u}\Big{)}\Big{]}\geq-\alpha(h(\mathbf{x})),$
(6)

with $\alpha:\mathbb{R}\rightarrow\mathbb{R}$ being an extended class
$\mathcal{K}_{\infty}$ function.

Using the condition in (6) and a possibly unsafe performance controller
$\pi_{\mathrm{perf}}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$, we can
construct a reactive controller by solving a quadratic program (QP) at each
time step

$\displaystyle\pi(\mathbf{x})=\operatorname*{arg\,min}_{\mathbf{u}\in\mathcal{U}}\
$ $\displaystyle\ \|\mathbf{u}-\pi_{\mathrm{perf}}(\mathbf{x})\|^{2}$ (7)
$\displaystyle\mathrm{subject\ to}\ $ $\displaystyle\ \Big{[}\frac{\partial
h(\mathbf{x})}{\partial\mathbf{x}}\Big{(}\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u}\Big{)}\Big{]}\geq-\alpha(h(\mathbf{x}))$

which is usually called a CBF-QP [17]. The CBF-QP can be seen as a safety
filter applied on top of $\pi_{\mathrm{perf}}(\mathbf{x})$, which finds the
closest control in the least-square sense that also enforces forward
invariance with respect to $\mathcal{C}$.

##  3 Problem Formulation

In this section, we present our assumptions on HCBFs and model uncertainty and
define the problem for learning a better CBF under uncertain dynamics. We
consider a set of state constraints in the form of

$\mathbf{c}_{i}(\mathbf{x})\leq 0,\ i=1,\cdots,r$ (8)

where $\mathbf{c}_{i}:\mathbb{R}^{n}\rightarrow\mathbb{R}$. We define
$\mathcal{S}_{i}$ as the 0-superlevel set of $-\mathbf{c}_{i}(\mathbf{x})$,
i.e.,

$\mathcal{S}_{i}=\\{\mathbf{x}\mid-\mathbf{c}_{i}(\mathbf{x})\geq 0\\}.$ (9)

We define the intersections of all $\mathcal{S}_{i}$’s as $\mathcal{S}$, i.e.,

$\mathcal{S}=\bigcap_{i=1}^{r}{\mathcal{S}_{i}}.$ (10)

The true safe set $\mathcal{C}$ under the constraints in (8) is defined as the
largest forward invariant set contained in $\mathcal{S}$ that can be expressed
as the 0-superlevel set of a continuously differentiable function. The notion
of forward invariance can be understood as the property that if the control
input satisfies (6), then if the initial state of the system $\mathbf{x}_{0}$
is within the set $\mathcal{C}$, then the state trajectory lies within
$\mathcal{C}$ for all $t\in\mathbf{I}(\mathbf{x}_{0})$. Thus, we have the
relationship

$\mathcal{C}\subseteq\mathcal{S}.$ (11)

We assume that an unknown continuously differentiable function $h$ is a valid
CBF on $\mathcal{C}$. In many cases, even though we cannot directly find a
continuously differentiable function with its 0-superlevel set being
$\mathcal{C}$, we are able to find another continuously differentiable
function $\widehat{h}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ such that its
0-superlevel set $\widehat{\mathcal{C}}$ is contained within $\mathcal{C}$

$\widehat{\mathcal{C}}=\\{\mathbf{x}\mid\widehat{h}(\mathbf{x})\geq
0\\}\subseteq\mathcal{C}.$ (12)

Assuming that we have access to $\widehat{h}$, without loss of generality, we
can write the relationship between $\widehat{h}$ and $h$ as

$h(\mathbf{x})=\widehat{h}(\mathbf{x})+\Delta h(\mathbf{x}),$ (13)

with $\Delta h:\mathbb{R}^{n}\rightarrow\mathbb{R}$ being a continuously
differentiable function. One assumption we make for $\widehat{h}$ is that it
has the same relative degree as $h$111The system has relative degree $r$ if,
in the neighborhood of the equilibrium,
$L_{\mathbf{g}}L_{\mathbf{f}}^{i-1}\mathbf{j}(\mathbf{x})=0$ for
$i=1,2,\cdots,r-1$ and
$L_{\mathbf{g}}L_{\mathbf{f}}^{r-1}\mathbf{j}(\mathbf{x})\neq 0$, where
$\mathbf{j}(\mathbf{x})$ is the output of the system.. This is a mild
assumption [36], given that the relative degree of a system represents the
actuation capabilities of the system dynamics and can often be inferred from
first principles. In this paper, we consider CBFs with relative degree one
because, without the loss of generality, we can always use the idea of
exponential CBFs [27] to create a CBF with relative degree one starting from a
CBF with a higher relative degree.

In the CBF-QP framework, the CBF is not the only source of uncertainty. In
practice, the system dynamics in (1) would be inaccurate because of unmodelled
dynamics and parametric errors. Instead of $\mathbf{f}$ and $\mathbf{g}$, we
would usually only have access to a nominal model

$\dot{\mathbf{x}}=\widehat{\mathbf{f}}(\mathbf{x})+\widehat{\mathbf{g}}(\mathbf{x})\mathbf{u},$
(14)

with locally Lipschitz continuous functions
$\widehat{\mathbf{f}}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and
$\widehat{\mathbf{g}}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times m}$.
Similar to the case in CBFs, without loss of generality, we have the
relationships

$\displaystyle\mathbf{f}(\mathbf{x})$
$\displaystyle=\widehat{\mathbf{f}}(\mathbf{x})+\Delta\mathbf{f}(\mathbf{x}),$
(15a) $\displaystyle\mathbf{g}(\mathbf{x})$
$\displaystyle=\widehat{\mathbf{g}}(\mathbf{x})+\Delta\mathbf{g}(\mathbf{x}),$
(15b)

with locally Lipschitz continuous functions
$\Delta\mathbf{f}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ and
$\Delta\mathbf{g}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times m}$. We assume
that the nominal dynamics have the same relative degree as the true dynamics,
which is a common assumption in the literature [32][35]. Since we only have a
conservative estimation of $h$ and the nominal dynamics, if we deploy CBF-QP
using these known functions, there would be no safety guarantees. Thus, the
main goal of this paper is to find an algorithmic approach to learning the
functions $\Delta h$, $\Delta\mathbf{f}$, and $\Delta\mathbf{g}$, which will
be discussed in Section 4.

##  4 Method

In this section, we propose an algorithmic approach to solve the problem
formulated in Section 3. The structure of this section is as follows. First,
we describe our proposed solution to the CBF learning problem. Then, we
describe how we learn the system dynamics. Finally, we show how we jointly
solve these two learning problems.

###  4.1 Learning the Control Barrier Function

Figure 1: Illustration of the effect of the distance function
$\mathbf{d}(\mathbf{x})$. These figures show the result of learning a CBF for
a target-reaching-obstacle-avoidance task on a unicycle (see Section 5.2 for
details). In the upper graph, the light blue region represents the unsafe set
under the HCBF, and the darker blue region represents the obstacle. The upper
graph shows the trajectory generated by a CBF-QP controller with a
proportional controller (see Section 5.2 for details) as the performance
controller and using the learned CBF-QP. The lower graph shows the evolution
of the CBF value along the trajectory. Figure 2: Illustration of the overall
training procedure. The “@” sign represents matrix multiplication. The gray
boxes represent the non-learning components. The purple box represents the
learned CBF, the blue box represents the learned system dynamics, the green
box represents the buffer storing safe interactions, and the red box
represents the buffer storing unsafe interactions.

Following the problem formulation in Section 3, we need to estimate $\Delta
h(\mathbf{x})$ in order to estimate the CBF. We propose to use a deep neural
network (DNN) to estimate $\Delta h(\mathbf{x})$, we write this DNN as
$\Delta\widehat{h}(\mathbf{x}\mid\theta)$, where $\theta$ represents the
weights of the DNN.

Given that $\Delta h(\mathbf{x})$ is a continuously differentiable function,
we also require $\Delta\widehat{h}(\mathbf{x}\mid\theta)$ to be a continuously
differentiable function with respect to $\mathbf{x}$. To achieve this, we use
a deep differential network with smooth activation functions, which we refer
the reader to [23] for a detailed description. The deep differential network
has two forward paths. One of the paths is the same as in standard fully-
connected DNNs. The other computes the Jacobian of the DNN with respect to its
input. Since it directly outputs the Jacobian, compared to performing an
additional numerical differentiation pass, using deep differential networks
increases the computational efficiency. A single layer within a deep
differential network has the form

$(\frac{\partial\bar{\mathbf{y}}}{\partial\mathbf{y}},\
\bar{\mathbf{y}})=\ell(\mathbf{y}),$ (16)

where $\mathbf{y}\in\mathbb{R}^{n_{i}\times 1}$ is the input of the layer,
$\bar{\mathbf{y}}\in\mathbb{R}^{n_{o}\times 1}$ is the output of the layer,
and the layer is represented by $\ell:\mathbb{R}^{n_{i}\times
1}\rightarrow(\mathbb{R}^{n_{o}\times n_{i}},\mathbb{R}^{n_{o}\times 1})$. The
Jacobian is computed as

$\frac{\partial\bar{\mathbf{y}}}{\partial\mathbf{y}}=\mathrm{diag}(\mathbf{g}^{\prime}(\mathbf{a}))\mathbf{W},$
(17)

where $\mathbf{W}\in\mathbb{R}^{n_{o}\times n_{i}}$ represents the weights of
that layer, $\mathbf{g}:\mathbb{R}^{n_{o}\times
1}\rightarrow\mathbb{R}^{n_{o}\times 1}$ represents the activation function,
$\mathbf{g}^{\prime}(\cdot)$ represents the derivative of the activation
function, and $\mathbf{a}=\mathbf{W}\mathbf{y}+\mathbf{bias}$.

To find the weights $\theta$, we would need to collect a dataset of features
and labels. However, since we do not have access to a CBF with its
0-superlevel set coinciding with $\mathcal{C}$, we do not have groundtruth
labels. A widely used approach [30] [28] is to learn a valid CBF without
groundtruth labels by utilizing the properties in (5) and write the loss
functions for learning $\theta$ as

$\displaystyle\mathcal{L}_{+}(\theta)$
$\displaystyle=\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{+}}\max\Big{(}0,-\tilde{h}(\mathbf{x}_{i}\mid\theta)\Big{)},$
(18a) $\displaystyle\mathcal{L}_{-}(\theta)$
$\displaystyle=\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{-}}\max\Big{(}0,\tilde{h}(\mathbf{x}_{i}\mid\theta)\Big{)},$
(18b)

with $\mathcal{L}_{+}$ representing the loss for safe states,
$\mathcal{L}_{-}$ representing the loss for unsafe states, $\mathcal{X}_{+}$
being the dataset containing safe interactions (state-control pair),
$\mathcal{X}_{-}$ being the dataset containing unsafe interactions, and

$\tilde{h}(\mathbf{x}\mid\theta)=\widehat{h}(\mathbf{x})+\Delta\widehat{h}(\mathbf{x}\mid\theta).$
(19)

We can see that for a safe state, $\mathcal{L}_{+}$ is only non-zero when the
estimated CBF
$\widehat{h}(\mathbf{x})+\Delta\widehat{h}(\mathbf{x}\mid\theta)$ is negative.
For unsafe states, $\mathcal{L}_{-}$ is only non-zero when the estimated CBF
is positive. Thus, in both cases, only when the sign of the estimated CBF is
wrong will there be a non-zero loss; otherwise, the loss is zero. However, one
trivial solution for $\Delta\widehat{h}$ that minimizes the losses is

$\Delta\widehat{h}(\mathbf{x})=-\widehat{h}(\mathbf{x}).$ (20)

While this is a minimizer for both $\mathcal{L}_{+}$ and $\mathcal{L}_{-}$, it
will also make the estimated CBF zero everywhere, making it an undesired
solution. Although, when combined with the loss derived from the CBF
constraint (which will be discussed later in this section), the learned CBF
will not constantly be zero, it will be close to zero for a large portion of
the state space, which makes it difficult to distinguish between safe and
unsafe states. This phenomenon can be seen in Fig. 1, where the “Without
$\mathbf{d}(x)$” case is trained using the losses in (18).

To deal with this issue, we note that the sign and trend of the CBF matters
while its magnitude is of less importance. Using this intuition, we establish
a simple heuristic, i.e., the further outside the safe set, the more negative
the CBF value should be, and the more inside the safe set, the more positive
the CBF value should be. We define the notion of “more inside” and “more
outside” using the state constraints in (8). For a single constraint
$\mathbf{c}(\mathbf{x})\leq 0$, we can compute the value of

$\mathbf{d}(\mathbf{x})=-\mathbf{c}(\mathbf{x}).$ (21)

When $\mathbf{d}(\mathbf{x})$ is positive, the larger it is, the more inside
the safe set $\mathbf{x}$ is. When $\mathbf{d}(\mathbf{x})$ is negative, the
smaller it is, the more outside the safe set $\mathbf{x}$ is. When there are
multiple constraints, we can compute

$\mathbf{d}_{i}(\mathbf{x})=-\mathbf{c}_{i}(\mathbf{x}),\ \forall i=1,\
\cdots,\ r.$ (22)

Then, $\mathbf{d}(\mathbf{x})$ is defined as

$\mathbf{d}(\mathbf{x})=\min\\{\mathbf{d}_{1}(\mathbf{x}),\
\cdots,\mathbf{d}_{r}(\mathbf{x})\\}.$ (23)

Using the $\mathbf{d}(\mathbf{x})$’s, we can write the new loss functions as

$\displaystyle\mathcal{L}_{+}(\theta)$
$\displaystyle=\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{+}}\max\Big{(}0,-\tilde{h}(\mathbf{x}_{i}\mid\theta)+\mathbf{d}_{+}(\mathbf{x}_{i})\Big{)}$
(24a) $\displaystyle\mathcal{L}_{-}(\theta)$
$\displaystyle=\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{-}}\max\Big{(}0,\tilde{h}(\mathbf{x}_{i}\mid\theta)-\mathbf{d}_{-}(\mathbf{x}_{i})\Big{)},$
(24b)

where $\mathbf{d}_{+},\mathbf{d}_{-}:\mathbb{R}^{n}\rightarrow\mathbb{R}$
represents the distance functions corresponding to the safe and unsafe set,
respectively. The effect of having $\mathbf{d}(\mathbf{x})$ in the loss
function can be seen in Fig. 1 (“With $\mathbf{d}(\mathbf{x})$” curve), where
the CBF value is no longer flat near the obstacle.

In addition to the CBF losses, we add another loss corresponding to the CBF
constraint in (6) to ensure the ability to generate safe control actions

$\displaystyle\mathcal{L}_{\mathbf{\nabla h}}(\theta)=$ $\displaystyle\
\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{+}}\max\Big{(}0,\frac{\partial\tilde{h}(\mathbf{x}_{i}\mid\theta)}{\partial\mathbf{x}}\dot{\tilde{\mathbf{x}}}_{i}$
$\displaystyle-\alpha(\tilde{h}(\mathbf{x}_{i}\mid\theta))\Big{)},$ (25)

where $\dot{\tilde{\mathbf{x}}}_{i}$ is modeled using the learned system
dynamics (see Section 4.2). Although the learned system dynamics would also be
parameterized by a set of weights, when performing gradient-based updates, the
gradient of $\mathcal{L}_{\mathbf{\nabla h}}$ is only calculated with respect
to $\theta$. We will defer the discussion of the learning procedure to Section
4.3. We also add a term in our loss function to regulate the amount of change
in $\tilde{h}(\mathbf{x}\mid\theta)$ induced by
$\Delta\widehat{h}(\mathbf{x}\mid\theta)$ as

$\mathcal{L}_{\Delta
h}(\theta)=\frac{1}{N}\sum_{\mathbf{x}_{i}\in\mathcal{X}_{+}\cup\mathcal{X}_{-}}{\Delta\widehat{h}^{2}(\mathbf{x}_{i}\mid\theta)}.$
(26)

By weighting this term against the other terms in the loss function, we can
add a prior on how confident the user is in the ability of the HCBF to recover
the safe set.

Using the terms defined above, the final loss function is given by

$\mathcal{L}_{\theta}(\theta)=\mathcal{L}_{+}+\lambda_{1}\mathcal{L}_{-}+\mathcal{L}_{\mathbf{\nabla
h}}+\lambda_{2}\mathcal{L}_{\Delta h},$ (27)

with $\lambda_{1},\lambda_{2}\in\mathbb{R}_{+}$ weighting the importance of
the individual loss terms. Since estimating part of the unsafe set as safe is
much more disastrous than estimating part of the safe set as unsafe,
$\lambda_{1}$ is usually larger than one.

###  4.2 Learning the System Dynamics

To learn the system dynamics, we use another neural network parameterized by
$\psi$, i.e., $\mathbf{F}(\mathbf{x}\mid\psi)$, to estimate both
$\Delta\mathbf{f}(\mathbf{x})$ and $\Delta\mathbf{g}(\mathbf{x})$. Using this
neural network, our estimated dynamics is defined as

$\dot{\tilde{\mathbf{x}}}(\mathbf{x},\mathbf{u}\mid\psi)=\widehat{\mathbf{f}}(\mathbf{x})+\widehat{\mathbf{g}}(\mathbf{x})\mathbf{u}+\mathbf{F}(\mathbf{x}\mid\psi)\begin{bmatrix}1\\\
\mathbf{u}\end{bmatrix}.$ (28)

Common methods in learning the system dynamics require obtaining data of
$\dot{\mathbf{x}}$ [23] or the next state (i.e., state at the “next” time
step) [36]. Using $\dot{\mathbf{x}}$ requires additional sensors, e.g.,
inertial measurement units. Using the next state is also not accurate, since
the commonly used integration schemes only approximate the true discrete-time
dynamics. Thus, instead of learning the system dynamics via a regression
problem on $\dot{\mathbf{x}}$ or the next state, we form a regression problem
on $\dot{h}(\mathbf{x})$ [32], which is given by

$\dot{h}(\mathbf{x})=\frac{\partial
h(\mathbf{x})}{\partial\mathbf{x}}\Big{(}\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u}\Big{)}.$
(29)

Using the estimated dynamics from (28) and the estimated CBF from (19), the
estimated $\dot{h}(\mathbf{x})$ is given as

$\dot{\tilde{h}}(\mathbf{x},\mathbf{u},\theta\mid\psi)=\frac{\partial\tilde{h}(\mathbf{x},\theta)}{\partial\mathbf{x}}\dot{\tilde{\mathbf{x}}}(\mathbf{x},\mathbf{u}\mid\psi).$
(30)

When learning the system dynamics, although the value of $\dot{\tilde{h}}$
does depend on $\theta$, the gradient is only calculated with respect to
$\psi$. Therefore, in the remainder of this section, we will omit
$\dot{\tilde{h}}$’s dependency on $\mathbf{u}$ and $\theta$. Additionally, we
can numerically estimate $\dot{h}(\mathbf{x})$ using the central difference
method

$\widehat{\dot{\tilde{h}}}(\mathbf{x})=\frac{\tilde{h}(\mathbf{x}_{+})-\tilde{h}(\mathbf{x}_{-})}{2\Delta{t}},$
(31)

where $\mathbf{x}_{+}$ represents the next state, and $\mathbf{x}_{-}$
represents the previous state. To learn the weights $\psi$, we use the loss
function defined as

$\mathcal{L}_{\psi}(\psi)=\sum_{\mathbf{x}_{i}\in\mathcal{X}_{+}\cup\mathcal{X}_{-}}\Big{(}\widehat{\dot{\tilde{h}}}(\mathbf{x}_{i})-\dot{\tilde{h}}(\mathbf{x}_{i}\mid\psi)\Big{)}^{2}.$
(32)

Note that when performing gradient-based updates for $\psi$, the gradient of
$\mathcal{L}_{\psi}$ is only calculated with respect to $\psi$. Given that

$\lim_{\Delta{t}\rightarrow
0}\widehat{\dot{\tilde{h}}}(\mathbf{x})=\frac{\partial\tilde{h}(\mathbf{x})}{\partial\mathbf{x}}\dot{\mathbf{x}},$
(33)

we can show that for a small $\Delta{t}$ and loss value, the error in the
learned dynamics is bounded. Assuming the loss is less than some positive
value, i.e.,

$\mathcal{L}_{\psi}(\psi)\leq\epsilon,$ (34)

where $\epsilon\in\mathbb{R}_{+}$, yields

$\widehat{\dot{\tilde{h}}}(\mathbf{x}_{i})-\dot{\tilde{h}}(\mathbf{x}_{i}\mid\psi)\leq\sqrt{\epsilon}.$
(35)

Given that the central difference method has a truncation error of
$\mathcal{O}(\Delta{t}^{2})$, we have

$\widehat{\dot{\tilde{h}}}(\mathbf{x})=\frac{\partial\tilde{h}(\mathbf{x})}{\partial\mathbf{x}}\dot{\mathbf{x}}+\mathcal{O}(\Delta{t}^{2}),$
(36)

which leads to

$\frac{\partial\tilde{h}(\mathbf{x}_{i})}{\partial\mathbf{x}}\Big{(}\dot{\mathbf{x}}_{i}-\dot{\tilde{\mathbf{x}}}(\mathbf{x}_{i},\mathbf{u}\mid\psi^{*})\Big{)}+\mathcal{O}(\Delta{t}^{2})\leq\sqrt{\epsilon}.$
(37)

Then, we have the following bound on the error of the learned dynamics, i.e.,
$\dot{\mathbf{x}}_{i}-\dot{\tilde{\mathbf{x}}}$:

$\dot{\mathbf{x}}_{i}-\dot{\tilde{\mathbf{x}}}(\mathbf{x}_{i},\mathbf{u}\mid\psi^{*})\leq\displaystyle\Big{[}\frac{\partial\tilde{h}(\mathbf{x}_{i})}{\partial\mathbf{x}}\Big{]}^{\dagger}\Big{(}\sqrt{\epsilon}+\mathcal{O}(\Delta{t}^{2})\Big{)}.$
(38)

This shows that with a small enough $\Delta{t}$ and loss value, our proposed
algorithm can learn a reasonably accurate model of the system dynamics.

###  4.3 Training Process

We train $\Delta\widehat{h}$ and $\mathbf{F}$ using a supervised learning
approach. For supervised learning, one key assumption for the training data is
that they are independently and identically distributed (i.i.d). Thus, instead
of only training the networks using data collected from the current episode,
we store the data in replay buffers [25] and only use randomly sampled data
from the replay buffer to train the network. We form two replay buffers, one
for safe data $\mathcal{X}_{+}$ and one for unsafe data $\mathcal{X}_{-}$.

The overall training procedure is as follows. At each time step, given the
current state, the performance controller computes a potentially unsafe action
$\mathbf{u}_{\mathrm{perf}}(\mathbf{x})$. Then, the unsafe action is passed
through the learned CBF filter, making it the estimated safe action. The
learned CBF-QP controller has the form

$\displaystyle\min_{\mathbf{u}\in\mathcal{U}}\ $ $\displaystyle\
\|\mathbf{u}-\mathbf{u}_{\mathrm{perf}}(\mathbf{x})\|$ (39)
$\displaystyle\mathrm{subject\ to}\ $ $\displaystyle\
\frac{\partial\tilde{h}(\mathbf{x})}{\partial\mathbf{x}}\dot{\tilde{\mathbf{x}}}(\mathbf{x},\mathbf{u}\mid\psi)\geq-\alpha(\tilde{h}(\mathbf{x})).$

Finally, the control action $\mathbf{u}$ is applied to the environment.
Additionally, the current state, the learned CBF, and the estimated safe
action are stored in the corresponding replay buffer at each time step. After
each episode ends, data sampled from the replay buffer are used to compute the
loss functions in (27) and (32). Then, using a stochastic gradient descent
algorithm, e.g., ADAM [21], the weights of the two networks
$\Delta\widehat{h}$ and $\mathbf{F}$ are updated. This procedure is repeated
until the two networks converge or if a predefined maximum episode number is
reached. A visual illustration of this procedure can be found in Fig. 2. For
our proposed approach, all of the learning is done offline, either in a
simulation environment or a specially designed experiment environment. After
the learning process converges, the learned CBF-QP can then be deployed to the
intended system.

##  5 Simulation Studies

In this section, we show the effectiveness of our approach using three
systems: double integrator, unicycle, and two-link arm. All experiments are
performed using PyTorch with the same neural network architecture. The deep
differential network $\widehat{h}$ consists of three layers with output sizes
$[128,128,1]$. The dynamics network $\mathbf{F}$ consists of two networks, one
for estimating $\Delta{\mathbf{f}}$, with output size $[64,64,n]$, and the
other estimates $\Delta{\mathbf{g}}$, with output size $[64,64,nm]$, which is
reshaped as a $n\times m$ matrix.

###  5.1 Double Integrator

Figure 3: Illustration of the state and control trajectory for the double
integrator system. The learned CBF-QP controller generates the trajectories
under two different initial guesses of the system dynamics. Figure 4: Contour
of the learned CBF for the double integrator system. The orange dashed line
denotes the zero-level line of the true CBF. The region above the orange
dashed line should be negative for the true CBF and positive below. The
contour values correspond to the CBF level sets. Figure 5: Comparison between
the trajectories generated by the estimated (learned) and groundtruth dynamics
starting from different initial guesses for the double integrator system.

The double integrator has the system dynamics given as

$\begin{bmatrix}\dot{x}\\\ \ddot{x}\end{bmatrix}=\begin{bmatrix}0&1\\\
0&0\end{bmatrix}\begin{bmatrix}x\\\ \dot{x}\end{bmatrix}+\begin{bmatrix}0\\\
1/m\end{bmatrix}\mathbf{u},$ (40)

with $x\in\mathbf{R}$ denoting the position, $\dot{x}\in\mathbf{R}$ denoting
the velocity, $\mathbf{u}\in\mathbb{R}$ denoting the control, and
$m\in\mathbf{R}_{+}$ denoting the mass. In our simulation environment, we set
$m=0.5$kg, however, we assume that $m$ is unknown. The system has a velocity
constraint

$\dot{x}\leq 3.$ (41)

We construct the HCBF as

$\widehat{h}(\mathbf{x})=2-\dot{x},$ (42)

which corresponds to the constraint

$\dot{x}\leq 2.$ (43)

Since this is a simple example, we can also get one of the CBFs that recovers
the entire safe set

$h(\mathbf{x})=3-\dot{x},$ (44)

which can be used to check the quality of the learned CBF. During training, we
use a PD controller as the performance controller

$\pi(\mathbf{x})=\mathbf{K}_{p}(x_{\mathrm{des}}-x)+\mathbf{K}_{d}(\dot{x}_{\mathrm{des}}-\dot{x}),$
(45)

with $\mathbf{K}_{p}=3$, $\mathbf{K}_{d}=1.0$, and
$[x_{\mathrm{des}},\dot{x}_{\mathrm{des}}]=[0,0]$. During training, we set
$\lambda_{1}=100.0$ and $\lambda_{2}=1.0$. The learning rate is $10^{-4}$. We
set

$\mathbf{d}_{+}(\mathbf{x})=\mathbf{d}_{-}(\mathbf{x})=3-\dot{x}.$ (46)

The class $\mathcal{K}_{\infty}$ function $\alpha$ is set to be

$\alpha(\mathbf{x})=\gamma\mathbf{x}.$ (47)

During training, the initial state of the system is uniformly sampled with
$x_{0}\in[-15,-5]$ and $\dot{x}_{0}=0$. We provide an initial guess of the
system dynamics by replacing the $m$ in (40) with our guess $\widehat{m}$.
Using our proposed algorithm, we trained for 100 epochs, and the trajectory
generated by the learned CBF-QP controller is shown in Fig. 3. It can be seen
that even though the initial guess is different, the trajectories generated by
the learned CBF-QP controller are very similar. Additionally, the state
trajectories are safe, despite the errors in the HCBF and the nominal
dynamics. The contour plot of the learned CBF is shown in Fig. 4. It can be
seen that the learned safe set almost recovers the true safe set, except for
$x$ values near zero. This is due to having little training data where the $x$
values are close to zero and $\dot{x}$ near $3$m/s. A comparison between the
learned (estimated) and groundtruth dynamics is shown in Fig. 5. We can see
that the learned dynamics are invariant to the initial guess and provides a
relatively accurate estimation of the groundtruth dynamics.

###  5.2 Unicycle

Figure 6: Comparison between the state trajectories of the groundtruth
dynamics and the estimated (learned) dynamics for the unicycle system under
different values of $\gamma$. For larger $\gamma$ values, the CBF-QP generates
control actions that approach the boundary of the safe set more aggressively.
Figure 7: The upper figure shows the motion generated by the learned CBF-QP
controller for the unicycle system under different values of $\gamma$. The
darker blue square represents the square obstacle. The light blue circle
represents the unsafe region corresponding to the HCBF. The lower figure shows
the CBF values along the trajectories for different values of $\gamma$. Figure
8: Illustration of how the partial derivative of the learned CBF changes with
respect to the distance between the unicycle and the obstacle.

The unicycle system has the dynamics

$\begin{bmatrix}\dot{x}\\\ \dot{y}\\\
\dot{\phi}\end{bmatrix}=\begin{bmatrix}\alpha_{v}\cos\phi&0\\\
\alpha_{v}\sin\phi&0\\\ 0&\alpha_{\omega}\end{bmatrix}\begin{bmatrix}v\\\
\omega\end{bmatrix},$ (48)

with $x$ denoting the position of the unicycle along the $x$ axis, $y$
denoting the position along the $y$ axis, and $\phi$ denoting the heading of
the unicycle. The terms $\alpha_{v}$ and $\alpha_{\omega}$ regulate how the
control input $v\in\mathbb{R}$ and $\omega\in\mathbb{R}$ affect the velocity
and angular velocity of the unicycle, respectively. We assume that the values
of $\alpha_{v}$ and $\alpha_{\omega}$ are unknown. The system performs an
obstacle avoidance task, where the obstacle is a square. Compared to a square,
it is easier to write an HCBF for a circular obstacle [15] as

$\widehat{h}(\mathbf{x})=x^{2}+y^{2}+2xl\cos\phi+2yl\sin\phi+l^{2}-r^{2},$
(49)

where $r$ is the radius of the constructed circular obstacle and $l$ is a
predefined lookahead distance. This choice of HCBF corresponds to the state
constraint

$x^{2}+y^{2}\geq r^{2}.$ (50)

This setup is shown in Fig. 7. During training, we use a proportional
controller as the performance controller

$\pi(\mathbf{x})=\begin{bmatrix}\mathbf{K}_{v}e\\\
\mathbf{K}_{\omega}(\beta-\phi)\end{bmatrix},$ (51)

where $\mathbf{K}_{v}=0.75$, $\mathbf{K}_{\omega}=3.0$, and

$\displaystyle e$
$\displaystyle=\sqrt{(x-x_{\mathrm{des}})^{2}+(y-y_{\mathrm{des}})^{2}}$ (52a)
$\displaystyle\beta$
$\displaystyle=\mathrm{atan}2(y_{\mathrm{des}}-y,x_{\mathrm{des}}-x).$ (52b)

For the loss parameters, we set $\lambda_{1}=10.0$ and $\lambda_{2}=0.0$. The
learning rate is set to be $10^{-5}$ and

$\mathbf{d}_{+}(\mathbf{x})=\mathbf{d}_{-}(\mathbf{x})=\max(|x|,|y|)-\ell_{s}/2.$
(53)

where $\ell_{s}$ represents the side length of the square. The class
$\mathcal{K}_{\infty}$ function $\alpha$ is the same as in (47). The system is
trained for 500 epochs, and the trajectory generated by the learned CBF-QP
controller is shown in Fig. 7. During training, we set $\gamma=5.0$. It can be
seen that after training if we change the value of $\gamma$, we can still
generate safe trajectories. Furthermore, as $\gamma$ gets smaller, the
controller gets more conservative, which is the expected behavior. This shows
that even using the learned CBF, we can tune the performance of the controller
without additional training. The difference between the estimated and learned
system dynamics is shown in Fig. 6. The groundtruth values for the control
regulation terms are $\alpha_{v}=0.75$ and $\alpha_{\omega}=0.75$; our initial
guess is $\alpha_{v}=1.0$ and $\alpha_{\omega}=1.0$. As we can see, the
learned (estimated) dynamics are different in many cases from the groundtruth
dynamics. However, the safety of the learned CBF-QP controller is not
violated. As shown in Fig. 8, the partial derivative of the learned CBF with
respect to $x$ is larger than the other two elements, which makes estimation
errors in $\dot{y}$ and $\dot{\phi}$ less significant.

###  5.3 Two-Link Arm

Figure 9: Illustration of the trajectory generated by the learned CBF-QP
controller. The solid blue curve represents the end-effector trajectory using
the learned dynamics. The dashed orange curve represents the end-effector
trajectory using the nominal dynamics. The wall is positioned at $x=3$. To
illustrate the motion of the arm, the two link arm illustrations are drawn
such that darker colors correspond to later in time.

The two-link arm has the system dynamics

$\begin{bmatrix}\dot{\mathbf{q}}\\\
\ddot{\mathbf{q}}\end{bmatrix}=\begin{bmatrix}\dot{\mathbf{q}}\\\
-\textbf{M}^{-1}(\mathbf{q})\textbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}\end{bmatrix}+\begin{bmatrix}\mathbf{0}\\\
\textbf{M}^{-1}(\mathbf{q})\end{bmatrix}\boldsymbol{\tau},$ (54)

where the joint angles are represented by $\mathbf{q}\in\mathbb{R}^{2}$, the
joint velocities by $\dot{\mathbf{q}}\in\mathbb{R}^{2}$, the joint
accelerations by $\ddot{\mathbf{q}}\in\mathbb{R}^{2}$, and the joint torques
by $\boldsymbol{\tau}\in\mathbb{R}^{2}$. The inertia matrix is represented by
$\mathbf{M}(\mathbf{q})\in\mathbb{R}^{2\times 2}$ and the Coriolis matrix is
represented by $\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\in\mathbb{R}^{2\times
2}$. Note that both $\mathbf{M}(\mathbf{q})$ and
$\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})$ are functions of the link masses
$m_{i}$’s and link lengths $\ell_{i}$’s, for $i=\\{1,2\\}$:

$\displaystyle\mathbf{M}(\mathbf{q})$
$\displaystyle=\begin{bmatrix}(m_{1}+m_{2})\ell_{1}^{2}&m_{2}\ell_{1}\ell_{2}\cos(\varphi)\\\
m_{2}\ell_{1}\ell_{2}\cos(\varphi)&m_{2}\ell_{2}^{2}\end{bmatrix},$ (55a)
$\displaystyle\textbf{C}(\mathbf{q},\dot{\mathbf{q}})$
$\displaystyle=\begin{bmatrix}0&m_{2}\ell_{1}\ell_{2}\dot{q}_{2}\sin(\varphi)\\\
-m_{2}\ell_{1}\ell_{2}\dot{q}_{1}\sin(\varphi)&0\end{bmatrix},$ (55b)

where $q_{1}$ represents the angle between the negative $y$ direction and the
first link counterclockwise, $q_{2}$ represents the angle between the negative
$y$ direction and the second link counterclockwise, and $\varphi=q_{1}-q_{2}$.
The end-effector position can be written as

$\begin{bmatrix}x_{ee}\\\
y_{ee}\end{bmatrix}=\begin{bmatrix}\ell_{1}\sin(q_{1})+\ell_{2}\sin(q_{2})\\\
-\ell_{1}\cos(q_{1})-\ell_{2}\cos(q_{2})\end{bmatrix}.$ (56)

In this example, we assume that the link lengths $\ell_{1}$ and $\ell_{2}$ are
unknown, and we have access to measurements of the end-effector position.
Although the link lengths can be found through inverse kinematics, we use our
proposed approach to estimate the system dynamics directly. The system starts
from the joint angles $[0,0]$ and needs to go to $[\pi,\pi]$ while avoiding
hitting a wall at $x=3$. We can write the corresponding HCBF as

$\displaystyle\widehat{h}(\mathbf{x})=$
$\displaystyle-(\widehat{\ell}_{1}\cos(q_{1})\dot{q}_{1}+\widehat{\ell}_{2}\cos(q_{2})\dot{q}_{2})+3\gamma$
$\displaystyle-\gamma(\widehat{\ell}_{1}\sin(q_{1})+\widehat{\ell}_{2}\sin(q_{2})).$
(57)

Figure 10: Illustration of the end-effector trajectory and CBF value over
time. The dashed line in the uppermost plot represents the position of the
wall. The dashed line in the lowermost plot represents the zero-CBF-value
line; everything below this line is considered unsafe by the CBF. Figure 11:
Comparison between the state trajectories of the groundtruth dynamics and the
estimated (learned) dynamics for the two-link arm system.

where $\widehat{\ell}_{1}=1.5$ and $\widehat{\ell}_{2}=1.5$ are the nominal
link lengths and the true link lengths used in the simulation are
$\ell_{1}=1.25\widehat{\ell}_{1}$ and $\ell_{2}=1.25\widehat{\ell}_{2}$.
During training, we use a PD controller as the performance controller

$\boldsymbol{\tau}=\begin{bmatrix}\mathbf{K}_{p}(q_{1}^{\mathrm{des}-q_{1}})+\mathbf{K}_{d}(\dot{q}_{1}^{\mathrm{des}-\dot{q}_{1}})\\\
\mathbf{K}_{p}(q_{2}^{\mathrm{des}-q_{2}})+\mathbf{K}_{d}(\dot{q}_{2}^{\mathrm{des}-\dot{q}_{2}})\end{bmatrix},$
(58)

with $\mathbf{K}_{p}=20.0$ and $\mathbf{K}_{d}=15.0$. The loss parameters are
chosen as $\lambda_{1}=100.0$ and $\lambda_{2}=0.0$. The learning rate is set
to $10^{-5}$. We set

$\mathbf{d}_{+}(\mathbf{x})=\mathbf{d}_{-}(\mathbf{x})=3-x_{ee}.$ (59)

The class $\mathcal{K}_{\infty}$ function $\alpha$ is the same as in (47). The
neural networks are trained for 1000 epochs, and the trajectory generated by
the learned CBF-QP controller is shown in Fig. 9. It can be seen that the
learned CBF-QP controller can render the system safe while ensuring task
completion.

To study the improvement in robustness attained by learning the system
dynamics, we now consider the same training procedure for $\Delta\hat{h}$, but
using only the nominal dynamics. In that case, the resulting trajectory is
also shown in Fig. 9, in which we can see the trajectory is unsafe. The end-
effector trajectory and the CBF value along the trajectory are shown in Fig.
10. It shows that when using the learned dynamics, as the end-effector
position gets closer to the wall, the CBF value goes to zero, and as it leaves
the wall, the CBF value increases, which is the expected behavior. When using
the nominal dynamics, although the learned CBF can recognize the states are
unsafe, the CBF-QP would not be able to generate control actions that pull the
system back into the safe set due to having inaccurate system dynamics. The
difference between the learned (estimated) dynamics and the groundtruth
dynamics is shown in Fig. 11. It can be seen that the state trajectories
generated by learned dynamics resemble the state trajectories generated by
groundtruth dynamics.

##  6 Conclusion

In this paper, we proposed an algorithmic approach to simultaneously learn a
CBF and the system dynamics, starting from an HCBF and nominal dynamics. The
CBF is learned using loss functions that enforce the CBF conditions and the
CBF constraint. We showed theoretically that our proposed approach could also
learn the system dynamics by only using the learned CBF and its time
derivative. The effectiveness of our proposed approach is demonstrated using
three simulation studies: double integrator target reaching under velocity
constraint, unicycle target reaching while avoiding a square obstacle, and
two-link arm target reaching while avoiding collision with a wall. In future
works, we plan to add a learned performance controller and perform experiments
on robotic systems in real life.

## References

  * [1] Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada.  Control barrier functions: Theory and applications.  In Proceedings of European Control Conference, Naples, Italy, pages 3420–3431, June 2019.
  * [2] Aaron D. Ames, Jessy W. Grizzle, and Paulo Tabuada.  Control barrier function based quadratic programs with application to adaptive cruise control.  In Proceedings of IEEE Conference on Decision and Control, Los Angeles, CA, pages 6271–6278, December 2014.
  * [3] Aaron D. Ames, Xiangru Xu, Jessy W. Grizzle, and Paulo Tabuada.  Control barrier function based quadratic programs for safety critical systems.  IEEE Transactions on Automatic Control, 62(8):3861–3876, 2017.
  * [4] Somil Bansal, Mo Chen, Sylvia L. Herbert, and Claire J. Tomlin.  Hamilton-Jacobi reachability: A brief overview and recent advances.  In Proceedings of IEEE Conference on Decision and Control, Melbourne, Australia, pages 2242–2253, December 2017.
  * [5] Somil Bansal and Claire J. Tomlin.  Deepreach: A deep learning approach to high-dimensional reachability.  In IEEE International Conference on Robotics and Automation, Xi’an, China, pages 1817–1824, May 2021.
  * [6] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba.  End to end learning for self-driving cars.  CoRR, abs/1604.07316, 2016.
  * [7] Jason J. Choi, Donggun Lee, Koushil Sreenath, Claire J. Tomlin, and Sylvia L. Herbert.  Robust control barrier-value functions for safety-critical control.  In Proceedings of IEEE Conference on Decision and Control, Austin, TX, pages 6814–6821, December 2021.
  * [8] Bolun Dai.  Adaptive identification of legged robotic kinematic structure.  CoRR, abs/2107.11836, 2021.
  * [9] Bolun Dai, Heming Huang, Prashanth Krishnamurthy, and Farshad Khorrami.  Data-efficient control barrier function refinement.  In Proceedings of the American Control Conference, San Diego, CA, pages 3675–3680, May 2023.
  * [10] Bolun Dai, Rooholla Khorrambakht, Prashanth Krishnamurthy, Vinícius Gonçalves, Anthony Tzes, and Farshad Khorrami.  Safe navigation and obstacle avoidance using differentiable optimization based control barrier functions.  IEEE Robotics and Automation Letters, 8(9):5376–5383, 2023.
  * [11] Bolun Dai, Rooholla Khorrambakht, Prashanth Krishnamurthy, and Farshad Khorrami.  Differentiable optimization based time-varying control barrier functions for dynamic obstacle avoidance.  CoRR, abs/2309.17226, 2023.
  * [12] Bolun Dai, Prashanth Krishnamurthy, and Farshad Khorrami.  Learning a better control barrier function.  In Proceedings of IEEE Conference on Decision and Control, Cancún, Mexico, pages 945–950, December 2022.
  * [13] Bolun Dai, Prashanth Krishnamurthy, Andrew Papanicolaou, and Farshad Khorrami.  State constrained stochastic optimal control for continuous and hybrid dynamical systems using DFBSDE.  Automatica, 155:111146, 2023.
  * [14] Bolun Dai, Virinchi Roy Surabhi, Prashanth Krishnamurthy, and Farshad Khorrami.  Learning locomotion controllers for walking using deep FBSDE.  CoRR, abs/2107.07931, 2021.
  * [15] Yousef Emam, Paul Glotfelter, Zsolt Kira, and Magnus Egerstedt.  Safe model-based reinforcement learning using robust control barrier functions.  CoRR, abs/2110.05415, 2021.
  * [16] Ruben Grandia, Andrew J. Taylor, Aaron D. Ames, and Marco Hutter.  Multi-layered safety for legged robots via control barrier functions and model predictive control.  In Proceedings of IEEE International Conference on Robotics and Automation, Xi’an, China, pages 8352–8358, May 2021.
  * [17] Thomas Gurriet, Andrew Singletary, Jacob Reher, Laurent Ciarletta, Eric Feron, and Aaron D. Ames.  Towards a framework for realizable safety critical control through active set invariance.  In Proceedings of ACM/IEEE International Conference on Cyber-Physical Systems, Porto, Portugal, pages 98–106, April 2018.
  * [18] Taylor A. Howell, Brian E. Jackson, and Zachary Manchester.  ALTRO: A fast solver for constrained trajectory optimization.  In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, China, pages 7674–7679, November 2019.
  * [19] Matthew Kelly.  An introduction to trajectory optimization: How to do your own direct collocation.  SIAM Review, 59(4):849–904, 2017.
  * [20] Hassan K Khalil.  Nonlinear Control.  Pearson New York, 2015.
  * [21] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Proceedings of International Conference on Learning Representations, San Diego, CA, May 2015.
  * [22] Cong Li, Zengjie Zhang, Ahmed Nesrin, Qingchen Liu, Fangzhou Liu, and Martin Buss.  Instantaneous local control barrier function: An online learning approach for collision avoidance.  CoRR, abs/2106.05341, 2021.
  * [23] Michael Lutter, Christian Ritter, and Jan Peters.  Deep lagrangian networks: Using physics as model prior for deep learning.  In Proceedings of International Conference on Learning Representations, New Orleans, LA, May 2019.
  * [24] Ian Michael Mitchell.  Application of level set methods to control and reachability problems in continuous and hybrid systems.  Stanford University, 2002.
  * [25] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [26] Quan Nguyen, Ayonga Hereid, Jessy W. Grizzle, Aaron D. Ames, and Koushil Sreenath.  3d dynamic walking on stepping stones with control barrier functions.  In Proceedings of IEEE Conference on Decision and Control, Las Vegas, NV, pages 827–834, December 2016.
  * [27] Quan Nguyen and Koushil Sreenath.  Exponential control barrier functions for enforcing high relative-degree safety-critical constraints.  In American Control Conference, Boston, MA, pages 322–328, July 2016.
  * [28] Zengyi Qin, Dawei Sun, and Chuchu Fan.  SABLAS: Learning safe control for black-box dynamical systems.  IEEE Robotics and Automation Letters, 7(2):1928–1935, 2022.
  * [29] Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan.  Learning safe multi-agent control with decentralized neural barrier certificates.  In Proceedings of International Conference on Learning Representations, Virtual Event, Austria, May 2021.
  * [30] Alexander Robey, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V. Dimarogonas, Stephen Tu, and Nikolai Matni.  Learning control barrier functions from expert demonstrations.  In Proceedings of IEEE Conference on Decision and Control, Jeju Island, South Korea, pages 3717–3724, December 2020.
  * [31] Matteo Saveriano and Dongheui Lee.  Learning barrier functions for constrained motion planning with dynamical systems.  In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, SAR, China, pages 112–119, November 2019.
  * [32] Andrew J. Taylor, Andrew Singletary, Yisong Yue, and Aaron D. Ames.  Learning for safety-critical control with control barrier functions.  In Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control, Berkeley, CA, volume 120, pages 708–717, June 2020.
  * [33] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg.  Recovery RL: safe reinforcement learning with learned recovery zones.  IEEE Robotics and Automation Letters, 6(3):4915–4922, 2021.
  * [34] Sander Tonkens and Sylvia L. Herbert.  Refining control barrier functions through hamilton-jacobi reachability.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, Kyoto, Japan, pages 13355–13362, October 2022.
  * [35] Chuanzheng Wang, Yiming Meng, Yinan Li, Stephen L. Smith, and Jun Liu.  Learning control barrier functions with high relative degree for safety-critical control.  In European Control Conference, Delft, The Netherlands, pages 1459–1464, June 2021.
  * [36] Tyler Westenbroek, David Fridovich-Keil, Eric Mazumdar, Shreyas Arora, Valmik Prabhu, S. Shankar Sastry, and Claire J. Tomlin.  Feedback linearization for uncertain systems via reinforcement learning.  In Proceedings of IEEE International Conference on Robotics and Automation, Paris, France, pages 1364–1371, May 2020.