Worrisome Properties of Neural Network Controllers and Their Symbolic
  Representations
We raise concerns about controllers' robustness in simple reinforcement
learning benchmark problems. We focus on neural network controllers and their
low neuron and symbolic abstractions. A typical controller reaching high mean
return values still generates an abundance of persistent low-return solutions,
which is a highly undesirable property, easily exploitable by an adversary. We
find that the simpler controllers admit more persistent bad solutions. We
provide an algorithm for a systematic robustness study and prove existence of
persistent solutions and, in some cases, periodic orbits, using a
computer-assisted proof methodology.
Jacek Cyranka, Kevin E M Church, Jean-Philippe Lessard
# Worrisome Properties of Neural Network Controllers

###### Abstract

We raise concerns about controllers' robustness in simple reinforcement learning benchmark problems. We focus on neural network controllers and their low neuron and symbolic abstractions. A typical controller reaching high mean return values still generates an abundance of persistent low-return solutions, which is a highly undesirable property, easily exploitable by an adversary. We find that the simpler controllers admit more persistent bad solutions. We provide an algorithm for a systematic robustness study and prove existence of persistent solutions and, in some cases, periodic orbits, using a computer-assisted proof methodology.

## 1 Introduction

The study of neural network (NN) robustness properties has a long history in the research on artificial intelligence (AI). Since establishing the existence of so-called adversarial examples in deep NNs in [14], it is well known that NN can output unexpected results by slightly perturbing the inputs and hence can be exploited by an adversary. Since then, the robustness of other NN architectures has been studied [44]. In the context of control design using reinforcement learning (RL), the robustness of NN controllers has been studied from the adversarial viewpoint [29, 42]. Due to limited interpretability and transparency, deep NN controllers are not suitable for deployment for critical applications. Practitioners prefer abstractions of deep NN controllers that are simpler and human-interpretable. Several classes of deep NN abstractions exist, including single layer or linear nets, programs, tree-like structures, and symbolic formulas. It is hoped that such abstractions maintain or improve a few key features: generalizability - the ability of the controller to achieve high performance in similar setups (e.g., slightly modified native simulator used in training); deployability - deployment of the controller in the physical world on a machine, e.g., an exact dynamical model is not specified and the time horizon becomes undefined; verifiability - one can verify a purported controller behavior (e.g., asymptotic stability) in a strict sense; performance - the controller reaches a very close level of average return as a deep NN controller.

In this work, we study the robustness properties of some symbolic controllers derived in [24] as well as deep NN with their a few neuron and symbolic abstractions derived using our methods. By robustness, we mean that a controller maintains its average return values when changing the simulator configuration (scheme/ time-step) at test time while being trained on some specific configuration. Moreover, a robust controller does not admit open sets of simulator solutions with extremely poor return relative to the average. In this regard, we found that NNs are more robust than simple symbolic abstractions, still achieving comparable average return values. To confirm our findings, we implement a workflow of a symbolic controller derivation: regression of a trained deep NN and further fine-tuning. For the simplest benchmark problems, we find that despite the controllers reaching the performance of deep NNs measured in terms of mean return, there exist singular solutions that behave unexpectedly and are persistent for a long time. In some cases, the singular solutions are persistent forever (periodic orbits). The found solutions are stable and an adversary having access to the simulation setup knowing the existence of persistent solutions and POs for specific setups and initial conditions may reconfigure the controlled system and bias it towards the bad persistent solutions; resulting in a significant performance drop, and if the controller is deployed in practice, may even lead to damage of robot/machine. This concern is critical in the context of symbolic controllers, which are simple abstractions more likely to be deployed on hardware than deep NNs. Two systems support the observed issues. First, the standard pendulum benchmark from OpenAI gym [5] and the cartpole swing-up problem.

Each instance of an persistent solution we identify is verified mathematically using computer-assisted proof (CAP) techniques based on interval arithmetic [27, 38] implemented in Julia [4]. Doing so, we verify that the solution truly exists and is not some spurious object resulting from e.g., finite arithmetic precision. Moreover, we prove the adversarial exploitability of a wide class of controllers. The existence of persistent solutions is most visible in the case of symbolic controllers. For deep NN, persistent solutions are less prevalent, and we checked that deep NN controllers' small NN abstractions (involving few neurons) somewhat alleviate the issue of symbolic controllers, strongly suggesting that the robustness is inversely proportional to the number of parameters, starkly contrasting with common beliefs and examples in other domains.

**Main Contributions.** Let us summarize the main novel contributions of our work to AI community below.

_Systematic controller robustness study._ In light of the average return metric being sometimes deceptive, we introduce a method for investigating controller robustness by designing an persistent solutions search and the penalty metric.

_Identification and proofs of abundant persistent solutions._ We sys
tematically find and prove existence of a concerning number of persistent orbits for symbolic controllers in simple benchmark problems. Moreover, we carried out a proof of a periodic orbit for a deep NN controller, which is of independent interest. To our knowledge, this is the first instance of such a proof in the literature.

_NN controllers are more robust than symbolic._ We find that the symbolic controllers admit significantly more bad persistent solutions than the deep NN and small distilled NN controllers.

### Related Work

_(Continuous) RL._ A review of RL literature is beyond the scope of this paper (see [34] for an overview). In this work we use state-of-the-art TD3 algorithms dedicated for continuous state/action spaces [12] based on DDPG [25]. Another related algorithm is SAC [16].

_Symbolic Controllers._ Symbolic regression as a way of obtaining explainable controllers appeared in [22, 20, 24]. Other representations include programs [39, 37] or decision trees [26]. For a broad review of explainable RL see [41].

_Falsification of Cyber Physical Systems (CPS)_ The research on falsification [3, 10, 40, 43] utilizes similar techniques for demonstrating the violation of a temporal logic formula, e.g., for finding solutions that never approach the desired equilibrium. We are interested in solutions that do not reach the equilibrium but also, in particular, the solutions that reach minimal returns.

_Verification of NN robustness using SMT_ Work on SMT like ReLUplex [6, 11, 21] is used to construct interval robustness bounds for NNs only. In our approach we construct interval bounds for solutions of a coupled controller (a NN) with a dynamical system and also provide existence proofs.

_Controllers Robustness._ Design of NN robust controllers focused on adversarial defence methods [29, 42].

_CAPs._ Computer-assisted proofs for ordinary differential equations (ODEs) in AI are not common yet. Examples include validation of NN dynamics [23] and proofs of spurious local minima [32].

### Structure of the Paper

Section 2 provides background on numerical schemes and RL framework used in this paper. Section 3 describes the training workflow for the neural network and symbolic controllers. The class of problems we consider is presented in Section 4. We describe the computer-assisted proof methodology in Section 5. Results on persistent periodic orbits appear in Section 6, and we describe the process by which we search for these and related singular solutions in Section 7.

## 2 Preliminaries

### Continuous Dynamics Simulators for AI

Usually, there is an underlying continuous dynamical system with control input that models the studied problem \(s^{\prime}(t)=f(s(t),a(t))\), where \(s(t)\) is the state, \(a(t)\) is the control input at time \(t\), and \(f\) is a vector field. For instance, the rigid body general equations of motion in continuous time implemented in robotic simulators like MuJoCo [36] are \(Mv^{\prime}+c=\tau+J^{T}f\), \(J,f\) is the constraint Jacobian and force, \(\tau\) is the applied force, \(M\) inertia matrix and \(c\) bias forces. For training RL algorithms, episodes of simulated rollouts \((s_{0},a_{0},r_{1},s_{1},\dots)\) are generated; the continuous dynamical system needs to be discretized using one of the available numerical schemes like the Euler or Runge-Kutta schemes [17]. After generating a state rollout, rewards are computed \(r_{k+1}=r(s_{k},a_{k})\). The numerical schemes are characterized by the approximation order, time-step, and explicit/implicit update. In this work, we consider the explicit Euler (E) scheme \(s_{k+1}=s_{k}+hf(s_{k},a_{k})\); this is a first-order scheme with the quality of approximation being proportional to time-step \(h\) (a hyperparameter). Another related scheme is the so-called semi-implicit Euler (SI) scheme, a two-step scheme in which the velocities are updated first. Then the positions are updated using the computed velocities. Refer to the appendix for the exact form of the schemes.

In the research on AI for control, the numerical scheme and time-resolution1 of observations \(h\) are usually fixed while simulating episodes. Assume we are given a controller that was trained on simulated data generated by a particular scheme and \(h\); we are interested in studying the controller robustness and properties after the zero-shot transfer to a simulator utilizing a different scheme or \(h\), e.g., explicit to semi-implicit or using smaller \(h\)'s.

Footnote 1: While in general time-resolution may not be equal to the time step, in this work we set them to be equal.

### Reinforcement Learning Framework

Following the standard setting used in RL, we work with a Markov decision process (MDP) formalism \((\mathcal{S},\mathcal{A},F,r,\rho_{0},\gamma)\), where \(\mathcal{S}\) is a state space, \(\mathcal{A}\) is an action space, \(F\colon\mathcal{S}\times\mathcal{A}\to\mathcal{S}\) is a deterministic transition function, \(r\colon\mathcal{S}\times\mathcal{A}\to\mathbb{R}\) is a reward function, \(\rho_{0}\) is an initial state distribution, and \(\gamma\in(0,1)\) is a discount factor used in training. \(\mathcal{S}\) may be equipped with an equivalence relation, e.g. for an angle variable \(\theta\), we have \(\theta\equiv\theta\in k2\pi\) for all \(k\in\mathbb{Z}\). In RL, the agent (policy) interacts with the environment in discrete steps by selecting an action \(a_{t}\) for the state \(s_{t}\) at time \(t\), causing the state transition \(s_{t+1}=F(s_{t},a_{t})\); as a result, the agent collects a scalar reward \(r_{t+1}(s_{t},a_{t})\), the (undiscounted) return is defined as the sum of discounted future reward \(R_{t}=\sum_{i=t}^{T}r(s_{i},a_{i})\) with \(T>0\) being the fixed episode length of the environment. RL aims to learn a policy that maximizes the expected return over the starting state distribution.

In this work, we consider the family of MDPs in which the transition function is a particular numerical scheme. We study robustness w.r.t. the scheme; to distinguish the _transition function used for training (also called native)_ from the _transition function used for testing_, we introduce the notation \(F_{train}\) and \(F_{test}\) resp. e.g. explicit Euler with time-step \(h\) is denoted \(F_{*}(\mathds{E},h)\), where \(*\in\{test,train\}\).

## 3 Algorithm for Training of Symbolic Controllers and Small NNs

Carrying out the robustness study of symbolic and small NN controllers requires that the controllers are first constructed (trained). We designed a three-step deep learning algorithm for constructing symbolic and small NN controllers. Inspired by the preceding work in this area the controllers are derived from a deep RL NN controller. The overall algorithm is summarized in Alg. 1.

### RL Training

First we train a deep NN controller using the state-of-the-art model-free RL algorithm TD3 [25, 12] - the SB3 implementation [30]. We choose TD3, as it utilizes the replay buffer and constructs deterministic policies (NN). Plots with the evaluation along the training procedure for studied systems can be found in App. C.
### Symbolic Regression

A random sample of states is selected from the TD3 training replay buffer. Symbolic abstractions of the deep NN deterministic policies are constructed using the symbolic regression over the replay buffer samples. Following earlier work [22, 20, 24] the search is performed by an evolutionary algorithm. For such purpose, we employ the PySR Python library [7, 8]. The main hyperparameter of this step is the complexity limit (number of unary/binary operators) of the formulas (\(k\) in Alg. 1). This procedure outputs a collection of symbolic representations with varying complexity. Another important hyperparameter is the list of operators used to define the basis for the formulas. We use only the basic algebraic operators (add, mul., div, and multip. by scalar). We also tried a search involving nonlinear functions like \(tanh\), but the returns were comparable with a larger complexity.

### Distilling Simple Neural Nets

Using a random sample of states from the TD3 training replay buffer we find the parameters of the small NN representation using the mean-squared error (MSE) regression.

### Controller Parameter Fine-tuning

Just regression over the replay buffer is insufficient to construct controllers that achieve expected returns comparable with deep NN controllers, as noted in previous works. The regressed symbolic controllers should be subject to further parameter fine-tuning to maximize the rewards. There exist various strategies for fine-tuning. In this work, we use the non-gradient stochastic optimization covariance matrix adaptation evolution strategy (CMA-ES) algorithm [19, 18]. We also implemented analytic gradient optimization, which takes advantage of the simple environment implementation, and performs parameter optimization directly using gradient descent on the model rollouts from the differentiable environment time-stepping implementation in PyTorch.

## 4 Studied Problems

We perform our experimental investigation and CAP support in the setting of two control problems belonging to the set of standard benchmarks for continuous optimization. First, the pendulum problem is part of the most commonly used benchmark suite for RL - OpenAI gym [5]. Second, the cart pole swing-up problem is part of the DeepMind control suite [35]. Following the earlier work [13] we used a closed-form implementation of the cart pole swing-up problem. While these problems are of relatively modest dimension, compared to problems in the MuJoCo suite, we find them most suitable to convey our message. The low system dimension makes a self-contained cross-platform implementation easier and eventually provides certificates for our claims using interval arithmetic and CAPs.

### Pendulum

The pendulum dynamics is described by a 1d \(2^{nd}\) order nonlinear ODE. We followed the implementation in OpenAI gym, where the ODEs are discretized with a semi-implicit (SI) Euler method with \(h=0.05\). For training we use \(F_{train}(\mathrm{SI},0.05)\). Velocity \(\omega\) is clipped to the range \([-8,8]\), and control input \(a\) to \([-2,2]\). There are several constants: gravity, pendulum length and mass \((g,l,m)\), which we set to defaults. See App. A.1 for the details. The goal of the control is therefore to stabilize the up position \(\theta=0\mod 2\pi\), with zero angular velocity \(\omega\). The problem uses quadratic reward for training and evaluation \(r=-\lfloor\theta\rfloor^{2}-0.1\omega^{2}-0.001a^{2}\), where \(\lfloor\theta\rfloor=\arccos(\cos(\theta))\) at given time \(t\) and action \(a\). The episode length is \(200\) steps. The max reward is \(0\), and large negative rewards might indicate long-term simulated dynamics that are not controlled.

### Cartpole Swing-up

The cartpole dynamics is described by a 2d \(2^{nd}\) order nonlinear ODEs with two variables: movement of the cart along a line (\(x,x^{\prime}\)), and a pole attached to the cart \((\theta,\theta^{\prime})\). We followed the implementation given in [15]. The ODEs are discretized by the explicit Euler (E) scheme with \(h=0.01\). As with the pendulum we use clipping on some system states, and several constants are involved, which we set to defaults. See B for details. The goal of the control is to stabilize the pole upwards \(\theta=0\mod 2\pi\) while keeping the cart \(x\) within fixed boundaries. The problem uses a simple formula for reward \(r=\cos\theta\), plus the episode termination condition if \(|x|\) is above threshold. The episode length is set to \(500\), hence the reward is within \([-500,500]\). Large negative reward is usually indicative of undesirable behaviour, with the pole continuously oscillating, the cart constantly moving, and escaping the boundaries fairly quickly.

## 5 Rigorous Proof Methodology

All of our theorems presented in the sequel are supported by a computer-assisted proof, guaranteeing that they are fully rigorous in a mathematical sense. Based on the existing body of results and our algorithm we developed in Julia, we can carry out the proofs for different abstractions and problems as long as the set of points of non-differentiability is small, e.g., it works for almost all practical applications: ReLU nets, decision trees, and all sorts of problems involving dynamical systems in a closed form. The input to our persistent solutions prover is a function in Julia defining the controlled problem, the only requirement being that the function can be automatically differentiated. To constitute a proof, this part needs to be carried out rigorously with interval arithmetic. Our CAPs are automatic; once our searcher finds a candidate for a persistent solution/PO, a CAP program attempts to verify the existence of the solution/PO by verifying the theorem (Theorem 1) assumptions. If the prover succeeds this concludes the proof.

### Interval Arithmetic

Interval arithmetic is a method of tracking rounding error in numerical computation. Operations on floating point numbers are instead done on _intervals_ whose boundaries are floating point num
bers. Functions \(f\) of real numbers are _extended_ to functions \(\overline{f}\) defined on intervals, with the property that \(\overline{f}(X)\) necessarily contains \(\{f(x):x\in X\}\). The result is that if \(y\) is a real number and \(Y\) is a thin interval containing \(y\), then \(f(y)\in\overline{f}(Y)\). For background, the reader may consult the books [27, 38]. Function iteration on intervals leads to the _wrapping effect_, where the radius of an interval increases along with composition depth. See Figure 1 for a visual.

### Computer-assisted Proofs of Periodic Orbits

For \(x=(x_{1},\ldots,x_{n})\), let \(||x||=\max\{|x_{1}|,\ldots,|x_{n}|\}\). The following is the core of our CAPs.

**Theorem 1**: _Let \(G:U\rightarrow\mathbb{R}^{n}\) be continuously differentiable, for \(U\) an open subset of \(\mathbb{R}^{n}\). Let \(\overline{x}\in\mathbb{R}^{n}\) and \(r^{*}\geq 0\). Let \(A\) be a \(n\times n\) matrix \({}^{2}\) of full rank. Suppose there exist real numbers \(Y\), \(Z_{0}\) and \(Z_{2}\) such that_

\[||AG(\overline{x})|| \leq Y, \tag{1}\] \[||I-ADG(\overline{x})|| \leq Z_{0}\] (2) \[\sup_{|\delta|\leq r^{*}}||A(DG(\overline{x}+\delta)-DG(\overline {x}))|| \leq Z_{2}, \tag{3}\]

_where \(DG(x)\) denotes the Jacobian of \(G\) at \(x\), and the norm on matrices is the induced matrix norm. If \(Z_{0}+Z_{2}<1\) and \(Y/(1-Z_{0}-Z_{2})\leq r_{*}\), the map \(G\) has a unique zero \(x\) satisfying \(||x-\overline{x}||\leq r\) for any \(r\in(Y/(1-Z_{0}-Z_{2}),r_{*}]\)._

A proof can be completed by following Thm 2.1 in [9]. In Sec. 5.3, we identify \(G\) whose zeroes correspond to POs. Conditions (1)-(3) imply that the Newton-like operator \(T(x)=x-AG(x)\) is a contraction on the closed ball centered at the _approximate zero_\(\overline{x}\) with radius \(r>0\). Being a contraction, it has a unique fixed point (\(x\) such that \(x=T(x)\)) by the Banach fixed point theorem. As \(A\) is full rank, \(G(x)=0\), hence an orbit exists. The radius \(r\) measures how close the approximate orbit \(\overline{x}\) is to the exact orbit, \(x\). The contraction is rigorously verified by performing all necessary numerical computations using interval arithmetic. The technical details appear in App. D.2.

### Set-up of the Nonlinear Map

A PO is a finite MDP trajectory. Let the step size be \(h\), and let the period of the orbit be \(m\). We present a nonlinear map that encodes (as zeroes of the map) POs when \(h\) is fixed. However, for technical reasons (see App. E), it is possible for such a proof to fail. If Alg. 2 fails to prove the existence of an orbit with a fixed step size \(h\), we fall back to a formulation where the step size is not fixed, which is more likely to yield a successful proof. This alternative encoding map \(G_{2}\) is presented in App. D.1. Given \(h\), pick \(g(h,\cdot)\in\{g_{\mathrm{E}},g_{\mathrm{SI}}\}\) one of the discrete dynamical systems used for numerically integrating the ODE. Let \(p\) be the dimension of the state space, so \(g(h,\cdot):\mathbb{R}^{p}\rightarrow\mathbb{R}^{p}\). We interpret the first dimension of \(\mathbb{R}^{p}\) to be the angular component, so that a periodic orbit requires a shift by a multiple of \(2\pi\) in this variable. Given \(h\), the number of steps \(m\) (i.e. period of the orbit) and the number of signed rotations \(j\) in the angular variable, POs are zeroes of the map (if and only if) \(G_{1}:\mathbb{R}^{pm}\rightarrow\mathbb{R}^{pm}\), defined by

\[G_{1}(X)=\begin{pmatrix}x_{0}-g(h,x_{m})+(j2\pi,\mathbf{0})\\ x_{1}-g(h,x_{0})\\ x_{2}-g(h,x_{1})\\ \vdots\\ x_{m}-g(h,x_{m-1})\end{pmatrix},\]

where \(\mathbf{0}\) is the zero vector in \(\mathbb{R}^{p-1}\), \(X=(x_{1},\ldots,x_{m})\) for \(x_{i}\in\mathbb{R}^{p}\), and \(x_{1},\ldots,x_{m}\) are the time-ordered states.

## 6 Persistent Orbits in Controlled Pendulum

When constructing controllers using machine learning or statistical methods, the most often used criterion for measuring their quality is the mean return from performing many test episodes. The mean return may be a deceptive metric for constructing robust controllers. More strongly, our findings suggest that mean return is not correlated to the presence of periodic orbits or robustness. One would typically expect a policy with high mean return to promote convergence toward states that maximize the return for any initial condition (IC) and also for other numerical schemes. Our experiments revealed reasons to believe this may be true for deep NN controllers. However, in the case of simple symbolic controllers, singular persistent solutions exist that accumulate large negative returns at a fast pace. By persistent solutions we mean periodic orbits that remain \(\varepsilon\) away from the desired equilibrium. This notion we formalize in Sec. 7.1. We emphasize that all of the periodic orbits that we prove are necessary stable in the usual Lyapunov sense, i.e., the solutions that start out near an equilibrium stay near the equilibrium forever, and hence feasible in numerical simulations. We find such solutions for controllers as provided in the literature and constructed by ourselves employing Alg. 1. We emphasize that our findings are not only numerical, but we support them with (computer-assisted) mathematical proofs of existence.

### Landajuela et. al [24] Controller

First, we consider the symbolic low complexity controller for the pendulum \(a=-7.08s_{2}-(13.39s_{2}+3.12s_{3})/s_{1}+0.27\), derived in [24] (with model given in App. A.1), where \(s_{1}=\cos\theta\), \(s_{2}=\sin\theta\), \(s_{3}=\omega=\theta^{\prime}\), and \(a\) is the control input. While this controller looks more desirable than a deep NN with hundreds thousand of parameters, its performance changes dramatically when using slightly different transition function at test-time, i.e., halved \(h\) (\(F_{test}(\mathrm{SI},0.025)\)) or the explicit Euler scheme (\(F_{test}(\mathrm{E},0.05)\)). Trajectories in Fig. 2 illustrate that some orbits oscillate instead of stabilizing at the equilibrium \(\hat{s}=\hat{\theta}=0\bmod 2\pi\). The average return significantly deteriorates for the modified schemes and the same ICs compared to \(F_{train}(\mathrm{SI},0.05)\); see Tab. 1. Such issues are present in deep NN controllers and small distilled NN to a significantly lower extent. We

Figure 1: Left: midpoint of interval enclosure of a proven persistent solution (see Appendix Tab. 23). Right: log-scale of radius of the interval enclosure. Calculations done at 163 bit precision, the minimum possible for this solution at episode length 1000.
associate the cause of the return deterioration with existence of 'bad' solutions - persistent periodic orbits (POs) (formal Def. 1). Using CAPs (c.f., Sec. 5) we obtain:

**Theorem 2**: _For \(h\in H=\{0.01,0.005,0.0025,0.001\}\), the nonlinear pendulum system with controller a from [24] described in the opening paragraph of Section 6.1 has a periodic orbit (PO) under the following numerical schemes; 1) (SI) with step size \(h\in H\), 2) (E) at \(h=0.05\) (native), and for all \(h\in H\)._

_The identified periodic orbits are persistent (see Def. 2) and generate minus infinity return for infinite episode length, with each episode decreasing the reward by at least 0.198._

### Our Controllers

The issues with robustness and performance of controllers of Sec. 6.1 may be an artefact of a particular controller construction rather than a general property. Indeed, that controller had a division by \(s_{1}\). To investigate this further we apply Alg. 1 for constructing symbolic controllers of various complexities (without divisions). Using Alg. 1 we distill a small NN (single hidden layer with \(10\) neurons) for comparison. In step 2 we use fine-tuning based on either analytic gradient or CMA-ES, each leading to different controllers. The studied controllers were trained using the default transition \(F_{train}(\mathrm{SI},0.05)\), and for testing using \(F_{test}(\mathrm{E},0.05)\), \(F_{test}(\mathrm{E},0.025)\), \(F_{test}(\mathrm{SI},0.05)\), \(F_{test}(\mathrm{SI},0.025)\).

Tab 1 reveals that the average returns deteriorate when using other numerical schemes for the symbolic controllers obtained using Alg. 1, analogous to the controller from [24]. The average return discrepancies are very large as well. We emphasize that all of the studied metrics for the symbolic controllers are far from the metrics achieved for the deep NN controller. Terminating Alg. 1 at step 2 results in a very bad controller achieving mean return only of \(-1061\), i.e., as observed in the previous works the symbolic regression over a dataset sampled from a trained NN is not enough to construct a good controller. Analogous to Theorem 2, we are able to prove the following theorems on persistent periodic orbits (Def. 1) for the controllers displayed in Table 1.

**Theorem 3**: _For \(h\in H=\{0.025,0.0125\}\), the nonlinear pendulum system with controller generated by analytic gradient refinement in Tab. 1 has POs under 1) (SI) with \(h\in H\) and at the native step size \(h=0.05\), 2) (E) with \(h\in H\)._

_The identified periodic orbits are persistent (see Def. 2) and generate minus infinity return for infinite episode length, with each episode decreasing the reward by at least \(0.18\)._

**Theorem 4**: _For \(h=0.0125\) and \(h=0.05\) (native) with scheme (E), the nonlinear pendulum system with controller generated by CMA-ES refinement in Tab. 1 has POs which generate minus infinity return for infinite episode length, with each episode decreasing the reward by at least 0.20._

## 7 Systematic Robustness Study

We consider a controller to be _robust_ when it has "good" return statistics at the native simulator and step size, which persist when we change simulator and/or decrease step size. If a degradation of return statistics on varying the integrator or step size is identified, we wish to identify the source.

### Background on Persistent Solutions and Orbits

Consider a MDP tuple \((\mathcal{S},\mathcal{A},F,r,\rho_{0},\gamma)\), a precision parameter \(\varepsilon>0\), a policy \(\pi\colon\mathcal{S}\to\mathcal{A}\) (trained using \(F_{train}\) and tested using \(F_{test}\)), a desired equilibrium \(\hat{s}\) (corresponding to the maximized reward \(r\)), and episode length \(N\).

**Definition 1**: _We call a persistent periodic orbit (PO) (of period n) an infinite MDP trajectory \((s_{0},a_{0},r_{1},s_{1},a_{1},\dots)\), such that \(s_{kn}=s_{0}\) for some \(n>1\) and all \(k\in\mathbb{N}\), and such that \(\|\hat{s}-s_{j}\|>\varepsilon\) for all \(j\geq 0\)._

**Definition 2**: _A finite MDP trajectory of episode length \(N\)\((s_{0},a_{0},p_{1},s_{1},a_{1},\dots,s_{N})\) such that \(\|\hat{s}-s_{j}\|>\varepsilon\) for all \(0\leq j\leq N\) is called a persistent solution._

Locating the objects in dynamics responsible for degradation of the reward is not an easy task, as they may be singular or local minima of a non-convex landscape. For locating such objects we experimented with different strategies, but found the most suitable the evolutionary search of _penalty maximizing solutions_. The solutions identified using such a procedure are necessarily stable. We introduce a measure of 'badness' of persistent solutions and use it as a search criteria.

**Definition 3**: _We call a penalty value, a function \(p\colon\mathcal{S}\times\mathcal{A}\to\mathbb{R}_{+}\), such that for a persistent solution/orbit the accumulated penalty value is bounded from below by a set threshold \(M\gg 0\), that is \(\sum_{i=0}^{N-1}p(s_{i},a_{i})\geq M\)._

**Remark 4**: _The choice of particular penalty in Def. 3 depend on the particular studied example. We choose the following penalties in the studied problems._

_1. \(p(s,a)=-r(s,a)\) for pendulum._

_2. \(p(s,a)=-r(s)+0.5(\theta^{\prime})^{2}+0.5(x^{\prime})^{2}\) for cartpole swing-up. Subtracting from the native reward value \(r(s)=\cos\theta\) the scaled sum of squared velocities (the cart and pole) and turning off the episode termination condition. This allows capturing orbits that manage to stabilize the pole, but are unstable and keep the cart moving. The threshold \(M\) in Def. 3 can be set by propagating a number of trajectories with random IC and taking the maximal penalty as \(M\)._

**Remark 5**: _For a PO, the accumulated penalty admits a linear lower bound, i.e. \(\sum_{m=0}^{n-1}p(s_{m},a_{m})\geq Cn\) for some \(C>0\). Thm. 2 implies \(C=0.14\) for the POs in Tab. 6 in the Appendix._

Figure 2: \(100\) numerical simulations with IC \(\omega=0\) and \(\theta\) sampled uniformly, time horizon set to \(T=6\), \(x\)-axis shows the (unnormalized) \(\omega\), and \(y\)-axis \(\theta\). In (a), all IC are attracted by an equilibrium at \(\omega=0\)mod\(2\pi\), \(\theta=0\). Whereas when applying different \(F_{test}\), (b) and (c) show existence of attracting periodic solutions (they can be continued infinitely as our theorems demonstrate).
### Searching for and Proving Persistent Orbits

We designed a pipeline for automated persistent/periodic orbits search together with interval proof certificates. By an interval proof certificate of a PO we mean interval bounds within which a CAP that the orbit exist was carried out applying the Newton scheme (see Sec. 5.2), whereas by a proof certificate of a persistent solution (which may be a PO or not) we mean interval bounds for the solution at each step, with a bound for the reward value, showing that it does not stabilize by verifying a lower bound \(\|\hat{s}-s_{t}\|>\varepsilon\). The search procedure is implemented in Python, while the CAP part is in Julia, refer Sec. 5 for further details.

```
1:\(F_{test}\); control policy \(\pi\); \(h\)-parameters of the evolutionary search; penalty function \(p\); trajectory length; search domain;
2:interval certificates of persistent/periodic orbits;
3:for each MDP do
4:for number of searches do
5: initialize CMA-ES search within specified bounds;
6: search for a candidate maximizing penalty \(p\) during the fixed episode length;
7:endfor
8:order found candidates w.r.t. their \(p\) value;
9:endfor
10:for each candidate do
11: search for nearby periodic orbit with Newton's method correction applied to suitable sub-trajectory;
12:if potential periodic orbit found then
13: attempt to prove existence of the orbit with Thm. 1;
14:if proof successful then
15: return an interval certificate of the orbit;
16:else
17: return proof failure;
18:endif
19:else
20: return periodic orbit not found;
21:endif
22: produce and return an interval certificate of the uncontrolled solution;
23:endfor
```

**Algorithm 2** Persistent Solutions/Orbits Search & Prove

### Findings: Pendulum

Changing simulator or step size resulted in substantial mean return loss (see Tab. 1), and simulation revealed stable POs (see Fig. 2). We proved existence of POs using the methods of Section 5.2-5.3. Proven POs are presented in tables in App. F. See also Fig. 3, where an persistent solution shadows an unstable PO before converging to the stable equilibrium. We present proven persistent solutions in the tables in App. F.

Comparing the mean returns in Tab. 1 we immediately see that deep NN controller performance does not deteriorate as much as for the symbolic controller, whereas the small net is located between the two extremes. This observation is confirmed after we run Alg. 2 for the symbolic controllers and NN. In particular, we did not identify any stable periodic orbits or especially long persistent solutions. However, the Deep NN controller is not entirely robust, admitting singular persistent solutions achieving returns far from the mean; refer to Tab. 4. On the other hand, the small \(10\) neuron NN also seems to be considerably more robust than the symbolic controllers. For the case \(F_{test}(\mathrm{E},0.05)\) the average returns are two times larger than for the symbolic controllers, but still two times smaller than for the deep NN. However, in the case \(F_{test}(\mathrm{E},0.05)\), the average returns are close to those of the deep NN contrary to the symbolic controllers. The small NN compares favorably to symbolic controllers in terms of E/SI return discrepancy metrics, still not reaching the level of deep NN. This supports our earlier conjecture (Sec. 1) that controller robustness is proportional to the parametric complexity.
dulum, the small NN sits between the symbolic and deep NN in terms of the studied metrics. We computed the mean accumulated shaped penalty \(p(s,a)=-r(s)+0.5(\theta^{\prime})^{2}+0.5(x^{\prime})^{2}\) for the selected controllers in Tab. 5. The contrast between the deep NN and the symbolic controller is clear, with the small NN being in between those two extremes. The mean penalty is a measure of the prevalence of persistent solutions. However, we emphasize that the Deep NN controller is not entirely robust and also admits singular persistent solutions with bad returns, refer to Tab. 4. Rigorously proving the returns for the deep NN was not possible in this case; see Rem. 6.
## 8 Codebase

Our full codebase is written in Python and Julia shared in a github repository [2]. The reason why the second part of our codebase is written in Julia is the lack of a suitable interval arithmetic library in Python. The Python part of the codebase consists of four independent parts - scripts: deep NN policy training, symbolic/small NN controller regression, regressed controller fine-tuning and periodic orbit/persistent solution searcher. All controllers that we use are implemented in Pytorch [28]. For the deep NN policy training we just use the Stable-baselines 3 library [30], which outputs a trained policy (which achieved the best return during training) and the training replay buffer of data. For the symbolic regression we employ the PySR lib. [7]. For the regressed controller fine-tuning we employ the pycma CMA-ES implementation [18]. Our implementation in Julia uses two external packages: IntervalArithmetic.jl [33] (for interval arithmetic) and ForwardDiff.jl [31] (for forward-mode automatic differentiation). These packages are used together to perform the necessary calculations for the CAPs.

## 9 Conclusion and Future Work

Our work is a first step towards a comprehensive robustness study of deep NN controllers and their symbolic abstractions, which are desirable for deployment and trustfulness reasons. Studying the controllers' performance in a simple benchmark, we identify and prove existence of an abundance of persistent solutions and periodic orbits. Persistent solutions are undesirable and can be exploited by an adversary. Future work will apply the developed methods to study higher dimensional problems often used as benchmarks for continuous control.

## 10 Acknowledgements

The project is financed by the Polish National Agency for Academic Exchange. The first author has been supported by the Polish National Agency for Academic Exchange Polish Returns grant no. PPN/PPO/2018/1/00029 and the University of Warsaw IDUB New Ideas grant. This research was supported in part by PL-Grid Infrastructure.
