Quantitative Verification with Neural Networks
We present a data-driven approach to the quantitative verification of
probabilistic programs and stochastic dynamical models. Our approach leverages
neural networks to compute tight and sound bounds for the probability that a
stochastic process hits a target condition within finite time. This problem
subsumes a variety of quantitative verification questions, from the
reachability and safety analysis of discrete-time stochastic dynamical models,
to the study of assertion-violation and termination analysis of probabilistic
programs. We rely on neural networks to represent supermartingale certificates
that yield such probability bounds, which we compute using a
counterexample-guided inductive synthesis loop: we train the neural certificate
while tightening the probability bound over samples of the state space using
stochastic optimisation, and then we formally check the certificate's validity
over every possible state using satisfiability modulo theories; if we receive a
counterexample, we add it to our set of samples and repeat the loop until
validity is confirmed. We demonstrate on a diverse set of benchmarks that,
thanks to the expressive power of neural networks, our method yields smaller or
comparable probability bounds than existing symbolic methods in all cases, and
that our approach succeeds on models that are entirely beyond the reach of such
alternative techniques.
Alessandro Abate, Alec Edwards, Mirco Giacobbe, Hashan Punchihewa, Diptarko Roy
# Quantitative Verification With Neural Networks For Probabilistic Programs and Stochastic Systems

###### Abstract.

We present a machine learning approach to quantitative verification. We investigate the quantitative reachability analysis of probabilistic programs and stochastic systems, which is the problem of computing the probability of hitting in finite time a given target set of states. This general problem subsumes a wide variety of other quantitative verification problems, from the invariant and the safety analysis of discrete-time stochastic systems to the assertion-violation and the termination analysis of single-loop probabilistic programs. We exploit the expressive power of neural networks as novel templates for indicating supermartingale functions, which provide general certificates of reachability that are both tight and sound. Our method uses machine learning to train a neural certificate that minimises an upper bound for the probability of reachability over sampled observations of the state space. Then, it uses satisfiability modulo theories to verify that the obtained neural certificate is valid over every possible state of the program and conversely, upon receiving a counterexample, it refines neural training in a counterexample-guided inductive synthesis loop, until the solver confirms the certificate. We experimentally demonstrate on a diverse set of benchmark probabilistic programs and stochastic dynamical models that neural indicating supermartingales yield smaller or comparable probability bounds than existing state-of-the-art methods in all cases, and further that the approach succeeds on models that are entirely beyond the reach of such alternative techniques.

Keywords:**Theory of computation Program verification; Probabilistic computation; Computing methodologies Machine learning; Neural networks.

## 1. Introduction

_Probabilistic Programs_. Probabilistic programs extend imperative programs with the ability to sample from probability distributions (Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996), which provide an expressive language for modelling applications such as randomised algorithms (Hastings and Ghahram, 1996; Hastings and Ghahram, 1996), robot motion planning (Kirkpatrick and Ghahram, 1996), cryptographic protocols (Kirkpatrick and Ghahram, 1996), and more generally for modelling and simulation of discrete-time stochastic processes (Hastings and Ghahram, 1996).

_Quantitative Reachability Analysis_. A fundamental problem in the formal verification of probabilistic programs is to compute _reachability probabilities_(Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996). More concretely, this requires us to compute the probability that an execution of a probabilistic program will reach a given subset of program configurations. By selecting the appropriate set of program configurations, we can express a variety of interesting quantitative verification questions in terms of computing reachability probabilities, such as computing the probability that a probabilistic loop terminates or violates an assertion (Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996), and computing the probability that a discrete-time stochastic system, such as a robot moving across an environment, remains within a set of its safe configurations (Hastings and Ghahram, 1996; Hastings and Ghahram, 1996; Hastings and Ghahram, 1996). However, as we might expect for a question of such generality, the problem of determining exact reachability probabilities for probabilistic loops over infinite state spaces is an uncomputable problem, as it can be instantiated to the decision problem of (deterministic) termination. Therefore, algorithmic methods for reachability analysis of probabilistic programs consider the problem of _bounding_ the probability of reaching a predetermined subset of states.
###### Abstract.

We consider the _nonnegative repulsing supermartingales_, which are the most important ones in the study of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expectation of the post-expect of the post-expectation of the post-expectation of the post-expect of the post-expectation of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expect of the post-expectation of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post-expect of the post post-expect of the post-expect of the post post-expect of the post-expect of the post-expect of the post post-expect of the post-expect of the post post-expect of the post post-expect of the post post-expect of the post
#### Variables

An ordered set Vars of real-valued variables with \(n=|\operatorname{Vars}|\), which in turn defines the state space \(\mathbb{R}^{n}\). Under this definition, a state \(s\in\mathbb{R}^{n}\) is an \(n\)-dimensional tuple of reals.

**Initial Condition**: A Boolean expression \(B_{0}\), which in turn defines the initial set \(S_{0}\subseteq\mathbb{R}^{n}\), as the set of states for which expression \(B_{0}\) evaluates to true.
**Update Command**: An update command \(C_{U}\) generated by the grammar of Fig. 1, where we consider every distribution associated with a probabilistic assignment as being obtained by applying the appropriate inverse transformation to a random variable uniformly distributed in the interval \([0,1]\). This defines the update function \(f\colon\mathbb{R}^{n}\times[0,1]^{k}\to\mathbb{R}^{n}\), where \(k\) is the number of syntactic probabilistic assignment statements occurring in \(C_{U}\).

Altogether, our probabilistic program defines a stochastic process, whose behavior is determined by the update equations

\[s_{t+1}=f(s_{t},r_{t}),\quad r_{t}\sim\mathbb{U}^{k},\qquad s_{0}\in S_{0}, \tag{1}\]

where \(\mathbb{U}^{k}\) denotes the uniform distribution over the \(k\)-dimensional hypercube \([0,1]^{k}\). The state at time \(0\) is determined by the initial condition, and each subsequent state is determined by the previous state and \(k\) random numbers sampled from the uniform distribution over \([0,1]\).

We remark that our model of infinitely running probabilistic programs encompasses state-dependent distributions (cf. Fig. 1, probability distributions). Probability distributions can be either discrete or continuous and may depend not only on constant parameters, but also on parameters that are determined from the system state. As such, parameters may depend on other distributions and thus define joint, multi-variate and hierarchically-structured distributions.

To formalise our quantitative verification questions, we first describe the semantics of our probabilistic programs as a Markov chain over the probability space of infinite words of random samples. Namely, this is the probability space defined as \((\Omega,\mathcal{F},\mathbb{P})\), where

* \(\Omega\) is the set of infinite sequences \(\big{(}[0,1]^{k}\big{)}^{\omega}\) of \(k\)-dimensional tuples of values in \([0,1]\), where \(k\) is the number of probabilistic assignments that occur in the update command;
* \(\mathcal{F}\) is the extension of the Borel \(\sigma\)-algebra over the unit interval \(\mathcal{B}([0,1])\) to \(\Omega\);
* \(\mathbb{P}\) is the extension of the Lebesgue measure on \([0,1]\) to \(\Omega\).

This extension of the Borel \(\sigma\)-algebra and Lebesgue measure to infinite sequences is a standard construction [12; 30]. Every state \(s\in\mathbb{R}^{n}\), when used as initial state, induces a stochastic process \(\{X_{t}^{s}(\omega)\}_{t\in\mathbb{N}}\) over the state space \(\mathbb{R}^{n}\). Let \(\omega=r_{0}r_{1}r_{2}\dots\) be an infinite sequence of random samples

Figure 1: Grammar of update commands, Boolean and arithmetic expressions.
in \([0,1]^{k}\), then the stochastic process is defined by the sequence of random variables

\[X^{s}_{t+1}(\omega)=f(X^{s}_{t}(\omega),r_{t}),\qquad X^{s}_{0}(\omega)=s, \tag{2}\]

This defines the natural filtration \(\{\mathcal{F}_{t}\}_{t\in\mathbb{N}}\) which is the smallest filtration to which the stochastic process \(X^{s}_{t}\) is adapted. In other words (Srivastava, 2017), this can be seen as a Markov chain with state space \(\mathbb{R}^{n}\) and transition kernel

\[T(s,S^{\prime})=\operatorname{Leb}\left(\left\{r\in[0,1]^{k}\mid f(s,r)\in S ^{\prime}\right\}\right). \tag{3}\]

where \(\operatorname{Leb}\) refers to the Lebesgue measure of a measurable subset of \([0,1]^{k}\). The transition kernel defines the _next time operator_\(\mathbb{X}\)(Srivastava, 2017, Definition 2.16) which can be applied to an arbitrary Borel-measurable function \(h\colon\mathbb{R}^{n}\to\mathbb{R}\) as

\[\mathbb{X}[h](s)=\int h(s^{\prime})T(s,\mathrm{d}s^{\prime}). \tag{4}\]

The next time operator applied to the function \(h\) defines the _post-expectation_\(\mathbb{X}[h]\) of \(h\).

### Quantitative Reachability Verification

We treat the general question of computing an upper bound on the probability that our stochastic process reaches a state in a target set \(A\in\mathcal{B}(\mathbb{R}^{n})\). This is the _reachability verification_ question. We define the event of reaching the target set, starting from state \(s\in\mathbb{R}^{n}\) after exactly \(t\) time steps as

\[Reach_{t}^{s}(A)=\left\{\omega\in\Omega\mid X^{s}_{0}(\omega)\notin A,\ldots,X^ {s}_{t-1}(\omega)\notin A,X^{s}_{t}(\omega)\in A\right\}. \tag{5}\]

Note that, under this definition, the special case \(t=0\) is defined as \(Reach_{0}^{s}(A)=\Omega\) when \(s\in A\) and \(Reach_{0}^{s}(A)=\emptyset\) when \(s\notin A\).

Henceforth, we use \(\mathbf{1}_{S}\) to denote the _indicator function_ of \(S\subseteq\mathbb{R}^{n}\), namely \(\mathbf{1}_{S}(s)=1\) if \(s\in S\) and \(\mathbf{1}_{S}(s)=0\) if \(s\notin S\). Below, we express the probability measure of the reachability event in exactly \(t\) steps, and at most \(t\) steps, in terms of indicator functions and the next time operator \(\mathbb{X}\).

**Lemma 2.1**.: \(Reach_{t}^{s_{0}}(A)\) _is measurable for every \(A\in\mathcal{B}(\mathbb{R}^{n})\), \(s_{0}\in\mathbb{R}^{n}\) and \(t\in\mathbb{N}\), and_

\[\mathbb{P}[Reach_{t+1}^{s_{0}}(A)] =\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_{0})\cdot\mathbb{X}[ \lambda s_{1}.\,\mathbb{P}[Reach_{t}^{s_{1}}(A)]](s_{0}), \tag{6a}\] \[\mathbb{P}[Reach_{0}^{s_{0}}(A)] =\mathbf{1}_{A}(s_{0}). \tag{6b}\]

Proof.: Measurability of \(Reach_{t}^{s_{0}}(A)\) is a standard result (Srivastava, 2017, Sect. 10.1.1). Eq. (6a) follows from the measurability of the transition kernel (A
**Lemma 2.3**.: \(Reach_{\mathrm{fin}}^{s}(A)\) _is measurable for every \(A\in\mathcal{B}(\mathbb{R}^{n})\) and \(s\in\mathbb{R}^{n}\), and_

\[\mathbb{P}[Reach_{\mathrm{fin}}^{s}(A)]=\lim_{t\to\infty}\mathbb{P}[ Reach_{\leq t}^{s}(A)]. \tag{10}\]

Proof.: \(Reach_{\mathrm{fin}}^{s}\) is measurable because it is a countable union of measurable sets. Furthermore, (10) follows from (9) and the properties of the measure of an increasing union ((7, Prop. 2.59)). 

As we show in Sect. 3, our method computes an upper bound for \(\mathbb{P}[Reach_{\mathrm{fin}}^{s}(A)]\) over the initial states \(s\in S_{0}\). Reachability verification subsumes several important verification questions, and infinitely running probabilistic programs generalise a wide variety of systems.

### Quantitative Verification of Probabilistic Loops

Infinitely running probabilistic programs are generalised models that subsume a wide variety of other probabilistic models. A common example are _probabilistic loops_(Nakamura and others, 2010), which refer to probabilistic programs with a single while-loop, a Boolean expression \(B_{G}\) that denotes the guard condition of the loop, and an update command \(C_{L}\) that denotes the body of the loop:

\[\textbf{while}\ B_{G}\ \textbf{do}\ C_{L}\ \textbf{od} \tag{11}\]

Since our model is infinitely running, we express probabilistic loops as infinitely running probabilistic programs with the initial condition \(B_{0}\) defined as \(B_{G}\), and the update command \(C_{U}\) defined as \(\textbf{if}\ B_{G}\ \textbf{then}\ C_{L}\ \textbf{else skip}\ \textbf{fi}\). This implies that the program executes \(C_{L}\) as long as \(B_{G}\) is satisfied, while it leaves all variables unchanged after leaving the loop. We remark that this model is restricted to single loops in isolation, without nesting. Common verification problems for probabilistic loops are the analysis of termination and of assertion-violation, which we express as follows.

**Termination Analysis**: Let \(G\in\mathcal{B}(\mathbb{R}^{n})\) be the guard set of a probabilistic loop, i.e., the set of states of states for which the guard condition evaluates to true. Then, the termination event is characterised by \(Reach_{\mathrm{fin}}^{s}(\mathbb{R}^{n}\setminus G)\). Our method computes an upper bound for the probability of termination \(\mathbb{P}[Reach_{\mathrm{fin}}^{s}(\mathbb{R}^{n}\setminus G)]\) over every state \(s\in G\) or, dually, a lower bound for the probability of non-termination.
**Assertion-violation Analysis**: Let \(G\in\mathcal{B}(\mathbb{R}^{n})\) the guard set of a single-loop probabilistic program and let \(A\in\mathcal{B}(\mathbb{R}^{n})\) be the satisfying set of an assertion at the beginning of the loop body. The assertion violation event is characterized by \(Reach_{\mathrm{fin}}^{s}(G\setminus A)\). Our method computes an upper bound for the probability of assertion violation or, dually, a lower bound for its satisfaction. Note that multiple assertions can be modelled similarly by adding variables.

### Quantitative Verification of Discrete-time Stochastic Systems

Instances of probabilistic systems that are subsumed by our definition of infinitely running probabilistic programs include stochastic (autonomous) dynamical systems in discrete time, which are widely studied in control theory ((4; 12)). Below, we instantiate these exemplars within our modeling framework. Discrete-time stochastic systems can be defined in terms of stochastic difference equations given a possibly nonlinear vector field \(g\colon\mathbb{R}^{n_{x}}\times\mathbb{R}^{n_{w}}\to\mathbb{R}^{n_{x}}\), a time-invariant input disturbance over \(\mathbb{R}^{n_{w}}\) with arbitrary distribution \(\mathcal{W}\), and an initial condition \(x_{0}\in\mathbb{R}^{n_{x}}\):

\[x_{t+1}=g(x_{t},w_{t}),\quad w_{t}\sim\mathcal{W}. \tag{12}\]

Under the assumption that \(g\) and the distribution \(\mathcal{W}\) can be defined as an update command generated by our program grammar, this is an infinitely running probabilistic program with \(n=n_{x}+n_{w}\) variables, where \(C_{U}\) encodes \(g\) as a composition of deterministic assignments, sampling from \(\mathcal{W}\) as probabilistic assignments from distributions with the respective constant parameters. Autonomous systems are naturally infinitely running and typical questions are the reachability analysis question
(cf. Sect. 2.2), the safety verification question, and the dual invariant verification question, which we phrase as follows.

**Safety Verification**: Let \(B\in\mathcal{B}(\mathbb{R}^{n})\) be a set of bad states. Then \(\Omega\setminus Reach^{s}_{\text{fin}}(B)\) characterises the event that the system never reaches a bad state when initialised in \(s\). Our method computes a lower bound over \(s\in S_{0}\) for the probability of safety, that is, \(1-\mathbb{P}[ Reach^{s}_{\text{fin}}(B)]\).
**Invariant Verification**: Let \(I\in\mathcal{B}(\mathbb{R}^{n})\) be a candidate invariant, then \(\Omega\setminus Reach^{s}_{\text{fin}}(\mathbb{R}^{n}\setminus I)\) characterises the event that \(I\) is an invariant, when the system is initialised in \(s\). Our method computes a lower bound over \(s\in S_{0}\) for the probability that \(I\) is invariant.

## 3. Neural Indicating Supermartingales

We provide the criteria for a certificate to ensure that the probability that a system, with update function \(f\), reaches a state in a target set \(A\in\mathcal{B}(\mathbb{R}^{n})\) from state in an initial set \(S_{0}\in\mathcal{B}(\mathbb{R}^{n})\) is bounded from above by a given value \(p\in[0,1]\). We call this certificate a _neural indicating supermartingale_. This we define as a neural network with \(n\) input neurons, an arbitrary number of hidden layers, and one output neuron, which encodes a function of type \(V\colon\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}\cup\{\infty\}\), whose output is guaranteed to be non-negative over the entire domain and satisfies the following three conditions:

(indicating condition) \[\forall s\in A\colon V(s)\geq 1,\] (13a) (non-increasing condition) \[\forall s\notin A\colon\mathbb{X}[V](s)\leq V(s),\] (13b) (bounding condition) \[\forall s\in S_{0}\colon V(s)<p.\] (13c)

Theorem 3.1 ().: _Let \(A\in\mathcal{B}(\mathbb{R}^{n})\) be a target set and \(s\in\mathbb{R}^{n}\) be an arbitrary initial state. Let \(V\colon\mathbb{R}^{n}\to\mathbb{R}_{\geq 0}\cup\{\infty\}\) be a non-negative function that satisfies (13a) and (13b). Then \(V(s)\geq\mathbb{P}[ Reach^{s}_{\text{fin}}(A)]\)._

Proof.: It follows from Lem. 2.2 and 2.3 and Kleene's fixpoint theorem over a CPO defined over the space of Borel measurable non-negative functions that satisfy (13a) and (13b), cf. Appendix A.3. We discard the requirement of an externally provided invariant from the soundness proof for non-negative repulsing supermartingales (S
yields probability bounds that are comparable or better than existing methods, and further succeeds on a wider range of probabilistic programs than previously possible.

### Training of Neural ISMs From Sample Observations

As previously stated, in the present work a neural ISM consists of a neural network with ReLU activation functions, with an arbitrary number of hidden layers. As with any ReLU-based neural network, we can train it using gradient descent and a finite set of states \(d\) sampled over \(\mathbb{R}^{n}\), which we denote as the finite sampling of the state space \(D=\{d^{(1)},\ldots,d^{(m)}\}\).

We seek to construct a loss function that guides the neural network to satisfy the specification set out Eq. (13). We therefore consider a loss function consisting of the summation of multiple terms, with each term corresponding to a condition for the ISM:

\[\mathcal{L}(V)=\beta_{1}\mathcal{L}_{\text{ind}}(V)+\beta_{2}\mathcal{L}_{ \text{bnd}}(V)+\beta_{3}\mathcal{L}_{\text{non-inc}}(V). \tag{15}\]

Here, \(\beta_{j}>0,j\in\{1,2,3\}\), denotes a constant coefficient for scaling the terms of each constituent loss. This reduces the task of learning a neural ISM to constructing a suitable loss function. The proposed loss function in Eq. (15) consists of three separate components, one for each condition in Eq. (13). Notably, we satisfy a typical non-negativity condition by construction (cf. Eq. (14)). Next, we consider each component in turn and provide a suitable loss that will encourage satisfaction of Eqs. (13a), (13c) and (13b).

#### 3.1.1. Indicating Loss

First, consider the condition in Eq. (13a), which we refer to as the indicating condition. For this, we use the following loss function:

\[\mathcal{L}_{\text{ind}}(V)=\mathbb{E}_{d\in D\cap A}\left[\text{ReLU}(1-V(d) )\right]. \tag{16}\]

This penalises all points not within the bad states which fail to satisfy the indicating condition, whilst ignoring any states which already satisfy it. In particular, we take the expectation of this over our finite data set \(S\).

#### 3.1.2. Bounding Loss

Secondly, we turn to the bounding condition, given by equation (13c). This condition determines the 'quality' of the ISM, as it provides the upper-bound \(p\). We can encourage to synthesis better ISMs by including a term in the loss that encourages

\[\mathcal{L}_{\text{bnd}}(V)=\mathbb{E}_{d\in D\cap S_{0}}\left[V(d)\right)]. \tag{17}\]

Finally, we consider the non-increasing condition, Eq. (13b), which relies on computing the post-expectation of the martingale. We consider two alternative approaches to this calculation: one which assumes information about the program, by leveraging a symbolic inference engine, and one which does not, relying instead on a data-driven approach to perform approximate inference. We label these approaches as _program-aware_ and _program-agnostic_ synthesis, and treat them next.

#### 3.1.3. Program-aware Non-increasing Loss

We begin by considering program-aware synthesis, which leverages knowledge of the program within the loss function itself. In particular, we implement the algorithm used by the exact symbolic inference engine PSI (Zhu et al., 2017; Zhang et al., 2018), along with symbolic representations of the program and candidate martingale \(V\) to calculate the post-expectation \(\mathbb{X}[V]\) of the martingale. This can then be used to construct a loss function which penalises samples for which the non-increasing condition is unsatisfied

\[\mathcal{L}_{\text{non-inc}}(V)=\mathbb{E}_{d\in D\setminus A}\left[\text{ReLU}( \mathbb{X}[V](d)-V(d))\right]. \tag{18}\]
#### 3.1.4. Program-agnostic Non-increasing Loss

While it is advantageous to incorporate as much knowledge of the program into the loss function as possible, the program-aware approach relies on a symbolic formula representing the program. For large programs this formula becomes more complex and possibly intractable. Furthermore, it relies on the smoothness of \(\mathbb{X}[V]\). We therefore consider an alternative formulation of the loss function which does not require knowledge of the program--rather, solely access to traces of it--and therefore is likely to perform better for more complex programs.

Ultimately, a program-agnostic approach differs from a program-aware one in that it does not exactly calculate the post-expectation of the ISM \(\mathbb{X}[V]\); instead, this is now approximately calculated using a data-driven approach. We can then use the same loss function as the program-aware case, Eq. (18), but using this approximation for the post-expectation. We emphasise that this does not impact the soundness of our scheme as an exact symbolic calculation of the post-expectation is used during verification. However we expect the probability bounds associated to the certificates to be less tight than those distilled from the program-aware ones. In practice, we utilise a Monte Carlo based approach to estimate the post-expectation. For each state, we sample its successor states by executing the program \(f\). To obtain an estimate of the post-expectation of \(V\) we extend every element \(d^{(i)}\) in our data set with a set of \(m^{\prime}\) randomly sampled successor states \(D^{\prime(i)}=\{d^{(i,1)},\ldots,d^{(i,m^{\prime})}\}\). Then \(\mathbb{X}[V](d)\) is estimated as:

\[\mathbb{X}[V](d)\approx\mathbb{E}_{d^{\prime}\in D^{\prime}}[V(d^{\prime})]. \tag{19}\]

#### 3.1.5. Smoothness of Activation Functions

Finally, we note that since ReLU is a non-smooth function, we found that in practice it is preferable to substitute it with a smooth approximation, Softplus, which takes the form

\[\text{Softplus}(s)=\log(1+\exp(s)). \tag{20}\]

However, Softplus can be a poor approximation to ReLU over the required \([0,1]\) interval for probabilities. Therefore, in practice we learn a function that instead represents a scaled probability \(\alpha p\), with the value \(\alpha>0\) being fixed to some large integer. As a result of this, the indicating component of the loss, Eq. (16) becomes \(\mathcal{L}_{\text{ind}}(V)=\mathbb{E}_{d\in D\cap\alpha}[\text{ReLU}(\alpha- V(d))]\). It is important to note that while Softplus is used as activation in the learning stage, during the verification stage we still verify using ReLU activation functions, ensuring soundness of the generated neural ISM.

### Verification of Neural ISMs Using SMT Solving

The purpose of the verification stage is to ensure that for no possible program trace are the conditions described in Eq. (13) violated. For this purpose, we rely on SMT solving. This involves providing an SMT solver with the negation of the specification described by Eq. (13). However, since the non-negativity condition is satisfied by construction of the networks, this part can be disregarded. Furthermore, the bounding condition does not impact the validity of the ISM, but instead its quality. The probability bound \(p\) is first optimised by the \(\mathcal{L}_{\text{bnd}}\) component in our loss function, then estimated using a binary search over \([0,1]\) and then checked over \(S_{0}\) as part of the verification step. The conditions given in Eq. (13) become

\[\forall s\in\mathbb{R}^{n}:\underbrace{(s\in A\to V(s)\geq 1)}_{\varphi_{ \text{ind}}}\wedge\underbrace{(s\notin A\to\mathbb{X}[V](s)\leq V(s))}_{ \varphi_{\text{non-inc}}}\wedge\underbrace{(s\in S_{0}\to V(s)<p)}_{ \varphi_{\text{ind}}}. \tag{21}\]

Therefore, the SMT solver is provided with the negation of Eq. (21), namely

\[\exists s\in\mathbb{R}^{n}:\underbrace{(s\in A\wedge V(s)<1)}_{\neg\varphi_{ \text{ind}}}\vee\underbrace{(s\notin A\wedge\mathbb{X}[V](s)>V(s))}_{\neg\varphi _{\text{non-inc}}}\vee\underbrace{(s\in S_{0}\wedge V(s)\geq p)}_{\neg\varphi _{\text{ind}}}. \tag{22}\]
The verifier seeks any assignment \(d_{\text{ex}}\) of \(s\) such that any of the formulae \(\neg\varphi_{\text{ind}}\), \(\neg\varphi_{\text{non-inc}}\) and \(\neg\varphi_{\text{{\sf{bnd}}}}\) are satisfiable (this check can be run in parallel over the three formulae). SMT solving is _sound_: if an assignment for \(s\) exists such that any of the provided formulae \(\neg\varphi_{\text{ind}}\), \(\neg\varphi_{\text{non-inc}}\) and \(\neg\varphi_{\text{{\sf{bnd}}}}\) are satisfiable then it will be found. Crucially, this means that if no counterexample is found, then the conditions in Eq. (13) are satisfied and the neural ISM is valid.

Alternatively, an assignment that satisfies any of \(\neg\varphi_{\text{ind}}\), \(\neg\varphi_{\text{non-inc}}\) or \(\neg\varphi_{\text{non-inc}}\) is a counterexample \(d_{\text{ex}}\): this represents a state for which the required conditions are invalidated and thus the candidate ISM is incorrect. The counterexample is then added to the data set \(D\) for synthesis to resume. Furthermore, we can determine which condition is invalidated by evaluating the clauses \(\neg\varphi_{\text{ind}}\), \(\neg\varphi_{\text{non-inc}}\) and \(\neg\varphi_{\text{{\sf{bnd}}}}\) separately at \(d_{\text{ex}}\) and determining which of these are satisfied.

### Counterexample-guided Inductive Synthesis

In this section we have detailed an inductive synthesis procedure to generate neural ISMs which utilises counterexamples. This procedure is known as counter-example guided inductive synthesis (CEGIS) (Steintein and Tschum, 2007; Steintein and Tschum, 2008) (cf. Fig. 2), and consists of of two components: a learner and a verifier. These components work in opposition to each other. On the one hand, the learner seeks to synthesise a candidate neural ISM that meets the desired specification (cf. Eq. (13)) over a finite set of samples from the state space. On the other, the verifier seeks to disprove the candidate generated by the learner by searching for counterexamples, i.e., instances of invalidation of the desired specification, over the original state space. If the verifier shows that no such counterexample exists, then the desired specification is met by the program and synthesis terminates successfully.

In this work, the learner takes the form of a ReLU-based neural network, trained over finite samples; meanwhile, the role of the verifier is assumed by the SMT solver. Several options exist for the latter choice, under the condition that the SMT solver can reason over nonlinear real arithmetic: in this work, we use Z3 (Zi et al., 2017).

#### 3.3.1. Generation of Additional Counterexamples

In order to better facilitate counterexample-guided synthesis, it is desirable to generate multiple counterexamples (if they exist). This can be achieved by calling the verifier with the condition \(\neg\varphi_{\text{ind}}\land\neg\varphi_{\text{non-inc}}\land\neg\varphi_{ \text{{\sf{bnd}}}}\land s\neq d_{\text{ex}}\), i.e. we now seek an assignment to \(\neg\varphi_{\text{ind}}\land\neg\varphi_{\text{non-inc}}\land\neg\varphi_{ \text{{\sf{bnd}}}}\) that is different to the one already obtained. However, in practice more useful counterexamples exist outside the neighbourhood of our current one. Suppose we have \(N\) counterexamples \(d_{\text{ex}}^{(1)},\ldots,d_{\text{ex}}^{(N)}\); we can instead construct

\[\neg\varphi_{\text{ind}}\land\neg\varphi_{\text{non-inc}}\land\neg\varphi_{ \text{{\sf{bnd}}}}\land\bigwedge_{1\leq i\leq N}(||d_{\text{ex}}^{(i)}-s||\geq \gamma), \tag{23}\]

where \(\gamma\in\mathbb{R}^{+}\) and \(||\cdot||\) denotes any norm of its input, to find further counterexamples that are at least \(\gamma\) from all current counterexamples (according to the chosen norm).

Figure 2. Overview of the counterexample-guided inductive synthesis procedure used to synthesise neural ISMs. Inputs to the procedure include a probabilistic program \(f\), initial set \(S_{0}\) and target set \(A\). The procedure generates an neural indicating supermartingale \(V(s)\) alongside a probability bound \(p\).
```
error=0; i=0; whilei<10do assert(error==0); p~Bernoulli(0.999); ifp==1then q~Bernoulli(0.5); ifq==1then i+=1 fi od else error=1 fi od
```

Listing 1: faulty_loop

## 4. Experimental Evaluation

The previous section develops a method for synthesising neural ISMs. This section presents an empirical evaluation of the method, by testing it against a series of benchmarks. Each benchmark is tested ten times. A test is successful if a valid ISM is synthesised, and the proportion of successful tests is recorded. Further, the average bound from the valid ISMs is also recorded, along with the average time taken by learning and verification. This procedure is applied separately for program-aware and program-agnostic synthesis. To compare our method against existing work, we perform template-based synthesis of linear ISMs using Farkas' Lemma.

It should be noted that our method is inherently stochastic. One source of randomness is the initialisation of template parameters in learning. In program-agnostic synthesis, an additional source of randomness is sampling successor states. So that the results accurately reflect the performance of our method, the random seed for these sources of randomness is different for each test. An additional source of randomness arises from the SMT solver Z3, when generating counterexamples. This cannot be controlled externally. Benchmarks are run on a machine with an Nvidia A40 GPU.

The benchmarks in this report are created using two patterns.

**Unreliable Hardware**: The first pattern models a program that executes on unreliable hardware. The goal is to upper bound the probability that the program fails to terminate due to a hardware fault. A simple example of such a program is faulty_loop, whose source code is presented in Listing 1. In this program, a variable i is initialised to 0. In each successful iteration, it is incremented by 1 with probability 0.5. A hardware fault is modelled with the error flag being set to 1.
**Robot Motion**: The second pattern models an agent that is moving through a physical environment. In these benchmarks, the uncertainty in control and sensing is modelled probabilistically. Part of the environment is designated as the target region, and part of the environment is designated as a hazardous region. The goal is to upper bound the probability that the robot enters the hazardous region. A simple example of such a program is repulse, whose source code is presented in Listing 2. The program starts at co-ordinate 10, in a one-dimensional environment. The target region is where \(x<0\), and hazardous region is where \(x>100\)
In each iteration, there is an equal probability of \(x\) being decremented by 2, and \(x\) being incremented by 1.

Several of the programs are based on benchmarks used in prior work focusing on other types of supermartingales (Beng et al., 2015; Beng et al., 2015). Additionally, there are several benchmarks that are entirely new.

### Results

The results are reported in Table 1. The table is divided into three sections. The first section shows the benchmarks where Farkas' Lemma can not be applied, and only our method is capable of producing a bound. The second section shows examples where both methods are able to produce a bound, but our method produces a notably better bound. The third section shows benchmarks where both methods produce comparable bounds.

The table has a column which describes the structure of the neural ISM that was trained. A single number \(h\) indicates that the ISM has a single hidden layer with \(h\) neurons. A pair \((h_{1},h_{2})\) indicates that there are two hidden layers with \(h_{1}\) and \(h_{2}\) neurons respectively.

Dashes in the table indicate experiments where a valid ISM could not be obtained. In the case of Farkas' Lemma, there are several cases, where there is no linear ISM for the benchmark. By contrast, program-agnostic and program-aware synthesis could be applied to almost all benchmarks. The one exception is \(\mathsf{repulse\_uniform}\) where only program-aware verification was unsuccessful. This is due to indicator functions in the post-expectation. These functions are not smooth expressions which posed a problem for the optimiser. This benchmark underscores the value of program-agnostic synthesis, since it does not require embedding the post-expectation in the loss function.

The first section of the benchmarks demonstrates that our method is capable of producing useful results on programs that are out-of-scope for previous methods. Furthermore, it is encouraging that the success ratio is high on all benchmarks. The second and third sections show benchmarks in which Farkas' Lemma can be applied. For these sections, the success ratio of our method is mostly 1.0. A high success ratio is to be expected, since these programs can also be solved by Farkas' Lemma.

The second section shows more complex benchmarks where our method was able to significantly improve upon Farkas' Lemma. The smallest improvement was over 0.04, and the largest improvement was over 0.39. The intuition here is that the neural structure allows more sophisticated ISMs to be learnt, that can better approximate how the reachability probability varies across the state space than is possible with linear templates, and thereby yield tighter probability bounds.

The third section shows relatively simple benchmarks. Here our method produces results that are marginally less tight in comparison to Farkas' Lemma. This is not surprising since our method uses neural networks consisting of a single neuron for these examples, owing to their simplicity. The expressive power of these networks is therefore similar to linear templates.

In summary, the results show our method does significantly better on more complex examples, and marginally worse on very simple examples. This is highlighted in Figure 3. Each point represents a benchmark. The position on the \(x\)-axis shows the probability bound obtained by our program-agnostic method, and the \(y\)-axis shows the probability bound obtained by Farkas' Lemma. Points above the line indicate benchmarks where neural ISMs outperform Farkas' Lemma, and vice versa. The scale is logarithmic to emphasise order-of-magnitude differences.

It is also worth considering the difference between the two loss functions used. Notice that the program-aware algorithm usually performs better than the program-agnostic algorithm, but the improvement is mostly marginal. This is in fact a strength of our method: our data-driven approach performs almost as well as one dependent on symbolic representations, which is promising in light of questions of scalability to more complex programs.
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Benchmark**} & \multirow{2}{*}{**Farkas’**} & \multicolumn{4}{c}{**Quantitative Neural Certificates**} & \multicolumn{2}{c}{**Network**} \\ \cline{3-6}  & & \multicolumn{2}{c}{Program-Agnostic} & \multicolumn{2}{c}{Program-Aware} & \multirow{2}{*}{**Arch.**} \\ \cline{3-6}  & & \multicolumn{1}{c}{\(p\)} & \multicolumn{1}{c}{Success Ratio} & \multicolumn{1}{c}{\(p\)} & \multicolumn{1}{c}{Success Ratio} \\ \hline persist\_2d & - & \(\leq\) 0.1026 & 0.9 & \(\leq\) 0.1175 & 0.9 & (3, 1) \\ faulty\_marbles & - & \(\leq\) 0.0739 & 0.9 & \(\leq\) 0.0649 & 0.8 & 3 \\ faulty\_unreliable & - & \(\leq\) 0.0553 & 0.9 & \(\leq\) 0.0536 & 1.0 & 3 \\ faulty\_regions & - & \(\leq\) 0.0473 & 0.9 & \(\leq\) 0.0411 & 0.9 & (3, 1) \\ \hline cliff\_crossing & \(\leq\) 0.4546 & \(\leq\) 0.0553 & 0.9 & \(\leq\) 0.0591 & 0.8 & 4 \\ repulsive & \(\leq\) 0.0991 & \(\leq\) 0.0288 & 1.0 & \(\leq\) 0.0268 & 1.0 & 3 \\ repulsive\_uniform & \(\leq\) 0.0991 & \(\leq\) 0.0344 & 1.0 & - & - & 2 \\ repulsive\_2d & \(\leq\) 0.0991 & \(\leq\) 0.0568 & 1.0 & \(\leq\) 0.0541 & 1.0 & 3 \\ faulty\_varying & \(\leq\) 0.1819 & \(\leq\) 0.0864 & 1.0 & \(\leq\) 0.0865 & 1.0 & 2 \\ faulty\_concave & \(\leq\) 0.1819 & \(\leq\) 0.1399 & 1.0 & \(\leq\) 0.1356 & 0.9 & (3, 1) \\ \hline fixed\_loop & \(\leq\) 0.0091 & \(\leq\) 0.0095 & 1.0 & \(\leq\) 0.0094 & 1.0 & 1 \\ faulty\_loop & \(\leq\) 0.0181 & \(\leq\) 0.0195 & 1.0 & \(\leq\) 0.0184 & 1.0 & 1 \\ faulty\_uniform & \(\leq\) 0.0181 & \(\leq\) 0.0233 & 1.0 & \(\leq\) 0.0221 & 1.0 & 1 \\ faulty\_rare & \(\leq\) 0.0019 & \(\leq\) 0.0022 & 1.0 & \(\leq\) 0.0022 & 1.0 & 1 \\ faulty\_easyl & \(\leq\) 0.0801 & \(\leq\) 0.1007 & 1.0 & \(\leq\) 0.0865 & 1.0 & 1 \\ faulty\_ndercr & \(\leq\) 0.0561 & \(\leq\) 0.0723 & 1.0 & \(\leq\) 0.0630 & 1.0 & 1 \\ faulty\_walk & \(\leq\) 0.0121 & \(\leq\) 0.0173 & 1.0 & \(\leq\) 0.0166 & 1.0 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results comparing neural ISMs with Farkas Lemma for different benchmarks. Here, \(p\) is the average probability bound generated by the certificate; success ratio is the number of successful experiments, out of 10 repeats, generated by CEGIS with neural ISM; ‘-’ means no result was obtained. We also denote the architecture of the network: \((h_{1},h_{2})\) denotes a network with 2 hidden layers consisting of \(h_{1}\) and \(h_{2}\) neurons respectively.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Benchmark**} & \multirow{2}{*}{**Farkas’**} & \multicolumn{4}{c}{**Quantitative Neural Certificates**} \\ \cline{3-6}  & & \multicolumn{2}{c}{Program-Agnostic} & \multicolumn{2}{c}{Program-Aware} \\ \cline{3-6}  & & Learn Time & Verify Time & Learn Time & Verify Time \\ \hline persist\_2d & - & 169.14 & 85.31 & 44.96 & 74.90 \\ faulty\_marbles & - & 114.24 & 29.23 & 15.86 & 28.68 \\ faulty\_unreliable & - & 123.85 & 45.48 & 18.34 & 33.97 \\ faulty\_regions & - & 17.92 & 35.85 & 17.55 & 32.38 \\ \hline cliff\_crossing & 0.11 & 134.61 & 19.02 & 21.27 & 29.07 \\ repulse & 0.19 & 16.65 & 5.00 & 6.49 & 3.74 \\ repulse\_uniform & 0.19 & 21.28 & 14.18 & - & - \\ repulsive\_2d & 0.12 & 122.92 & 64.54 & 15.75 & 47.70 \\ faulty\_varying & 0.36 & 21.74 & 5.06 & 4.71 & 3.28 \\ faulty\_concave & 0.39 & 49.12 & 13.37 & 13.49 & 7.82 \\ \hline fixed\_loop & 0.15 & 14.16 & 3.14 & 3.34 & 2.43 \\ faulty\_loop & 0.16 & 25.52 & 3.81 & 3.73 & 2.66 \\ faulty\_uniform & 0.34 & 20.20 & 1.91 & 6.75 & 1.33 \\ faulty\_rare & 0.27 & 25.52 & 4.27 & 3.71 & 2.96 \\ faulty\_easyl & 0.31 & 104.20 & 12.78 & 4.95 & 7.51 \\ faulty\_ndercr & 0.33 & 104.89 & 9.06 & 5.37 & 4.66 \\ faulty\_walk & 0.32 & 15.08 & 4.00 & 6.97 & 3.33 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results showing the time taken in seconds to synthesise ISMs by our method and Farkas’ Lemma. For our method, we show the time taken during learning and verification.
The time taken to run the benchmarks are shown in Table 2. This is separated into the time taken for learning and the time taken for verification. Unsurprisingly, the time taken for synthesis varies, depending the complexity of the program. Time taken for verification is similar for both program-aware and program-agnostic synthesis. However, time taken for learning is significantly higher in the program-agnostic algorithm. This is not surprising, since the ISM needs to be evaluated many times for each state in program-agnostic synthesis, whereas the post-expectation needs to be evaluated once for each state in program-aware synthesis. Also, note that Farkas' Lemma is faster than our method. This is expected, as when the ISM certificate is instantiated to the setting of linear templates and programs, the conditions give rise to a convex optimisation problem that can be solved by linear programming (Farkas, 2018). By contrast, learning neural certificates is a non-convex optimisation problem for which we must resort to gradient descent.

### Case Studies

Having presented the experimental results, we shall look at some benchmarks in more detail.

Recall the repulse program introduced earlier (Listing 2). While this is a small program, our method is still able to produce a significantly better result than Farkas' Lemma. The linear and neural ISMs are illustrated in Figure 3(a). This neural ISM has a single hidden layer consisting of three ReLU components that are summed together. This allows a convex piecewise linear function to be learnt.

The cliff_crossing program (Listing 3) is an interesting benchmark, for which our method is capable of producing a significantly better bound. This program models a robot crossing a road that

Figure 3. Probability bounds generated using program-agnostic neural ISMs and using Farkas’ Lemma. We also show the line \(y=x\) to indicate which approach outperforms and by how much: above the line indicates that neural ISMs outperform Farkas’ Lemma, below the line for the opposite. This demonstrates that neural ISMs can significantly outperform linear templates when a better bound exists, but otherwise is able to achieve similar results. Note that our approach with program-aware neural ISMs provides even better outcomes, compared with Farkas’ Lemma.
is adjacent to a cliff. The x-coordinate models how far along the road the robot is, with \(x=0\) being the beginning, and \(x=50\) being the end. Then the y-coordinate models how close to the cliff the robot is, with \(y=0\) being the furthest point away from the cliff, and \(y=10\) being the closest point to the cliff still on the road. Here the neural ISM consists of four ReLU components, and the benefits of a piecewise linear function occur in two dimensions, leading to a clear improvement compared to the linear ISM. These two ISMs are illustrated in Figure 4. Notice the probability bound at the initial state \((0,0)\) is significantly lower in the neural ISM.

So far, these examples have used neural ISMs with a single hidden layer. An example that uses two hidden layers is faulty_concave. The source code can be found in Listing 4. Two hidden layers are necessary in order to obtain performance that is superior to Farkas' Lemma. The benchmark faulty_concave is similar to faulty_loop, which has already been introduced. The variable \(i\) is initialised to \(1\). In each successful iteration, it is incremented with probability \(0.5\) until \(10\) is reached. In each iteration, there is a small probability of a hardware fault. Unlike faulty_loop, the probability of failure changes depending on the value of \(i\). When \(1\leq i\leq 3\), the probability of failure is \(0.001\). However, when \(4\leq i\leq 9\), the probability of failure is \(0.01\), an order of magnitude higher. A linear template cannot account for this conditional behaviour, so Farkas' Lemma produces an overly conservative ISM, which behaves as if the probability of failure were \(0.01\) in every iteration. Indeed, a neural ISM with a single hidden layer also cannot account for this branching behaviour. This is because such an ISM would always be convex. A concave ISM is needed, which requires at least two hidden layers. This is illustrated in Figure 3(b).

## 5. Related Work

### Martingales

Martingales have already been applied extensively in the context of verifying probabilistic programs.
_Ranking supermartingales_. RSMs have previously been studied to prove positive almost-sure termination [13]. This is a qualitative reachability property, in contrast to the quantitative properties studied here. In particular, [3] use a CEGIS-based approach to synthesise neural RSMs.

_Repulsing supermartingales_. RepSMs are another kind of martingale used to prove quantitative reachability properties [16]. They produce a bound using Azuma's inequality. A weakness of RepSMs is that they can only be applied to a narrow range of problems [59]. In many other cases, a valid RepSM does not exist, or such a RepSM produces a trivial bound. By contrast, ISMs can be applied to a broad range of probabilistic problems.

Figure 4. Indicating supermartingales for the repulse benchmark and faulty_concave as generated using Farkas’ Lemma and using neural ISMs. The initial state (dashed line) shows that tighter upper bounds are generated using neural ISMs compared to Farkas’ Lemma.

Figure 5. Indicating supermartingales for the cliff_crossing benchmark as generated using Farkas’ Lemma (5a) and using neural ISMs (5b). The right hand figure illustrates the tighter bounds obtainable through the use of neural templates.
_Indicating supermartingales_. This work introduces ISMs to verify quantitative reachability properties. Similar certificates have been introduced earlier under the names _non-negative repulsive supermartingales_ and _stochastic invariant indicators_(Han and Krapivsky, 2015; Krapivsky, 2016; Krapivsky, 2016). However, existing methods rely on symbolic synthesis approaches which impose further, strong conditions.

Specifically, they require the identification of deterministic invariants that are stronger than \(\mathbb{R}^{n}\). Invariants are required by methods for the synthesis of linear certificates based on Farkas' Lemma to enforce non-negativity, and it is required for the synthesis of polynomial certificates, which additionally imposes compactness to leverage Putinar's Positivstellensatz.

By contrast, neural ISMs do not rely on further conditions beyond (13a), (13b), and (13c), and leverage the expressivity of neural networks to represent certificates without requiring a deterministic invariant to restrict the domain of the function. While we ensure the soundness of our approach, this additional expressive power limits its theoretical completeness. Still, we have experimentally shown that our method is highly likely to succeed in practice over a broad variety of benchmarks.

_Cost martingales_. Previous work has used martingales and similar structures to bound the expected value of a variable in program state, when a probabilistic program terminates. This variable can be interpreted as a reward or a cost (Krapivsky, 2016; Krapivsky, 2016). This work has been extended to higher moments (Krapivsky, 2016). These techniques uses linear and semidefinite programming for synthesis, unlike this work which uses learning in a CEGIS architecture for synthesis.

### Probabilistic Model Checking

A well-developed approach to reasoning about probabilistic systems is probabilistic model checking (Krapivsky, 2016; Krapivsky, 2016). A well established probabilistic model checking tool is PRISM (Krapivsky, 2016). This approach encodes properties using probabilistic extensions to temporal logics such as LTL and CTL. Verification occurs through a combination of graph analysis and numerical approximation. They can be applied to systems modelled as discrete-time Markov chains (DTMCs), continuous-time Markov chains (CTMCs) and Markov decision processes (MDPs). Probabilistic model checking is usually applied to finite state spaces, whereas this work looks at programs with infinite state spaces, and programs that combine discrete and continuous variables.

### Pre-Expectation Calculus

The weakest pre-expectation calculus is a formalism for reasoning about probabilistic programs (Krapivsky, 2016; Krapivsky, 2016; Krapivsky, 2016). It is an extension of predicate transformer semantics for classical programs (Krapivsky, 2016). The calculus relates expressions to their expected values after a program runs. The pre-expectation calculus is connected to martingales, and this is studied in (Krapivsky, 2016). In order to reason about the weakest pre-expectations of loops, invariants are used.

Recent work concentrates on automatically finding weakest pre-expectations. (Krapivsky, 2016) present a method where finding an invariant from a template is reduced to first-order logic constraints, that are given to a constraint solver to handle. (Krapivsky, 2016) use counterexample refinement and Lagrange interpolation to find polynomial invariants. (Krapivsky, 2016) introduce a method for the static analysis of probabilistic programs, based on algebraic structures. This method was applied to generate linear invariants.

Most recently, a CEGIS-like method has been used to learn invariants (Krapivsky, 2016). This work is similar to the work presented here in that it is a data-driven approach using learning, uses a computer algebra system, and uses counterexamples to explore the state space. One difference is that the work uses regression trees as models, as opposed to neural networks. A limitation of this work is that it only supports discrete distributions, whereas this work also looks at probabilistic programs, that sample from continuous distributions.
### Dynamical Systems

While our work looks at discrete-time stochastic systems, CEGIS-based methods have also been used to verify continuous-time systems, specified with differential equations. CEGIS has been used to verify stability for such systems by synthesising Lyapunov functions with polynomial and neural templates (Bauer and Sigmond, 2001; Sigmond, 2002; Sigmond, 2002). Further, safety has been verified with barrier certificates (Sigmond, 2002; Sigmond, 2002).

## 6. Conclusion

Our method is the first to use neural networks as certificates for quantitative reachability analysis of probabilistic programs, to upper bound the probability that a target set is reached in finitely many steps. We exploit the expressive power of neural networks as templates for indicating supermartingales to obtain tight approximations of the reachability probability, without the need for supporting deterministic invariants. Our program-agnostic synthesis method uses gradient descent optimisation of a loss function defined over program executions to discover neural certificates, whose soundness we verify using satisfiability modulo theories in a counterexample-guided inductive synthesis loop. We provide a prototype implementation, and show the advantages of neural certificates over a diverse set of benchmarks, including their applicability to programs that lack linear/polynomial certificates, and their ability to attain tighter probability bounds on reachability. Our approach is suited to extension to further quantitative verification questions, such as obtaining probability bounds on a wider range of temporal properties beyond reachability (Bauer and Sigmond, 2002), and bounds on other types of quantities such as expected accrued costs (Sigmond, 2002; Sigmond, 2002; Sigmond, 2002).

## Appendix A. Detailed Proofs

### Proof of Lemma 2.1

From measurability of the transition kernel and (Bauer and Sigmond, 2002, Lem. 1) we have that

\[\mathbb{P}[Reach_{t+1}^{s_{0}}(A)] =\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_{0})\cdot\int_{ \mathbb{R}^{n}}\mathbb{P}[Reach_{t}^{s_{1}}(A)]\ T(s_{0},\mathrm{d}s_{1}) \text{By \@@cite[cite]{[\@@bibref{}{Bauer and Sigmond, 2002}{}{}]}} \tag{4}\] \[=\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_{0})\cdot\mathbb{X}[ \lambda s_{1}.\mathbb{P}[Reach_{t}^{s_{1}}(A)]] \text{By \@@cite[cite]{[\@@bibref{}{Bauer and Sigmond, 2002}{}{}]}}\]

Case \(t=0\) follows from the definition of the reachability event in (5).

### Proof of Lemma 2.2

Since the sets in the family \(\{Reach_{t}^{s_{0}}\colon t\in\mathbb{N}\}\) are disjoint, we note that the union in (7) that defines \(Reach_{\leq t}^{s_{0}}\) is a union of disjoint sets. By countable additivity of measure and disjointness of the union in (7), we see that:

\[\mathbb{P}[Reach_{\leq t}^{s_{0}}(A)]=\sum_{i=0}^{t}\mathbb{P}[ Reach_{t}^{s_{0}}(A)] \tag{24}\]

By Lem. 2.1, we have that

\[\mathbb{P}[Reach_{\leq t+1}^{s_{0}}(A)] =\mathbb{P}[Reach_{0}^{s_{0}}(A)]+\sum_{i=1}^{t+1}\mathbb{P}[ Reach_{i}^{s_{0}}(A)] \text{By \@@cite[cite]{[\@@bibref{}{Bauer and Sigmond, 2002}{}{}]}} \tag{25}\] \[=\mathbf{1}_{A}(s_{0})+\sum_{i=1}^{t+1}\mathbb{P}[ Reach_{i}^{s_{0}}(A)] \text{By \@@cite[cite]{[\@@bibref{}{Bauer and Sigmond, 2002}{}{}]}}\] \[=\mathbf{1}_{A}(s_{0})+\sum_{i=0}^{t}\mathbb{P}[ Reach_{i+1}^{s_{0}}(A)]\]
\[=\mathbf{1}_{A}(s_{0})+\sum_{i=0}^{t}\mathbf{1}_{\mathbb{R}^{n} \setminus A}(s_{0})\cdot\mathbb{X}[\lambda s_{1}.\,\mathbb{P}[Reach_{t}^{s_{1}}( A)]](s_{0})\] By ( 6a ) \[=\mathbf{1}_{A}(s_{0})+\sum_{i=0}^{t}\left(\mathbf{1}_{\mathbb{R}^ {n}\setminus A}(s_{0})\cdot\int_{\mathbb{R}^{n}}\mathbb{P}[Reach_{t}^{s_{1}}( A)]\ T(s_{0},\mathrm{d}s_{1})\right)\] By ( 4 ) \[=\mathbf{1}_{A}(s_{0})+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_{0 })\cdot\int_{\mathbb{R}^{n}}\left(\sum_{i=0}^{t}\mathbb{P}[Reach_{t}^{s_{1}}( A)]\right)\ T(s_{0},\mathrm{d}s_{1})\] \[=\mathbf{1}_{A}(s_{0})+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_ {0})\cdot\int_{\mathbb{R}^{n}}\mathbb{P}[Reach_{\leq t}^{s_{1}}(A)]\ T(s_{0}, \mathrm{d}s_{1})\] By ( 24 ) \[=\mathbf{1}_{A}(s_{0})+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_ {0})\cdot\mathbb{X}[\lambda s_{1}.\,\mathbb{P}[Reach_{\leq t}^{s_{1}}(A)]](s_{0})\] By ( 4 )

Furthermore, we have that

\[\mathbb{P}[Reach_{\leq 0}^{s_{0}}(A)]=\mathbb{P}[Reach_{0}^{s_{0}}(A)]= \mathbf{1}_{A}(s_{0})\] By ( 24 ) and ( 6b ) ( 25 )

### Proof of Theorem 3.1

We adapt the soundness proof of non-negative repulsing supermartingales [59] to neural certificates (which do not require a externally-provided deterministic invariant to restrict their domain) in our setting of probabilistic systems (i.e. we do not consider non-deterministic transition dynamics).

We consider the set of Borel measurable functions

\[\mathcal{V}=\mathbb{R}^{n}\rightarrow\mathbb{R}_{\geq 0}\cup\{\infty\} \tag{26}\]

and give it the structure of a complete partially ordered set (CPO), [19, p. 175]:

* we extend the natural ordering relation on \(\mathbb{R}_{\geq 0}\cup\{\infty\}\) to the pointwise partial order over functions: \[U\sqsubseteq V\Leftrightarrow\forall s\in\mathbb{R}^{n}.\,U(s)\leq V(s)\] (27)
* we define the bottom element \(\bot\) as the function that always returns zero, \[\bot(x)=0,\] (28)
* for a countable increasing chain of functions \(\{V_{i}\}_{i\in\mathbb{N}}\) we define its least upper bound (lub) \(\sqcup\{V_{i}\colon i\in\mathbb{N}\}\) argumentwise: \[(\sqcup\{V_{i}\colon i\in\mathbb{N}\})\ (s)=\sqcup\{V_{i}(s)\colon i\in \mathbb{N}\}\] (29)

We introduce a higher-order function \(\Phi:\mathcal{V}\rightarrow\mathcal{V}\) defined as

\[\Phi(V)(s)=\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s)\cdot \mathbb{X}[V](s) \tag{30}\]

We now show that if a \(V\in\mathcal{V}\) satisfies conditions (13a) and (13b) then it is a pre-fixpoint of \(\Phi\), namely that:

\[\Phi(V)\sqsubseteq V \tag{31}\]

Let us recall that \(\Phi(V)(s)=\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s)\cdot \mathbb{X}[V](s)\) by definition (30). Then, we let \(s\in\mathbb{R}^{n}\) be arbitrary, and proceed by cases.

\[\text{Case }s\in A\colon\ \Phi(V)(s)=\mathbf{1}_{A}(s)+\mathbf{1}_{ \mathbb{R}^{n}\setminus A}(s)\cdot\mathbb{X}[V](s)=1\leq V(s)\] By ( 13a ) \[\text{Case }s\notin A\colon\ \Phi(V)(s)=\mathbf{1}_{A}(s)+\mathbf{1}_{ \mathbb{R}^{n}\setminus A}(s)\cdot\mathbb{X}[V](s)=\mathbb{X}[V](s)\leq V(s)\] By ( 13b )

We now aim to show that the function \(\lambda s_{0}\in\mathbb{R}^{n}.\,\mathbb{P}[Reach_{\mathrm{fin}}^{s_{0}}(A)]\) is the least fixpoint of \(\Phi\), using Kleene's Fixpoint Theorem [19, CPO Fixpoint Theorem 1, Theorem 8.15, p. 183], and to do this we have to show that \(\Phi\) is continuous, that is,

* \(\Phi\) is monotone, that is, \(U\sqsubseteq V\) implies \(\Phi(U)\sqsubseteq\Phi(V)\) and
* \(\Phi\) preserves least upper bounds, that is, \(\Phi(\sqcup\{V_{i}\colon i\in\mathbb{N}\})=\sqcup\{\Phi(V_{i})\colon i\in\mathbb{N}\}\)

_Monotonicity._ To show that that \(\Phi\) is monotone, let \(s\in\mathbb{R}^{n}\) be arbitrary and assume \(U\sqsubseteq V\):

\[\Phi(U)(s) =\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s) \cdot\mathbb{X}[U](s)\] \[=\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s)\cdot \int_{\mathbb{R}^{n}}U(s^{\prime})\ T(s,\mathrm{d}s^{\prime})\] By ( 4 ) \[\leq\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s) \cdot\int_{\mathbb{R}^{n}}V(s^{\prime})\ T(s,\mathrm{d}s^{\prime})\] By \[U\sqsubseteq V\] and ( 27 ) \[=\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s) \cdot\mathbb{X}[V](s)\] By ( 4 ) \[=\Phi(V)(s)\]

therefore \(\Phi(U)\sqsubseteq\Phi(V)\).

_Least upper bound preservation._ We first note that \(\{\Phi(V_{i}):i\in\mathbb{N}\}\) is a countable increasing chain because \(\Phi\) is monotone. Then, to show that \(\Phi\) preserves least upper bounds, we show that for any countable increasing chain \(\{V_{i}:i\in\mathbb{N}\}\):

\[\Phi(\sqcup\{V_{i}\colon i\in\mathbb{N}\})=\sqcup\{\Phi(V_{i})\colon i\in \mathbb{N}\} \tag{32}\]

To this end, we first note that by Eq. 29, and properties of monotone increasing real sequences in the extended non-negative reals (54, Theorem 15, p.21), for arbitrary \(s\in\mathbb{R}^{n}\) we have:

\[(\sqcup\{V_{i}\colon i\in\mathbb{N}\})(s)=\sqcup\{V_{i}(s):i\in\mathbb{N}\}= \lim_{i\to\infty}V_{i}(s) \tag{33}\]

Then, we show that (33) implies (34) defined as

\[\mathbb{X}[\sqcup\{V_{i}:i\in\mathbb{N}\}](s)=(\sqcup\{\mathbb{X}[V_{i}]:i\in \mathbb{N}\})(s) \tag{34}\]

which holds because

\[\mathbb{X}[\sqcup\{V_{i}:i\in\mathbb{N}\}](s) =\int_{\mathbb{R}^{n}}(\sqcup\{V_{i}:i\in\mathbb{N}\})(s^{\prime })\ T(s,\mathrm{d}s^{\prime})\] By ( 4 ) \[=\int_{\mathbb{R}^{n}}\left(\lim_{i\to\infty}V_{i}(s^{\prime}) \right)\ T(s,\mathrm{d}s^{\prime})\] By ( 33 ) \[=\lim_{i\to\infty}\int_{\mathbb{R}^{n}}V_{i}(s^{\prime})\ T(s, \mathrm{d}s^{\prime})\] By MCT \[=\lim_{i\to\infty}\mathbb{X}[V_{i}](s)\] By ( 4 ) \[=(\sqcup\{\mathbb{X}[V_{i}]:i\in\mathbb{N}\})(s)\] By ( 33 )

This is an application of the Monotone Convergence Theorem (MCT) (7, p.78) because \(\{V_{i}\}_{i\in\mathbb{N}}\) is a countable increasing chain of non-negative functions. We now use (34) to show that \(\Phi\) preserves least upper bounds. Let \(s\in\mathbb{R}^{n}\) be arbitrary:

\[\Phi(\sqcup\{V_{i}\colon i\in\mathbb{N}\})(s) =\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s) \cdot\mathbb{X}[\sqcup\{V_{i}\colon i\in\mathbb{N}\}](s)\] By ( 30 ) \[=\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s)\cdot \sqcup\{\mathbb{X}[V_{i}](s)\colon i\in\mathbb{N}\}\] By ( 34 ) and ( 29 ) \[=\sqcup\{\mathbf{1}_{A}(s)+\mathbf{1}_{\mathbb{R}^{n}\setminus A }(s)\cdot\mathbb{X}[V_{i}](s)\colon i\in\mathbb{N}\}\] \[=(\sqcup\{\Phi(V_{i})\colon i\in\mathbb{N}\})\ (s)\] By ( 29 ) and ( 30 )
_Least fixpoint of \(\Phi\)._ Next, we define \(\Phi^{0}(V)=V\) and \(\Phi^{t+1}(V)=\Phi(\Phi^{t}(V))\), and show that the function that maps a state to its finite time reachability probability is equal to a particular least upper bound

\[\lambda s_{0}\in\mathbb{R}^{n}.\,\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s_{0 }}(A)]=\sqcup\{\Phi^{t}(\bot)\colon t\in\mathbb{N}\} \tag{35}\]

which by Kleene's Fixpoint Theorem is the least fixpoint of \(\Phi\). To achieve this, we show that

for all

\[t\in\mathbb{N}\]

 and

\[s_{0}\in\mathbb{R}^{n}\]

, we have \[\mathbb{P}[\mathit{Reach}_{\leq t}^{s_{0}}(A)] =\Phi^{t+1}(\bot)(s_{0});\] (36) for all

\[s_{0}\in\mathbb{R}^{n}\]

, we have \[\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s_{0}}(A)] =(\sqcup\{\Phi^{t}(\bot)\colon t\in\mathbb{N}\})(s_{0})\] (37)

_Proof of (36)._ We proceed by induction on \(t\in\mathbb{N}\), that is, we show

* \(\mathbb{P}[\mathit{Reach}_{\leq 0}^{s_{0}}(A)]=\Phi^{1}(\bot)(s_{0})\),
* and, assuming \(\mathbb{P}[\mathit{Reach}_{\leq t}^{s_{0}}(A)]=\Phi^{t+1}(\bot)(s_{0})\) we show that \(\mathbb{P}[\mathit{Reach}_{\leq t+1}^{s_{0}}(A)]=\Phi^{t+2}(\bot)(s_{0})\)

For the base case, we recall that \(\forall s.\,\bot(s)=0\) and therefore \(\mathbb{X}[\bot](s_{0})=0\). Then, we have

\[\mathbb{P}[\mathit{Reach}_{\leq 0}^{s_{0}}(A)] =\mathbf{1}_{A}(s_{0}) \text{By \eqref{eq:1}}\] \[=\mathbf{1}_{A}(s_{0})+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_ {0})\cdot\mathbb{X}[\bot](s_{0}) \text{By \eqref{eq:2} and \eqref{eq:2}}\] \[=\Phi^{1}(\bot)(s_{0})\]

For the inductive case, we leverage Lem. 2.2 as follows:

\[\mathbb{P}[\mathit{Reach}_{\leq t+1}^{s_{0}}(A)] =\mathbf{1}_{A}(s_{0})+\mathbf{1}_{\mathbb{R}^{n}\setminus A}(s_ {0})\cdot\mathbb{X}[\lambda s_{1}.\mathbb{P}[\mathit{Reach}_{\leq t}^{s_{1}}(A )]](s_{0}) \text{By \eqref{eq:2}}\] \[=\Phi(\lambda s_{1}.\mathbb{P}[\mathit{Reach}_{\leq t}^{s_{1}}(A )])(s_{0}) \text{By \eqref{eq:2}}\] \[=\Phi(\Phi^{t+1}(\bot))(s_{0}) \text{By IH}\] \[=\Phi^{t+2}(\bot)(s_{0})\]

_Proof of (37)._ We leverage Lem. 2.3 and (36):

\[\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s_{0}}(A)] =\lim_{t\to\infty}\mathbb{P}[\mathit{Reach}_{\leq t}^{s_{0}}(A)] \text{By \eqref{eq:2}}\] \[=\lim_{t\to\infty}\Phi^{t+1}(\bot)(s_{0}) \text{By \eqref{eq:2}}\] \[=\lim_{t\to\infty}\Phi^{t}(\bot)(s_{0})\] \[=(\sqcup\{\Phi^{t}(\bot)\colon t\in\mathbb{N}\})(s_{0}) \text{By \eqref{eq:2}}\]

_Conclusion._ By (37) we conclude (35). By (35), and continuity of \(\Phi\), we invoke Kleene's Fixpoint Theorem [19, CPO Fixpoint Theorem 1, Theorem 8.15, p. 183] to conclude that \(\lambda s_{0}\in\mathbb{R}^{n}.\,\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s_ {0}}(A)]\) is the least fixpoint of \(\Phi\). Therefore, \(\lambda s_{0}\in\mathbb{R}^{n}.\,\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s_ {0}}(A)]\) is also the least pre-fixpoint of \(\Phi\). This implies that \(\left(\lambda s_{0}\in\mathbb{R}^{n}.\,\mathbb{P}[\mathit{Reach}_{\mathrm{fin }}^{s_{0}}(A)]\right)\sqsubseteq V\) and therefore \(V(s)\geq\mathbb{P}[\mathit{Reach}_{\mathrm{fin}}^{s}(A)]\) follows.
