# Developing Optimal Causal Cyber-Defence Agents
via Cyber Security Simulation

Alex Andrew  Sam Spillard  Joshua Collyer  Neil Dhir

###### Abstract

In this paper we explore cyber security defence, through the unification of a
novel cyber security simulator with models for (causal) decision-making
through optimisation. Particular attention is paid to a recently published
approach: dynamic causal Bayesian optimisation (Aglietti et al., 2021, dcbo).
We propose that dcbo can act as a blue agent when provided with a view of a
simulated network and a causal model of how a red agent spreads within that
network. To investigate how dcbo can perform optimal interventions on host
nodes, in order to reduce the cost of intrusions caused by the red agent.
Through this we demonstrate a complete cyber-simulation system, which we use
to generate observational data for dcbo and provide numerical quantitative
results which lay the foundations for future work in this space.

Machine Learning, ICML

##  1 Introduction

In a recent paper Dhir et al. (2021) voice concern regarding cyber criminals
developing new and complex malicious tools which defenders cannot rapidly
adapt to, or rapidly counter with existing methods. They argue that the
sophistication of these new tools warrants the entry of new advanced cyber-
defence techniques to counter these threats. For the purposes of active cyber
defence (acd), they propose the rapid introduction of reinforcement learning
(rl) and causal inference. We focus on the latter in this paper due to the
much larger, existing, body of work that has hitherto focused on rl for cyber
security applications see e.g. (Ridley, 2018) and (Sewak et al., 2021) for a
review.

The actions that a blue agent (ba) takes when acting as a defender of a
network node can be viewed as a solution to an optimal decision making
problem. To have the best chance of survival at each time step, the ba acting
as a defender must make the optimal intervention to protect itself. The ba is
effectively trying to minimise the probability that it will be compromised,
given the current state of the network. This becomes particularly complex in
the computer network setting, as the causal relationships between network
properties and the chance that a node will be compromised vary over time, as a
red agent (ra) takes sequential malicious actions on the network.

To find the optimal interventions which the ba should take to counter the
actions of the ra, we propose the use of dcbo which was introduced to identify
optimal interventions in precisely this kind of setting. Furthermore, it was
shown to converge faster than competing methods in a variety of similar
dynamic scenarios (Aglietti et al., 2021). dcbo is able to provide the optimal
sequence of interventions, accounting for causal temporal dynamics, in a
dynamical system such as that of a network under attack. This optimisation is
performed in relation to a specific cost function, such as minimising network
down-time or a per-node appraisal. In order to investigate interventional
dynamics, we apply dcbo to data simulated by Yawning Titan (yt) – a novel
cyber security simulator developed by the UK Government to test new solutions
to existing problems within the cyber security domain.

For our simulations, the ba has two available actions. They can either (1)
restore a compromised node (thereby removing ra from that node) or (2) isolate
the node (removing the connections from that node to other nodes, thereby
making it impossible for the ra to reach it) for a number of time steps. In
order to avoid trivial solutions in this setting, we impose a cost to both
actions, as well as a cost to nodes in the network being compromised. The
optimal set of interventions will then be those that minimise the total cost
to the agent – keeping the network free from ra infection while minimising the
use of expensive actions.

The paper is organised as follows; in section 2 we review some necessary
background material for the optimisation problem. This leads into related work
on cyber simulators in section 3, allowing us to introduce yt in section 4.
Having gone through yt, we demonstrate how we use it with models for (causal)
decision-making through optimisation, with our methodology in section 5. We
present experiments, results and conclusion in section 6, section 7 and
section 8 respectively.

##  2 Preliminaries and problem setup

#### Notation.

Random variables are denoted by upper-case letters and their values by lower-
case letters. Sets of variables and their values are noted by bold upper-case
and lower-case letters respectively. We make extensive use of the do-calculus,
for details see (Pearl, 2009, §3.4). Samples (observational) drawn from a
system or process unperturbed over time are contained in $\mathcal{D}^{O}$.

#### Structural causal model.

Structural causal models (scms) (Pearl, 2009, ch. 7) are used as the semantics
framework to represent an underlying environment. For the exact definition as
used by Pearl see (Pearl, 2009, def. 7.1.1). An scm is parametrised by the
quadruple
$\left\langle\mathbf{U},\mathbf{V},\mathbf{F},p(\mathbf{U})\right\rangle$.
Here, $\mathbf{U}$ is a set of exogenous variables which follow a joint
distribution $p(\mathbf{U})$ and $\mathbf{V}$ is a set of endogenous
(observed) variables. Within $\mathbf{V}$ we distinguish between three types
of variables: $\mathbf{X}$, manipulative (treatment); $\mathbf{Z}$, non-
manipulative covariates and $\mathbf{Y}$ the target (output) variables. The
sets of parents, children, ancestors, and descendants in a graph $\mathcal{G}$
will be denoted by PA, CH, AN and DE respectively.

Further, endogenous variables are determined by a set of functions
$\mathbf{F}\subset\mathcal{F}$. Let
$\mathbf{F}\triangleq\\{f_{i}\\}_{V_{i}\in\mathbf{V}}$ (Lee & Bareinboim,
2020, §1) s.t. each $f_{i}$ is a mapping from (the respective domains of)
$U_{i}\cup\textrm{PA}_{i}$ to $V_{i}$ – where $U_{i}\subseteq\mathbf{U}$ and
$\textrm{PA}_{i}\subseteq\mathbf{V}\setminus V_{i}$. Graphically, each scm is
associated with a causal diagram (a directed acyclical graph, dag for short)
$\mathcal{G}=\left\langle\mathbf{V},\mathbf{E}\right\rangle$ where the edges
are given by $\mathbf{E}$. Each vertex in the graph corresponds to a variable
and the directed edges point from members of $\textrm{PA}_{i}$ and $U_{i}$
toward $V_{i}$ (Pearl, 2009, ch. 7). A directed edge is s.t. $V_{i}\leftarrow
V_{j}\in\mathbf{E}$ if $V_{i}\in\textrm{PA}_{j}$ (i.e. $V_{j}$ is a child of
$V_{i}$).

###  2.1 Problem statement

Formally we can pose our problem setting as version of the _control problem_
as defined by Pearl & Robins (1995, §2). The setting consists of a dag,
associated vertex set $\mathbf{V}$ partitioned into four disjoint sets
$\mathbf{V}=\\{\mathbf{X},\mathbf{Z},\mathbf{U},Y\\}$ _for each_ time-slice.
There exists a temporal order on the disjoint sets so that
$\mathbf{V}=\bigcup_{t=0}^{T}\mathbf{V}_{t}$ so that each slice is an ancestor
of $\mathds{1}_{t>0}\cdot\mathbf{V}_{t}$ in $\mathcal{G}$. Each slice contains
_one_ outcome variable with index $t$. Each slice is a sub-graph of
$\mathcal{G}$ and each sub-graph is a directed rooted tree where the root node
is the response variable $Y_{t}$. In the offline setting we are privy to
observational samples contained in $\mathcal{D}^{\text{O}}$ and from that
information we seek a _plan_ which is an ordered sequence of interventions
which when implemented, jointly maximise (minimise) the associated sequence of
response variables $\\{Y_{0},Y_{1},\ldots,Y_{T}\\}$.

Using causal inference notation we seek the interventional plan:
$\\{\operatorname{do}\\!\left(\mathbf{X}^{*}_{0}=\mathbf{x}^{*}_{0}\right),\operatorname{do}\\!\left(\mathbf{X}^{*}_{1}=\mathbf{x}^{*}_{1}\right),\ldots,\operatorname{do}\\!\left(\mathbf{X}^{*}_{T}=\mathbf{x}^{*}_{T}\right)\\}$
which is found by solving a _dynamic causal global optimisation_ (Aglietti et
al., 2021) problem of the form:

$\displaystyle\mathbf{X}_{t}^{*},\mathbf{x}_{t}^{*}$
$\displaystyle=\operatorname*{arg\,min}_{\mathbf{X}_{t}\in\mathcal{P}(\mathbf{X}_{t}),\mathbf{x}_{t}\in\text{dom}(\mathbf{X}_{t})}$
$\displaystyle\mathbb{E}{\left[\mathbf{Y}_{t}\mid\operatorname{do}\\!\left(\mathbf{X}_{t}=\mathbf{x}_{t}\right),\mathds{1}_{t>0}\cdot\bm{\mathcal{X}}_{t-1}\right]}$
(1)

where
$\bm{\mathcal{X}}_{t-1}\triangleq\bigcup_{i=0}^{t-1}\operatorname{do}\\!\left(\mathbf{X}_{i}=\mathbf{x}_{i}\right)$
and $\mathcal{P}(\cdot)$ is the powerset over the interventional variables at
time $t$.

This setting makes a number of assumptions which are required to a sequence of
optimal actions within a causal framework (the details of which can be found
in (Aglietti et al., 2021, Assumptions 1)). Notwithstanding, we seek, at every
time step, to construct models for different intervention sets (the
expectation in section 2.1) by integrating various sources of data while
accounting for past interventions. With this construction we are able to query
them to find an optimal intervention at each time-index given the history.

##  3 Related work

As noted in section 1 much of the literature has focused on the use of rl in
cyber security. We will instead consider relevant literature on cyber
simulators (section 3.1) and the few studies which have used concepts from
causal inference for cyber security research (section 3.2).

###  3.1 Cyber simulators

There are many cyber simulators that exist or are currently in development, so
why the need for yt in this work? We consider our contemporaries to motivate
the use of yt.

FARLAND (Molina-Markham et al., 2021) aims to support the development of rl
agents within a hybrid simulation and emulation system. This increased
fidelity provides a means of developing rl cyber defence agents capable of
defending real systems but hinders the research of fundamental aspects such as
algorithm development. yt is more abstract and flexible which enables us to
easily test different and novel approaches without having to significantly re-
engineer the underlying code base. CybORG (Standen et al., 2021),
comparatively, shares many features with yt. Both are simulators and are
controlled by YAML configuration files that define the scenario for a given
environment. The fundamental difference however is the fidelity of node
representation. As CybORG contains both a simulation and emulation component,
the simulation models real features of hosts, such as the operating systems,
hardware architectures, users and passwords. All of this information is
intentionally abstracted away within yt. This level of abstraction could be
viewed negatively but is the very reason new and novel decision making
approaches can be rapidly integrated and experimented with whilst also
providing a cyber relevant environment. Another recent development is
Microsoft’s Cyber Battle Simulator (Team., 2021, cbs). Similar to yt, cbs
abstracts away from many of the details of real life however still
incorporates host information such a specific host vulnerabilities and in-
depth environment setup. When designing yt we wanted to focus on a very simple
setup where the hosts only have a few shared attributes and it is easy to spin
up new network models. This would allow someone to quickly set up an
abstraction of a network topology, without having to worry about host-specific
details, and easily integrate it into their own research.

###  3.2 Causality for cyber security

Causal inference methods have yet to find their place within the cyber
security domain. There have been some previous attempts to apply causal
decision making to historic cyber data by Abel et al. (2020) and Mueller et
al. (2019). However, these attempts use causal models without a temporal
dimension (the main feature of dcbo) and only focus on predicting and alerting
rather than selecting optimal defensive actions. Since we are using a
simulator to collect our data, we can easily test new scenarios and model
performance differences that one would not be able to with a single historic
data set.

In another strand of work, Shi et al. (2017) use transfer entropy to quantify
the causal relationship between any two variables in their cyber-physical
system. In other words they undertake causal-discovery of their complex
network in order to not a priori have to rely on a model of the underlying
dynamical system.

##  4 Yawning titan

We introduce Yawning Titan (yt). yt is an abstract, highly flexible, cyber
security simulator that is capable of simulating a range of cyber security
scenarios111A Python implementation is available:
https://github.com/dstl/YAWNING-TITAN.. We provide a formal definition for
completeness.

######  Definition 1.

Cyber security simulator. Simulation in the cyber context is the process of
modelling a real-world computer-system environment to predict outcomes of
actions from a number of agents where the goals of said agents are to control
either system or data from the system.

There are a number of these simulators (see section 3 for related work), who
broadly work on the same principle as described in the above definition. yt is
built to train cyber defence agents to defend arbitrary network topologies in
highly customisable configurations. It was developed with the following design
principles in mind:

  1. 1.

Simplicity over complexity

  2. 2.

Minimal hardware requirements

  3. 3.

Support for a wide range of algorithms

  4. 4.

Enhanced agent/policy evaluation support

  5. 5.

Flexible environment and game-rule setup

Design principles 1, 2 and 5 make yt an ideal environment to test and evaluate
new and novel approaches to cyber security decision making and provides a
means of transitioning approaches such as dcbo into a cyber defence context.
Principally we will be modelling the actions of one blue agent (ba) and one
red agent (ra).

Let
$\mathcal{G}_{\textrm{net}}=\left\langle\mathbf{V}_{\textrm{net}},\mathbf{E}_{\textrm{net}}\right\rangle$,
with vertex set $\mathbf{V}_{\textrm{net}}$ and edge set
$\mathbf{E}_{\textrm{net}}$, be the undirected graph which simulates a
computer network in yt. Before going further, take care not to confuse the two
types of graphs we employ in this work; $\mathcal{G}$ represents the directed
causal diagram and $\mathcal{G}_{\textrm{net}}$ the undirected associated
computer network – an example instance is shown in fig. 1. Each node
$V_{i}\in\mathbf{V}_{\textrm{net}}$ represent the hosts on the network and arc
$E_{j}\in\mathbf{E}_{\textrm{net}}$ represents possible connectivity between
the two connected hosts (grey edges in fig. 1). This approach allows us to
easily model a large variety of different types of network topologies in their
simplest form without worrying about protocols or different types of hosts.
This basic approach to the network allows us to abide by our first design
principle.

yt allows for considerable customisation through the use of customised
configurations. The configuration summary contains a number of different
settings that the user can control in order to change how the game is played.
From these settings you can change success conditions for the agents and how
each of the different agents can act by toggling actions and action
probabilities.

The machines in the network each have their own attributes that affect how
they behave and how they are affected by the ra and the ba.

Vulnerability score

Affects how easy it is for the ra to compromise the node. It is automatically
generated at the start of each scenario based on settings in the configuration
file and there are blue actions that can modify the vulnerability scores of
nodes.

Isolation Status

Indicates whether a node has been isolated. This means that all of the
incoming and outgoing connections are disabled. This can be modified by blue
actions.

True Compromised Status

Indicates whether a node has been infected by the ra. If the ra has control of
the node then it can use the node as a foothold to spread to other connected
nodes.

Blue seen Compromised Status

Represents if the node is compromised and the ba is aware of the intrusion.
Depending on the scenarios configuration, the ba may only see an obscured view
of the network and see this instead of the true value, effectively simulating
the differences between perfect and imperfect detection.

Figure 1: Example output from yt on a 25 node network
$\mathcal{G}_{\textrm{net}}$. The ba is defending a high value target node
(purple node). The ra controls a large portion of the network, indicated by
yellow nodes with known and unknown compromise (indicated by the edge colour
of the yellow nodes), unbeknownst to the ba. There are three entry nodes into
the system, two of which the ra has made use of (marked by an ‘E’). The ba is
in the process of removing the ra from the node adjacent to the high value
target node. The ra is attacking a node in the bottom left hand corner of the
network. Best viewed in colour.

Using the aforementioned configuration settings, the user is able to modify
the subset of actions that are available to ra and ba. The ba has actions that
allow it to:

  * •

reduce the vulnerability of nodes,

  * •

scan the network for red intrusions,

  * •

remove ra from a node,

  * •

reset a node back to its initial state,

  * •

deploy deceptive nodes,

  * •

isolate a node and

  * •

reconnect a previously isolated node.

The ra has a variety of different attacks that it can use based on the
settings in the configuration file. Unless using a guaranteed attack, eq. 2
will be used to determine if the ra (with a skill level $0<RS\leq 1$) will
compromise node $V_{i}\in\mathbf{V}_{\textrm{net}}$:

$AS=\frac{100\times RS^{2}}{RS+(1-\textrm{vuln}(V_{i}))}.$ (2)

The attack will succeed if:

$AS\geq u$ (3)

where $u\sim\mathcal{U}(0,100)$ is sampled from a uniform distribution, $AS$
is the attack score (a measure of how powerful an attack is) and
$\textrm{vuln}(V_{i})$ is the vulnerability of node $V_{i}$ (a measure of how
exposed a host is to being compromised). For example, a computer without a
firewall or a user that has no security training, could be modelled as having
high vulnerability. Since $0\leq\textrm{vuln}(V_{i})\leq 1$ and $0\leq AS\leq
100$, the use of this formula ensures that the likelihood for the ra to
compromise a node increases proportionately with the skill of the agent and
decreases proportionately with the defence of the node
$(1-\textrm{vuln}(V_{i}))$.

At each time step, each agent is allowed one action, chosen from the subset of
activated actions, to affect the environment. The order of yt execution is as
follows:

  1. 1.

The ra performs their action

  2. 2.

The Environment checks if the ra has won

  3. 3.

The ba performs their action

  4. 4.

The Environment returns the reward of the ba’s action

  5. 5.

The Environment checks if the ba has won.

As yt is built upon the OpenAI Gym framework (Brockman et al., 2016), the code
is simple to understand and highly decoupled. The agents can be any function
that picks an action based on an observation matrix. Consequently, although yt
was built to run with a Stable-Baselines3 (Raffin et al., 2021) it is simple
to model the actions of the ba using a causal sequential decision-making
agent.

##  5 Methodology

In this section we characterise the various building-blocks required for the
optimal intervention-recommendation part of this study. First and foremost, we
consider the different decision-making methods under investigation – for
continuous action spaces. All are based on Bayesian optimisation (Močkus,
1975; Garnett, 2022, bo) which is our weapon of choice to solve section 2.1.

bo is an optimisation method used for global optimisation of black-box
functions (Garnett, 2022, §1) – i.e. those which do not assume a specific
functional form (Mockus, 2012) and are only distinguished by their inputs and
outputs. It finds use in numerous domains (Garnett, 2022), but is principally
used for optimising expensive black-box functions (where the cost of an
evaluation can be e.g. monetary, time-dependent or societal). bo has key
components:

  1. 1.

the objective function is modelled with a Gaussian process (gp) (Williams &
Rasmussen, 2006) – the expectation in section 2.1;

  2. 2.

each new evaluation of the objective function is incorporated via a Bayesian
update procedure and

  3. 3.

an acquisition function is used to determine the next, high utility, point of
evaluation of the objective.

Numerous advancements and improvements to bo have been made over the years,
but here we focus on those that extend the original framework to the
explicitly causal setting. The first causal extension to bo is given by the
causal Bayesian optimisation (cbo) model, introduced by Aglietti et al.
(2020). cbo is used in settings where the response variable $Y$, is part of a
scm in which a sequence of interventions can be performed. cbo is designed for
_static_ environments. It does not account for the temporal evolution of the
system, consequently breaking the temporal dependency structure which exists
among variables – see fig. 2 for the causal diagram used in this study.

In order to handle problem spaces with explicit temporal dynamics, Aglietti et
al. (2021) introduced dynamic causal Bayesian optimisation (dcbo). dcbo is
useful in the kind of scenarios considered in this paper; ones where all
causal effects in the causal diagram are changing over time. At every time
step dcbo identifies a local optimal intervention by integrating both
observational and past interventional data collected from the system.
Precisely, a dcbo agent seeks to minimise the objective function while
accounting for the cost of intervention. It is important to note that
interventions typically have a large financial, societal, ethical or other
cost associated with them. This is true in the cyber setting as well where a
potentially optimal intervention could be e.g. shutting down a node but which
has huge financial cost due to the services running on that node. Hence an
agent needs to optimise the objective function whilst taking _that_
interventional cost into account.

Using observational data from yt and the three methods described above, we
investigate optimal action policies which reduce and prevent intrusions into a
network from a ra, by solving the dcgo problem in section 2.1. We now outline
relevant causal components.

###  5.1 Causal diagram

The causal diagram induced by the scm is shown in fig. 2 and is in accordance
with causal sufficiency (no unobserved confounders) as per the assumptions
made in (Aglietti et al., 2021, assumptions 1). Further, the graph topology is
homogeneous within a time-slice and remains so across time. Simply, the dag
prescribes the causal relationships between endogenous variables within and
across time. Variable descriptions are given in table 1.

$P_{0}$$I_{0}$$S_{0}$$C_{0}$$H_{0}$$A_{0}$$T_{0}$$P_{1}$$I_{1}$$S_{1}$$C_{1}$$H_{1}$$A_{1}$$T_{1}$$P_{2}$$I_{2}$$S_{2}$$C_{2}$$H_{2}$$A_{2}$$T_{2}$ Figure 2: Causal diagram $\mathcal{G}$ shown for the first three time-steps. The target variable is $T$ with manipulative variables given by $\mathbf{X}_{t}=\\{P_{t},I_{t}\\}$. The rest are non-manipulative variables. The within slice topology repeats for 25 time-steps in our experimental setup and the variable connectivity repeats as shown for the same length of time. For variable descriptions see table 1. Table 1: scm variable descriptions. For their causal relationship see fig. 2. Variable |  Description
---|---
$P$ |  Probability that a ba will restore (RES) a node, thereby removing the ra from that node.
$I$ |  Probability that a ba will isolate (ISO) a node, thereby making both inbound and outbound attacks from a ra impossible.
$S$ |  Available attack surface for the ra. Number of nodes that are connected to the network and not yet compromised.
$C$ |  Operating cost of a node being compromised. We assume that all nodes have equal operating cost of compromise.
$H$ |  Likelihood of further compromise. This is equal to the total vulnerability of all nodes that are directly connected to any compromised node.
$A$ |  Operating cost of taking a given action. We define different costs for both RES and ISO.
$T$ |  Total operating cost. Sum of $C$ and $A$.

This dag is chosen as we believe it represents the simplest causal structure
of how compromise spreads through a network, whilst accounting for the
specific operating costs of taking actions to prevent further intrusion. It is
important to factor in these costs to avoid trivial solutions, such as
disconnecting every node from the network (despite that sometimes being the
best option in a hostile cyber environment)! Additionally, in order to ensure
the temporal functional form of the ra is captured in our model, we choose to
explicitly dynamically model $S$ and $H$, the attack surface and likelihood of
additional compromise. This allows the results of actions that the ra takes in
between time steps, as well as the causal relationships, to be explicitly
modelled. Due to our total control of the yt environment, we are able to make
certain assumptions that would not hold in an equivalent real-world setting,
such as; the only ba actions being taken are RES or ISO, and only one at a
time; the ra can only spread through the network from nodes it already
controls, and cannot create additional entry points; and the topology of the
network remains consistent across time, apart from those cases where a node is
disconnected by the ba. These are strong assumptions but critical to ensure
our dag accurately represents the data generating process.

###  5.2 Structural equation model

The dag is chosen to represent the most important properties of the
environment that could also be used in a real world scenario. The nodes
represent the chance that a system can become compromised, how that compromise
spreads, and costs associated with that compromise.

We setup yt as a data-generator in order to collect observational data, used
to fit the causal relationships prescribed by the directed edges in
$\mathcal{G}$ in fig. 2. For this, we initialise a ba with a two-dimensional
action space of $\\{\text{restore }(\textrm{RES}),\text{isolate
}(\textrm{ISO})\\}$ and random action probabilities – full description in
table 1.

The functional relationship between variables, the true sem, are provided in
eq. 4 to eq. 10. Here $K_{t}$ is the subset of all nodes that are compromised
at time $t$ and $\phi_{t}$ the subset of all nodes that are isolated at time
$t$. Where $\phi^{c}$ denotes the complement of subset $\phi$, $N^{+}(V_{i})$
denotes all nodes that can be reached via a single edge from node $V_{i}$ (see
fig. 1 for an example), $\Gamma_{\\{c,\ \textrm{RES},\ \textrm{ISO}\\}}$
represents the cost of compromise, restore and isolate respectively,
$\mathcal{A}_{t}$ represents the action the ba took at time $t$,
$\textrm{vuln}(V_{i})$ represents the vulnerability of node $V_{i}$ and
$[\cdot]$ represents the Iverson bracket. The exponent in eq. 7 is implemented
in order to improve convergence of an rl agent trained on a similar
environment, so we include it here for consistency.

$\displaystyle P_{t}$ $\displaystyle=p_{t}(\mathcal{\textrm{RES}})$ (4)
$\displaystyle I_{t}$ $\displaystyle=p_{t}(\mathcal{\textrm{ISO}})$ (5)
$\displaystyle S_{t}$ $\displaystyle=|K_{t}^{c}\cap\phi_{t}^{c}|$ (6)
$\displaystyle C_{t}$ $\displaystyle=\left(\sum_{n=1}^{n=N}\Gamma_{c}[n\in
K_{t}]\right)^{1.5}$ (7) $\displaystyle H_{t}$ $\displaystyle=\sum_{n\in
K_{t}}\sum_{v\in N^{+}(n)}(\textrm{vuln}(v)[v\notin\phi_{t}])$ (8)
$\displaystyle A_{t}$
$\displaystyle=\begin{cases}\Gamma_{\textrm{RES}}&\mathcal{A}_{t}=\textrm{RES}\\\
\Gamma_{\textrm{ISO}}&\mathcal{A}_{t}=\textrm{ISO}\end{cases}$ (9)
$\displaystyle T_{t}$ $\displaystyle=C_{t}+A_{t}.$ (10)

These structural equation models allow us to transfer observable information
from yt by manipulating properties of the environment so that we receive an
expression for each variable in the dag. The time dependence of the two
observable variables that have transition edges, namely $S_{t}$ and $C_{t}$,
is implicit in the above equations. As $C_{t-1}$ increases (due to an
increased number of compromised nodes), there is an implicit change in $K_{t}$
and therefore in $S_{t}$. Similarly, as $H_{t-1}$ increases due to more
vulnerability in the network, there is an implicit increase in $C_{t}$ as
additional nodes get compromised between times $t-1$ and $t$.

As noted, having access to observational data means we are able to estimate
the sem (since we do not have access to the true sem):

$\displaystyle P_{t}$ $\displaystyle=f_{P}(t)+\epsilon_{P}$ $\displaystyle
I_{t}$ $\displaystyle=f_{I}(t)+\epsilon_{I}$ $\displaystyle S_{t}$
$\displaystyle=f_{S}(C_{t-1},I_{t})+\epsilon_{S}$ $\displaystyle C_{t}$
$\displaystyle=f_{C}(H_{t-1})+\epsilon_{C}$ $\displaystyle H_{t}$
$\displaystyle=f_{H}(P_{t},C_{t})+\epsilon_{H}$ $\displaystyle A_{t}$
$\displaystyle=f_{A}(P_{t},I_{t})+\epsilon_{A}$ $\displaystyle T_{t}$
$\displaystyle=f_{T}(C_{t},A_{t})+\epsilon_{T}$

by placing Gaussian process estimators on all functions
$f_{i}(\cdot),i\in\\{P,I,S,C,H,A,T\\}$. This sem is then used to find the
ground truth optimal intervention for the system at each time step,
$\\{y^{\star}_{t}\mid t=0,\dots,24\\}$, given it has full access to the data
from yt and the causal graph in fig. 2.

Figure 3: Experimental results of different optimisation methods applied to yt
data, showing convergence of dcbo and competing methods (cbo and bo) across
five replicates. The black dotted line shows the optimal response value
$\\{y^{*}_{t}\mid t=22,23,24\\}$. Shaded areas are $\pm$ one standard
deviation. The $x$-axis shows the total cumulative cost of the intervention
set over 50 trials - effectively how many times the optimisation had to probe
the underlying function in order to build up an approximation of the causal
relationship.

##  6 Experiments

To fit our initial sem, we use yt to generate observational data. This
involves setting up a number of yt environments each with a different dummy
agent that has a probabilistic chance to take each action. We then step
through the environments to see how well the chosen probabilities
perform222Python code required to reproduce this experiment:
https://github.com/alan-turing-institute/causal-cyber-defence.

We configure a simple blue agent (ba) that can take two actions, restore (RES)
or isolate (ISO), with probabilities $p(\mathcal{\textrm{RES}})$ and
$p(\mathcal{\textrm{ISO}})$. Additionally, if a node is isolated, it will
automatically be reconnected to the network after five time steps.

We also configure a simple red agent (ra) that can only perform a single
action, a basic attack, that uses eq. 2 and eq. 3 with a skill level of
$25\%$. A $25\%$ skill level is rather low and means that on average the
weakest nodes will take four turns to compromise. The ra chooses its target by
picking a random node $V_{i}$ from the set of all nodes
$V_{i}\in\mathbf{V}_{\textrm{net}}$ such that $V_{i}$ is an entry node or
$V_{i}$ is connected to a compromised node. In fig. 1 the ra can target any
node with a link to an orange node or any node marked with an ‘E’. In fig. 1,
the ra has chosen a node in the bottom left.

We initialise ten identical network environments with identical ra’s. We pick
a simple network topology with ten network nodes. Combined with the basic
action spaces of both the ba and ra, this simple setup allows the relatively
simple dag in fig. 2 to be constructed. Integrating dcbo with more complex
topologies and agents would be easy to do, but would require re-drawing the
dag such that all causal relationships are accounted for. In the setup we
randomly assign a node in the network to be the entry node into the network
for the ra. We then randomly select a single node to be a high value target
(hvt) from the set of nodes that are furthest away from the entry node – the
purple node in fig. 1. The configuration selected for this experiment was
chosen to create a non-trivial environment that may not necessarily reflect
any real world systems but still contains features and challenges that agents
would have to face in these systems. For each environment, we draw a single
random value for $p(\mathcal{\textrm{RES}})$ and $p(\mathcal{\textrm{ISO}})$
from a Gaussian distribution centred around $0.5$, and instantiate a ba using
these values. We then run the simulation for 25 time steps, or until the hvt
is compromised, allowing both ra and ba to sequentially take actions according
to their configurations. At each time step, we calculate the values for the
nodes in fig. 2 according to the sem in eq. 4 to eq. 10.

This provides an array of observational data for each node of size $10\times
25$. As in (Aglietti et al., 2021, 4.2 Real experiments), cbo and dcbo use
this data to fit a non-parametric simulation of the relationships in fig. 2
and to compute a causal prior.

We provide both cbo and dcbo this observational data and run them, along with
bo, for 50 trials each, allowing them to intervene on the manipulative
variables $P$ and $I$ in the domain $[0,1]$, to minimise $T$ at time steps
$t=22,23,24$. The convergence of the models on the optimal outcome value is
shown in fig. 3, against the cost of intervention. Note that cbo and dcbo
converge much more consistently and more efficiently than bo.

##  7 Results and discussion

Figure 3 shows that dcbo and cbo provide a way of efficiently evaluating the
optimal decisions to make in a complex cyber environment, with a time-
dependent causal structure. They are able to take historic data, along with a
causal model of the data generating process, and find an optimal intervention
set, choosing actions with the least associated cost from both the action, and
the effect of the ra.

A large part of both cbo and dcbo’s efficiency, and an additional hurdle to
implementing these causal methods in a real-world cyber setting, is the
creation of the underlying dag on which these methods rely for inference. For
a controlled setting, such as those within simulators such as yt, the creation
of this dag and corresponding observational data is relatively trivial. On the
contrary, in a real-world setting, the dag creation must be knowledge-driven
and can often grow to unmanageable scales when considering all the factors at
play in a potentially hostile cyber environment. However, with sufficient
domain expertise and an understanding of the data-generating process, we have
shown that a causal view of activities within a complex cyber simulator is
possible and indeed more powerful than a naïve data-centric view. We leave it
as an exercise to the reader to think of dag’s for real-world networks that
they are familiar with, as we believe it can shed light on relationships that
might otherwise go un-noticed.

The results show that cbo generally converges to the optimal decision the
quickest with dcbo varying in its relative efficiency. This implies that in
this specific network topology, the temporal relationships do little to aid in
the convergence on the optimal solution. However, the causal relationships
within time slices, exploited by both cbo and dcbo, do significantly improve
the convergence over traditional bo.

One of the main advantages of cbo and dcbo is speed and their handling of
data-sparsity. In addition, provided that the training environment remains
broadly stationary, dcbo and similar approaches are able to be rapidly re-
trained and re-calibrate defensive policies as the threats and risk
environment changes. Compare this to rl where a major disadvantage of those
methods is their sample inefficiency and training time. As a means of
comparison, a Proximal Policy Optimisation (ppo) agent (Schulman et al., 2017)
was trained using the stable-baselines3 (Raffin et al., 2021) rl library
within the same environment used to gather data for our comparison models. It
took significantly longer to converge to its optimal interventions (see fig.
4) and its optimal solution was far from the true optimal. This shows our
possible need for another method to choose optimal decisions and dcbo and cbo
have been shown to quickly hone in on the true solution. These results are not
conclusive. We will need to perform further analysis with rl to fully
understand the pros and cons.

Figure 4: ppo agent trained in the same environment as the other decision
making algorithms. The reward on the horizontal axis is the cumulative reward
over 25 time steps. The reward ($R$) at each timestep is $R=-T$ where $T$ is
the total cost calculated in eq. 10. The ppo manages to reach its optimal
solution within a couple of minutes but it should be noted that we encountered
a considerable spread in performance when testing ppo in this context.

##  8 Conclusion and future work

We have demonstrated the utility of using causal inference in a cyber-
simulation framework where the goal is to provide a blue agent with optimal
action recommendations. In that process we have made a number of operative
assumptions, discussed throughout the paper. Due to the preliminary nature of
the work, there are many avenues going forward.

On the causal inference side, we have assumed _causal sufficiency_ i.e. the
absence of any unmeasured confounders. This is a strong and oftentimes
unrealistic assumption. There is a host of literature which focuses on causal
decision-making in the rl domain, which we aim to bring across to the bo
sphere, such as the work by Lee & Bareinboim (2018). Moreover, it is equally
true that we have only focused on continuous interventions in this paper but
cyber-security data is often discrete, hence future work will incorporate
options for handling discrete data.

On the yt side, we hope to further explore causal decision making algorithms
and their integration into yt, increasing the complexity of the environments
and number of actions available. We also hope to use these models in an on-
line setting, where causal decision making is able to provide real-time
recommendations of interventions, which are then fed directly into yt’s
simulation.

With the on-line setting in place, we then plan to gather more results on how
algorithms such as dcbo fare against traditional reinforcement learning
algorithms, such as ppo, in these different and more complex environments.
What is more, dcbo is _always_ active but this may not be necessary. It would
be better if dcbo activated only _if_ a threat has been detected upon which an
optimal causal decision-making agent is deployed. Early work on one such (non-
causal) approach was recently published by Hammar & Stadler (2022) using
optimal stopping (Bertsekas, 2012). Finally, we hope that a systematic
comparison of agents trained using a variety of different algorithms will show
the value that a causal understanding of the cyber environment can bring to
optimal cyber decision making.

## References

  * Abel et al. (2020) Abel, S., Tang, Y., Singh, J., and Paek, E.  Applications of causal modeling in cybersecurity: An exploratory approach.  _Advances in Science, Technology and Engineering Systems Journal_ , 5(3):380–387, 2020.
  * Aglietti et al. (2020) Aglietti, V., Lu, X., Paleyes, A., and González, J.  Causal Bayesian Optimization.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pp. 3155–3164. PMLR, 26–28 Aug 2020.
  * Aglietti et al. (2021) Aglietti, V., Dhir, N., González, J., and Damoulas, T.  Dynamic Causal Bayesian Optimization.  In _Advances in Neural Information Processing Systems_ , volume 35, 2021.
  * Bertsekas (2012) Bertsekas, D.  _Dynamic programming and optimal control: Volume I_ , volume 1.  Athena Scientific, 2012.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym, 2016.
  * Dhir et al. (2021) Dhir, N., Hoeltgebaum, H., Adams, N., Briers, M., Burke, A., and Jones, P.  Prospective artificial intelligence approaches for active cyber defence.  _arXiv preprint arXiv:2104.09981_ , 2021.
  * Garnett (2022) Garnett, R.  _Bayesian Optimization_.  Cambridge University Press, 2022.  in preparation.
  * Hammar & Stadler (2022) Hammar, K. and Stadler, R.  Intrusion prevention through optimal stopping.  _IEEE Transactions on Network and Service Management_ , pp. 1–1, 2022.  doi: 10.1109/TNSM.2022.3176781.
  * Lee & Bareinboim (2018) Lee, S. and Bareinboim, E.  Structural causal bandits: where to intervene?  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Lee & Bareinboim (2020) Lee, S. and Bareinboim, E.  Characterizing optimal mixed policies: Where to intervene and what to observe.  _Advances in neural information processing systems_ , 33, 2020.
  * Močkus (1975) Močkus, J.  On bayesian methods for seeking the extremum.  In _Optimization techniques IFIP technical conference_ , pp. 400–404. Springer, 1975.
  * Mockus (2012) Mockus, J.  _Bayesian approach to global optimization: theory and applications_ , volume 37.  Springer Science & Business Media, 2012.
  * Molina-Markham et al. (2021) Molina-Markham, A., Miniter, C., Powell, B., and Ridley, A.  Network environment design for autonomous cyberdefense.  _arXiv preprint arXiv:2103.07583_ , 2021.
  * Mueller et al. (2019) Mueller, W. G., Memory, A., and Bartrem, K.  Causal discovery of cyber attack phases.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pp. 1348–1352, 2019.  doi: 10.1109/ICMLA.2019.00219.
  * Pearl (2009) Pearl, J.  _Causality: Models, Reasoning and Inference_.  Cambridge University Press, USA, 2nd edition, 2009.
  * Pearl & Robins (1995) Pearl, J. and Robins, J. M.  Probabilistic evaluation of sequential plans from causal models with hidden variables.  In _UAI_ , volume 95, pp. 444–453. Citeseer, 1995.
  * Raffin et al. (2021) Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 22(268):1–8, 2021.  URL http://jmlr.org/papers/v22/20-1364.html.
  * Ridley (2018) Ridley, A.  Machine learning for autonomous cyber defense.  _The Next Wave_ , 22(1):7–14, 2018.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sewak et al. (2021) Sewak, M., Sahay, S. K., and Rathore, H.  Deep reinforcement learning for cybersecurity threat detection and protection: A review.  In _International Conference On Secure Knowledge Management In Artificial Intelligence Era_ , pp. 51–72. Springer, 2021.
  * Shi et al. (2017) Shi, D., Guo, Z., Johansson, K. H., and Shi, L.  Causality countermeasures for anomaly detection in cyber-physical systems.  _IEEE Transactions on Automatic Control_ , 63(2):386–401, 2017.
  * Standen et al. (2021) Standen, M., Lucas, M., Bowman, D., Richer, T. J., Kim, J., and Marriott, D.  Cyborg: A gym for the development of autonomous cyber agents.  _arXiv preprint arXiv:2108.09118_ , 2021.
  * Team. (2021) Team., M. D. R.  Cyberbattlesim.  https://github.com/microsoft/cyberbattlesim, 2021.  Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei.
  * Williams & Rasmussen (2006) Williams, C. K. and Rasmussen, C. E.  _Gaussian processes for machine learning_ , volume 2.  MIT press Cambridge, MA, 2006.