# Reinforcement Learning for Industrial Control Network Cyber Security
Orchestration

John Mern
Department of Aeronautics and Astronautics
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>
&Kyle Hatch
Department of Computer Science
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>
&Ryan Silva
Applied Physics Laboratory
Johns Hopkins University
Laurel, MD 20723
&Jeff Brush
Applied Physics Laboratory
Johns Hopkins University
Laurel, MD 20723
&Mykel J. Kochenderfer
Department of Aeronautics and Astronautics
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>
PhD Candidate, Stanford Intelligent Systems LaboratoryUndergraduate Student,
Stanford UniversityAssociate Professor, Stanford Intelligent Systems
Laboratory

###### Abstract

Defending computer networks from cyber attack requires coordinating actions
across multiple nodes based on imperfect indicators of compromise while
minimizing disruptions to network operations. Advanced attacks can progress
with few observable signals over several months before execution. The
resulting sequential decision problem has large observation and action spaces
and a long time-horizon, making it difficult to solve with existing methods.
In this work, we present techniques to scale deep reinforcement learning to
solve the cyber security orchestration problem for large industrial control
networks. We propose a novel attention-based neural architecture with size
complexity that is invariant to the size of the network under protection. A
pre-training curriculum is presented to overcome early exploration difficulty.
Experiments show in that the proposed approaches greatly improve both the
learning sample complexity and converged policy performance over baseline
methods in simulation.

##  1 Introduction

Cyber attacks have been increasingly targeting computer networks that are
integrated with industrial control systems (ICS) [1]. Attack techniques
focusing on stealth and redundant access make them difficult to detect and
stop [2]. Intrusion detection systems monitor networks and alert security
orchestrators to potential intrusions, though high false-alarm rates cause
many alerts to receive no security response. Advanced persistent threat (APT)
attackers take advantage of this by spreading across target networks
undetected over long periods to prepare an eventual attack [3]. In addition to
the theft of sensitive data common in APT attacks on standard networks,
attacks on ICS can additionally result in disruption of the controlled system,
physical destruction of equipment, and even loss of life [4, 5, 6].

It is not feasible for human security teams to respond to every potential
intrusion alert and to reliably mitigate all threats. This work seeks to
demonstrate the feasibility of developing an automated cyber security
orchestrator (ACSO) to assist human analysts by automatically investigating
and mitigating potential attacks on ICS networks. We first present a model of
this task as a discrete-time sequential decision making problem. The proposed
model captures many of the challenges of the underlying decision problem while
remaining agnostic to specific network configuration and communication
dynamics. To support this, we implemented a cyber attack simulator that can
efficiently generate large quantities of trial data.

Many sequential decision methods cannot solve this problem because explicit
models of APT attack dynamics and alert behaviors are not generally known.
Reinforcement learning (RL) may be used to train policies for complex tasks
without explicit models [7, 8], however the ACSO problem poses several
challenges to existing reinforcement learning (RL) solvers. The observation
and action spaces grow with the number of nodes on the protected network,
leading to very high-dimensional, discrete observation and action spaces. Deep
reinforcement learning is known to struggle to learn over large, discrete
action spaces due to the difficulty of generalizing over actions [9].
Similarly, large input spaces have been found to decrease the sample
efficiency of stochastic learning [10].

APT attacks are designed to be difficult to detect, causing limited
observability of the true compromise state of nodes on the network. Partially
observable problems are known to be significantly more difficult to solve than
their fully observable counterparts, requiring learning over sequences of
observations to infer unseen states [11]. Effective defense against APTs
requires timely response to compromises, though intrusion campaigns can last
several months. Modeling the decision process requires fine resolution time-
steps, leading to very long time horizons. Long time-horizons and sparse
rewards can drastically increase the sample complexity of learning, both
through exploration difficulty and temporally delayed credit assignment [12,
13].

This paper proposes a neural network architecture and training curriculum that
scales to the large problem space. We present an attention-based architecture
that effectively learns over many cyber network nodes without growth in the
number of required parameters. The lower parameter complexity network is shown
to significantly accelerate training while converging to a higher performing
solution than a larger baseline architecture. To overcome exploration
difficulty, we introduce a supervised pretraining step that minimizes a
composite large-margin classification and temporal-difference loss over a set
of expert trajectories.

We tested the proposed solution methods against baseline neural network
architectures and training approaches, as well as against expert-designed
policies. These experiments demonstrate how each component contributes to the
proposed method’s scalability and suggest that deep RL is a viable method to
implement an autonomous network security agent. The methods presented may be
generalized to scale RL to other problems with large input spaces, output
spaces, or long time horizons.

##  2 Background

###  2.1 Reinforcement Learning

In sequential decision problems, the environment is modeled by states $s$ that
evolve according to potentially stochastic dynamics. An agent takes actions
$a$ that condition state transition distributions $T(s^{\prime}\mid s,a)$ and
generate rewards $r(s,a,s^{\prime})$. In many problems, the state of the world
is not known. In these partially observable domains, agents receive noisy
observations according to $o\sim Z(o\mid s,a)$.

A sequential decision problem is solved by an action sequence that maximizes
the expected value
$V(s)=\mathbb{E}\big{[}\sum_{t}\gamma^{t}r(s_{t},a_{t})\big{]}$ for all states
in the trajectory. Reinforcement learning methods learn a policy
$\pi:o_{t-\tau:t}\mapsto a_{t}$ that maps a history of observations to actions
through repeated trial-and-error with the environment. In each trial, actions
are taken according to the current policy until a terminal condition is
reached. At the end of the trial, the policy is updated to improve the
expected performance.

Reinforcement learning methods that use neural networks to represent the
learned policy are known as _deep_ reinforcement learning. Deep Q-Network
(DQN) learning is a popular method in which the learned neural network
predicts the expected value
$Q(o_{t-\tau:t},a)=\mathbb{E}\big{[}r(s_{t},a)+\gamma V(s_{t+1})\big{]}$ of
taking each action in the action space for a given input history [14]. The
Q-network is used as a policy by taking the action with the highest predicted
value $a^{*}=\mathrm{argmax}_{a}Q(o_{t-\tau:t},a)$.

Deep RL training typically requires large amounts of data [15]. The amount of
trials required to solve a task tends to grow with the task complexity and
neural network size, with more complex tasks often requiring larger neural
networks. Reinforcement learning agents improve by stochastically exploring
new trajectories with each trial. Tasks with very large input spaces, output
spaces, or very long time horizons tend to require much more exploration
because the odds of finding a successful trajectory through random exploration
are low [16].

###  2.2 Networked Industrial Control Systems

Industrial control systems are networks of devices used to monitor and control
a coordinated physical process, such as an assembly line or power plant.
Simple operational technology computing devices such as programmable logic
controllers (PLCs) are installed on the industrial equipment to receive sensor
readings or to issue control commands. These PLCs are networked with
information technology networks to enable remote access and control [17].

Integration of ICS networks with internet-connected networks has made them
vulnerable to cyber attack [18]. Despite this increased risk, networking is
often an operational requirement for processes where system components are
difficult to access or geographically distant. For example, electrical power
grid management requires coordination of substations that may be distributed
over very large areas. To attempt to mitigate this vulnerability, many
networks are organized into firewall-separated levels, with low-privilege
nodes on less restricted and internet-accessible levels, and process-critical
nodes on more isolated subnets [17].

Many recent, high-profile infrastructure attacks were executed by advanced
persistent threats (APTs). Advanced persistent threats comprise the top two
tiers of the Defense Science Board threat taxonomy and are typically well-
funded and willing to spend significant time to achieve their goal [19]. APTs
overcome the layered ICS network structure by first compromising a low-
privilege node in the target network [20]. Using this node, the APT will then
conduct internal network reconnaissance and take control over more privileged
nodes until it gains the authority needed to achieve its primary objective,
for example to destroy equipment. A typical APT attack process, based on the
tactics of the MITRE ATT&CK framework [21], is shown in fig. 1.

Figure 1: APT Attack Progression: This shows the typical progression of an APT
attack at the level of tactics in the MITRE ATT&CK framework. The process
starts on the left with initial intrusion, proceeding over several months to
eventual execution.

During reconnaissance and lateral movement, APT activity tends to be very
difficult to detect with automated intrusion detection systems [3]. The
lateral movement phase also comprises the majority of the time of the attack
campaign, often taking several months to complete. Once an attack is staged,
however, it can usually be executed in a matter of hours. Because of this,
successfully defending against an APT attack requires detecting and securing
the compromised nodes in the staging phases.

Machine learning methods have been applied to many aspects of cyber security,
most extensively to perception tasks such as malware recognition and anomaly
detection [22, 20, 23, 24]. Reinforcement learning has been applied to cyber
security in various capacities. [25] provide a survey of recent work in this
area. The scope of each work referenced in this survey is limited in the type
of network under protection, the actions available to the defender agent,
and/or to the type of attack against which it is defending. For example, [26]
propose a system restricted to filtering signals spoofing networked sensor
measurements.

##  3 Problem Formulation

The objective is to prevent an APT from disrupting a set of PLCs while
minimizing how much the defensive actions interfere with network operations.
This work is restricted to defending the engineering level (2) and plant level
(1) of the Purdue enterprise reference architecture (PERA) [27] as shown in
fig. 2. Level 2 contains exclusively workstations and servers, and level 1
contains workstations and all PLCs. The network studied in this work has
twenty-five workstations in level 2, five workstations in level 1, and fifty
PLCs for a total of eighty-three nodes.

Figure 2: Simulated Network Architecture: The problem simulates level 2 and
level 1 of the PERA model. Each level contains an operations VLAN and a
nominally empty quarantine VLAN. Level 2 contains twenty-five workstation
nodes and three servers. Level 1 has five local human-machine interface nodes
and fifty networked PLCs. All network message traffic is simulated through
virtual switches, routers, and firewalls. The computing node for the ACSO is
not explicitly modeled.

The state space for the problem is the joint space of the ICS network states
and the APT attacker states. The ICS network state is the joint state of all
nodes on the network. Server and workstation node states define how much
access and control the APT has and where on the network the node is located. A
PLC state defines whether or not it has been disrupted by an attack. The APT
state defines the current APT sub-goal, knowledge, and available resources.

The ACSO agent has full knowledge of the network configuration and PLC states,
but observes node compromise state through alerts. Alerts are actively
generated by investigative actions taken by the ACSO, which may generate false
negatives, and passively by an intrusion detection system triggered by APT
actions on compromised nodes with many false alarms. Each episode begins with
a single level 2 workstation compromised by an APT. The episode terminates
when 25 PLCs have been disrupted by APT attacks or after 5,000 hours have been
simulated. Each decision step is one hour of simulation world time and the
discount factor is 0.999.

Each step, the ACSO can choose to take an action on a single node or do
nothing. There are seven valid action types for workstation nodes, six action
types for server nodes, and two action types for PLCs. A total of 329 unique
actions may be taken across the nodes. With every step, the APT also takes
actions based on its state and a stochastic attacker model. Actions are not
necessarily instantaneous and some actions will complete over several time-
steps. During this time, agents can initiate additional actions.

Reward each step is based on the fraction of PLCs that are operating nominally
and the inconvenience caused by the ACSO actions taken. The reward function is
defined as

$\displaystyle\leavevmode\nobreak\ r(s,a)$
$\displaystyle=r_{\text{PLC}}(s,a)+r_{\text{IT}}(s,a)+r_{\text{Term}}(s,a)$
(1) $\displaystyle=\Big{(}1-0.04\sum_{p\in\text{PLCs}}1\\{p\mathrm{\
disrupted}\\}\Big{)}\Big{(}1.0-\sum_{a\in
A_{t}}\mathrm{cost}(a)\Big{)}+\frac{1}{\gamma}1\\{s_{\text{time}}\geq
t_{\text{max}}\\}$ (2)

where $A_{t}$ is the set of all actions completing at time step $t$.

The first term scales reward by the number disrupted PLCs, where disruption of
twenty five PLCs results in process shutdown. The second term imposes a
penalty for ACSO actions, where each action is assigned a cost based on its
perceived burden to network operations. Action costs were elicited from
network and security experts. For example the low-disruption action of
rebooting a workstation has a cost of 0.01, while the more disruptive action
of re-imaging a server has a cost of 0.05. The final term rewards the agent
for reaching the episode time limit $t_{\max}$. The $1/\gamma$ magnitude of
the terminal reward ensures the optimal state value does not drift with
episode time.

To facilitate this and future research, we developed an ICS network attack
simulation (INASIM) in the Julia language. To the authors knowledge, this is
the first open-source APT attack simulation capable of generating data in
quantities required for deep reinforcement learning on a single workstation.
INASIM implements a configurable network simulation and a baseline APT
attacker model, as well as an API for custom attacker agents. The simulation
is compatible with the POMDPs.jl [28] framework as well as with the OpenAI Gym
[29] framework through supplied Python bindings. The simulation and baseline
attacker model are described briefly here. A description of states, actions,
and environment dynamics is provided in the appendix along with numerical
settings used for the studies in this work.

The network simulation defines the structure of the network, the state of the
nodes, and the actions available to the APT and ACSO agents. The ACSO can take
actions to investigate node states or to secure compromised nodes. The APT can
take actions to gain further control over a node it has already compromised,
or to use a compromised node to find and infect additional nodes. The APT can
only act from nodes it has already compromised, though the ACSO can act on any
node in the network. Each node’s state defines how much it has been
compromised by the APT. Different types of compromise enable the APT to take
different actions from or on that node. Compromise levels also change the way
the defensive ACSO actions affect the node. For example, an APT can take
actions on a node in the “escalated privileges” state to resist ACSO actions
to change the password.

The baseline APT agent is modeled as a stochastic finite state machine. The
machine states correspond to the attack-phase tactics shown in fig. 1. Every
time-step, the APT first updates its machine state based on its belief over
the network. Each machine-state defines a set of exit-criteria and a
stochastic rules-based sub-policy. The APT may have one of two goals and may
use one of two attack vectors for each episode. The APT goal may be to disrupt
the PLC process under control or to destroy the PLC-controlled equipment.

##  4 Solution Methods

The proposed solution uses an augmented DQN algorithm to learn a policy
defined by our attention-based neural network. We implemented several
extensions to the baseline algorithm, based on studies of Rainbow DQN [30].
The included extensions are double-DQN [31], prioritized experience replay
[32], and $n$-step TD loss [33]. The training loss for a given step is

$\leavevmode\nobreak\
\mathcal{L}_{TD}\big{(}Q_{\pi}(h_{t},a)\big{)}=\Big{\|}\Big{(}\sum_{\tau=t+1}^{t+n}\gamma^{\tau-t}r_{\tau}+\gamma^{n}Q_{\phi}\big{(}h_{t+n},\mathrm{argmax}_{a^{\prime}}Q_{\pi}(h_{t+n},a^{\prime})\big{)}\Big{)}-Q_{\pi}(h_{t},a)\Big{\|}$
(3)

where $h_{t}$ is the history of observations at time $t$, $Q_{\pi}$ is the
action value estimate of the policy network and $Q_{\phi}$ is the action value
estimate of the target network. The $\|\cdot\|$ represents the Huber-loss
norm. This loss is calculated over batches of importance-weighted samples from
an experience replay buffer and used to estimate the gradient for each network
update.

In addition to the task reward presented in eq. 1, a shaping reward was
defined based on the formulation of [34]. This reward was designed to
incentivize the agent to secure compromised nodes. The shaping function was
defined as

$\leavevmode\nobreak\
r_{shape}(s,a,s^{\prime})=\gamma(A\delta_{W}+B\delta_{S})$ (4)

where $\delta_{W}$ and $\delta_{S}$ are changes in the number of workstations
and servers compromised by the APT from state $s$ to $s^{\prime}$,
respectively, and $A$ and $B$ are weight factors. The weighted sum of eq. 1
and eq. 4 were used for training. Only eq. 1 was used for evaluation.

Training hyper-parameters were tuned by a grid search training on a smaller
ICS network with ten level 2 workstation nodes, three level 1 workstations
nodes, and thirty PLCs. We searched over the shaping reward weight, the
observation-history interval, the target network update frequency, and the
$\epsilon$-greedy exploration decay schedule. The parameter set leading to the
highest average performing policy over several seeds after 500 episodes was
selected. We used the PyTorch framework for neural network implementation and
training [35]. Numerical values for the training parameters as well as
additional practical details on the training process can be found in the
appendix.

###  4.1 Neural Network

Figure 3: Attention Network: Inputs for each network node are passed into
attention sub-graphs based on node type. These sub-graphs embed each node’s
history to a latent vector representation. These vectors are stacked and
passed through a self attention sub-graph, to provide global context to each
latent vector. These vectors are then passed through fully connected sub-
graphs to output action value estimates for each node.

The ACSO problem has input and output spaces with dimensions that scale with
the number of nodes in the ICS network. Node observations are represented as
vectors $o^{(i)}_{t}$, where each element of the vector indicates whether the
corresponding alert or action was observed on that node $i$ during that step
$t$. Workstation node observations have 16 elements, server node observations
have 14 elements, and PLC observations have 7 elements. The complete
observation for a given time-step contains the observation vector for each
node in the ICS network, with 792 total elements.

Because the problem is partially observable, a history of prior observations
was used as the network input each step. At time $t$, a history of the
previous $\tau$ steps was input to the network, such that the complete input
was $H_{t}=O_{t-\tau:t}$. In this work, 256 prior steps were used, for a total
input size of 202,752. The output space consists of all valid actions at each
time-step, for a total of 329 elements.

Specialized neural architectures can take advantage of structure inherent in
input data to reduce complexity. For example, convolutional nets leverage
spatial invariance in images and recurrent networks encode correlation in
sequences [36, 37]. We designed the neural network shown in fig. 3 using
attention mechanisms [38] to accommodate the large input space without
significant degradation of explanatory capacity or intractable growth in size.
Attention mechanisms have been shown to improve learning efficiency on tasks
with exchangeable inputs [39, 40].

Each node history is input to its own sub-graph which embeds its history array
to a single latent state vector. All sub-graphs of a given node type share the
same parameter set, so a growth in the number of nodes does not cause a growth
in the total number of network parameters. The latent states of each
individual node are then stacked and input to a global attention sub-graph.
This sub-graph allows the network to learn which features of neighboring nodes
are relevant to the value function of actions on a given node. These
contextualized node vectors are then passed to feed-forward output sub-graphs.
Like the input sub-graphs, network parameters are shared between nodes of the
same type.

The temporal attention and global attention sub-graphs are shown in fig. 4.
The temporal attention sub-graphs alternate 1D convolution in the time
dimension with multi-headed, dot-product attention layers so that the network
may learn to attend to elements of the input sequence that are important to
solving the problem. The global self-attention sub-graph alternates multi-
headed dot-product attention mechanisms with affine layers. This structure
allows the output representation corresponding to a node $i$ to include
relevant information from any other node in the network. Including this
learned global context in the latent representation of each node allows the
output space to be factored to individual sub-graphs for each node.

(a) Temporal Attention

(b) Global Attention

Figure 4: Attention Sub-Graphs: Figure (a) shows the temporal attention graph,
which alternates layers of multi-headed attention with 1D temporal
convolution. Figure (b) shows the global attention sub-graph, which alternates
fully connected layers with multi-headed attention. A skip connection is
included to facilitate gradient back-propagation.

The proposed neural network architecture had a total of 683,108 parameters.
This size does not necessarily vary with the number of ICS nodes considered. A
convolutional neural network was implemented as a baseline for comparison. The
baseline network had four 1D temporal convolutional layers, followed by a
fully connected layer. The number of parameters of the convolutional network
grows with the with the number of network nodes nodes, with 1,240,329 total
parameters for the tested network. We designed the baseline to be as close in
size to the baseline network as possible without shrinking the latent
dimensions too quickly. Details of the convolutional network architecture are
given in the appendix.

###  4.2 Pre-Training Method

The problem was difficult to effectively explore using $\epsilon$-greedy
random exploration. To accelerate the learning process, we implemented a
supervised pre-training step similar to the AlphaStar expert training [7]. We
used a stochastic rules based expert policy to generate a batch of example
trajectories. The agent was then trained to minimize a composite loss term
over this data.

The pre-training loss term was based on a composite loss for distributional
DQN [41]. The loss is defined as

$\displaystyle\mathcal{L}_{P}(Q)=\mathcal{L}_{TD}(Q)+\lambda\mathcal{L}_{LM}(Q)$
(5)

where $\mathcal{L}_{TD}$ is the temporal difference loss defined in eq. 3,
$\mathcal{L}_{LM}$ is a large-margin classification loss, and $\lambda$ is a
weighting hyper-parameter. The margin loss term is defined as

$\displaystyle\leavevmode\nobreak\ \mathcal{L}_{LM}(Q)$
$\displaystyle=\max_{a\in\mathcal{A}}\big{[}Q(h,a)+l(a_{E},a)\big{]}-Q(h,a_{E})$
(6) $\displaystyle l(a_{E},a)$
$\displaystyle=\begin{cases}Q(h,a)-Q(h,a_{E})+\delta,&\text{if }a_{E}\neq a\\\
0,&\text{otherwise}\end{cases}$ (7)

where $a_{E}$ is the action selected by the expert policy and $\delta$ is the
desired margin. This term encourages the values of actions taken by the expert
policy to have a value higher than all others by at least $\delta$. Including
the TD-loss term in the pre-training loss ensures that the value estimates are
approximately Bellman-consistent, reducing forgetting early in RL learning.

##  5 Experiments

We trained our neural network to solve the ACSO task using the proposed pre-
training method and DQN algorithm. Trained policy performance was evaluated
over a batch of 100 episodes, with APT goal and attack vector fixed across
episodes. We ran experiments to evaluate the contribution of each novel
element by comparing it to baseline policies and training methods. These
experiments measured the performance of the converged policies and the sample
efficiency of the learning.

Both network architectures were pre-trained using the proposed composite loss.
These pre-trained networks were then tuned with RL for 500 episodes each. RL
training was conducted using the hyper-parameters found during grid-search.
Each architecture was also trained using DQN without pre-training, using 5
randomly initialized networks to test the impact of pre-training. Training
details are in the appendix.

(a) TD-Loss

(b) Discounted Return

Figure 5: Training Results: The left figure shows the average TD-loss per-
episode for each architecture and pre-training combination. The right figure
shows the time discounted sum of rewards for each episode. Both graphs show
curves of the original data with an exponentially smoothed curve in bold.

Figure 5 shows the learning performance of each experiment configuration. For
the randomly initialized networks, the best-performing seed is shown. The
episode average TD-loss of eq. 3 and discounted task return for 1.25 million
episode steps are shown. As can be seen, our neural network TD loss decays
much faster than the convolutional baseline both with and without pre-
training, though pre-training tended to slow loss decay. The action-value
margin of eq. 6 causes the agent to more often select actions similar to the
pre-trained policy. This reduces how much the policy encounters other
observation-action pairs, slowing learning of their values. For the
convolutional network, the TD-decay was initially accelerated with pre-
training, however, the network quickly diverged after approximately 200,000
steps. This may be due to the the convolutional neural network being more
sensitive to inconsistent action-value estimates induced by the pre-training.

The learning performance of the discounted return shows similar trends to the
TD-loss, with our architecture generally reaching convergence faster the
convolutional network. Additionally, our network’s converged training
performance was generally higher than the performance of the convolutional
network. Unlike the TD-loss trends, pre-training accelerated the improvement
in discounted return, allowing our architecture to approach converged
performance much more quickly. The pre-training negatively impacted the
performance of the convolutional architecture.

The evaluation performance of the best trained policy from each configuration
is shown in table 1. The mean performance over $100$ episodes along with one
standard error bounds are given for four performance metrics. Task return is
the time discounted sum of rewards defined by eq. 1. The action cost is
$10\times$ the average cost per-step of defensive actions taken by the ACSO.
PLC downtime is the total number of hours that PLCs were off-nominal over each
5,000 hour episode. Compromise time is the average number of IT nodes with an
off-nominal state per simulation hour. The performance of the expert policy
used to generate the pre-training data is also given.

Network | Pre-Train | Return | Action Cost | PLC Downtime | Compromise Time
---|---|---|---|---|---
Ours | Yes | $940.4\pm 0.0$ | $0.59\pm 0.01$ | $\mathbf{0.0\pm 0.0}$ | $\mathbf{0.2\pm 0.0}$
No | $\mathbf{941.0\pm 0.4}$ | $0.59\pm 0.01$ | $\mathbf{0.0\pm 0.0}$ | $1.0\pm 0.0$
Conv | Yes | $791.5\pm 2.7$ | $\mathbf{0.26\pm 0.01}$ | $6.2\pm 0.2$ | $3.9\pm 0.0$
No | $738.6\pm 5.0$ | $0.57\pm 0.01$ | $5.0\pm 0.3$ | $3.5\pm 0.1$
Expert | – | $906.9\pm 4.5$ | $0.72\pm 0.01$ | $0.2\pm 0.0$ | $1.3\pm 0.2$
Table 1: Evaluation Performance: This table shows the results of evaluating
each trained policy over 100 trials. Evaluations of the expert policy used for
pre-training trajectories is also given. The best performance for each metric
is shown in bold.

As can be seen, our neural network greatly outperformed the convolutional
network and the expert policy, both with and without pre-training. Our pre-
trained network discounted return and action cost approximately equal those of
the randomly initialized network. Interestingly, the pre-trained policy had a
significantly lower average number of compromised nodes per hour than any
other policy, suggesting that it was able to benefit from the increased
learning rate to find a more effective policy. Both convolutional networks
failed to outperform the expert policy, despite the expert policy trajectories
having been used to pre-train the convolutional network. The novel networks
outperformed the convolutional on task return despite having a higher action
cost due to the fact that it successfully prevented more PLC downtime. This
suggests that the actions it learned to take were both more aggressive and
more effective at impeding APT progress. Conversely, the expert policy
incurred a higher average action cost than both novel networks. This suggests
that though the expert policy was successful in preventing PLCs from going
down, it was inefficient compared to the learned strategies.

##  6 Conclusions

This work presented a POMDP formulation of the cyber network security
orchestration problem for ICS networks. To enable this work and future
research in this domain, we implemented a low computational cost ICS network
attack simulation that is efficient enough for deep RL methods. A novel DQN
architecture was shown to enable deep reinforcement learning to accommodate
the large observation and action spaces of the problem. The proposed pre-
training improved the learning performance of our architecture.

We ran several experiments to measure the effect each contribution had on
learning efficiency and converged policy performance. The experiments show
that both our proposed network architecture and the pre-training improve
learning rate and converged performance. The results suggest that deep
reinforcement learning may be a viable method to implement automated security
agents to mitigate the threat of APT attacks. The proposed contributions may
also be applied to solve other large-scale, real world problems with discrete
spaces. Results suggest that a neural architecture encoding the correct
implicit bias can not only improve training efficiency, but may be required to
learn an acceptable policy.

To build on this work, a simulation with increased fidelity to real-world
network dynamics is needed. Some future directions this may take are to tune
parametric network models to real-world network data or to implement
simulators with virtual machine kernels and real, known exploits. The former
is likely the more promising path, as the latter poses significant computation
and security challenges. Including a model of an actual ICS process and
linking the RL reward to process efficiency is also another path to
investigate. More robust attacker models with a wider set of attack vectors
such as IDS signal spoofing or compromise of the ACSO compute assets should be
investigated.

More solution methods will also be explored with the proposed network
architecture. Recurrent neural networks and variational networks are
alternatives to sliding window histories that may provide better beliefs over
the network state. Integrating attention mechanisms with graph neural networks
[42] may enable application of learned policies to ICS networks with
dissimilar architectures. Policy gradient methods, such as phasic policy
gradient [43], will also be explored as an alternative to DQN.

The current work only considers a fixed, stochastic attacker model. To improve
the robustness of learned policies, adversarial learning will be explored to
generate more challenging attacker strategies. Similarly, the alert and
transition dynamics were modeled as stationary, though these may evolve in
real-world network operations. Stress testing of these dynamics will also be
considered. Assurance and verification of ACSO actions, especially on safety-
critical ICS will be investigated.

## Acknowledgments and Disclosure of Funding

This material is based upon work supported by the Johns Hopkins University
Applied Physics Laboratory. The work was supported by the Stanford University
Institute for Human-Centered AI (HAI) and Google Cloud.

## References

  * [1] Resul Das and Muhammet Zekeriya Gündüz  “Analysis of cyber-attacks in IoT-based critical infrastructures”  In _International Journal of Information Security Science_ 8.4, 2020, pp. 122–133
  * [2] Tejasvi Alladi, Vinay Chamola and Sherali Zeadally  “Industrial Control Systems: Cyberattack trends and countermeasures”  In _Computer Communications_ 155, 2020, pp. 1–8  DOI: 10.1016/j.comcom.2020.03.007
  * [3] Meicong Li et al.  “The study of APT attack stage model”  In _2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS)_ , 2016, pp. 1–5  DOI: 10.1109/ICIS.2016.7550947
  * [4] Ralph Langner  “Stuxnet: Dissecting a Cyberwarfare Weapon”  In _IEEE Security and Privacy_ 9.3, 2011, pp. 49–51  DOI: 10.1109/MSP.2011.67
  * [5] Gaoqi Liang et al.  “The 2015 Ukraine blackout: Implications for false data injection attacks”  In _IEEE Transactions on Power Systems_ 32.4, 2016, pp. 3317–3318
  * [6] Alessandro Di Pinto, Younes Dragoni and Andrea Carcano  “TRITON: The first ICS cyber attack on safety instrument systems”  In _Black Hat USA_ , 2018, pp. 1–26
  * [7] Oriol Vinyals et al.  “Grandmaster level in StarCraft II using multi-agent reinforcement learning”  In _Nature_ 575.7782, 2019, pp. 350–354  DOI: 10.1038/s41586-019-1724-z
  * [8] Carl-Johan Hoel et al.  “Combining planning and deep reinforcement learning in tactical decision making for autonomous driving”  In _IEEE Transactions on Intelligent Vehicles_ 5.2, 2019, pp. 294–305
  * [9] Gabriel Dulac-Arnold, Richard Evans, Peter Sunehag and Ben Coppin  “Reinforcement Learning in Large Discrete Action Spaces”  In _Computing Research Repository_ , 2015  arXiv:1512.07679
  * [10] Thanh Thi Nguyen, Ngoc Duy Nguyen and Saeid Nahavandi  “Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications”  In _IEEE Transactions on Cybernetics_ 50.9, 2020, pp. 3826–3839  DOI: 10.1109/TCYB.2020.2977374
  * [11] Matthew J. Hausknecht and Peter Stone  “Deep Recurrent Q-Learning for Partially Observable MDPs”  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2015, pp. 29–37
  * [12] Marcin Andrychowicz et al.  “Hindsight Experience Replay”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5048–5058
  * [13] Jose A. Arjona-Medina et al.  “RUDDER: Return Decomposition for Delayed Rewards”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 13544–13555
  * [14] Volodymyr Mnih et al.  “Playing Atari with Deep Reinforcement Learning”  In _Computing Research Repository_ , 2013  arXiv:1312.5602
  * [15] Colin Wei and Tengyu Ma  “Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 9722–9733
  * [16] Adrien Ecoffet et al.  “First return, then explore”  In _Nature_ 590.7847, 2021, pp. 580–586
  * [17] Keith Stouffer, Joe Falco and Karen Scarfone  “Guide to industrial control systems (ICS) security”  In _NIST special publication_ 800.82, 2011, pp. 16–16
  * [18] Colleen Glenn, Dane Sterbentz and Aaron Wright  “Cyber threat and vulnerability analysis of the US electric sector”  In _Idaho National Laboratory Technical Report_ , 2016, pp. 6–16
  * [19] James R Gosler and Lewis Von Thaer  “Resilient Military Systems and the Advanced Cyber Threat”  In _Defense Science Board Technical Report_ Office of the Undersecretary of Defense for Acquisition, Technology,Logistics, 2013, pp. 30–31
  * [20] Steven Meckl et al.  “Collaborative Cognitive Assistants for Advanced Persistent Threat Detection”  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2017, pp. 171–178
  * [21] Blake E Strom et al.  “MITRE ATT&CK®: Design and Philosophy”  In _Mitre Technical Report_ , 2016
  * [22] Giovanni Apruzzese et al.  “On the effectiveness of machine and deep learning for cyber security”  In _IEEE Conference on Cyber Conflict (CyCon)_ , 2018, pp. 371–390
  * [23] Sadegh Momeni Milajerdi et al.  “HOLMES: Real-Time APT Detection through Correlation of Suspicious Information Flows”  In _IEEE Symposium on Security and Privacy_ , 2019, pp. 1137–1152  DOI: 10.1109/SP.2019.00026
  * [24] Sheraz Naseer et al.  “Enhanced network anomaly detection based on deep neural networks”  In _IEEE Access_ 6, 2018, pp. 48231–48246
  * [25] Thanh Thi Nguyen and Vijay Janapa Reddi  “Deep Reinforcement Learning for Cyber Security”  In _Computing Research Repository_ , 2019  arXiv:1906.05799
  * [26] Abhishek Gupta and Zhaoyuan Yang  “Adversarial Reinforcement Learning for Observer Design in Autonomous Systems under Cyber Attacks”  In _Computing Research Repository_ , 2018  arXiv:1809.06784
  * [27] Theodore J. Williams  “The Purdue Enterprise Reference Architecture”  In _Information Infrastructure Systems for Manufacturing_ B-14, 1993, pp. 43–64
  * [28] Maxim Egorov et al.  “POMDPs.jl: A Framework for Sequential Decision Making under Uncertainty”  In _Journal of Machine Learning Research_ 18, 2017, pp. 26:1–26:5
  * [29] Greg Brockman et al.  “OpenAI Gym”  In _Computing Research Repository_ , 2016  arXiv:1606.01540
  * [30] Matteo Hessel et al.  “Rainbow: Combining Improvements in Deep Reinforcement Learning”  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2018, pp. 3215–3222
  * [31] Hado Hasselt  “Double Q-learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2010, pp. 2613–2621
  * [32] Tom Schaul, John Quan, Ioannis Antonoglou and David Silver  “Prioritized Experience Replay”  In _International Conference on Learning Representations (ICLR)_ , 2016
  * [33] Richard S. Sutton and Andrew G. Barto  “Reinforcement Learning: An Introduction”  The MIT Press, 2018, pp. 141–145
  * [34] Andrew Y. Ng, Daishi Harada and Stuart J. Russell  “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping”  In _International Conference on Machine Learning (ICML)_ , 1999, pp. 278–287
  * [35] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 8024–8035
  * [36] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton  “ImageNet Classification with Deep Convolutional Neural Networks”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2012, pp. 1106–1114
  * [37] Kyunghyun Cho et al.  “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation”  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1724–1734  DOI: 10.3115/v1/d14-1179
  * [38] Ashish Vaswani et al.  “Attention is All you Need”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5998–6008
  * [39] John Mern, Dorsa Sadigh and Mykel J. Kochenderfer  “Object Exchangability in Reinforcement Learning”  In _International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ , 2019, pp. 2126–2128
  * [40] John Mern, Dorsa Sadigh and Mykel J Kochenderfer  “Exchangeable Input Representations for Reinforcement Learning”  In _American Control Conference (ACC)_ , 2020, pp. 3971–3976
  * [41] Todd Hester et al.  “Deep Q-learning From Demonstrations”  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2018, pp. 3223–3230
  * [42] Petar Velickovic et al.  “Graph Attention Networks”  In _International Conference on Learning Representations (ICLR)_ , 2018
  * [43] Karl Cobbe, Jacob Hilton, Oleg Klimov and John Schulman  “Phasic Policy Gradient”  In _Computing Research Repository_ , 2020  arXiv:2009.04416


# We need to talk about random seeds

Steven Bethard
University of Arizona
<EMAIL_ADDRESS>

###### Abstract

Modern neural network libraries all take as a hyperparameter a random seed,
typically used to determine the initial state of the model parameters. This
opinion piece argues that there are some safe uses for random seeds: as part
of the hyperparameter search to select a good model, creating an ensemble of
several models, or measuring the sensitivity of the training algorithm to the
random seed hyperparameter. It argues that some uses for random seeds are
risky: using a fixed random seed for “replicability” and varying only the
random seed to create score distributions for performance comparison. An
analysis of 85 recent publications from the ACL Anthology finds that more than
50% contain risky uses of random seeds.

##  1 Introduction

Modern neural network libraries all take as a hyperparameter a random seed, a
number that is used to initialize a pseudorandom number generator. That
generator is typically used to determine the initial state of model
parameters, but may also affect optimization (and inference) in other ways,
such as selecting which units to mask under dropout, or selecting which
instances of the training data go into each minibatch during gradient descent.
Like any hyperparameter, neural network random seeds can have a large or small
impact on model performance depending on the specifics of the architecture and
the data. Thus, it is important to optimize the random seed hyperparameter as
we would any other hyperparameter, such as learning rate or regularization
strength.

Such tuning is especially important with the pre-trained transformer
architectures currently popular in NLP (BERT, Devlin et al., 2019; RoBERTa Liu
et al., 2019; etc.), which are quite sensitive to their random seeds Risch and
Krestel (2020); Dodge et al. (2020); Mosbach et al. (2021). Several solutions
to this problem have been proposed, including specific optimizer setups
Mosbach et al. (2021), ensemble methods Risch and Krestel (2020), and
explicitly tuning the random seed like other hyperparameters Dodge et al.
(2020).

The NLP community thus has some awareness of the problems that random seeds
present, but it is inconsistent in its approaches to solving those problems.
The remainder of this opinion piece first presents a taxonomy of different
ways that neural network random seeds are used in the NLP community,
explaining which uses are safe and which are risky. It then reviews 85
articles published in the ACL Anthology, categorizing their random seed uses
based on the taxonomy. This analysis finds that more than 50% of the articles
include risky uses of random seeds, suggesting that the NLP community still
needs a broader discussion about how we approach random seeds.

##  2 A taxonomy of random seed uses

This section highlights five common uses of neural network random seeds in the
NLP community, and categorizes them as either safe or risky.

###  2.1 Safe use: Model selection

The random seed is a hyperparameter of a neural network architecture that
determines where in the model parameter space optimization should begin. It
may also affect optimization by determining the order of minibatches in
gradient descent, or through mechanisms like dropout’s random sampling of unit
activations. As the random seed is a hyperparameter, it can and should be
optimized just as other hyperparameters are. Unlike some other
hyperparameters, there is no intuitive explanation of why one random seed
would be better or worse than another, so the typical strategy is to try a
number of randomly selected seeds. For example:

> Instead, we compensate for the inherent randomness of the network by
> training multiple models with randomized initializations and use as the
> final model the one which achieved the best performance on the validation
> set… Björne and Salakoski (2018)
>
> The test results are derived from the 1-best random seed on the validation
> set. Kuncoro et al. (2020)

###  2.2 Safe use: Ensemble creation

Ensemble methods are an effective way of combining multiple machine-learning
models to make better predictions Rokach (2010). A common approach to creating
neural network ensembles is to train the same architecture with different
random seeds, and have the resulting models vote Perrone and Cooper (1995).
For example:

> In order to improve the stability of the RNNs, we ensemble five distinct
> models, each initialized with a different random seed. Nicolai et al. (2017)
>
> Our model is composed of the ensemble of 8 single models. The
> hyperparameters and the training procedure used in each single model are the
> same except the random seed. Yang and Wang (2019)

###  2.3 Safe use: Sensitivity analysis

Sometimes it is useful to demonstrate how sensitive a neural network
architecture is to a particular hyperparameter. For example, Santurkar et al.
(2018) shows that batch normalization makes neural network architectures less
sensitive to the learning rate hyperparameter. Similarly, it may be useful to
show how sensitive neural network architectures are to their random seed
hyperparameter. For example:

> We next (§3.3) examine the expected variance in attention-produced weights
> by initializing multiple training sequences with different random seeds…
> Wiegreffe and Pinter (2019)
>
> Our model shows a lower standard deviation on each task, which means our
> model is less sensitive to random seeds than other models. Hua et al. (2021)

###  2.4 Risky use: Single fixed seed

NLP articles sometimes pick a single fixed random seed, claiming that this is
done to improve consistency or replicability. For example:

> An arbitrary but fixed random seed was used for each run to ensure
> reproducibility… Le and Fokkens (2018)
>
> For consistency, we used the same set of hyperparameters and a fixed random
> seed across all experiments. Lin et al. (2020)

Why is this risky? First, fixing the random seed does not guarantee
replicability. For example, the tensorflow library has a history of producing
different results given the same random seeds, especially on GPUs Two Sigma
(2017); Kanwar et al. (2021). Second, not optimizing the random seed
hyperparameter has the same drawbacks as not optimizing any other
hyperparameter: performance will be an underestimate of the performance the
architecture is capable of with an optimized model.

What should one do instead? The random seed should be optimized as any other
hyperparameter. Dodge et al. (2020), for example, show that doing so leads to
simpler models exceeding the published results of more complex state-of-the-
art models on multiple GLUE tasks Wang et al. (2018). The space of
hyperparameters explored (and thus the number of random seeds explored) can be
restricted to match the availability of compute resources with techniques such
as random hyperparameter search Bergstra and Bengio (2012) where $n$
hyperparameter settings are sampled from the space of all hyperparameter
settings (with random seeds treated the same as all other hyperparameters). In
an extremely resource-limited scenario, random search might select only a
single value of some hyperparameter (such as random seed), which might be
acceptable given the constraints, but should probably be accompanied by an
explicit acknowledgement of the risks of underestimating performance.

###  2.5 Risky use: Performance comparison

It is a good idea to compare not just the point estimate of a single model’s
performance, but distributions of model performance, as comparing performance
distributions results in more reliable conclusions Reimers and Gurevych
(2017); Dodge et al. (2019); Radosavovic et al. (2020). However, it has
sometimes been suggested that such distributions can be obtained by training
the same architecture and varying only the random seed. For example:

> We re-ran both implementations multiple times, each time only changing the
> seed value of the random number generator…we observe a statistically
> significant difference between these two distributions… Reimers and Gurevych
> (2017)
>
> Indeed, the best approach is to stop reporting single-value results, and
> instead report the distribution of results from a range of seeds. Doing so
> allows for a fairer comparison across models… Crane (2018)

(Note the difference between sections 2.5 and 2.3: sensitivity analysis
describes only how sensitive the model is to the random seed; performance
comparision makes claims about whether one model is better than another.)

Why is this risky? If the goal is to compare the best possible model trainable
from one architecture to the best possible model trainable from another
architecture, as in the case of leaderboard comparisons, then varying random
seeds is generating a bunch of suboptimal models for that comparison. If the
goal is to compare the family of models that result from training one
architecture to the family of models that result from training another
architecture, then varying only the random seed is generating a small biased
slice of the family, since the family consists of the model variations across
all hyperparameter settings, not just random seeds.

What should one do instead? If the goal is to compare the best possible models
trainable from different architectures, then the random seed needs to be
optimized just as we would for any other hyperparameter. It’s still a good
idea to compare distributions, rather than point estimates, so standard
statistical techniques can be applied. For example, bootstrap samples may be
drawn from the test set, and evaluating a model on each of those samples will
give a distribution over the model’s expected performance Dror et al. (2018).
Comparing these distributions will give a statistically sound estimate of
whether the best model found for one neural network architecture outperforms
the best model found for another. If the goal is instead to compare families
of models, then it makes sense to train many versions of the same
architecture, but they should be sampled to vary across all hyperparameters,
not just the random seed hyperparameter111Occasionally, the random seed might
be the only hyperparameter, e.g., an extreme black box machine-learning
scenario where the only way to get model variants is to vary the order in
which training data instances are fed to the model. In such cases, it would be
acceptable to vary only the random seed.. Comparing these distributions will
give a statistically sound estimate of whether one architecture (and not just
the best-optimized instance of that architecture) is better than another.

##  3 Random seed uses in ACL

Having introduced both safe and risky uses of neural network random seeds, we
now turn to the current state of NLP with respect to such seeds. The following
analysis is illustrative, not exhaustive, providing a conservative estimate of
the prevalence of the problem.

On 29 Jun 2021, I searched the ACL Anthology for articles containing the
phrases “random seed” and “neural
network”222https://www.aclweb.org/anthology/search/?q=%22random+seed%22+%22neural+network%22.
The ACL Anthology search interface returns a maximum of 10 pages of results,
with 10 results per page, so I collected all 100 search results. Non-articles
(entire proceedings, author pages, supplementary material) were excluded, as
were articles where the random seeds were not used to initialize a neural
network (e.g., only for dataset selection). The result was 85 articles, from
publications between 2015 and 2021.

I read each of the articles and categorized its use of random seeds into one
of the five purposes introduced in section 2. While it is conceptually
possible for an article to fall into more than one category (e.g., having both
ensembles and sensitivity analysis) the vast majority of articles I read fell
into a single category, typically with just a single sentence where random
seed was used. For the tiny fraction of articles where more than one category
applied, since my goal only was to get a rough distribution of random seed
use, I selected a “primary” category arbitrarily from the categories present.
A spreadsheet detailing each article reviewed, its category of random seed
use, and a snippet of English text from the article justifying my assignment
of that category is
available333https://docs.google.com/spreadsheets/d/1ab7TlW4xgOdydZJuUAQc4XQ6Us7cAQrQG8eC2OvGeaM/edit.

Type | Purpose | Count
---|---|---
Safe | Model selection | 12
Safe | Ensemble creation | 13
Safe | Sensitivity analysis | 12
Safe sub-total | 37
Risky | Fixed seed | 24
Risky | Performance comparison | 24
Risky sub-total | 48
Table 1: Uses of neural network random seeds for 85 ACL Anthology articles.
$2015$$2016$$2017$$2018$$2019$$2020$$2021$$-10$$0$$10$model selectionensemble
creationsensitivity analysisfixed seedperformance comparison Figure 1: Uses of
neural network random seeds by year for 85 ACL Anthology articles.

Table 1 shows the distribution of articles across the different random seed
purposes. More than half of the articles (48) include a risky use of random
seeds, with 24 using a single fixed seed and 24 using only random seeds to
generate distributions for performance comparisons. This suggests that NLP
researchers are often using neural network random seeds without the necessary
care.

One might wonder if the NLP community is getting better over time, that is, if
risky uses are on the decline as NLP researchers become more familiar with
neural networks research. Figure 1 shows that this was not the case: though
the volume of articles that matched the query varies from year to year, for
most years the number of risky uses of random seeds is similar to the number
of safe uses. This suggests that NLP researchers continue to have trouble
distinguishing safe from risky uses of neural network random seeds.

##  4 Discussion

We have seen that risky uses of neural network random seeds – using only a
fixed seed or generating performance distributions for model comparisons by
varying only random seeds – are still widespread within the NLP community. The
analysis in section 3 is a conservative estimate of the problem. The query
used in the analysis matched articles only if they had the explicit phrases
“neural network” and “random seed” both within the article. That means the
search did not return articles on neural networks where no “random seed” was
mentioned, yet in such cases it is likely that a single fixed seed was used.
Therefore the proportion of fixed seed papers in our sample is likely an
underestimate of the proportion in the true population444This query also can’t
address another interesting issue that is out of scope for the current
article: quantifying how many articles don’t optimize any hyperparameters at
all..

How do we move the NLP community away from risky uses of neural network random
seeds? Hopefully, this article can help to start the necessary conversations,
but clearly it is not an endpoint in and of itself. Part of the responsibility
must fall on mentors in the NLP community, such as university faculty and
industry research leads, to ensure that they are training their mentees about
these topics. Part of the responsibility will fall on reviewers of NLP
articles, who can identify misuses of neural network random seeds and flag
them for revision. And of course part of the responsibility falls on NLP
authors themselves to make sure they understand the nuances of neural network
hyperparameters like random seeds and the ways in which they should and should
not be used.

##  5 Conclusion

This opinion piece has introduced a simple taxonomy of common uses for neural
network random seeds in the NLP literature, describing three safe uses (model
selection, ensemble creation, and sensitivity analysis) and two risky uses
(single fixed seed and varying only the random seed to generate distributions
for performance comparison). An analysis of 85 articles from the ACL Anthology
showed that more than half of these NLP articles include risky uses of neural
network random seeds. Hopefully, highlighting this issue can help the NLP
community to improve our mentorship and training and move away from risky uses
of neural network random seeds in the future.

## Ackmnowledgements

Thanks to all of the anonymous reviewers; addressing their suggestions has
improved this opinion piece. However, despite interesting discussions with
those reviewers, it seems that ACL Rolling Review is not conducive for papers
that seek to start a discussion rather than report empirical results. Hence,
this arXiv submission.

## References

  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio. 2012.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2).
  * Björne and Salakoski (2018) Jari Björne and Tapio Salakoski. 2018.  Biomedical event extraction using convolutional neural networks and dependency parsing.  In _Proceedings of the BioNLP 2018 workshop_ , pages 98–108, Melbourne, Australia. Association for Computational Linguistics.
  * Crane (2018) Matt Crane. 2018.  Questionable answers in question answering research: Reproducibility and variability of published results.  _Transactions of the Association for Computational Linguistics_ , 6:241–252.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019\.  Show your work: Improved reporting of experimental results.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2185–2194, Hong Kong, China. Association for Computational Linguistics.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _CoRR_ , abs/2002.06305.
  * Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.  The hitchhiker’s guide to testing statistical significance in natural language processing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.
  * Hua et al. (2021) Hang Hua, Xingjian Li, Dejing Dou, Chengzhong Xu, and Jiebo Luo. 2021.  Noise stability regularization for improving BERT fine-tuning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3229–3241, Online. Association for Computational Linguistics.
  * Kanwar et al. (2021) Pankaj Kanwar, Reed Wanderman-Milne, and Duncan Riach. 2021.  RFC: Random numbers in TensorFlow 2.0 by wangpengmit - pull request #38 - tensorflow/community.
  * Kuncoro et al. (2020) Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and Phil Blunsom. 2020.  Syntactic structure distillation pretraining for bidirectional encoders.  _Transactions of the Association for Computational Linguistics_ , 8:776–794.
  * Le and Fokkens (2018) Minh Le and Antske Fokkens. 2018.  Neural models of selectional preferences for implicit semantic role labeling.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Lin et al. (2020) Yan-Jie Lin, Hong-Jie Dai, You-Chen Zhang, Chung-Yang Wu, Yu-Cheng Chang, Pin-Jou Lu, Chih-Jen Huang, Yu-Tsang Wang, Hui-Min Hsieh, Kun-San Chao, Tsang-Wu Liu, I-Shou Chang, Yi-Hsin Connie Yang, Ti-Hao Wang, Ko-Jiunn Liu, Li-Tzong Chen, and Sheau-Fang Yang. 2020.  Cancer registry information extraction via transfer learning.  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , pages 201–208, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Nicolai et al. (2017) Garrett Nicolai, Bradley Hauer, Mohammad Motallebi, Saeed Najafi, and Grzegorz Kondrak. 2017.  If you can’t beat them, join them: the University of Alberta system description.  In _Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection_ , pages 79–84, Vancouver. Association for Computational Linguistics.
  * Perrone and Cooper (1995) Michael P. Perrone and Leon N. Cooper. 1995.  _When networks disagree: Ensemble methods for hybrid neural networks_, pages 342–358. World Scientific.
  * Radosavovic et al. (2020) Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. 2020.  Designing network design spaces.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10428–10436.
  * Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017.  Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.
  * Risch and Krestel (2020) Julian Risch and Ralf Krestel. 2020.  Bagging BERT models for robust aggression identification.  In _Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying_ , pages 55–61, Marseille, France. European Language Resources Association (ELRA).
  * Rokach (2010) Lior Rokach. 2010.  Ensemble-based classifiers.  _Artificial Intelligence Review_ , 33(1):1–39.
  * Santurkar et al. (2018) Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018.  How does batch normalization help optimization? In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Two Sigma (2017) Two Sigma. 2017.  A workaround for non-determinism in TensorFlow.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Yang and Wang (2019) Liner Yang and Chencheng Wang. 2019.  The BLCU system in the BEA 2019 shared task.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 197–206, Florence, Italy. Association for Computational Linguistics.


11institutetext: Adapt Centre, Dublin City University, Ireland
11email<EMAIL_ADDRESS>22institutetext: Cork Institute of
Technology, Ireland
22email<EMAIL_ADDRESS>
# Neural Architecture Search using Particle Swarm and Ant Colony Optimization

Séamus Lankford 11 Diarmuid Grimes 22

###### Abstract

Neural network models have a number of hyperparameters that must be chosen
along with their architecture. This can be a heavy burden on a novice user,
choosing which architecture and what values to assign to parameters. In most
cases, default hyperparameters and architectures are used. Significant
improvements to model accuracy can be achieved through the evaluation of
multiple architectures. A process known as Neural Architecture Search (NAS)
may be applied to automatically evaluate a large number of such architectures.

A system integrating open source tools for Neural Architecture Search
(OpenNAS), in the classification of images, has been developed as part of this
research. OpenNAS takes any dataset of grayscale, or RBG images, and generates
Convolutional Neural Network (CNN) architectures based on a range of
metaheuristics using either an AutoKeras, a transfer learning or a Swarm
Intelligence (SI) approach.

Particle Swarm Optimization (PSO) and Ant Colony Optimization (ACO) are used
as the SI algorithms. Furthermore, models developed through such
metaheuristics may be combined using stacking ensembles.

In the context of this paper, we focus on training and optimizing CNNs using
the Swarm Intelligence (SI) components of OpenNAS. Two major types of SI
algorithms, namely PSO and ACO, are compared to see which is more effective in
generating higher model accuracies. It is shown, with our experimental design,
that the PSO algorithm performs better than ACO. The performance improvement
of PSO is most notable with a more complex dataset. As a baseline, the
performance of fine-tuned pre-trained models is also evaluated.

###### Keywords:

AutoML NAS Swarm Intelligence PSO ACO CNN

##  1 Introduction

The area of Auto Machine Learning (AutoML) [1] is a growing area of interest
in recent years. This is reflected in the development of several open source
AutoML libraries among which include Auto-WEKA [2], Hyperopt-Sklearn [3],
AutoKeras [4], Auto-Sklearn [5, 6] and TPOT [7].

Despite a renewal of interest in AutoML, many of these open source solutions
focus on creating simpler neural architectures. Libraries which concentrate on
generating more complex architectures, such as CNNs, are at early stages of
development. Consequently they are poorly documented and often unreliable [4].
In addition, the alternative of using commercial platforms is expensive and
therefore users are left with few practical or viable options.

The development of OpenNAS integrates several metaheuristic approaches in a
single application used for the neural architecture search of more complex
neural architectures such as convolutional neural networks. Furthermore, the
effectiveness of NAS in generating good neural architectures for image
classification is evaluated. Standard approaches to NAS, using the AutoKeras
framework, are also incorporated into the system design.

A key aspect of the study is to contrast Swarm Intelligence (SI) algorithms
for NAS. Consequently, Particle Swarm Optimization (PSO) [8] and Ant Colony
Optimization (ACO) [9] have been chosen as metaheuristics for creating high
performing CNN architectures for grayscale and RGB image datasets.

##  2 Background

###  2.1 Convolutional Neural Networks

CNNs are feed-forward Deep Neural Networks (DNNs) used for image recognition.
The original CNN architecture was proposed by LeCun [10] and consisted of two
convolution layers, two pooling layers, two fully connected (FC) layers and an
output layer. Subsequently, numerous models were developed including popular
ones such as ResNet [11] and VGG [12]. In this study, custom CNN architectures
are created by using SI heuristics to find better combinations of
convolutional, pooling and FC layers.

###  2.2 Auto ML

AutoML involves the automation of the entire machine learning pipeline
including data augmentation, feature engineering, model selection, choice of
hyperparameters and finally neural architecture selection and creation. By
constrast, NAS has a more narrow focus in that it concentrates on neural
architecture selection and creation [13].

Tree-based Pipeline Optimization Tool (TPOT) is an open source python package
that uses genetic programming in optimizing the machine learning pipeline [7].
The library performs well on simple NAS tasks involving the scikit-learn API.
Given this study involves generating more complex CNNs, rather than developing
optimal pipelines, it was decided not to use TPOT as part of the initial
solution architecture. However, as part of future work, it may have a role in
optimizing hyper parameter selection.

AutoKeras [4] is an open source AutoML system using Bayesian optimization and
network morphism for efficient neural architecture search.

###  2.3 Neural Architecture Search

Neural architecture search is the process of automatically finding and tuning
DNNs. It has been shown that DNNs have made remarkable progress in solving
many real world problems such as image recognition, speech recognition and
machine translation[14]. In general, NAS systems consist of three main
components: a search space, a search algorithm and an evaluation strategy. The
search space sets out which architectures can be used in principle whereas the
search strategy outlines how the search space is explored. Finally the
evaluation strategy determines which architectures yield the best results on
unseen data.

A basic approach to NAS is the brute force training and evaluation of all
possible model combinations. On completion, the best performing model is
selected. However, this is impractical due to the combinatorics of the
problem. Using metaheuristics, such as swarm intelligence, is an alternative
which seeks the best model within reasonable time constraints.

###  2.4 Swarm Intelligence

Swarm Intelligence, a category of Evolutionary Computing, has been used for
classification problems in the following forms: Particle Swarm Optimization
(PSO) [15, 16] and Ant Colony Optimization (ACO) [17].

####  2.4.1 Particle Swarm Optimization

PSO belongs to the class of swarm intelligence techniques and is a population-
based stochastic technique for solving optimization problems developed in
1995. An open source python library, for CNN optimization using the PSO
algorithm was developed by Fernandes et al [18]. The results demonstrate that
their approach, psoCNN, quickly finds CNN architectures which offer
competitive performance for any given dataset.

####  2.4.2 Ant Colony Optimization

ACO, modelled on the activities of real ant colonies, involves moving through
a parameter space of all potential solutions to find the optimal weights for a
neural network.

Using ACO, a system known as DeepSwarm was developed by Byla and Pang [19] to
find high performing neural architectures for CNNs. They showed that it offers
competitive performance when tested on well-known image datasets.

##  3 Approach

With artificial neural networks, there are many parameters to choose from such
as the number of hidden network layers, number of neurons per layer, type of
activation function, choice of optimizer and so on. The final network design
often depends on the problem domain and is typically achieved in a time
consuming trial and error fashion.

Similar problems exist with CNNs but these problems are exacerbated by the
length of time, and amount of computational resources required to train such
networks. Clearly, a core objective of NAS is to find good network performance
within acceptable time limits through the reduction of both the number of
networks tested and the length of time required for their evaluation. The
implementation of NAS can be achieved through a variety of approaches
including transfer learning using pre-trained networks, network morphism or
swarm intelligence. Using these approaches as its pillars, a NAS system
(OpenNAS) has been built which tackles such problems 111
https://github.com/seamusl/OpenNAS-v1. OpenNAS does not enforce a particular
architecture but rather it allows novel and interesting architectures to be
discovered.

In this work we focus on the swarm intelligence component of the OpenNAS
system. The swarm optimization techniques currently used are Particle Swarm
Optimization and Ant Colony Optimization.

Table 1: Parameters for Particle Swarm Optimization | Config A | Config B
---|---|---
Swarm
Number of iterations | 10 | 20
Swarm size | 20 | 10
Cg | 0.5 | 0.5
CNN architecture
Minimum outputs from a Conv layer | 3 | 3
Maximum outputs from a Conv layer | 256 | 256
Maximum neurons in a FC layer | 300 | 300
Minimum size of a Conv kernel | 3 x 3 | 3 x 3
Maximum size of a Conv kernel | 7 x 7 | 7 x 7
Minimum layers | 3 | 3
Maximum layers | 20 | 20
CNN Training
# epochs for particle evaluation | 5 | 5
# epochs for global best | 100 | 100
Dropout rate | 0.5 | 0.5
Batch normalize layer outputs | Yes | Yes
Probability Settings
probability_convolution | 0.6 | 0.6
probability_pooling | 0.3 | 0.3
probability_fully_connected | 0.1 | 0.1

The PSO algorithm determines how the principal CNN layer types, and their
associated hyperparameters, are connected together. The generated models
consist of architectures using a mix of convolutional, average pooling, max
pooling and fully connected layers. In addition, dropout layers and batch
normalization layers are also added to alleviate overfitting. The
hyperparameters associated with each layer type are indicated in Table 1.

Particle architectures, i.e. model architectures, are compiled for a number of
epochs and evaluation is carried out using the standard loss function of
cross-entropy loss. Particle architectures with the smallest loss are selected
by the algorithm. The number of epochs parameter for pBest must be carefully
chosen since it is the main driver of both run time and model accuracy.

Using an ACO approach, the parameters used for model training in the
exploration process are highlighted in Table 2. Two test configurations are
considered. In the first case, 8 ants are used with 30 epochs and in the
second case, 16 ants are used with 15 epochs. The depth parameter was fixed at
20.

Table 2: Parameters for Ant Colony Optimization | Config A | Config B
---|---|---
Ant Colony
Number of Ants | 8 | 16
Number of Epochs | 30 | 15
Search Depth | 20 | 20
CNN architecture
Kernel Sizes | 1, 3, 5 | 1, 3, 5
Minimum layers | 1 | 1
Maximum layers | 20 | 20
CNN Training
Dropout rate | 0.1, 0.3, 0.5 | 0.1, 0.3, 0.5
Batch normalize layer outputs | Yes | Yes
Probability Settings
pheromone start, decay, evaporation | 0.1 | 0.1
greediness | 0.5 | 0.5

Fine-tuning was implemented by initially removing the fully connected layers
from the top of the model. Two blocks are then added, each of which has a
fully connected layer, a batch normalization layer and a dropout layer. The
hybrid structure is then trained with the new dataset. Fine-tuning of a VGG16
network is illustrated in Figure 1.

Figure 1: Fine-tuned VGG16 model

##  4 Design

Figure 2: Open source Neural Architecture Search (OpenNAS) system design

The high level view of the system architecture is presented in Figure 2. The
system is organized into the following python modules: OpenNAS, pre-processor,
trainer, ensemble, super stacker, sysconfig and loader. The pre-train function
uses transfer learning as either a feature extractor or to fine-tune the pre-
trained networks of VGG16, VGG19, MobileNet or ResNet50.

With the swarm function, PSO or ACO can be used to search for the best neural
architecture. Existing open source python libraries were customized for both
PSO and ACO functionality. Particle swarms were implemented using a psoCNN
library[18] whereas ant colonies used the DeepSwarm library [19]. The
environment required Python 3.7, Tensorflow 1.14, Keras 2.2.4, Numpy 1.16.4
and Matplotplib 3.1.0.

Existing NAS tools, such as AutoKeras, were also integrated into the OpenNAS
system. AutoKeras is a powerful open source library which provides functions
to automatically search for optimal architectures for deep learning models.
However, this library is still in beta development and the associated
documentation is quite poor.

With the ensemble module, there are options to build stacked ensembles using
either homogeneous or heterogeneous base learners. These learner outputs are
subsequently passed to a suite of meta learner algorithms. The system
generates the optimal neural architecture model using the chosen heuristic.

##  5 Evaluation

Two datasets were chosen for the experimental design, namely CIFAR10 [20] and
Fashion_Mnist [21]. A primary research objective is the development of a
neural architecture search tool which generates high performing architectures
for generic datasets of either grayscale (one channel) or color (triple
channel) images. The CIFAR10 dataset meets this requirement in that it is a
challenging dataset of color images. The Fashion_Mnist dataset is also
suitable since it a well-tested and well-understood dataset of black and white
images. For reference, the state of the art (SOA) accuracy achieved on CIFAR10
is 98.5%[22] whereas with Fashion_Mnist, the SOA accuracy is 94.6%[23].

###  5.1 Particle Swarm Optimization

In order to test variance and reproducibility, each configuration was run 5
times on both CIFAR10 and Fashion_Minst which resulted in the evaluation of
4000 CNN architectures for this phase of the study.

####  5.1.1 Evaluation of models trained on CIFAR10 dataset

Validation accuracy was used to evaluate the performance of both PSO
configurations. It is clear from Table 3 that the PSO model trained on swarm
settings of a lower population and higher number of iterations (population of
10 and 20 iterations) performed significantly better. In terms of accuracy,
the mean performance was 3.5% better.

Table 3: Performance of PSO models on CIFAR10 Model |  | Acc
---
Max
| Acc
---
Mean
| Acc
---
StDev
| Time
---
(min)
Layers
| Population 10
---
Iterations 20
0.900 | 0.853 | 0.044 | 1316 | 30
| Population 20
---
Iterations 10
0.883 | 0.818 | 0.053 | 1119 | 38

Both configurations have a very low standard deviation for model accuracy
indicating a high level of reproducibility between test runs. At a mean run
time of 21.9 hours for the first configuration and 18.6 hours for the second
configuration, the PSO search for CNN architectures is a slow process
considering high performance workstations, with NVIDIA GeForce GTX 1080 Ti
graphic cards, were used.

####  5.1.2 Evaluation of models trained on Fashion_Mnist dataset

PSO models trained on the Fashion_Mnist dataset (Table 4), achieved much
higher accuracy compared with models developed using CIFAR10 data. Similar to
CIFAR10, the low standard deviation associated with both implementations of
Fashion_Mnist models indicate the PSO approach produces consistent results
between different test runs.

Table 4: Performance of PSO models on Fashion_Mnist Model |  | Acc
---
Max
| Acc
---
Mean
| Acc
---
StDev
| Time
---
(min)
Layers
| Population: 10
---
Iterations: 20
0.943 | 0.932 | 0.009 | 994 | 30
| Population: 20
---
Iterations: 10
0.943 | 0.935 | 0.008 | 1319 | 38

The stochastic nature of metaheuristics impacts the run times associated with
PSO for both Fashion_Mnist and CIFAR10. In all tests, no clear pattern emerged
with regard to run times: CIFAR10 was faster using a population of 10 with 20
iterations whereas Fashion_Mnist was faster with a population of 20 with 10
iterations. Therefore, in terms of run time, no clear conclusion could be
drawn by doubling the population and halving the iterations.

With regard to the impact of swarm settings on model accuracy for
Fashion_Mnist, again there is little to separate the configurations. With a
mean accuracy of 93.5% for a population of 20 with 10 iterations and a
corresponding mean model accuracy of 93.2% using a population of 10 with 20
iterations, no clear conclusion can be drawn.

Therefore, unlike CIFAR10, changing the swarm settings by doubling population
and halving iterations does not impact model accuracy in the case of
Fasion_Mnist. Both configurations for the PSO algorithm perform well on this
dataset.

###  5.2 Ant Colony Optimization

Similar to other metaheuristics, there are several parameters which can be
tuned for optimal neural architecture search using Ant Colony Optimization
[17]. With OpenNAS, users may select the options of depth, number of ants and
number of epochs in directing how the neural architecture search is conducted.

####  5.2.1 Evaluation of models trained on CIFAR10 dataset

With CIFAR10 data, the results from Table 5 indicate that a greater number of
ants leads to higher model accuracy. The improvement in max model accuracy
achieved, through doubling the number of ants and halving the number of
epochs, was modest at just 1.2%. The impact on run time for a small increase
in accuracy was severe. Doubling the number of ants effectively doubled the
run time (even though the number of epochs was halved). The standard deviation
for accuracy is very low indicating good reproducibility between the various
test runs.

Table 5: Performance of ACO models on CIFAR10 Model |  | Acc
---
Max
| Acc
---
Mean
| Acc
---
StDev
| Time
---
(min)
Layers
| Ants: 8
---
Epochs: 30
0.848 | 0.822 | 0.025 | 541 | 18
| Ants: 16
---
Epochs: 15
0.836 | 0.821 | 0.014 | 1004 | 16

####  5.2.2 Evaluation of models trained on Fashion_Mnist dataset

The performance of ACO models using Fashion_Mnist data is highlighted Table 6.
It can be seen that both configurations perform well resulting in accuracies
greater than 93%. The difference in mean model accuracy between configuration
A (8 ants and 30 epochs) is trivial when compared to configuration B (16 ants
and 15 epochs). However, similar to ACO on CIFAR10, the difference in run time
is very significant for configuration B. Effectively it took over 7 hours
longer to achieve an accuracy improvement of 0.1%.

Clearly in the case of a simpler dataset such as Fashion_Mnist, using a number
of ants in excess of 8 is not worth doing. This finding is similar to that
seen with the more complex CIFAR10 dataset, above. Therefore choosing the
number of ants, used for this ACO implementation, is an important
consideration impacting run time performance. As anticipated, the standard
deviation for accuracy is also very low indicating good reproducibility
between the various test runs.

Table 6: Performance of ACO models on Fashion_Mnist Model |  | Acc
---
Max
| Acc
---
Mean
| Acc
---
StDev
| Time
---
(min)
Layers
| Ants: 8
---
Epochs: 30
0.934 | 0.931 | 0.002 | 375 | 7
| Ants: 16
---
Epochs: 15
0.934 | 0.932 | 0.004 | 837 | 19

##  6 Discussion

Figure 3: OpenNAS performance of all models on CIFAR10 and Fashion_Mnist

The OpenNAS performance of all models across both datasets is illustrated in
Figure 3. The results demonstrate performance comparable to that achieved by
the psoCNN [18] approach and better than that of DeepSwarm [19].

The highest accuracy of OpenNAS in CIFAR-10 classification was 90.0%. This was
achieved using using a PSO-derived model. By comparison, DeepSwarm achieved a
top accuracy of 88.7%.

With Fashion_Mnist data, the highest performing model for OpenNAS is again a
PSO derived model with an accuracy of 94.3%. This result compares very
favourably with the SOA accuracy of 94.6%. The highest performing model for
DeepSwarm achieved an accuracy of 93.56%.

In the case of psoCNN, experiments were conducted on Fashion_Mnist but not on
CIFAR-10. The best performing psoCNN model on Fashion_Mnist was 91.9% without
dropout and 94.5% with dropout.

The findings clearly show that a PSO approach leads to higher model accuracies
given that DeepSwarm is exclusively based on an ACO approach.

The pre-trained networks of MobileNet and RestNet50 delivered the poorest
performance with CIFAR10. The other pre-trained networks, using VGG
architectures, performed very well on the same dataset.

With a more complex dataset, such as CIFAR10, the mean performance improvement
of the PSO algorithm is significant when compared with ACO. With
configurations used in this study, PSO achieved a mean accuracy of 85.3% on
CIFAR10 compared with an ACO mean accuracy of 82.2%.

The approach taken by ACO, in determining the best architecture is very
different to the PSO approach. With ACO, simpler models are initially
evaluated at lower depths with progressively more complex models being
evaluated at deeper search levels. Therefore at search depth 1, there is
essentially just a single hidden layer being evaluated. The number of ants
specified creates new architectures which simply vary the hyper parameters
used for that layer. With each new depth being explored, an additional layer
is added to the architecture being explored.

Furthermore, the ACO approach enables the targeting of hyper parameter
optimization within a given layer type rather than optimizing at the overall
architecture level. Specifying a large number of ants, with a reduced depth,
ensures the search space is restricting to studying the effects of layer hyper
parameters rather than model depth and the constituent layers. By comparison,
the number of layers in PSO generated models is entirely stochastic.

##  7 Conclusion

The OpenNAS approach identifies the hyperparameters within each layer of
networks used for image classification of grayscale and color datasets. In
addition the number and type of layers for the neural architecture are also
identified. This combined approach generates model architectures which achieve
competitve accuracies when classifying the CIFAR10 and Fashion_Mnist datasets.

The results of swarm intelligence algorithms, in the context of this study,
have generated impressive performances. However, in many cases, their
performance is only marginally better than fine-tuned pre-trained VGG models.
The accuracies of PSO derived models have been shown to exceed those of ACO
derived models in the image classification of grayscale and color datasets.

In addition, the OpenNAS integrated approach, using both PSO and ACO
algorithms, yields higher accuracies when compared with DeepSwarm which relies
on a single metaheuristic.

## References

  * [1] F. Hutter, L. Kotthoff, and J. Vanschoren, _Automated machine learning: methods, systems, challenges_. Springer Nature, 2019.
  * [2] L. Kotthoff, C. Thornton, H. H. Hoos, F. Hutter, and K. Leyton-Brown, “Auto-weka 2.0: Automatic model selection and hyperparameter optimization in weka,” _The Journal of Machine Learning Research_ , vol. 18, no. 1, pp. 826–830, 2017.
  * [3] B. Komer, J. Bergstra, and C. Eliasmith, “Hyperopt-sklearn,” in _Automated Machine Learning_. Springer, Cham, 2019, pp. 97–111.
  * [4] H. Jin, Q. Song, and X. Hu, “Auto-keras: An efficient neural architecture search system,” in _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 1946–1956.
  * [5] M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter, “Auto-sklearn: efficient and robust automated machine learning,” in _Automated Machine Learning_. Springer, Cham, 2019, pp. 113–134.
  * [6] M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter, “Auto-sklearn 2.0: The next generation,” _arXiv preprint arXiv:2007.04074_ , 2020.
  * [7] R. S. Olson and J. H. Moore, “Tpot: A tree-based pipeline optimization tool for automating,” _Automated Machine Learning: Methods, Systems, Challenges_ , p. 151, 2019.
  * [8] B. A. Garro and R. A. Vázquez, “Designing artificial neural networks using particle swarm optimization algorithms,” _Computational intelligence and neuroscience_ , vol. 2015, 2015.
  * [9] M. Mavrovouniotis and S. Yang, “Training neural networks with ant colony optimization algorithms for pattern classification,” _Soft Computing_ , vol. 19, no. 6, pp. 1511–1522, 2015.
  * [10] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [12] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [13] T. Elsken, J. H. Metzen, F. Hutter _et al._ , “Neural architecture search.” 2019.
  * [14] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” _Proceedings of the IEEE_ , vol. 105, no. 12, pp. 2295–2329, 2017.
  * [15] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in _Proceedings of ICNN’95-International Conference on Neural Networks_ , vol. 4. IEEE, 1995, pp. 1942–1948.
  * [16] R. C. Eberhart and Y. Shi, “Comparison between genetic algorithms and particle swarm optimization,” in _International conference on evolutionary programming_. Springer, 1998, pp. 611–616.
  * [17] M. Dorigo and L. M. Gambardella, “Ant colony system: a cooperative learning approach to the traveling salesman problem,” _IEEE Transactions on evolutionary computation_ , vol. 1, no. 1, pp. 53–66, 1997.
  * [18] F. E. F. Junior and G. G. Yen, “Particle swarm optimization of deep neural networks architectures for image classification,” _Swarm and Evolutionary Computation_ , vol. 49, pp. 62–74, 2019.
  * [19] E. Byla and W. Pang, “Deepswarm: Optimising convolutional neural networks using swarm intelligence,” in _UK Workshop on Computational Intelligence_. Springer, 2019, pp. 119–130.
  * [20] A. Krizhevsky, V. Nair, and G. Hinton, “The cifar-10 dataset,” _online: http://www. cs. toronto. edu/kriz/cifar. html_ , vol. 55, 2014.
  * [21] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” _arXiv preprint arXiv:1708.07747_ , 2017.
  * [22] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 113–123.
  * [23] B. Ma, X. Li, Y. Xia, and Y. Zhang, “Autonomous deep learning: A genetic dcnn designer for image classification,” _Neurocomputing_ , vol. 379, pp. 152–161, 2020.