# Talking After Lights Out: An Ad Hoc Network for Electric Grid Recovery

Jan Janak1, Dana Chee2, Hema Retty3, Artiom Baloian1, Henning Schulzrinne1
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>This research was developed with
funding from the Defense Advanced Research Projects Agency (DARPA). The views
and conclusions contained in this document are those of the authors and should
not be interpreted as representing the official policies, either expressed or
implied, of the Defense Advanced Research Projects Agency or the U.S.
government. Distribution statement A. Distribution approved for public
release, distribution unlimited. Not export controlled per ES-FL-020821-0013.
1Department of Computer Science, Columbia University, USA  2Perspecta Labs,
USA  3FAST Labs, BAE Systems, USA

###### Abstract

When the electric grid in a region suffers a major outage, e.g., after a
catastrophic cyber attack, a “black start” may be required, where the grid is
slowly restarted, carefully and incrementally adding generating capacity and
demand. To ensure safe and effective black start, the grid control center has
to be able to communicate with field personnel and with supervisory control
and data acquisition (SCADA) systems. Voice and text communication are
particularly critical. As part of the Defense Advanced Research Projects
Agency (DARPA) Rapid Attack Detection, Isolation, and Characterization Systems
(RADICS) program, we designed, tested and evaluated a self-configuring mesh
network architecture and prototype called the Phoenix Secure Emergency Network
(PhoenixSEN). PhoenixSEN is designed as a drop-in replacement for primary
communication networks, combines existing and new technologies, can work with
a variety of link-layer protocols, emphasizes manageability and auto-
configuration, and provides a core set of services and applications for
coordination of people and devices including voice, text, and SCADA
communication. The PhoenixSEN prototype was evaluated in the field through a
series of DARPA-led exercises. The same system is also likely to support
coordination of recovery efforts after large-scale natural disasters.

###### Index Terms:

Ad hoc networks, network architecture, network security.

##  I Introduction

Most electric power outages are locally-contained and recovery can rely on the
public or utility-owned communications infrastructure to coordinate
restoration and energizing parts of the electric grid. Large-scale electric
power outages, a.k.a blackouts, are rare but do happen [1, 2]. Recovering from
a large-scale outage typically follows a special procedure known as black
start. “A total or partial shutdown of the national electricity transmission
system (NETS) is an unlikely event. However, if it happens, we are obliged to
make sure there are contingency arrangements in place to ensure electricity
supplies can be restored in a timely and orderly way. Black start is a
procedure to recover from such a shutdown.” [3]

In the United States (U.S.), the black start procedure is usually managed by
regional transmission organizations (RTOs) that coordinate several electric
utilities. For example, PJM, a large RTO, describes its black start operation
as follows: “Black Start capability is necessary to restore the PJM
transmission system following a blackout. Black Start Service shall enable
PJM, in collaboration with the Transmission Owners, to designate specific
generators whose location and capabilities are required to re-energize the
transmission system.” [4]

Even if sufficient black start generating capability is available, a
successful black start requires coordination of electricity supply and demand,
typically by incrementally adding both generating capacity and load. Such
coordination usually takes place either via phone calls to substation
personnel, or via real-time control of supervisory control and data
acquisition (SCADA) devices. Both cases require network connectivity. Grid
operators often rely on internet service providers (ISPs) for network services
[5]. If the ISPs are also impaired by the blackout, network connectivity may
be difficult to guarantee.

If the blackout is caused by a network-based cyber attack, the attacker may
also attempt to actively thwart or delay power restoration, making a bad
situation worse. The Defense Advanced Research Projects Agency (DARPA) has
recognized the danger network-based cyber attacks represent for the U.S.
critical power grid infrastructure and launched the Rapid Attack Detection,
Isolation, and Characterization Systems (RADICS) program [6]. The goal of the
program is to create a set of tools that will aid the power distribution
industry in recovering from a hypothetical large-scale blackout triggered by a
network-based cyber attack.

In this paper, we present the design, prototype implementation, and
experimental evaluation of the Phoenix Secure Emergency Network (PhoenixSEN)
designed by BAE Systems and Columbia University as part of the DARPA RADICS
tool set. PhoenixSEN consists of a hybrid, isolated, self-forming network and
services specifically designed to enable the coordination of power restoration
amidst an ongoing network-based cyber attack. It combines existing and new
technologies, can work with a variety of link-layer protocols, and provides
applications for rapid coordination of people and devices. The network is
designed as a drop-in replacement for primary communication networks that are
likely to be severely impaired during a large-scale blackout.

We begin by discussing the motivation and the problem being addressed by our
work (and the DARPA RADICS program in general) in Section II. Section III
presents a simplified model of the U.S. electrical grid with an emphasis on
networking infrastructure. We then discuss the general architecture and
features of the Phoenix network and node in Section IV. The subsequent
sections describe various building blocks, applications, and services in
detail: naming and service discovery (Section V), voice and chat support
(Section VI), network monitoring (Section VII), and insider attack mitigation
(Section VIII). We conclude in Section X and discuss potential future work and
applicability of PhoenixSEN to other scenarios, e.g., natural disaster
response.

##  II Motivation & Problem Statement

Our work is primarily motivated by the need of the power distribution industry
for backup network infrastructure that could be used to recover from a large-
scale blackout caused by a network-based cyber attack [7]. We assume that the
grid control SCADA devices, as well as the network infrastructure itself maybe
be compromised and could act maliciously. Therefore, the backup infrastructure
should provide a means to reconnect healthy devices and keep those isolated
from potentially compromised or malicious devices. This would allow for an
incremental approach where devices are connected to the temporary (isolated)
network only after they have been inspected and deemed healthy.

Like most complex networked systems today, power grid infrastructure relies,
at least partially, on networks managed by external ISPs for remote command,
control, and coordination of both machines (SCADA) and operators (voice). It
is likely that the network infrastructure itself will be severely affected in
case of a large-scale blackout. This presents an interesting “chicken or the
egg” dilemma. The network needs power to connect the grid, but the grid cannot
operate without the network. Clearly, there needs to be infrastructure in
place that will allow the grid to be temporarily self-sufficient, at least
during the initial restart phase when the grid is not yet fully operational.
We envision a mostly self-configuring network infrastructure that can take
advantage of various existing link layer technologies and can be deployed to
geographically dispersed sites used by the power grid infrastructure after the
blackout. We assume the personnel setting up the infrastructure will have
technical background, but not necessarily in computer or network engineering.
The network would provide the minimum set of services and bandwidth necessary
to black-start the power grid in a secure manner.

A recent series of large scale blackouts illustrates that such events are not
merely a hypothetical possibility. The 2019 blackout in Argentina, Uruguay,
and Paraguay left 48 million people without power [1]. For comparison, the
Boston-Washington metropolitan area has 50 million inhabitants. In 2019, a
large-scale power outage in England affected more than a million homes and
severely disrupted the public transit system [2, 8].

Network-based cyber attacks on critical power grid infrastructure can
potentially have even more disastrous and long-lasting consequences, leaving
tens of millions of people in large metropolitan areas without power for
extended periods of time. An ongoing cyber attack on power grid systems may
attempt to thwart any restart attempts, leaving a large number of people
without power for days or weeks. Depending on the state of the power grid
infrastructure, a full black-start recovery after a cyber attack may also take
a long time, e.g., days or weeks, depending on the sophistication of the
attack.

While some of the critical grid infrastructure may be temporarily powered by
on-site backup diesel generators, the situation will get progressively worse
as those generators begin malfunctioning or start running out of fuel.
Communication networks stop working, potentially bringing down other critical
services such as the Global Positioning System (GPS). Water and gas systems,
hospitals, nursing homes, and waste treatment facilities will soon begin
shutting down, transportation infrastructure can be severely affected. The
situation gets progressively worse as more critical services shut down [9].

In a large-scale catastrophic scenario like this, with a substantial and
prolonged disruption of electric power, time is of the essence. The operation
of the electrical grid must be restored within a few days to prevent other
critical facilities from shutting down. A full black-start recovery of the
electric grid requires communication and coordination. However, communication
networks are unlikely to remain operational after a substantial power
disruption event.

While the main use case for the work presented in this paper is black-start
recovery of the power grid, a similar architecture could also be used to
create temporary isolated networks for emergency communications in various
disaster relief scenarios, e.g., the hurricanes that ravaged Puerto Rico in
2017 [10]. Similar technology is being used by some disaster relief
organizations such as the Red Cross [11].

##  III Architectural Model of U.S. Electrical Grid

(a) Distributed grid model where electricity market participants use shared
transmission infrastructure coordinated by a regional transmission operator
(RTO) or independent system operator (ISO). (b) A detailed overview of the
communications infrastructure required to keep the grid operational and
reliable. The diagram shows a simplified architectural model. The existing
grid exhibits considerable variety across geographic and political boundaries.

Figure 1: The U.S. electrical grid evolves towards a more distributed model
which requires an increasing amount of coordination, which in turn requires
extensive data communications infrastructure. Modern-day grid infrastructure
resembles a geographically dispersed cyber-physical system (CPS), where a
computer program (EMS) manages the flow of electric power through the system
via real-time SCADA-based remote instrumentation of physical processes.

The U.S. electrical grid is a complex, heterogeneous, geographically dispersed
system that combines physical infrastructure for producing and delivering
electric power with computer-based monitoring, management, and control. As
essential infrastructure that has been evolving for more than a century and is
subject to extensive government regulation, the grid has seen incremental
upgrades and organic growth, resulting in considerable variability across
geographical and political boundaries. The grid’s architecture is moving from
a model with a small number of vertically-integrated111A vertically-integrated
utility controls all stages of electric power supply chain, from generation to
distribution among consumers. utility monopolies towards an interconnected
model with a multitude of utility and non-utility companies coordinating to
use shared transmission infrastructure. In the interconnected model,
networking infrastructure plays an increasingly important role and is critical
for reliable grid operation.

In the U.S., hundreds of companies participate in the production and
transmission of electric power. To ensure reliable grid operation, the Federal
Energy Regulation Commission (FERC) has designated the North American Electric
Reliability Corporation (NERC) to develop and enforce operational standards.
Regional system operators coordinate the generation, transmission, and
distribution of electric power. In some regions the system operator is
affiliated with a particular utility company. In other regions the system
operator is an independent entity known as the RTO that coordinates multiple
utilities. Some regions have an independent system operator (ISO) instead with
a similar role. The differences are subtle and beyond the scope of this paper.
The RTO/ISO operates a wholesale electricity market, guarantees non-
discriminatory access to shared grid infrastructure, and ensures reliable grid
operation and compliance with NERC standards. The actual electric power
generation, distribution, metering, and billing is provided by utility and
non-utility companies coordinated by the RTO/ISO.

The major elements of an electric grid are the devices that produce and
transmit electric power, information technology (IT), industrial control
systems (ICSs), and the underlying network infrastructure. Since electric
power is generated and consumed almost instantaneously, the grid must be
coordinated to match power generation with demand in real-time. Figure 1
provides a simplified architectural model of the U.S. grid with a focus on the
communications infrastructure.

NERC operational standards provide high-level guidance primarily aimed at
ensuring reliable grid operation. The actual implementation details of power
grid systems are left to the RTOs/ISOs and utilities. As a result, existing
grid systems are heterogeneous and often use a multitude of devices that
communicate with mutually incompatible protocols. Early substation automation
devices communicated using proprietary protocols over industry-specific buses
or serial links. Modern-day substation systems tend to use standardized
process automation protocols and reuse existing wired and wireless network
technologies.

The flow of electric power through the grid is managed by an energy management
system (EMS) program. The primary purpose of the EMS to keep the grid
operational and reliable in response to varying conditions such as the
available generator pool, transmission capacity, and instantaneous load.
Ideally, a single instance of the EMS with the ability to remotely control
critical grid devices would be provided by the RTO/ISO for the entire region.
In practice, individual transmission operators typically run their own EMS to
monitor and protect their assets.

The EMS obtains the data about available generation resources and transmission
capacity for its scheduling and planning algorithms from the Open Access Same-
Time Information System (OASIS), a standardized web-based management system
[12] that serves as an interface between electricity market participants,
transmission providers, and the RTO/ISO. FERC requires that each RTO/ISO must
provide an OASIS node. Clients typically interact with the OASIS system by
invoking Hypertext Transfer Protocol (HTTP) application programming interfaces
(APIs) over the internet.

The operators of infrastructure deemed critical for the reliability of the
grid by the RTO/ISO are required to provide the RTO/ISO with remote access to
selected components for monitoring and control purposes. This is accomplished
by integrating the RTO/ISO’s and the operator’s SCADA and synchrophasor
subsystems over a redundant wide area network (WAN) provided for this purpose
by the RTO/ISO. The data obtained from these subsystems is used by the EMS to
build a global view of the state of the grid. Furthermore, the EMS can use
SCADA to remotely control grid devices in the field, e.g., circuit breakers.
Not all grid participants need to be SCADA-capable and connected to the
RTO/ISO WAN. Smaller entities sometimes rely on the internet for all
communication with the RTO/ISO.

(a) A Phoenix node at each substation connects SCADA and backend devices to a
virtual network spanning all substations of the utility. Per-utility virtual
networks share common physical infrastructure but are isolated from one
another. Each Phoenix node helps route packets for other utilities. A
dedicated forensic access port is provided on each node.

(b) The Phoenix node consists of an Intel NUC with a uniform software
installation and peripherals in a weather resistant enclosure. One-time
deployment configuration is performed via a memory card (distributed
separately) or the included smartphone.

Figure 2: PhoenixSEN is a drop-in replacement for ISP networks used by
utilities. The OLSR-based network consists of uniform Phoenix nodes deployed
at substations and interconnected by a variety of links (long, short, fast,
slow). Each utility is provided with an isolated virtual network spanning all
its substations.

From the previous paragraphs it follows that many different types of data
networks are involved in managing the flow of electricity through the grid.
The RTO/ISO operates a redundant WAN used to connect to the control centers of
grid infrastructure providers critical for overall system reliability. The
typical RTO/ISO WAN is a redundant Multiprotocol Label Switching (MPLS)
network based on links leased from several external ISPs. The synchrophasor
subsystem, if deemed critical for future grid systems, will most likely use a
dedicated WAN with stricter latency and bandwidth guarantees.

The use of the public switched telephone network (PSTN) for human-to-human
voice communications between the RTO/ISO and utility personnel is required by
NERC. Despite an overall increase in remote instrumentation capabilities
across the electrical grid, human-to-human voice communication remains the
most important communication modality in emergency situations, e.g., after a
blackout. To meet NERC’s strict reliability requirements, the phone system
typically uses cellular or satellite phones as backup.

Each utility also operates a dedicated WAN spanning its (sometimes large)
service area that connects the utility’s control center with all substations.
The typical utility WAN is Internet Protocol-based (IP) and uses a combination
of public (ISP-owned) and non-public (utility-owned) network infrastructure. A
substation with connected devices (e.g., SCADA) must also provide a field area
network (FAN). The FAN connects substation automation devices as well as any
remote devices (metering, data collection) within the substation’s service
area, e.g., a neighborhood. Due to the large variety in deployed automation
devices, the FAN is perhaps the most heterogeneous network type and is
typically based on a combination of wired and wireless technologies. The
public internet (not pictured) is typically used for other communication,
e.g., to access the RTO/ISO’s OASIS portal, or to transfer metering or billing
information between the utility and its customers.

SCADA interactions between the RTO/ISO and utility control centers typically
use standardized protocols such as the Distributed Network Protocol 3 (DNP3)
[13], the Inter-Control Center Communications Protocol (ICCN) [14], or IEC
61850 [15] carried over TCP/IP. The SCADA subsystem is a hierarchically
organized system where the RTO/ISO’s master terminal unit (MTU) communicates
with the MTUs at utility control centers, which in turn communicate with
remote terminal units (RTUs) deployed at substations. The synchrophasor
subsystem is based on a hierarchy of phasor data concentrators (PDCs) that
aggregate and process IEEE C37.118 [16] data streams coming from phasor
measurement units (PMUs) deployed across the grid.

##  IV Phoenix Secure Emergency Network

The PhoenixSEN is a self-configuring ad hoc network architecture designed to
provide a drop-in replacement for the grid’s primary (ISP) communication
networks. The network requires minimal deployment configuration and offers
essential services for human-to-human (voice, text) and device-to-device
(SCADA) coordination. Uniform hardware and software architecture allows rapid
deployment from a storage facility to substations. Nodes are designed for
compatibility with a variety of link technologies, e.g., radio, fiber, or
powerline. Fig. 2a illustrates the overall network architecture.

PhoenixSEN consists of a designated control center (CC) and Phoenix nodes
deployed to substations or relay points. The nodes are interconnected with
short-distance and long-distance links. Some of the links can be provided by
third-parties, e.g., the National Guard. PhoenixSEN is designed to be deployed
into geographic areas served by multiple utility companies. The network
provides an isolated virtual network to each utility on top of shared physical
infrastructure. The virtual network spans all utility’s substations. Each
Phoenix node connect its substation local area networks (LANs) to the
appropriate virtual network and also route packets for virtual networks of
other utilities. The CC provides additional equipment and services to
facilitate network monitoring and management, and to support one-way broadcast
communication across the deployment area.

Each Phoenix node provides multiple virtual LANs (VLANs) to its substation.
Typically, there will be one VLAN for SCADA devices and another VLAN for IT
(backend) systems. The addressing architecture of each VLAN is configurable,
allowing it to match the original ISP network in order to minimize the need to
reconfigure existing equipment. Each Phoenix node also provides essential
network services locally to enable communication within the substation even
when disconnected from PhoenixSEN. These services include, among others,
Domain Name System (DNS), Dynamic Host Configuration Protocol (DHCP), Network
Time Protocol (NTP), and Voice over IP (VoIP) signaling.

The primary purpose of PhoenixSEN is to restore connectivity in a grid under
network-based cyber attack. We assume the grid’s devices (SCADA and IT) might
be compromised and may contribute to the attack. For this reason, Phoenix
nodes provide a dedicated access port for a forensic team and services through
which portions of the network or individual devices can be isolated from the
rest of the network. PhoenixSEN also comes with a built-in intrusion detection
system (IDS) that attempts to automatically mitigate certain types of insider
attacks.

###  IV-A Phoenix Node

The Phoenix node is designed to be deployed from a storage facility to
substations by ground transportation or via air lift shortly after a blackout.
All hardware comes in a weather-resistant enclosure which contains all
essential components such as Intel Next Unit of Computing (NUC), Ethernet
switches and cables, GPS, Wi-Fi access points, and VoIP clients. All Phoenix
nodes use a uniform hardware and software architecture to simplify deployment
and installation. Fig. 2b illustrates the hardware architecture, Fig. 3
provides an overview of the software architecture. Fig. 9 shows a small-scale
prototype built for field evaluations.

Figure 3: Phoenix node software architecture. An isolated network environment
(grey) with with all required services is created for each substation LAN. The
environments that belong to the same utility are connected across PhoenixSEN.

The Phoenix node is based on the Intel NUC small form-factor computer. The
operating system (OS) is Ubuntu Linux 16.04 in a minimal configuration. All
custom software for PhoenixSEN is pre-installed in the form of Docker
containers. The containers are built from source code and are specifically
designed to support re-building in the field without internet access, e.g.,
after modifications to fix critical bugs or vulnerabilities.

A newly deployed Phoenix node starts in a minimal configuration. In this
state, the node starts only a deployment manager process and a management
VLAN. Utility and substation specific services are not yet provided. The
deployment manager provides a user interface (UI) for the substation crew to
perform one-time deployment configuration, i.e., to enter the utility name and
substation number. The management VLAN can be used by the CC for remote
support, configuration, or maintenance.

The configuration for utility and substation specific services is generated by
a custom configuration synthesis program out of a model of the whole
PhoenixSEN. The model contains parameters such as number of utilities, number
of substations per utility, desired IP addressing architecture, types of VLANs
and links, etc. In most cases the model can be created prior to PhoenixSEN
deployment based on information collected from utilities in the deployment
region. In that case, the generated configuration library can be pre-loaded
onto each Phoenix node. If the model is unavailable prior to deployment, the
synthesis can be performed at the CC and the generated configuration library
can be distributed to substations on a Universal Serial Bus (USB) memory card,
or the CC can run the synthesis program on Phoenix nodes remotely over the
management network.

Upon receiving the utility name and substation number, the node selects the
appropriate configuration from the configuration library and starts utility
and substation specific services and VLANs. A fully configured Phoenix node
runs one or more isolated virtual network environments. The purpose of the
network environment is to emulate the primary ISP network that connected the
substation to the internet before blackout. Nodes with multiple links also
serve as transparent routers for other utilities sharing the PhoenixSEN. The
virtual network environments are configured for a particular substation and
typically correspond to the substation’s VLANs, e.g., IT systems, VoIP devices
(phones), and SCADA. Each environment is connected to a subset of the ports on
the Ethernet switch. One port connects to the substation LAN, the other ports
connect to link modems or radios. The network environments are implemented
using the Linux networking namespace capability with services provided by
Docker containers.

Each network environment runs its own instance of the Optimized Link State
Routing Protocol (OLSR) agent which publishes the environment’s IP subnet
information to PhoenixSEN. This information is exchanged only with OLSR agents
that belong to the same utility and environment type (e.g., SCADA). For
example, utility A’s SCADA environment on one Phoenix node connects to utility
A’s SCADA environments on all other Phoenix nodes, but not to utility B’s
SCADA environments.

The utilities served by the same PhoenixSEN can use conflicting or overlapping
IP subnets (e.g., 192.168.x.y). Supporting such scenarios natively would
require VLAN support on all PhoenixSEN links. In order to stay compatible with
a wide variety of link layer technologies (including IP-only point-to-point
links), the node does not require VLAN support on links. Instead, it passes
all traffic through a Virtual Extensible LAN (VxLAN) [17] gateway which
encapsulates Ethernet frames in User Datagram Protocol (UDP) packets prior to
transmission. A unique VxLAN virtual network identifier (VNI) is generated for
each utility-VLAN combination during configuration synthesis.

Each network environment provides fully isolated elementary network services
to the corresponding substation VLAN such as DHCP, DNS, or NTP. The DNS
service is described in Section V. Each environment also runs a custom Netmon
agent to discover and identify devices connected to the substation LAN. The
Netmon service is described in more detail in Section VII.

The type of the network environment determines what additional network
services are created. For example, the VoIP environment provides a Session
Initiation Protocol (SIP) [18] server on each Phoenix node to keep the
substation’s phones operational and to provide a means of communication
between the substation personnel and the CC. Please refer to Section VI for
more detail. The SCADA environment provides additional services for intrusion
detection and mitigating insider attacks by compromised devices (Section
VIII).

###  IV-B Deployment & Network Formation

The deployment of PhoenixSEN begins with the deployment of the CC. The CC then
coordinates the deployment of the rest of the network and provides remote
assistance to substations while they are setting up Phoenix nodes. During the
initial deployment phase, the CC may only have one-way broadcast capability,
e.g., a high-power high frequency (HF) radio with voice or low-speed data
support. The CC crew can use the broadcast channel to transmit substation-
specific setup instructions, just enough to get the substation connected. Once
the substation is connected, the CC crew can help configure the substation’s
Phoenix node remotely.

Upon Phoenix node delivery, the substation crew begins setting up the node. We
assume the crew has technical background (e.g., in electric grid engineering),
but not necessarily in IT, communications, or networking. The node includes a
playbook and detailed installation instructions. All parts and connectors are
clearly labeled. The instructions are detailed enough to allow the crew to
independently setup the Phoenix node in a minimal configuration with a low-
bandwidth control connection to the CC. To facilitate this critical first
step, the node enclosure may include a pre-configured HF receiver to receive
broadcast communication from the CC.

When powered on, the Phoenix node begins to search for other Phoenix nodes on
its link interfaces. Each interface has an instance of the OLSR daemon
configured to automatically discover other OLSR-enabled [19] nodes reachable
over the interface. Gradually, Phoenix nodes form an OLSR-based mobile ad hoc
network (MANET) that eventually spans all substations and the CC. Once the
network is formed, the CC can further configure and coordinate deployed
Phoenix nodes over the network.

While forming, PhoenixSEN provides connectivity in phases, gradually providing
additional modes of communication as the system transitions from one phase to
another. The phases are as follows.

  1. 1.

Low-speed, broadcast-only communication from the CC to substations in the
process of deploying a Phoenix node.

  2. 2.

Low-speed command and control connection between the CC and each substation.
The connection provides just enough bandwidth for the CC crew to configure the
Phoenix node remotely.

  3. 3.

The Phoenix node is fully connected to the network, but the overlay may not
have yet fully formed or the node may be in the process of resolving an
addressing conflict. The substation may not be able to reach all other
substations of the same utility yet.

  4. 4.

The node is fully connected and provides VoIP and text communication between
the CC and the substation crew.

Finally, the substation or CC crew perform one-time deployment configuration
by entering the utility name and substation number into the node. This step
can be performed locally at the substation or remotely over the network. The
node uses the entered information to create utility and substation specific
services and VLANs. The process is described in more detail in Section IV-A.

##  V Naming and Service Discovery

For compatibility with existing devices and applications, PhoenixSEN provides
naming and service discovery services based on the DNS. The requirements
outlined in Section II and the fact that PhoenixSEN must remain usable when
disconnected from the internet make the traditional DNS architecture with a
single authoritative master server difficult to use in an ad hoc network. In
PhoenixSEN, each node should be able to independently contribute its own
records into a flat namespace based on a DNS zone shared by all nodes. In this
section, we describe the design of a hybrid peer-to-peer DNS system used in
PhoenixSEN.

The two primary use-cases for DNS in PhoenixSEN are discovering services
offered by Phoenix nodes and supporting third-party tools with pre-existing
Transport Layer Security (TLS) certificates. The first use-case is implemented
in the VoIP and chat service, as described in Section VI, which uses DNS
service discovery to map phone numbers to Phoenix nodes in a distributed
manner. The second case refers to forensic and cyber-security activities
performed with third-party tools. During live exercises, external participants
often needed to be able to connect their tools with mandatory TLS server
certificate validation to PhoenixSEN. The DNS service allows registering
domain names that match pre-existing TLS certificates on a first-come first-
serve basis and resolving those names across PhoenixSEN.

The DNS service takes advantage of the following: 1) a weak eventual
consistency model is sufficient for DNS; 2) in a network like PhoenixSEN, the
impact of conflicts and inconsistencies can be minimized network-wide through
design. Point 1) expands the range of protocols that could be used for DNS
database replication to gossip (epidemic), flooding, and multicast-based
protocols. Point 2) means that if applications and devices are configured
appropriately, the probability of conflicts due to a weak consistency model
will be negligible.

PhoenixSEN provides such DNS subsystem, with no dedicated master server and
without any single point of failure. The subsystem is a hybrid of regular DNS
and multicast DNS [20]. Devices and services can use the subsystem for
network-wide DNS Service Discovery (DNS-SD) [21]. In an ad hoc network such as
PhoenixSEN, built-in service discovery provides a means for applications to
discover services dynamically rather than relying on static configuration.

Dynamic service discovery improves robustness in scenarios where the network
is impaired or only partially formed. A DNS-based service discovery mechanism
allows the reuse of higher layer protocols and services, e.g., TLS server
certificate validation.

Fig. 4 illustrates the architecture of the DNS subsystem. Each Phoenix node
runs the full set of services that make up the DNS subsystem. Thus, DNS is
always available on substation LANs, even if the Phoenix node itself is
isolated from the rest of the network. As more Phoenix nodes join the network,
DNS records gathered from other nodes will become automatically available to
the devices. Where possible, the subsystem uses standardized protocols and
existing (unmodified) open source software.

Figure 4: DNS subsystem architecture. Each Phoenix node runs the same set of
services. The block with gray background shows the VoIP service using the DNS
subsystem for service discovery. Custom-made blocks are highlighted in blue.
Communication via proprietary protocols is highlighted in purple.

Each Phoenix node provides a recursive DNS resolver to substation LANs based
on the Unbound resolver. We chose Unbound because its behavior can be
customized with an external Python program. We use that feature to resolve
ordinary DNS queries via multicast DNS (mDNS) [20] (described later).

Unbound first forwards the query to a local authoritative DNS server
implemented with Knot DNS server. The DNS server is authoritative for the
domain phxnet.org and for a portion of the in-addr.arpa space corresponding to
the IP subnet allocated to the Phoenix node. The domain phxnet.org represents
a namespace shared by all Phoenix nodes. Any node can register a record in the
shared namespace. If multiple nodes register the same record, all such records
are merged when a client attempts to resolve the record. It is left up to the
application to resolve the potential conflict.

If no record is found, Unbound forwards the query to an external Python
module. The module attempts to resolve the query via mDNS using a local Avahi
daemon instance222Avahi is the de facto standard implementation of mDNS in
Linux based OSes.. The Python module and Avahi daemon communicate via a D-Bus
API.

In a typical configuration, the Avahi daemon multicasts DNS queries over LAN
interfaces, e.g., local Ethernet or Wi-Fi interfaces. Since multicast DNS
packets are not forwarded by layer 3 routers, only nodes connected to the same
link can be reached this way. PhoenixSEN is, however, a routed network managed
by an OLSR [19] daemon running on each node. The daemon includes the Basic
Multicast Forwarding (BMF) plugin that enables multicast communication for the
overlay network. We use the plugin to propagate Avahi’s mDNS packets across
the OLSR network. The BMF plugin forwards multicast packets along a spanning
tree covering the entire network, i.e., all Phoenix nodes.

The Avahi daemon on each Phoenix node publishes all DNS records from the local
DNS server via mDNS. This is accomplished with a custom program called “Avahi
Publisher”. The program behaves as a secondary DNS server for the shared zone.
When the zone is updated, Knot sends a DNS NOTIFY request to the program. The
program issues a DNS zone transfer to Knot to download all records and
publishes those records via Avahi’s D-BUS API. Thus, LAN clients that resolve
DNS records via Unbound receive records not only from the local Knot DNS
server, but also from all DNS servers found anywhere in the network.

Devices that obtain an IP address via DHCP are automatically assigned
hostnames by the DHCP server. The DHCP server uses the standard dynamic DNS
(DNS UPDATE) protocol to publish records to the local DNS server. For example,
a DHCP client with the name “foo” will be assigned the hostname
foo.phxnet.org. Having an automatically generated hostname for each device has
two main benefits: 1) the device can be reached by its chosen name across the
network; 2) services running on the device can use TLS certificates with a
wild-card certificate issued by Letsencrypt.

Figure 5: Internal architecture of the VoIP and chat subsystem. The subsystem
can integrate a variety of SIP (red) and WebRTC (green) based VoIP clients.
Each Phoenix node runs a full set of services. VoIP clients register at the
nearest node. Communication sessions that traverse slow links (blue) are
optionally transcoded and compressed. VoIP clients and servers rely on DNS-SD
for server federation and client discovery. IP multicast (used by DNS-SD)
helps make the subsystem fully decentralized and with no single point of
failure.

###  V-A Service Discovery Example

PhoenixSEN provides a SIP-based decentralized VoIP service (highlighted in
gray in Fig. 4) designed to help substation personnel coordinate during a
black-start recovery. The service relies on the PhoenixSEN DNS subsystem for
network-wide service discovery. We discuss how the VoIP service uses the DNS
subsystem in this section.

The VoIP clients at a substation register with the nearest SIP server. Since
each Phoenix node runs a dedicated SIP server, the VoIP clients usually
register with the SIP at their substation’s Phoenix node. To discover the SIP
server, a VoIP client performs the standard procedure described in [22]. The
DNS queries issued by the VoIP client will be forwarded by Unbound to the
local Knot DNS server. The DNS server has the corresponding DNS PTR and SRV
resource records in its zone and those records point the VoIP client to the
SIP server on the local Phoenix node. This way, all VoIP clients across the
whole network can have identical configuration and can, in case of
smartphones, roam between substations.

Upon VoIP client registration, the SIP server publishes a custom DNS record
mapping the VoIP client’s number to the hostname of the SIP server where it
has registered, for example

    4822._voip.phxnet.org. IN CNAME voip-phx23.phxnet.org.

where 4822 is a VoIP client’s phone number and voip-phx23.phxnet.org is the
hostname of the Phoenix node where the client is registered. A custom program
called “DNS Publisher“ (Fig. 4) publishes the record to the local DNS server
via DNS UPDATE and the record is subsequently disseminated across the network
with multicast DNS.

When a remote SIP server receives a SIP INVITE [18] for the VoIP client, it
extracts the called number from the Request-URI and constructs a domain name
from the number within the _voip.phxnet.org suffix. The SIP server then
queries the DNS for the corresponding CNAME record. If a match is found, the
call is forwarded to the SIP identified by the record.

Note that the multicast DNS subsystem is configured to pro-actively
disseminate the requests across the network. Thus, that for existing
(registered) numbers, the query will usually be answered by the local DNS
server from its cache fast. Queries for non-existing (unregistered) numbers
take a few seconds until multicast DNS times out.

Both VoIP clients and servers rely on the DNS service discovery feature to
locate each other. This helps make the VoIP architecture fully decentralized
and scalable, allows (mobile) VoIP clients to roam between Phoenix nodes, and
makes swapping hardware components, e.g., VoIP clients, easy in the case of
compromise or malfunction.

##  VI Voice and Chat

One of the purposes of PhoenixSEN is to facilitate the coordination of people
during an emergency. To that end, the network provides built-in support for
SIP-based [18] real-time voice and text communication. The service supports
the following modalities: two-party calls, multi-party conferencing, text
messaging, and multi-party group chat (with a persistent log). Fig. 5 shows
the internal architecture of the subsystem.

The VoIP subsystem has been designed to support a variety of VoIP clients. IP
digital enhanced cordless communications (DECT) phone represents VoIP clients
that come bundled with the Phoenix node hardware. Each Phoenix node comes with
a couple of IP DECT phones pre-configured for use with the network. The client
labeled as “PC with a web browser” could be any tablet, laptop, or a PC with a
recent web-browser with Web Real-Time Communication (WebRTC) support. The SIP
server includes a JavaScript WebRTC application that can turn any such device
into a fully-featured VoIP and chat client without the need to install any
software. The desk phone icon represents VoIP clients that had existed at the
substation before the Phoenix node was installed. The smartphone VoIP client
represents mobile Android devices with a SIP client application installed. The
mobile clients are connected via a Wi-Fi network created by the Phoenix node
and can roam between substations while keeping the same number.

The VoIP and chat subsystem uses a simple four-number dialing plan where each
substation is allocated a fixed prefix. The substation’s VoIP clients are
assigned numbers from that prefix. Each Phoenix node runs a full set of VoIP
services (SIP and WebRTC servers, conference server, chat server). The clients
register at the nearest SIP server, typically the one located on the Phoenix
node installed in the client’s substation. Having a full set of VoIP services
on each Phoenix node allows the node to function in isolation. Even if the
node is disconnected from PhoenixSEN, calls and chats within its substation
should remain possible.

The subsystem supports ordinary two-way calls between any two VoIP clients,
multi-party conference calls with conference rooms created on demand, peer-to-
peer chat on compatible devices (Android clients), and a web-based group chat
service accessible through the JavaScript WebRTC client. The VoIP service
supports persistent logging and archiving of conversations (both voice and
text), if needed. The SIP server on each Phoenix node performs transparent
media transcoding and enforces authentication and encryption on calls that
traverse PhoenixSEN, i.e., two or more Phoenix nodes. Transcoding helps
establish an upper bound on the VoIP bandwidth on slow (long-haul) links and
improves robustness of calls traversing such links.

A simple four-digit dialing plan with prefixes allocated to substations was
used during live exercises, however, the VoIP subsystem has no fixed dialing
plan hard-coded. Instead, it relies on DNS-SD to discover available SIP
servers and VoIP clients, and to discover the SIP server where a particular
VoIP client is registered. Section V-A describes in detail how the VoIP
subsystem uses DNS-SD. Having a fully dynamic VoIP subsystem has important
benefits: 1) Phoenix nodes have uniform hardware and software configuration
and are interchangeable; 2) mobile VoIP clients roaming from one substation to
another are supported; and 3) the subsystem is fully decentralized with no
single point of failure.

##  VII Network Monitoring

In a geographically distributed ad hoc network architecture such as
PhoenixSEN, real-time network monitoring and situation awareness is key for
successful deployment and operation. In this section, we describe Netmon, a
network monitoring service designed and developed for PhoenixSEN. Netmon
provides near real-time information about the state of the network through a
web-based UI. The UI lets the control center see the topology of the formed
OLSR network and provides alerts when it detects security-related events and
incidents. Fig. 6 shows the UI displaying the network topology graph of a
PhoenixSEN prototype during a live exercise.

Figure 6: A PhoenixSEN topology graph shown by Netmon during a live exercise.
Elements that might require attention are highlighted in red. Hexagonal nodes
represent substation Phoenix nodes. The menus on the right show a list of a
substation’s devices (top) and detail about a particular device (bottom).
Figure 7: Netmon internal architecture. Each Phoenix node runs an agent
process that streams collected data to a backend server in the control center.
Network operators interact with Netmon via a web-based UI implemented in
JavaScript. The architecture provides a near real-time overview of the
network.

The internal architecture of Netmon is shown in Fig. 7. Every Phoenix node
runs a custom Netmon agent process. The agent collects data about the state of
the node using OS-level instrumentation and about directly attached substation
(LAN) devices using active network scanning. It also provides an API for other
processes on the same node, e.g., an IDS. The collected data includes
statistics from selected network interfaces, the state of the node’s OLSR
links, and information about discovered devices.

To discover devices attached to the node’s substation LAN interfaces, the
agent periodically issues Address Resolution Protocol (ARP) requests for all
IP addresses that belong to the interface’s configured IP subnet. A device
that responds to the ARP request is recorded and will be displayed in the
network topology graph, as shown in Fig. 6. Once a LAN device has been
discovered, it is periodically contacted to determine the device’s
reachability. The agent also probes the state of selected UDP and Transmission
Control Protocol (TCP) ports on all discovered LAN devices to see what
services might be offered by the device.

A separate IDS process, running on the same Phoenix node, provides additional
data to the Netmon agent. The data represents potential intrusion detection
incidents and security threats. Netmon transmits the data to the backend where
it is then presented to the network operator by the frontend (UI).

Each agent maintains a persistent WebSocket [23] connection to an agent server
discovered via DNS SRV. The control center Phoenix node provides one agent
server instance for each VLAN. As long as the agent is connected to the
server, data is streamed to the agent server in near real-time. When an agent
gets disconnected from the server, it temporarily stores the collected data in
a local cache. All locally cached data will be uploaded to the server later
once the agent has reconnected.

The agent server stores all the data from all connected agents in a persistent
database. The data is processed and indexed to allow time-based addressing and
aggregation, i.e., retrieving the state of the network at a particular time.
This feature allows debugging or post-mortem analysis after an exercise when
the physical network infrastructure is no longer operational. A backend
server, also running on the control center Phoenix node, serves the data to
the frontend (UI) via a RESTful and WebSocket based API. The API allows the
frontend code to retrieve any data generated by the agents, and to receive
asynchronous (push) updates as new data becomes available.

The UI is a JavaScript application running in a web browser on a laptop in the
control center. The application is specifically designed to always show the
most recent network state without the need to reload the browser window. The
UI is automatically updated based on push notifications received from the
backend server. The UI provides a quick “at a glance” overview of the entire
network, as well as detailed information about individual network components.
The operator can use the UI to quickly scan the network for faulty (red) nodes
and links, or see which parts of the network require immediate attention.

The Netmon agent is implemented in Python, uses Scapy for network probing and
device discovery, and SQLite for the local cache. The agent and backend
servers are both implemented in JavaScript running in NodeJS. All collected
data is stored in a MongoDB database on the control center Phoenix node. The
frontend is a JavaScript application implemented with Vue.js and running in a
web browser.

##  VIII Mitigating Insider Attacks

Insider attacks are of particular concern in large physical infrastructures
such as the electric grid. An attacker could use compromised devices in a
substation LAN to attempt to thwart recovery attempts. If the attacker also
gains physical access to the substation, they could plant a malicious device
in the substation’s network (“device-in-a-closet”) [24] and use the device to
launch man-in-the-middle (MITM) attacks. Lack of Ethernet security combined
with ever shrinking form factor of embedded devices makes such attacks
practical and very hard to discover.

To help mitigate the risk of network-based insider attacks, we designed and
built a prototype device called EtherShield. EtherShield is a “bump in the
wire” embedded device with two Ethernet ports: internal and external. The
internal port is connected to a trusted (not compromised) SCADA device,
preferably, as close to the SCADA device as possible to minimize the risk of
malicious actors getting access to that segment of the network. The external
port is connected to the untrusted substation LAN. Until activated, the device
operates as a regular Ethernet switch. Once activated from the nearest Phoenix
node, the device transparently authenticates all communication between the
SCADA device and the network. Fig. 8 shows the architecture of the EtherShield
subsystem.

Figure 8: The architecture of the EtherShield subsystem. An Ethernet switch-
like device is inserted between a trusted SCADA device and untrusted network.
The device transparently authenticates all Ethernet frames received and sent
by the device. The diagram shows one possible configuration for networks that
support Port-based Network Access Control (IEEE 802.1X).

Each EtherShield device will have been paired with the nearest Phoenix node
via a MITM-resistant USB connection prior to deployment into the LAN. During
pairing, the EtherShield device and the Phoenix node generate the
cryptographic material for secure communication over an untrusted LAN. Until
activated, the EtherShield device passes all traffic between its two ports
unmodified. Once a SCADA device protected with an EtherShield is deemed
secure, the network operator can activate a secure mode in the EtherShield to
isolate the SCADA device from potentially malicious traffic. In a secure mode,
the EtherShield transparently authenticates all packets between the SCADA
device, the nearest Phoenix node, and (optionally) other devices within the
substation LAN.

The authentication is based on either IEEE 802.1X [25] or IPsec [26]. The
EtherShield can be configured to only let authenticated traffic through to the
SCADA device, or to also allow traffic from other LAN devices not protected by
an EtherShield device. This gives the network operator some flexibility while
incrementally securing the substation SCADA network. Critical SCADA devices
can be incrementally protected and isolated from malicious traffic without
disrupting the operation of the rest of the substation LAN.

We built a prototype EtherShield device based on the Raspberry Pi running Open
vSwitch. In the default “open” mode of operation, Open vSwitch is configured
to function as a simple learning Ethernet switch. When switched into a secure
mode, the device starts either an IEEE 802.1X supplicant or an IPsec tunnel
and re-configures Open vSwitch to redirect a portion of the traffic to those
services. For example, in the IEEE 802.1X mode only Extensible Authentication
Protocol over LAN (EAPoL) frames are redirected to the supplicant process.
Frames received from the protected device are transparently augmented with an
authentication header (IEEE 802.1X or IPsec authentication header) as they
pass through the switch. A frame received from the untrusted LAN for the
protected device is first authenticated before the frame is forwarded to the
device. A custom controller application provides a secure HTTP API. This API
can be used to configure and control the device from the nearest Phoenix node.

The Phoenix node provides a standards-based set of services to support the
EtherShield subsystem. The cryptographic material generated during pairing is
kept in a PostgreSQL database, along with account information. The node
provides an IEEE 802.1X authenticator to the substation LAN based on the
Remote Authentication Dial-In User Service (RADIUS) protocol [27]. The
authentication is implemented with FreeRADIUS. An IPsec-based authenticator
based on strongSwan is also provided.

##  IX Related Work

Disaster Communications. The critical role of communication in the aftermath
of a large-scale disaster event became apparent after hurricane Katrina in
2005 [28], hurricane Irene in 2011, and during the 2017 Atlantic hurricane
season [29]. In Puerto Rico, over 11 million people lost electricity for 11
days in the largest blackout in U.S. history [30]. These events rendered
existing communication networks across large geographic areas unusable [31,
10]. Additionally, significant challenges caused by non-interoperable
communication systems were reported [9], limiting situational awareness and
preventing communication across jurisdictions and organizations [32, 33]. A
2016 National Institute of Standards and Technology (NIST) report [34] found
communication failures to be a major obstacle in all recent disaster recovery
efforts.

Industrial Control Systems. Modern industrial control systems, including the
power grid, require communication to function. As critical infrastructure,
such systems are increasingly targets of cyber attacks [35, 36]. A 2011
vulnerability analysis performed by the Idaho National Laboratory found that
applying traditional IT system protective measures to real-time energy
delivery control systems is inadequate and could lead to a power disruption
[7]. The Stuxnet worm [37] and the Maroochy water breach [38] incidents
illustrate the danger network-based cyber attacks pose for industrial control
systems. Even ordinary planned upgrades can have devastating consequences in
systems that control physical processes [39].

Network Architectures. A network architecture suitable for emergency scenarios
will inevitably be ad hoc and temporary [11]. Such architectures have been the
subject of active research for decades and many promising ideas have been
proposed including peer-to-peer, mesh, mobile, delay-tolerant, opportunistic,
and hybrid network architectures [40, 41, 42]. Despite the wide range of
communication architectures, basic interoperability between organizations is
still a problem.

There have been attempts to repurpose existing technology to provide services
where communication networks are unavailable. The Village Telco project [43]
designed and built a do it yourself (DIY) toolkit for voice and text
communications based on wireless mesh networking and VoIP. The Osmocom project
[44] provides the essential building blocks for temporary cellular network
infrastructures.

Disaster Management Systems. Recently, specialized disaster management systems
have been gaining interest. Several organizations specializing in emergency
response have made their solutions available [11, 45, 46, 47]. The core
functionality of these systems centers around crowd-sourced field data
gathering, incident tracking, and visualization. Such systems typically place
no special requirements on the underlying communication networks. There have
been attempts to use advanced technology in emergencies, for example, Panacea
Glass proposes to use Google Glass and cloud computing for situation awareness
and effective triage of patients [48, 49].

Network Monitoring Systems. The ad hoc nature of temporary emergency networks
calls for some form of real-time monitoring for topology discovery, bandwidth
estimation, intrusion detection, and troubleshooting in the field. A vast
number of network measurement and monitoring tools have been developed [50].
The tools can be broadly classified into passive, active, and hybrid (a
combination of active and passive) [51].

Passive monitoring relies on packet capture [52, 53], OS instrumentation [54],
or sampling [55] to perform measurements. Passive methods require existing
network traffic to function, the ability to observe data flows within the
network, and are generally less intrusive than active methods [56]. Notable
passive monitoring tools include CoralReef [57], Bro [58], Wireshark [59],
Snort [60], and sFlow [61].

Active monitoring estimates network properties by observing the handling of
special-purpose data injected into the network by the monitoring tool. Active
methods are generally intrusive, generate variable amounts of artificial
network load, and may interfere with application traffic. Existing active
methods rely on IP options (Internet Control Message Protocol (ICMP) echo
[62], traceroute [63]), TCP congestion control (iPerf [64]), bandwidth and
transmission time probing (BWPing [65]), or packet dispersion measurements
[66, 67]. Scapy [68] is a popular packet manipulation tool for active network
discovery and monitoring.

An important aspect of any network monitoring system is its data processing
architecture. The architecture determines the number and locations of
collection points, types of collected data, collection protocols, and data
processing algorithms. If data collection takes place over the network being
monitored, care must be taken to minimize the generated overhead and its
impact on the monitored network. Well-known monitoring data collection
protocols include Cisco NetFlow [69], the IPFix protocol [70], and Simple
Network Monitoring Protocol (SNMP) [71]. SNMP is widely supported by existing
networking equipment.

The following monitoring tools inspired Netmon, the monitoring tool presented
in this paper: Moloch [72], MRTG [73], OpenNMS [74], Cacti [75], and Zabbix
[76].

##  X Conclusion

We presented the design and prototype implementation of PhoenixSEN, an ad hoc
network architecture specifically designed to enable real-time coordination of
people and (SCADA) devices during an emergency, while the primary network
infrastructure may be inoperable. Our network architecture is designed with
sufficient flexibility to be used a drop-in replacement for network
infrastructure and services typically provided by third-party ISPs. Our work
is primarily motivated by the needs of the power distribution industry during
a hypothetical large-scale blackout triggered by a network-based cyber attack.
We believe PhoenixSEN has the potential to speed up power grid recovery and
service restoration, in particular during a cyber attack that is persistent
and ongoing.

Several iterations of the PhoenixSEN prototype described in this paper were
tested and evaluated through a series of DARPA-led cyber security exercises on
Plum Island, NY [77]. During the exercise, PhoenixSEN was used together with a
variety of power grid recovery tools contributed by other exercise participant
to asses the readiness of the infrastructure to recover from a simulated
large-scale cyber attack on the electrical grid. Photos in Fig. 9 show the
evaluated PhoenixSEN prototype.

(a)

(b)

(c)

Figure 9: Photos of the PhoenixSEN prototype evaluated through DARPA-led
exercises. Left to right: Phoenix node components (Intel NUC, POE switch,
Yealink W52P phone), Phoenix node in weather-resistant enclosure (Intel NUC,
Yealink W52P phone, Netgear Ethernet switch, Tripp Lite UPS), small-scale
Phoenix SEN prototype installation in a lab. The photos were taken by Hema
Retty at the University of Illinois at Urbana-Champaign in September 2018.

This paper describes the most recent iteration of PhoenixSEN. Some of the
features described in this paper ended up not being evaluated in live
exercises. The EtherShield prototype device described in Section VIII was
developed and tested in a lab at Columbia University. The hardware deployed on
Plum Island did not have GPS receivers since the precise location of each
Phoenix node was known in advance. One-time deployment configuration (Section
IV-A) was performed from a laptop attached to the Phoenix node via an Ethernet
port. The configuration synthesis approach was only tested on a Columbia
University testbed. The chat feature (Section VI), which was originally
developed to support team coordination during the exercise, was eventually
replaced with a third-party application the participants were already familiar
with.

###  X-A Future Work

Although the design of PhoenixSEN was primarily motivated by specific needs of
the power distribution industry, we believe the resulting architecture is more
general and flexible, and might be suitable for other emergency scenarios as
well. As part of future work, PhoenixSEN could be extended with features and
services for first responders, disaster recovery, and other emergency
scenarios where communication and coordination is critical even if the primary
communication networks are not operational. We envision supporting a variety
of common emergency and disaster scenarios on top a general-purpose hardware
and software architecture with interchangeable service profiles. Each profile
will activate services designed to meet the needs of a particular emergency
scenario, e.g., incident management, ticketing, collaborative document
editing, and others.

The VoIP subsystem in PhoenixSEN provides all the usual modes of real-time
communication: two-way calls, multi-party conference calls, direct chat, and
group chat. It is likely that PhoenixSEN might get deployed in challenging
scenarios where ordinary calls and chats may not be the most efficient means
of communication. As part of an effort to support more emergency scenarios,
further research into alternative means of (near) real-time communication
might be warranted. Specifically, good support for push-to-talk (PTT)
communication seems to be a promising direction for first responder
communication systems.

Recent advances in automated speech-to-text and text-to-speech technologies
might provide a means to automatically transcribe and index emergency voice
communications. In addition, such technologies might provide better support
for voice communication over long-haul and slow links such as HF band links,
satellite links, and power line communication (PLC) systems. A combination of
Robust Header Compression (RoHC) [78], PTT, transcoding, and voice-to-text
could be potentially used to design a near real-time, highly robust, and
resilient communication system suitable for very challenged communication
networks and scenarios.

We plan to continue developing the Netmon software with features specifically
designed for mobile ad hoc networks. We believe that no other existing network
monitoring solution supports ad hoc or highly dynamic networks well. In a
future version of Netmon, we plan to remove the restriction to have only one
instance of backend services per network. We will aim to make Netmon fully
decentralized, with the backend services and data sharded across all Phoenix
nodes. To build a global view of the network, Netmon will obtain and merge
data obtained from all Phoenix nodes.

## Acknowledgment

The work described in this paper was very much a team effort. The authors
developed the PhoenixSEN prototype as part of a joint team with members from
BAE Systems and Perspecta Labs.

We would also like to thank Clifton Lin, Charles Tao, Defu Li, Ranga Reddy,
and James Dolan, all members of the BAE Systems team, for fruitful discussions
and help in developing and testing the prototype.

We are indebted to Frafos GmbH, the developers of the ABC session border
controller (SBC), who generously provided us with a free license to use the
SBC in our prototype. Furthermore, Frafos engineers were instrumental in
helping us to install and configure the SBC during the early stages of our
project.

## References

  * [1] CNN. (2019, Aug.) Massive failure leaves Argentina, Paraguay and Uruguay with no power. [Online]. Available: https://edition.cnn.com/2019/06/16/world/power-outage-argentina-uruguay-paraguay
  * [2] BBC. (2019, Aug.) Major power failure affects homes and transport. [Online]. Available: https://www.bbc.com/news/uk-49300025
  * [3] National Grid ESO. Black start. [Online]. Available: https://www.nationalgrideso.com/balancing-services/system-security-services/black-start
  * [4] PJM. PJM manual 12: Balancing operations. [Online]. Available: https://www.pjm.com/~/media/documents/manuals/m12.ashx
  * [5] National Institute of Standards and Technology, “NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 3.0,” Sep. 2014. [Online]. Available: http://dx.doi.org/10.6028/NIST.SP.1108r3
  * [6] Defense Advanced Research Projects Agency (DARPA). Rapid Attack Detection, Isolation, and Characterization Systems (RADICS) program. [Online]. Available: https://www.darpa.mil/program/rapid-attack-detection-isolation-and-characterization-systems
  * [7] Idaho National Laboratory. (2011, Sep.) Vulnerability Analysis of Energy Delivery Control Systems (INL/EXT-10-18381).
  * [8] National Grid ESO. (2019, Aug.) Interim Report into the Low Frequency Demand Disconnection (LFDD) following Generator Trips and Frequency Excursion on 9 Aug 2019. [Online]. Available: https://www.nationalgrideso.com/document/151081/download
  * [9] Applied Technology Council, “Critical Assessment of Lifeline System Performance: Understanding Societal Needs in Disaster Recovery,” National Institute of Standards and Technology, Tech. Rep., Apr. 2016. [Online]. Available: http://dx.doi.org/10.6028/NIST.GCR.16-917-39
  * [10] FCC. (2018, Aug.) 2017 Atlantic Hurricane Season Impact on Communications Report and Recommendations Public Safety Docket No. 17-344. [Online]. Available: https://docs.fcc.gov/public/attachments/DOC-353805A1.pdf
  * [11] Red Cross. (2019, Aug.) Red Cross Disaster Services Technology. [Online]. Available: http://www.tucsoncert.info/Red_Cross_DST_slides.pdf
  * [12] F. E. R. Commission, “Order No. 889: Open Access Same-Time Information System and Standards of Conduct,” 1996. [Online]. Available: https://www.ferc.gov/legal/maj-ord-reg/land-docs/order889.asp
  * [13] “IEEE Standard for Electric Power Systems Communications-Distributed Network Protocol (DNP3),” _IEEE Std 1815-2012 (Revision of IEEE Std 1815-2010)_ , pp. 1–821, 2012.
  * [14] International Electrotechnical Commission, “Telecontrol equipment and systems - Part 6-503: Telecontrol protocols compatible with ISO standards and ITU-T recommendations - TASE.2 Services and protocol,” _IEC Std 60870-6-503 TASE.2 Services and protocol_ , 2014.
  * [15] ——, “Communication networks and systems for power utility automation,” _IEC Std 61850_ , 2013.
  * [16] “IEEE Standard for Synchrophasor Measurements for Power Systems,” _IEEE Std C37.118.1-2011_ , 2011.
  * [17] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright, “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” RFC 7348, Internet Engineering Task Force, Aug. 2014. [Online]. Available: http://www.ietf.org/rfc/rfc7348.txt
  * [18] J. Rosenberg, H. Schulzrinne, G. Camarillo, A. Johnston, J. Peterson, R. Sparks, M. Handley, and E. Schooler, “SIP: Session Initiation Protocol,” RFC 3261, Internet Engineering Task Force, Jun. 2002. [Online]. Available: http://www.ietf.org/rfc/rfc3261.txt
  * [19] T. Clausen and P. Jacquet, “Optimized Link State Routing Protocol (OLSR),” RFC 3626, Internet Engineering Task Force, Oct. 2003. [Online]. Available: http://www.ietf.org/rfc/rfc3626.txt
  * [20] S. Cheshire and M. Krochmal, “Multicast DNS,” RFC 6762, Internet Engineering Task Force, Feb. 2013. [Online]. Available: http://www.ietf.org/rfc/rfc6762.txt
  * [21] ——, “DNS-Based Service Discovery,” RFC 6763, Internet Engineering Task Force, Feb. 2013. [Online]. Available: http://www.ietf.org/rfc/rfc6763.txt
  * [22] J. Rosenberg and H. Schulzrinne, “Session Initiation Protocol (SIP): Locating SIP Servers,” RFC 3263, Internet Engineering Task Force, Jun. 2002. [Online]. Available: http://www.ietf.org/rfc/rfc3263.txt
  * [23] I. Fette and A. Melnikov, “The WebSocket Protocol,” RFC 6455, Internet Engineering Task Force, Dec. 2011. [Online]. Available: http://www.ietf.org/rfc/rfc6455.txt
  * [24] C. Haschek. The curious case of the raspberry pi in the network closet. [Online]. Available: https://blog.haschek.at/2018/the-curious-case-of-the-RasPi-in-our-network.html
  * [25] IEEE. (2020) 802.1X: Port-Based Network Access Control. [Online]. Available: https://1.ieee802.org/security/802-1x/
  * [26] A. Farrel and P. Resnick, “Sanctions Available for Application to Violators of IETF IPR Policy,” RFC 6701, Internet Engineering Task Force, Aug. 2012. [Online]. Available: http://www.ietf.org/rfc/rfc6701.txt
  * [27] C. Rigney, S. Willens, A. Rubens, and W. Simpson, “Remote Authentication Dial In User Service (RADIUS),” RFC 2865, Internet Engineering Task Force, Jun. 2000\. [Online]. Available: http://www.ietf.org/rfc/rfc2865.txt
  * [28] U.S. House of Representatives. (2006) A Failure of Initiative: Final Report of the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina. [Online]. Available: https://www.congress.gov/congressional-report/109th-congress/house-report/377/1
  * [29] Fitzpatrick, Leo and Scurato, Carmen and Torres, Joseph. (2019, May) Connecting the Dots. The Telecommunications Crisis in Puerto Rico. Free Press. [Online]. Available: https://www.freepress.net/sites/default/files/2019-05/connecting_the_dots_the_telecom_crisis_in_puerto_rico_free_press.pdf
  * [30] Campbel, Alexia Fernandez. It took 11 months to restore power to Puerto Rico. Vox. [Online]. Available: https://www.vox.com/identities/2018/8/15/17692414/puerto-rico-power-electricity-restored-hurricane-maria
  * [31] Federal Government of the United States. (2006) The Federal Response to Hurricane Katrina: Lessons Learned. [Online]. Available: https://georgewbush-whitehouse.archives.gov/reports/katrina-lessons-learned/
  * [32] M. Stute, M. Maass, T. Schons, M.-A. Kaufhold, C. Reuter, and M. Hollick, “Empirical insights for designing information and communication technology for international disaster response,” _International Journal of Disaster Risk Reduction_ , p. 101598, 2020.
  * [33] L. K. Comfort and T. W. Haase, “Communication, Coherence, and Collective Action: The Impact of Hurricane Katrina on Communications Infrastructure,” _Public Works Management & Policy_, vol. 10, no. 4, pp. 328–343, 2006. [Online]. Available: https://doi.org/10.1177/1087724X06289052
  * [34] L. Johnson, T. D. O’Rourke, S. Chang, C. A. Davis, L. D.-O. I. Robertson, H. Schulzrinne, and K. Tierney, “Critical assessment of lifeline system performance: Understanding societal needs in disaster recovery,” National Institute of Standards and Technology, Gaithersburg, MD, Report NIST CGR 16-917-39, Apr. 2016. [Online]. Available: https://www.nist.gov/sites/default/files/documents/el/resilience/NIST-GCR-16-917-39.pdf
  * [35] The White House. (2009) Remarks by the President on Securing Our Nation’s Cyber Infrastructure. [Online]. Available: https://obamawhitehouse.archives.gov/the-press-office/remarks-president-securing-our-nations-cyber-infrastructure
  * [36] National Electric Sector Cybersecurity Organization Resource (NESCOR. Analysis of Selected Electric Sector High Risk Failure Scenarios. [Online]. Available: https://smartgrid.epri.com/NESCOR.aspx
  * [37] S. Karnouskos, “Stuxnet worm impact on industrial cyber-physical system security,” in _IECON 2011 - 37th Annual Conference of the IEEE Industrial Electronics Society_ , Nov 2011, pp. 4490–4494.
  * [38] J. Slay and M. Miller, “Lessons Learned from the Maroochy Water Breach,” in _International Conference on Critical Infrastructure Protection_. Springer, 2007, pp. 73–82.
  * [39] National Transportation Safety Board. (2019, Aug.) Over-pressure of a Columbia Gas of Massachusetts Low-pressure Natural Gas Distribution System. [Online]. Available: https://www.ntsb.gov/investigations/AccidentReports/Pages/PLD18MR003-preliminary-report.aspx
  * [40] F. Legendre, T. Hossmann, F. Sutton, and B. Plattner, “30 years of wireless ad hoc networking research: what about humanitarian and disaster relief solutions? what are we still missing?” in _Proceedings of the 1st International Conference on Wireless Technologies for Humanitarian Relief (ACWR’11)_ , 2011, pp. 217–217.
  * [41] G. V. Kumar, Y. V. Reddyr, and D. M. Nagendra, “Current research work on routing protocols for MANET: a literature survey,” _international Journal on computer Science and Engineering_ , vol. 2, no. 03, pp. 706–713, 2010\.
  * [42] goTenna. (2019, Aug.) goTenna Mesh Network. [Online]. Available: https://gotenna.com/
  * [43] M. Adeyeye and P. Gardner-Stephen, “The Village Telco project: a reliable and practical wireless mesh telephony infrastructure,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2011, no. 1, p. 78, 2011.
  * [44] The Osmocom Project. (2020) Open Source Mobile Communications. [Online]. Available: http://osmocom.org/
  * [45] NetHope, Inc. NetHope. [Online]. Available: https://nethope.org
  * [46] American Radio Relay League. Amateur Radio Emergency Service (ARES). [Online]. Available: http://www.arrl.org/ares
  * [47] Sahana Foundation. (2020) Open Source Disaster Management Software. [Online]. Available: https://sahanafoundation.org/
  * [48] J. Gillis, P. Calyam, A. Bartels, M. Popescu, S. Barnes, J. Doty, D. Higbee, and S. Ahmad, “Panacea’s glass: Mobile cloud framework for communication in mass casualty disaster triage,” in _2015 3rd IEEE International Conference on Mobile Cloud Computing, Services, and Engineering_. IEEE, 2015, pp. 128–134.
  * [49] D. Jiang, R. Huang, P. Calyam, J. Gillis, O. Apperson, D. Chemodanov, F. Demir, and S. Ahmad, “Hierarchical cloud-fog platform for communication in disaster incident coordination,” in _2019 7th IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud)_ , April 2019, pp. 1–7.
  * [50] Les Cottrell. (2020) Network Monitoring Tools. [Online]. Available: https://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html
  * [51] B. B. Lowekamp, “Combining active and passive network measurements to build scalable monitoring systems on the grid,” _SIGMETRICS Perform. Eval. Rev._ , vol. 30, no. 4, p. 19–26, Mar. 2003. [Online]. Available: https://doi.org/10.1145/773056.773061
  * [52] S. McCanne and V. Jacobson, “The bsd packet filter: A new architecture for user-level packet capture,” in _USENIX winter_ , vol. 46, 1993.
  * [53] The Tcpdump Group. (2020) tcpdump and libpcap. [Online]. Available: https://www.tcpdump.org/
  * [54] J. Salim, H. Khosravi, A. Kleen, and A. Kuznetsov, “Linux Netlink as an IP Services Protocol,” RFC 3549, Internet Engineering Task Force, Jul. 2003. [Online]. Available: http://www.ietf.org/rfc/rfc3549.txt
  * [55] P. Phaal, S. Panchen, and N. McKee, “InMon Corporation’s sFlow: A Method for Monitoring Traffic in Switched and Routed Networks,” RFC 3176, Internet Engineering Task Force, Sep. 2001. [Online]. Available: http://www.ietf.org/rfc/rfc3176.txt
  * [56] W. John, S. Tafvelin, and T. Olovsson, “Passive internet measurement: Overview and guidelines based on experiences,” _Computer Communications_ , vol. 33, no. 5, pp. 533–550, 2010.
  * [57] K. Keys, D. Moore, R. Koga, E. Lagache, M. Tesch, and k. claffy, “The architecture of CoralReef: an Internet traffic monitoring software suite,” in _Passive and Active Network Measurement Workshop (PAM)_. Amsterdam, Netherlands: RIPE NCC, Apr 2001.
  * [58] V. Paxon, “Using bro to detect network intruders: experiences and status,” in _Proceedings of First International Workshop on the Recent Advances in Intrusion Detection_ , 1998.
  * [59] The Wireshark Foundation. (2020) Wireshark. [Online]. Available: https://www.wireshark.org/
  * [60] The Snort Team. (2020) Snort: Network Intrusion Detection and Prevention System. [Online]. Available: https://www.snort.org/
  * [61] InMon Corp. (2020) sFlow. [Online]. Available: https://sflow.org/
  * [62] J. Postel, “Internet Control Message Protocol,” RFC 792, Internet Engineering Task Force, Sep. 1981. [Online]. Available: http://www.ietf.org/rfc/rfc792.txt
  * [63] G. Malkin, “Traceroute Using an IP Option,” RFC 1393, Internet Engineering Task Force, Jan. 1993. [Online]. Available: http://www.ietf.org/rfc/rfc1393.txt
  * [64] iPerf Authors. (2020) iPerf. [Online]. Available: https://iperf.fr/
  * [65] BWPing Authors. (2020) BWPing. [Online]. Available: https://bwping.sourceforge.io/
  * [66] C. Dovrolis, P. Ramanathan, and D. Moore, “What do packet dispersion techniques measure?” in _Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No. 01CH37213)_ , vol. 2. IEEE, 2001, pp. 905–914.
  * [67] R. Kapoor, L.-J. Chen, L. Lao, M. Gerla, and M. Y. Sanadidi, “CapProbe: A Simple and Accurate Capacity Estimation Technique,” in _Proceedings of the 2004 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications_ , ser. SIGCOMM ’04. New York, NY, USA: Association for Computing Machinery, 2004, p. 67–78. [Online]. Available: https://doi.org/10.1145/1015467.1015476
  * [68] The Scapy Community. (2020) Scapy. [Online]. Available: https://scapy.net/
  * [69] B. Claise, “Cisco Systems NetFlow Services Export Version 9,” RFC 3954, Internet Engineering Task Force, Oct. 2004. [Online]. Available: http://www.ietf.org/rfc/rfc3954.txt
  * [70] B. Claise, B. Trammell, and P. Aitken, “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information,” RFC 7011, Internet Engineering Task Force, Sep. 2013. [Online]. Available: http://www.ietf.org/rfc/rfc7011.txt
  * [71] J. Case, M. Fedor, M. Schoffstall, and J. Davin, “Simple Network Management Protocol (SNMP),” RFC 1157, Internet Engineering Task Force, May 1990. [Online]. Available: http://www.ietf.org/rfc/rfc1157.txt
  * [72] Moloch Developers. (2020) Moloch Full Packet Capture. [Online]. Available: https://molo.ch/
  * [73] Tobi Oetiker. (2020) The Multi Router Traffic Grapher. [Online]. Available: https://oss.oetiker.ch/mrtg/
  * [74] The OpenNMS Group. (2020) The OpenNMS Platform. [Online]. Available: https://www.opennms.com/
  * [75] The Cacti Group. (2020) Cacti. [Online]. Available: https://www.cacti.net/
  * [76] Zabbix LLC. (2020) Zabix. [Online]. Available: https://www.zabbix.com
  * [77] Lily Hay Newman. The hail mary plan to restart a hacked US electric grid. [Online]. Available: https://www.wired.com/story/black-start-power-grid-darpa-plum-island
  * [78] K. Sandlund, G. Pelletier, and L.-E. Jonsson, “The RObust Header Compression (ROHC) Framework,” RFC 5795, Internet Engineering Task Force, Mar. 2010. [Online]. Available: http://www.ietf.org/rfc/rfc5795.txt

This research was developed with funding from the Defense Advanced Research
Projects Agency (DARPA). The views and conclusions contained in this document
are those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the Defense Advanced
Research Projects Agency or the U.S. government. Distribution statement A.
Distribution approved for public release, distribution unlimited. Not export
controlled per ES-FL-020821-0013.