††DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
unlimited. This material is based upon work supported by the National
Aeronautics and Space Administration under Air Force Contract No.
FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the National Aeronautics and Space Administration. © 2024
Massachusetts Institute of Technology and the United States Government as
represented by the Administrator of the National Aeronautics and Space
Administration. Delivered to the U.S. Government with Unlimited Rights, as
defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any
copyright notice, U.S. Government rights in this work are defined by DFARS
252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other
than as specifically authorized by the U.S. Government may violate any
copyrights that exist in this work.

# Tradeoffs When Considering Deep Reinforcement Learning for Contingency
Management in Advanced Air Mobility

Luis E<EMAIL_ADDRESS>Technical Staff, Surveillance
Systems Group, 244 Wood Street, Lexington, MA 02421, AIAA Senior Member
Massachusetts Institute of Technology Lincoln Laboratory, Lexington, MA,
02421, USA  Marc W. Brittain222Formerly MIT Lincoln Laboratory, Surveillance
Systems Group, 244 Wood Street, Lexington, MA 02421, AIAA Senior Member
Massachusetts Institute of Technology Lincoln Laboratory, Lexington, MA,
02421, USA  Steven D. Young 333Aerospace Research Engineer, Safety Critical
Avionics System Branch, MS 234, AIAA Fellow. NASA Langley Research Center,
Hampton, VA, 23681, USA

###### Abstract

Air transportation is undergoing a rapid evolution globally with the
introduction of Advanced Air Mobility (AAM) and with it comes novel challenges
and opportunities for transforming aviation. As AAM operations introduce
increasing heterogeneity in vehicle capabilities and density, increased levels
of automation are likely necessary to achieve operational safety and
efficiency goals. This paper focuses on one example where increased automation
has been suggested. Autonomous operations will need contingency management
systems that can monitor evolving risk across a span of interrelated (or
interdependent) hazards and, if necessary, execute appropriate control
interventions via supervised or automated decision making. Accommodating this
complex environment may require automated functions (autonomy) that apply
artificial intelligence (AI) techniques that can adapt and respond to a
quickly changing environment. This paper explores the use of Deep
Reinforcement Learning (DRL) which has shown promising performance in complex
and high-dimensional environments where the objective can be constructed as a
sequential decision-making problem. An extension of a prior formulation of the
contingency management problem as a Markov Decision Process (MDP) is presented
and uses a DRL framework to train agents that mitigate hazards present in the
simulation environment. A comparison of these learning-based agents and
classical techniques is presented in terms of their performance, verification
difficulties, and development process.

##  1 Introduction

The air transportation system is currently undergoing a rapid evolution with
the introduction of novel concepts and the demand for an increase in
operational efficiency. One of those novel concepts is Advanced Air Mobility
(AAM), which foresees the transport of cargo and passengers across cities, as
well as communities currently under-served by aviation. These AAM operations
may include electric vertical take-off and landing (eVTOL) aircraft, high-
density operations, and a combination of piloted (i.e., remote or on-board)
and autonomous flights. Across the world, organizations such as the FAA, NASA,
and EASA expect these operations to increase in density from a single
operation per hour to over 100 simultaneous operations per hour over a local
region [1, 2, 3, 4]. The FAA and NASA refer to the increasing phases of
complexity for AAM operations as Urban Air Mobility (UAM) Maturity Levels
(UML). Low-density human piloted airspace operations are classified as UML 1,
while highly autonomous and dense operations are described as UML 4 and
beyond. Evident in NASA’s and the FAA’s concepts of operations, is an
expectation that some types of operations will transition from human piloted
to fully autonomous operations. Artificial Intelligence (AI) techniques are
viewed as one way to enable higher levels of autonomy because the operations
will need to make safety critical decisions within dynamic, high-density
environments and in the presence of complex hazards, where other algorithmic
approaches are inflexible and difficult to scale. To abate the expected types
of hazards, autonomous systems will need to implement Contingency Management
(CM), a capability described by NASA within the In-Time Aviation Safety
Management Systems (IASMS) concept [5]. Contingency management is the ability
to (1) plan a set contingencies prior to and if necessary during flight, (2)
decide which contingency to execute based on available information and context
(e.g., aircraft or system states, hazard level), and (3) execute the
contingency444Re-planning may be required at various rates depending on
several factors (including the changing flight context).

Contingencies may be defined at various levels of detail, for example, (1) a
procedure to be followed by a pilot or air traffic controller; (2) a change to
the destination (i.e., divert); (3) a revised set of waypoints to traverse;
and (4) a change in airspeed, altitude, or heading. The latter three are often
wrapped into path planning algorithms; which essentially create updated
trajectories in real time. Thus, the prospect of aircraft entering unsafe
states can be mitigated by using path planning algorithms or playbooks
constructed from Mixed Integer Linear Programming (MILP) Optimization. Within
these approaches, a fixed time horizon and dynamic constraints are used to
explore optimal paths. Fundamental to prior work is the use of algorithms such
as Dijkstra, A*, Probabilistic Roadmap, Potential Fields, Rapidly-Exploring
Random Tree (RRT) with approximation of the trajectories through discrete
nodes (i.e., grid or graph nodes), Dubins paths, or trajectories generated
through a fully defined dynamics model [6, 7, 8, 9, 10, 11, 12]. Considering
uncertainty within the path planning algorithms poses several challenges. For
example, A* and Dijkstra rely on deterministic cost functions and heuristics
to provide their convergence guarantees and adaptations for uncertainty can
reduce the quality of solutions. Potential fields are rooted on determinism as
they rely on scalar fields with sources and sinks representing objects in the
environment (e.g., buildings, intruding aircraft, destination, and origin
sites, that each exert a repulsive or attractive force). Modeling dynamic
obstacles (e.g., intruding aircraft, or Temporary Flight Restriction) or
sensor noise requires the transformation of precise position or velocity to be
represented as probability distributions and causes an ever-increasing
computational complexity for the algorithm. At its root, RRT assumes
deterministic state transitions, and research into adaptations of RRT has
provided solutions to include uncertainty; however, computation time may still
be an issue when re-planning in real time [13]. Playbooks derived from MILP
optimization have been extensively investigated, but can also suffer from the
computational time required and symmetric solutions requiring heuristics to
break ties [14, 15]

In recent years, Reinforcement Learning (RL), a subset of Machine Learning
(ML), has shown equivalent or better performance than previously introduced
path planning techniques by directly learning a policy through interaction
with a simulated environment that contains uncertainty in its modeling. A key
enabler leading to the rise in the use of RL to solve challenging problems is
in part due to the ability to synthetically create large amounts of data
through simulated environments. Specifically, in sequential decision-making
problems, a RL approach incorporates uncertainty into the learning process by
modeling the environment as a Markov Decision Process (MDP) and allowing
agents to learn through interaction with the environment. In aviation, RL has
seen applications in air traffic management, for maintaining the separation
distance between aircraft (known as conflict resolution and separation
assurance), scheduling, capacity balancing, and Detect and Avoid (DAA) [16].

Inspired by these achievements, the authors had two primary objectives: (1) to
create a simulation environment or framework in which AI-based CM techniques
may be evaluated, and (2) to implement and evaluate an initial set of
techniques from literature. A prior publication focused on creating the
simulation environment and framework [17]. In this paper, we focus primarily
on the second objective, extending the prior simulation framework and
evaluating trained AI-based versus heuristic path planning techniques. The
remainder of the paper is organized as follows. Section 2 discusses background
information on RL in aviation and their associated challenges. Section 3
outlines the formulation of the training and evaluation simulation
environment. Section 4 provides a description of the algorithm design and
reward modeling of the agents under test. Section 5 discusses results from the
evaluation of each contingency management agent under test. Section 6 provides
a summary of observations made during training and evaluation of the agents.
Section 7 makes recommendation on additional work necessary to refine the
agents developed and path forward.

##  2 Background

Prior research has demonstrated Deep Reinforcement Learning (DRL) systems can
outperform classical approaches in a variety of complex decision-making tasks.
Unlike heuristic-based systems, which use predefined rules and expert
knowledge, DRL systems learn and adapt to new situations through experience
and interaction with the environment. As an example, to support traffic
separation assurance in commercial aviation, researchers have implemented a
variety of approaches to train AI agents to resolve traffic conflicts. Each
implementation models uncertainty as an MDP where the actions are composed of
speed changes, lateral movements, or vertical flight level changes. The
defining differences between implementations, outside of simulator and
uncertainty modeling, are the structure of the state space, action space, and
algorithm used for the learning-based logic. Moreover, when compared to MILP
approaches, which are limited to offline execution due to extensive
computational resources and are constrained by their formulation, DRL systems
offer more flexible and scalable solutions. These advantages make DRL a
promising approach for addressing the dynamic and unpredictable nature of
real-world problems [18, 19, 20, 21, 22].

In the case of DAA, the Federal Aviation Administration (FAA) introduced the
use of machine learning approaches in safety critical systems within the next
generation family of collision avoidance systems known as Airborne Collision
Avoidance System X (ACAS X) for commercial (ACAS Xa), large unmanned (ACAS
Xu), small unmanned (ACAS sXu), and rotor-craft (ACAS Xr) [23, 24, 25, 26].
This family of collision avoidance systems modeled the collision problem as an
MDP where the transition model between states is fully defined and Q-Tables
are solved through value iteration. Due to the “curse of dimensionality”
increasing the memory footprint of these Q-tables, the FAA TCAS Program Office
also conducted research into table compression and fully defining the three-
dimensional collision problem through neural network implementations. However,
these solutions were unable to achieve the equivalent safety and alerting
balance as the published standards with the same training environment and
encounter modeling [27, 28, 29]. Alternative research approaches have used
Proximal Policy Optimization (PPO), Deep Q-Networks (DQN), and double deep
Q-network (DDQN) which have also shown promising results in providing
collision free trajectories with a return to waypoint optimization strategy
[30, 31].

Due to the promising results observed in prior studies, the authors set out to
meet two key objectives:

  1. 1.

Create a simulation environment and overarching framework for rapidly
developing, evaluating, and verifying AI/ML-based techniques as applied to CM
for future In-time Aviation Safety Management Systems (IASMS) [5].

  2. 2.

Use the framework/tool to evaluate a specific set of AI/ML techniques as part
of specific CM designs and associated performance metrics.

If achieved, this will help lead to research-informed standards for
simulator(s)/testbed(s) and performance metrics to evaluate AI approaches that
may be developed in academia, government, and industry. This may also help to
standardize the training and validation of AI/ML-based algorithms for other
functions implemented in aviation [32].

Prior work has led to an initial development framework (i.e., the AAM-Gym
testbed [33, 17]) that is extended here. AAM-Gym allows the user to rapidly
prototype, simulate, and analyze use-cases where novel AI/ML-based safety
management solutions are being investigated. In this work the use-case (or
application) is automated risk mitigation (i.e., auto-CM) for representative
AAM/UAM scenarios. AAM-Gym contains a set of AI/ML-based algorithms that may
be used as-is or tailored by the user, and it allows for inclusion of new
algorithms or logic primitives. Once AL/ML-based algorithms are selected (or
defined), the framework allows the user to choose a set of performance metrics
of interest; define assumed human roles and procedures (if any); and define
the degree of interaction with DAA or other automation that may independently
mitigate other types of hazards not directly considered by the CM design. The
framework allows for consideration of AI/ML-techniques both as a means of
performing CM, and as a tool to help evaluate other means of implementing CM
(e.g., more traditional structured logic-based methods). For example, in this
paper we use AAM-Gym to help assess various means of choosing and executing
contingencies, and we use AAM-Gym to help assess various AI/ML techniques to
help in understanding their limitations and benefits. Lastly, the framework
and simulation tests can help in understanding interoperability issues within
systems of systems (e.g., DAA and CM operating together); this topic is to be
addressed in future work. We feel such a framework is critical to
comprehensive verification and validation of AI/ML-based applications used in
safety-critical aviation systems.

Because of the prior work and lessons-learned on applications of AI/ML to DAA,
it seemed a good starting point for the work described here on CM. Namely,
testing a simulation capability and evaluating an initial set of DRL-based
techniques. The remainder of this paper focuses on work toward Objective 2;
however several lessons-learned during implementation and testing have led to
improvements in the framework (Objective 1).

##  3 Problem Formulation

This section explains the approach to establish the algorithms, models,
scenarios, and metrics used for the testing of the selected CM agents.

###  3.1 DRL Algorithms

Basing the DRL contingency management framework on prior AAM-Gym developments
[33, 17] allowed the research team to quickly adapt and test algorithms that
adopt the OpenAI [34] interface. Two algorithms, D2MAV [35] and Soft Actor
Critic with Discrete actions (SACD) [36] extended with attention, are tested
for training a DRL agent. These algorithms have demonstrated efficacy in
supporting separation assurance in AAM environments [18, 19, 37, 38]. These
algorithms were initially used in the development of the simulation framework
and described in [17].

###  3.2 Baseline Simulation and Heuristic Algorithm

As baselines, this paper uses a heuristic based agent inspired by [39] and
nominal simulations where no agent interacts with the aircraft. The heuristic
based agent has equivalent state space information as the DRL with the
additional knowledge of the nodes that define the corridor network. The
heuristic based agent uses estimates of the energy reserves during flight and
constantly monitors the energy required to continue the flight plan as well as
the probability of loss-of-control111For the tested scenarios, we use the term
loss-of-control to represent the case where there is degraded navigation
system performance such that the vehicle is no longer able to navigate to the
next waypoint within an acceptable safety margin.. The criteria for taking
control of the aircraft is defined by Algorithm 1 in the Appendix and is
summarized as follows:

  1. 1.

Calculate the remaining flight time available and required flight time to
continue route, based on energy reserves.

  2. 2.

Reroute aircraft if the energy required to complete flight plan is greater
than energy reserves.

  3. 3.

Reroute aircraft if it experiences or encounters any hazard along the flight
path.

  4. 4.

Reroute aircraft if there is a hazardous situation at the destination that
would preclude landing (i.e., $P_{H_{dest}}$ is greater than zero).

  5. 5.

Reroute utilizing corridor network if aircraft can sustain flight, and there
is no hazard found along the path.

  6. 6.

If flight path through corridor network is not possible, reroute directly to
closest vertiport whose direct path does not cross a hazardous area or has the
lowest total risk.

###  3.3 Hazard Modeling

The framework allows for the introduction of a variety of hazard types that
may impact aircraft dynamics and state. In this paper the hazards of interest
are 1) wind, 2) no fly zones, 3) regions of high-density population (modeled
as risk fields) 4) increased energy consumption due to the charge cycle of the
battery, and 5) navigation system performance where aircraft experience a
loss-of-control. 3D wind fields are generated using a back-end simulator
(BlueSky [38]). For training initial tasks, a constant zero-wind is used.
Then, as tasks and training get more complex, randomly generated winds fields
are used. This is further discussed in Sec 4.4. A New York City population
density model is extracted from the Oak Ridge National Library population
database and merged with the temporal component of the New York City Taxi and
Limousine Trip Record Data [40, 41]. This population data is used with a
ballistic trajectory model, defined by [39], to estimate the probability of
casualties when aircraft reaches zero energy.

###  3.4 Aircraft Energy Modeling

The aircraft available energy is modeled by the linear energy model defined in
[17]. The model is modified to directly initialize the aircraft with an energy
level defined by a uniform distribution [20, 250] KWh. The energy model uses
three properties to calculate the energy consumed for a given time step: 1)
the battery life cycle for the battery assigned to the aircraft, 2) a constant
energy consumption rate, and 3) a battery failure probability assigned to each
aircraft. The probability of battery failure is an additional feature added to
the energy model to further introduce uncertainty in the likelihood of
aircraft reaching their desired destination even with high energy levels
assigned at the beginning of flight. During agent training these factors on
average lead to total flight times between 2–41 minutes. This range in flight
time allows the DRL algorithms to experience complete energy failure of the
aircraft and learn to divert aircraft to alternative vertiports or return to
the departure vertiport. During evaluation of the algorithms, the range of
initial energy is modified to explore the behavior learned.

###  3.5 Scenario Generation

Training and evaluation of DRL agents requires an accrual of experience by
interacting with a simulation environment to learn the optimal policy. For the
testing reported in this paper, the environment is initialized by randomly
sampling from 40 days of synthetic AAM traffic generated with UAMToolKit, an
event driven AAM traffic capacity and demand modeling tool [42]. The toolkit
assumes a total fleet size of 100 aircraft. Since the focus of this research
is on the cruise/enroute phase of the flight, the UAMToolKit parameters for
capacity at vertiports, passenger capacity, turn-around times, and wind data
are selected to maximize the number of airborne aircraft. Thus, the constraint
on vertiport and passenger capacity is removed by setting it to unlimited
capacity. While wind field information is ignored for traffic generation, and
a one-minute turn-around time is selected to maximize aircraft utilization.
All operations are uniformly distributed in between eight altitude lanes from
1,000-5,000 ft AGL traversing between vertiports through a pre-existing
helicopter network designed by [42]. A uniform distribution is used to assign
an aircraft type with its associated dynamic limits from a set of
representative UAM vehicles whose names are obscured for anonymity. Table 1,
summarizes the aircraft dynamics of each vehicle type.

Table 1: Vehicle power and speed ranges. Aircraft Type | Battery Charge (KWh) | Speed Range (knots)
---|---|---
UAM-A | 20 - 250 | 20 - 174
UAM-B | 20 - 250 | 20 - 156
UAM-C | 20 - 250 | 20 - 148
UAM-D | 20 - 250 | 20 - 130

###  3.6 Metrics

All aircraft state and environment information was recorded for post
processing and evaluation. In addition, the following metrics were calculated
during run time: total reward experienced by each aircraft, aircraft reaching
each terminal state (i.e., reaching destination vertiport, rerouted to
alternate vertiport, return to departure vertiport, experience loss-of-
control, energy depleted), time spent in flight route, contingency actions
taken, latitudes and longitudes traversed, and the projected probability of
casualty when reaching the terminal state.

##  4 Algorithm Design

This section discusses design and training of the CM agents using two
previously developed DRL algorithms. We extend a previous DRL framework [17]
by introducing additional state information, rewards, and an implementation of
a heuristic based approach.

###  4.1 Curriculum Training

Undertaking the development of a DRL agent requires understanding the use case
objectives and fine tuning each reward parameter to achieve the desired
behavior. Inspired by the human learning process, we adopt a curriculum
learning strategy where the agent is sequentially trained through a set of
increasingly complex tasks supportive of CM for UAM operations. Each task
builds upon the previous one, allowing the agents to learn robust contingency
management strategies, as shown in Table 2.

Each agent is pre-loaded with a set of possible vertiport landing sites and
the planned destination. At each time step in a simulated flight, the agent
must decide whether to intervene and reroute the aircraft to an alternate
vertiport, return to the departure vertiport, or continue to its original
destination. In this paper, the factors that influence an agent’s CM decisions
are available power, wind, navigation system performance, no-fly zones, and
underlying population density.

Table 2: Curriculum Learning Tasks. Task | Description
---|---
T1 | Agent learns to reach destination.
T2 | T1 and selects to reroute to alternative vertiport due to reduced energy levels.
T3 | T2 and is cognizant of probability of loss of aircraft control fields and no-fly zones.
T4 | T3 and wind fields are included in the simulations with agent cognizant of wind speeds at current location.
T5 | T4 and is cognizant of population density and selects to navigate through a low population zone as flight conditions worsen.

###  4.2 State Space

Each aircraft, referred to as ownship, in the simulation environment is
controlled by an independent instance of the contingency management agent. The
ownship’s state $s$ at time $t$ is defined by Eq 1.

$s_{t}=O_{t}+V_{t}+h_{t}$ (1)

$O_{t}=(\psi,z,\dot{v_{xy}},v_{xy},E,\dot{E},P_{c},W_{y},W_{x},(x^{(j)}_{\text{wpt}}-x,y^{(j)}_{\text{wpt}}-y,P^{(j)}_{H},\rho^{(j)}_{dest})\quad\forall\;j\in[1,N_{\text{wpt}}]$
(2)

where $O_{t}$ represents ownship’s heading ($\psi$), altitude ($z$),
horizontal acceleration ($\dot{v_{xy}}$), horizontal true air speed
($v_{xy}$), remaining energy ($E$), and energy used within a time step
($\dot{E}$). $P_{c}$ is the probability of casualties in the event of loss of
aircraft control due to complete battery failure or a loss-of-control field.
$W_{y}$ and $W_{x}$ represent the north and east wind speed respectively.
Additionally, $N_{\text{wpt}}$ number of waypoints along the original flight
plan are provided with relative east (X) and north (Y) location to ownship
$(x^{(j)}_{\text{wpt}}-x,y^{(j)}_{\text{wpt}}-y)$, hazard function output at
waypoint ($P^{(j)}_{H}$), and distance to destination from waypoint
($\rho^{(j)}_{dest}$).

$V_{t}=\theta_{dest},\rho_{dest},P_{H_{dest}},(P^{(j)}_{H_{\theta}})\quad\forall\;j\in[1,N_{\theta}+1],$
(3)

$V_{t}$ contains the ownship relative bearing to destination
($\theta_{dest}$), distance to destination ($\rho_{dest}$), hazard function
output at destination ($P_{H_{dest}}$), and $P^{(j)}_{H_{\theta}}$, the hazard
function output of $N_{\theta}$ points within a radius $R_{field}$ from the
ownship, such that these points are evenly distributed in a circle around
ownship and an additional point at the current location of ownship. The angle
between each of these adjacent points is given by:

$\Delta\theta=\frac{360^{\circ}}{N_{\theta}}$

$h_{t}=(\rho^{(i)},\theta^{(i)},P^{(i)}_{H})\quad\forall\;i\in[1,N_{vertiports}]$
(4)

The vertiport state information $h^{(i)}$ at time $t$ includes the relative
distance from ownship to the vertiport ($\rho^{(i)}$), relative bearing to
ownship ($\theta^{(i)}$) and probability of loss of aircraft control at the
vertiport ($P^{(i)}_{H}$), defined by a hazard function.

###  4.3 Reward Model

In prior research we presented a reward model formulated for contingency
management DRL agents [17]. This section discusses extensions of the reward
model and the observed or expected behavior that led to changes in the reward
model. Due to the curriculum training approach undertaken, not all rewards are
active in each curriculum task, and their specific use will be highlighted in
the results. The action space is extended from prior research to include two
additional heading change actions of $\pm$1$\degree$ for the agent to fine
tune the desired heading towards a vertiport, see Table 3.

Table 3: Actions available for agents. Description | Value
---|---
Heading change | -5°, -1°, 0°, 1°, 5°
No action | True, False
Use assigned flight route | True, False

The available actions are defined as follows:

  1. 1.

Heading change - The agent commands a left or right turn by a pre-defined
magnitude (i.e., $\pm$5$\degree$, $\pm$1$\degree$, 0$\degree$) and continues
this heading change until a new action is chosen.

  2. 2.

No action - The agent does not command an action and the aircraft continues on
its current heading or active flight plan.

  3. 3.

Use assigned flight route - The aircraft continues on its planned flight route
or returns to the nearest waypoint of the flight plan, if prior actions caused
a deviation, and continuous along flight route.

The reward function utilized by the MDP is defined by Eq.5, where $R(s_{t})$,
$R(a_{t})$ are defined by Eq. 6, 15.

$R(s_{t},a_{t},p_{t},t)=R(s_{t})+R(a_{t}),$ (5)

$R(s_{t})=\delta{T_{energy}}+\delta{T_{H}}+\delta{T_{vertiport}}+\delta{T_{population}+\Omega}$
(6)

$R(s_{t})$ encompasses the state-dependent penalties and rewards triggered by
an aircraft reaching a specific state, as defined by Eq. 6. One penalty is
applied when an aircraft’s energy reserves, $E$, are below 90 KWh or
completely depleted, as defined by Eq. 7. In addition, when an aircraft’s
energy reserves are depleted, the aircraft is terminated, and also incur a
penalty of $\delta{T_{population}}$, defined by Eq. 13. When an aircraft
enters a loss-of-control field (i.e. navigation performance hazard) or no-fly
zone the probability of terminating the flight is calculated with a hazard
function and compared against a zero-mean Gaussian distribution or a threshold
criteria and incurs a penalty of $\delta{T_{H}}$, as defined by Eq. 8. Where
$D_{H}$ is the distance of the aircraft to the center of the hazard.

$\delta{T_{energy}}=\begin{cases}-2.0&\text{if $E=0.0$ KWh}\\\
-0.0015&\text{if $E<90.0$ and $E>0.0$ KWh}\\\ 0&\text{else}\\\ \end{cases}$
(7)

$\delta{T_{H}}=\\\ \begin{cases}-0.24&\text{if $D_{H}$ < 1,000 m and
$P_{H_{center}}>\beta_{H}$}\\\ -0.12&\text{if $D_{H}$ < 1,000 m and
$P_{H_{center}}\leq\beta_{H}$}\\\ 0&\text{else}\\\ \end{cases}$ (8)

If the hazard region is representing a loss-of-control field, the value of Eq.
9 is compared against Eq. 10 (i.e., Eq. 9 > Eq. 10) to assess whether the
flight should be considered as terminated. In addition, the value of
$\beta_{H}$ in Eq. 8 takes on the output of Eq. 10 and $P_{H_{center}}$ is the
value of Eq. 9 at the aircraft location. The process is equivalent if the
hazard region is representing a no-fly zone, with $\beta_{H}$ taking on the
value of $\beta_{H_{no-fly}}$ to assess whether the flight should be
considered terminated, and the penalty. Lastly, when an aircraft is within a
specified range of a vertiport, $d_{goal}$, the aircraft is terminated and
awarded $\delta{T_{vertiport}}$. The value $\delta{T_{vertiport}}$ is
dependent on the vertiport reached: the destination, an alternative vertiport,
or departure vertiport. The value of $\delta{T_{vertiport}}$ is calculated by
Eq. 11 and Eq. 12. In Eq. 12, $T_{straight\\_path\\_dest}$ represents the time
required to fly directly to the destination, and
$P_{H_{straight\\_path\\_dest}}$ denotes the accumulated risk of flying
directly to the destination.

The multivariate Gaussian distribution defining the hazard region is dependent
on the relative position of the aircraft to the center of the distribution, as
seen in Eq. 9 where the variables are in units of meters defined in the local
east, north, up coordinate system.

The population penalty incurred during energy depletion is calculated by 1)
projecting the aircraft impact location assuming a ballistic trajectory, 2)
querying the population database for the expected individuals at projected
location and 3) assuming a uniform distribution of population within the
100x100 m bin provided by the population database to output a probability of
casualties $P_{c}$, and the fraction of the population impacted $P_{p}$. The
penalty is then calculated using Eq. 13. Lastly, a step penalty, $\Omega$, is
applied that decreases as the aircraft approaches the destination, defined by
Eq. 14.

$P_{H}(X,Y)=e^{\frac{(X-\mu_{xi})^{2}+(Y-\mu_{yi})^{2}}{-2\sigma^{2}}}$ (9)

$\beta_{H_{loss}}=\mathcal{N}(\mu=0.0,\sigma=0.26903^{2})$ (10)

$\delta_{vertiport}=\\\ \begin{cases}1.0&\text{if vertiport = destination}\\\
0.5&\text{if vertiport $\neq$ destination and $\neq$ takeoff location}\\\
\delta_{vertiport\\_return\\_home}&\text{if vertiport = takeoff location}\\\
\end{cases}.$ (11)

$\delta_{vertiport\\_return\\_home}=\\\ \begin{cases}-\frac{E}{1000}&\text{if
$\frac{E}{\dot{E}}>T_{straight\\_path\\_dest}$ and
$P_{H_{straight\\_path\\_dest}}==0$}\\\ 0.5&\text{else}\\\ \end{cases}.$ (12)

$\delta{T_{population}}=\frac{-1}{\log\left(\frac{b-1}{b}\right)}\log(\frac{b-P_{c}P_{p}}{b})$
(13)

$\Omega=\begin{cases}-\delta_{step}\cdot
e^{\frac{-4\cdot\rho_{dest}}{d_{max}}}&\text{if $\rho_{dest}\leq d_{max}$}\\\
-\delta_{step}\cdot e^{-4}&\text{else}\\\ \end{cases}$ (14)

$R(a_{t})$ is defined by

$R(a_{t})=\\\ \begin{cases}\delta_{action\\_penalty}&\text{if $a_{t}\neq$ ``No
action'' or $a_{t}\neq$''Use assigned flight route''}\\\ 0&\text{else}\\\
\end{cases},$ (15)

where $\delta_{action\\_penalty}$ is a penalty applied when any action is
chosen other than “No action” or “Use assigned flight route”.

The final reward model parameters, as shown in Table 4, were individually
tuned by using a manual coordinate descent search.

Table 4: Final Reward Parameters and Values. Parameter | Value
---|---
$N_{\text{wpt}}$ | 2
$N_{\theta}$ | 20
$N_{vertiports}$ | 29
$d_{\text{max}}$ | 647,391.47 m
$\delta_{\text{action\\_penalty}}$ | -0.001
$\delta_{\text{step}}$ | -0.0001
$\delta{T_{energy}}$ | -2.0
$\delta{p_{max}}$ | -0.00015
$\sigma$ | 269.023
$\beta_{H_{no-fly}}$ | 0.2
$P_{H_{threshold}}$ | 0.2
$b$ | 0.001
$R_{field}$ | 500 m
$d_{goal}$ | 500 m

###  4.4 Experiment Setup

A span of tests were designed to help understand and advance the overarching
framework (i.e., Objective 1) as well as to evaluate the set of CM agents
previously described (Objective 2). The Lincoln Laboratory Supercomputer
Center (LLSC) was utilized in this study to meet the substantial data
requirements for training and evaluating the DRL agents. The compute
architecture comprised 16 Intel Xeon Gold 6248 2.5 GHz compute nodes, each
equipped with two NVIDIA Tesla V100 graphics processing units (GPUs) [43].
Each curriculum task listed in Table 2 necessitated the careful introduction
of rewards and state inputs to achieve satisfactory performance. This
performance was attained by training a DRL agent with 40 distinct parallel
simulations, each independently running 3-hour windows of operations. Training
at each curriculum task involved 20,000 iterations, with each iteration
consisting of 64 training steps (equivalent to 5 simulation seconds) per 40
distinct parallel simulation. Consequently, each curriculum task required an
average of 68 hours of wall-clock time.

In curriculum task T3, the centers of the loss-of-control and no-fly zone
hazard regions are randomly initialized to align with a node in the network as
designed in [42]. Wind speed and direction are randomly sampled from
independent uniform distributions ranging in [0, 10] knots and [0, 360]
degrees. The velocity of aircraft is initialized using a uniform distribution
with velocities ranging between [5, 65] knots. A total of 100 simulated days
of operations are used for evaluation of each algorithm. Each evaluation
contains a maximum of 100 aircraft simultaneously operating in the airspace.
The results show the mean value of the metrics and two standard deviations
($\sigma_{2}$) from the mean (bootstrapped). The loss-of-control hazard region
is held static in the evaluation to force a majority of the nominal traffic to
traverse through the hazard, as seen in Figure 1.

Figure 1: Network utilized for evaluation with loss-of-control hazard to force
the majority of nominal traffic to traverse through hazard. Vertiport
locations displayed with red circles, and nodes defining network segments
displayed with green circles. Colormap indicates the distribution of hazard
severity within the hazard region (red, more severe; blue, less severe).*

##  5 Results

44footnotetext: Map data from OpenStreetMap openstreetmap.org/copyright

In this section we discuss the observed performance of each agent versus
nominal operations (i.e. unequipped). Figures 2 to 5 summarize collected data;
which includes, for each agent, 100 evaluations repeatedly simulated with a
unique random seed used for all agents. Over 798 million operations providing
over 266 million hours of flight data were collected for the DRL agents to
achieve the performance through the curriculum learning. Overall when the
aircraft fleet is initialized with energy reserves sampled from a uniform
distribution equivalent to the training scenarios, 20–250 KWh, the DRL based
agent successfully reroutes an average of 95.2% ($\sigma_{2}=~{}\pm$ 1.08%)
and 91.6% ($\sigma_{2}=~{}\pm$ 1.29%) of aircraft to vertiports, for SACD-A
and D2MAV-A respectively. The heuristic agent outperforms the nominal
(unequipped) operations, however it only successfully reroutes 76.1%
($\sigma_{2}=~{}\pm$ 6.07%) to a vertiport as compared to (46.2%) for the
nominal operations. Closer inspection of Fig. 2a reveals that each agent
maintains a comparable number of aircraft reaching their destination as in the
nominal case with SACD-A rerouting more aircraft to their destination if the
original flight plan is unfeasible. Since all agents only have
$N_{\text{wpt}}$ of flight path information they must utilize state
information near ownship to sense regions of loss-of-control (i.e.,
$P^{(j)}_{H}$, $P_{H_{dest}}$ as defined in Section 4). While the heuristic
agent allows for tracks to use routes with a risk of loss-of-control below
$P_{H_{threshold}}$ to reduce situations where no route is possible, more
aircraft are found to experience loss-of-control than $P_{H_{threshold}}$ and
reducing this value leads to a tradeoff in more aircraft experiencing energy
depletion due to the longer flight paths selected. This tradeoff is due to a
lack of global information resulting in sub-optimal routing causing 21.8%
($\sigma_{2}=~{}\pm$ 5.612%) of aircraft to experience loss-of-control. The
SACD-A nearly eliminates loss-of-control risk with only 0.07%
($\sigma_{2}=~{}\pm$ 0.119%) of aircraft encountering it. This is achieved by
SACD-A exploiting the state space information in training, Fig. 3, and
rerouting aircraft around the hazard region, as seen in Fig. 4. Even with
intensive training, Fig. 4 shows the flight trajectories produced by the DRL
agents can contain unexpected characteristics that may not make them
operationally suitable. The sampling efficiency of SACD-A also leads it to
outperform the D2MAV-A algorithm, a similar outcome to prior comparison [38].
To further evaluate the differences in performance, an evaluation of scenarios
where aircraft are initialized with 20–90 KWh was conducted (i.e., less energy
reserves). Figure 2b demonstrates that the performance of the DRL agent is
robust to the reduction in aircraft energy as the results are similar to those
of Fig. 2a with an increase in aircraft experiencing an "Out of Energy"
terminal state.

(a) Evaluations with full range of energy reserves.

(b) Stressing evaluations with reduced energy reserves.

Figure 2: Comparison of the average performance of 100 evaluations with the
heuristic, SACD-A, D2MAV-A, and unequipped agents. Subplot (a) includes
aircraft initialized with 20-250 KWh, while (b) includes only aircraft
initialized with 20-90 KWh.

(a) Curriculum T4 learning curves of SACD-A agent.

(b) Curriculum T4 learning curves of D2MAV-A agent.

Figure 3: Final training of (a) SACD-A, and (b) D2MAV-A agents for curriculum
task T4 where wind is introduced. High fraction of aircraft reaching a
vertiport ($\sim$0.8) at iteration 0 is due to the application of transfer
learning from previous curriculum tasks. Rolling mean of 100 iterations used
to remove noise in plotting of data. Figure 4: Example case where DRL reroutes
may be operationally unsuitable.*

Although the DRL agents outperform the heuristic agent in terms of aircraft
reaching the original destination and leading more aircraft to a vertiport,
the reward function required careful tuning to balance each additional
objective as the agents progressed through the curriculum learning tasks. A
known issue with DRL agents is called “catastrophic forgetting”, where an
agent loses previously acquired knowledge while learning a new objective. This
phenomenon is prevalent in complex, dynamic environments, where continuous
adaptation is required. This phenomenon was experienced between curriculum
task T2–T3 when introducing loss-of-control regions as the logic transitioned
from optimizing reduction in loss-of-control to further reduce the fraction of
“out of energy” aircraft, as seen in Fig. 5. A common way to reduce the
likelihood of catastrophic forgetting is to establish an experience replay,
where a simulation buffers past experience [44]. To reduce the instance of
catastrophic forgetting, a replay buffer was implemented in this study,
storing originally 4 million simulations steps when the phenomenon occurred,
and increased to 8 million simulation steps thereafter. This was only possible
due to the 364 GB of random-access memory available in the LLSC compute nodes.
Moreover, during training of an agent, a rolling average of the reward
function was used to assess the performance of the agent in training as well
as a full evaluation of 10 days of simulation every 1000 training iterations.
Network weights were only updated when these evaluation and training
iterations would produce a higher average reward from a previous checkpoint.
44footnotetext: Map data from OpenStreetMap openstreetmap.org/copyright

Figure 5: Example of catastrophic forgetting experienced during transition
from T2 to T3 curriculum task training at iteration $\sim$13000.

##  6 Conclusion

A set of learning-based in-time decision-making agents were trained and
evaluated as supportive of automation CM for AAM operations in the presence of
hazards. The learning-based agents show DRL can meet complex safety objectives
and are robust to uncertainty in the environment. When compared to a heuristic
based approach, the DRL agents have a lower standard deviation in decision-
making variability; with results within two standard deviations varying by
less than 2% as compared to 6% indicating a higher level of understanding of
the uncertainties in the environment. These results show promise for agents
that can achieve both safety and operational efficiency objectives. To achieve
the performance shown, over 798 million operations providing over 266 million
hours of flight experience were collected for the DRL agents throughout the
entire curriculum. This large amount of training experience highlights a
tradeoff with expert knowledge-based heuristic systems. Careful tuning is also
required to avoid catastrophic forgetting during training. Additionally, a
notable tradeoff of using learning-based approaches is the challenge of
achieving safety-critical assurance levels for ML-based agents. Rigorous
verification and validation processes are needed, both for the agents
themselves and the tools used to test them (e.g., the simulation environment).
To help with this challenge, this work presents a framework to design, test,
and evaluate a variety of algorithm approaches at scales previously not
possible; and collect evidence such as suggested by [32]. As aviation moves
towards increasingly autonomous systems, the development of comprehensive
validation frameworks is essential to ensure safety and reliability.

##  7 Future Work

Future research will focus on enhancing the robustness and efficacy of
learning-based approaches in contingency management systems for AAM. Firstly,
we will integrate probability of casualty and the real-time risk assessment
method suggested by [39] to the heuristic agent and compare the performance
versus the DRL agents. Additionally, we intend to extend the range of hazards
to include dynamic hazards and multiple hazards within the environment.
Lastly, we aim to evaluate the interoperability of the agents with collision
avoidance agents. This is needed to ensure seamless and appropriate
coordination when independent autonomous agents may affect flight.

## Acknowledgment

The work was funded by NASA’s System-Wide Safety Project via an Interagency
Agreement with the US Air Force (IA 80LARC23TA002, Project 10384).

## Appendix

Algorithm 1 Heuristic Agent Algorithm

1:Input: Environment State Space

2:Input: Corridor Network Nodes (latitude, longitude, altitudes)

3:Output: Reroute or Continue Flight Path

4:Environment State Space: $S_{t}$ $\triangleright$ see Eq. 1 for definitions

5:Flight Time Remaining: $T_{remaining}\leftarrow E/\dot{E}$

6:Flight Time Required: $T_{required}\leftarrow\rho_{dest}/v_{xy}$

7:Probability of Loss of Control: $P_{H}\leftarrow
any(P^{(j)}_{H}>P_{H_{threshold}},P_{H_{dest}}>P_{H_{threshold}})$

8:$IsReRouteRequired(S_{t},network,T_{remaining},T_{required},P_{H})$

9:function RerouteInNetwork($S_{t}$, network)

10: $\rho_{\text{min}}\leftarrow\infty$ $\triangleright$ Initialize the
shortest route’s distance to infinity

11: $r_{P_{H}}\leftarrow\infty$ $\triangleright$ Initialize the shortest
route’s probability of loss-of-control

12: $r_{selected}\leftarrow None$ $\triangleright$ Initialize route selected
as None

13: for $vpt\in Veriports$ do

14: $R,R_{\rho},R_{P_{H}}\leftarrow Dijkstra(\text{closest node to aircraft
location},vpt,network)$ $\triangleright$ Find route to vertiport $R$, path
distance $R_{\rho}$ and output probability of loss-of-control of aircraft
$R_{P_{H}}$

15: if $R_{P_{H}}\leq P_{H_{threshold}}$ and $R_{\rho}<\rho_{\text{min}}$ then

16: $\rho_{\text{min}}\leftarrow R_{\rho}$

17: $r_{P_{H}}\leftarrow R_{P_{H}}$

18: $r_{selected}\leftarrow R$ return $r_{selected}$

19:function StraightReroute($S_{t}$, network) $\triangleright$ Find shortest
closest vertiport with no hazard in straight path

20: $\rho_{\text{min}}\leftarrow\infty$ $\triangleright$ Initialize the
shortest distance to infinity

21: $r_{P_{H}}\leftarrow\infty$ $\triangleright$ Initialize path’s probability
of loss-of-control

22: $vpt_{selected}\leftarrow None$ $\triangleright$ Initialize route selected
as None

23: for $vpt\in Veriports$ do

24: $R,R_{\rho},R_{P_{H}}\leftarrow Distance(\text{aircraft location},vpt)$
$\triangleright$ Straight route $R$, distance $R_{\rho}$ and Hazard in route
$R_{P_{H}}$

25: if $R_{P_{H}}\leq P_{H_{threshold}}$ and $R_{\rho}<\rho_{\text{min}}$ then

26: $\rho_{\text{min}}\leftarrow R_{\rho}$

27: $r_{P_{H}}\leftarrow R_{P_{H}}$

28: $r_{selected}\leftarrow R$ return $r_{selected}$

29:function IsRerouteRequired($S_{t}$, network, $T_{remaining}$,
$T_{required}$, $P_{H}$)

30: if $T_{required}>T_{remaining}$ or $P_{H}$ then

31: $r_{selected}\leftarrow RerouteInNetwork(S_{t},network)$

32: if $r_{selected}\textbf{is}None$ then

33: $r_{selected}\leftarrow StraightReroute(S_{t},network)$

34: Reroute using $r_{selected}$

35: else

36: Don’t Take Control

## References

  * National Aeronautics and Space Administration [2020] National Aeronautics and Space Administration, “National Aeronautics and Space Administration (NASA) UAM Vision Concept of Operations (ConOps) UAM Maturity Level (UML)4,” https://ntrs.nasa.gov/api/citations/20205011091/downloads/UAM%20Vision%20Concept%20of%20Operations%20UML-4%20v1.0.pdf, 2020\.  Accessed on 2023-10-04.
  * Federal Aviation Administration [2022] Federal Aviation Administration, “Urban Air Mobility Concept of Operations version 2.0,” , 2022.  URL https://www.faa.gov/sites/faa.gov/files/UrbanAirMobilityUAMConceptofOperations2.0_0.pdf.
  * Barrado et al. [2020] Barrado, C., Boyero, M., Brucculeri, L., Ferrara, G., Hately, A., Hullah, P., Martin-Marrero, D., Pastor, E., Rushton, A. P., and Volkert, A., “U-space concept of operations: A key enabler for opening airspace to emerging low-altitude operations,” _Aerospace_ , Vol. 7, No. 3, 2020, p. 24.
  * Ministry of Land, Infrastructure, Transport and Tourism, Committee for Advanced Air Mobility [2023] Ministry of Land, Infrastructure, Transport and Tourism, Committee for Advanced Air Mobility, “Concept of Operations for Advanced Air Mobility,” , March 2023.  URL https://www.mlit.go.jp/common/001611491.pdf.
  * Ellis et al. [2021] Ellis, K. K., Krois, P., Koelling, J., Prinzel, L. J., Davies, M., and Mah, R., “A Concept of Operations (ConOps) of an in-time aviation safety management system (IASMS) for Advanced Air Mobility (AAM),” _AIAA Scitech 2021 Forum_ , 2021, p. 1978.
  * Dreyfus [1969] Dreyfus, S. E., “An appraisal of some shortest-path algorithms,” _Operations research_ , Vol. 17, No. 3, 1969, pp. 395–412.
  * Lozano-Perez [1990] Lozano-Perez, T., _Spatial planning: A configuration space approach_ , Springer, 1990.
  * Barraquand et al. [1992] Barraquand, J., Langlois, B., and Latombe, J.-C., “Numerical potential field techniques for robot path planning,” _IEEE transactions on systems, man, and cybernetics_ , Vol. 22, No. 2, 1992, pp. 224–241.
  * Melchior and Simmons [2007] Melchior, N. A., and Simmons, R., “Particle RRT for path planning with uncertainty,” _Proceedings 2007 IEEE International Conference on Robotics and Automation_ , IEEE, 2007, pp. 1617–1624.
  * Lugo-Cárdenas et al. [2014] Lugo-Cárdenas, I., Flores, G., Salazar, S., and Lozano, R., “Dubins path generation for a fixed wing UAV,” _2014 International conference on unmanned aircraft systems (ICUAS)_ , IEEE, 2014, pp. 339–346.
  * Kim et al. [2021] Kim, J., Sharma, P., Atkins, E., Neogi, N., Dill, E., and Young, S., “Assured Contingency Landing Management for Advanced Air Mobility,” _2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC)_ , 2021, pp. 1–12.  10.1109/DASC52595.2021.9594498.
  * Haghighi et al. [2022] Haghighi, H., Delahaye, D., and Asadi, D., “Performance-based emergency landing trajectory planning applying meta-heuristic and Dubins paths,” _Applied Soft Computing_ , Vol. 117, 2022, p. 108453.  https://doi.org/10.1016/j.asoc.2022.108453, URL https://www.sciencedirect.com/science/article/pii/S1568494622000242.
  * Jiang et al. [2021] Jiang, C., Hu, Z., Mourelatos, Z. P., Gorsich, D., Jayakumar, P., Fu, Y., and Majcher, M., “R2-RRT*: Reliability-based robust mission planning of off-road autonomous ground vehicle under uncertain terrain environment,” _IEEE Transactions on Automation Science and Engineering_ , Vol. 19, No. 2, 2021, pp. 1030–1046.
  * Saber and Feron [2023a] Saber, S., and Feron, E., “Optimized Escape Path Planning for Commercial Aircraft Formations,” _Journal of Guidance, Control, and Dynamics_ , 2023a, pp. 1–16.
  * Saber and Feron [2023b] Saber, S. I., and Feron, E. M., “Robust Emergency Aircraft Avoidance Solutions for Commercial Aircraft Formations,” _AIAA AVIATION 2023 Forum_ , 2023b, p. 3260.
  * Razzaghi et al. [2022] Razzaghi, P., Tabrizian, A., Guo, W., Chen, S., Taye, A., Thompson, E., Bregeon, A., Baheri, A., and Wei, P., “A survey on reinforcement learning in aviation applications,” _arXiv preprint arXiv:2211.02147_ , 2022\.
  * Alvarez et al. [2023] Alvarez, L. E., Brittain, M., and Breeden, K., “Towards a Standardized Reinforcement Learning Framework for AAM Contingency Management,” _2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC)_ , IEEE, 2023, pp. 1–6.
  * Brittain and Wei [2019] Brittain, M., and Wei, P., “Autonomous Separation Assurance in An High-Density En Route Sector: A Deep Multi-Agent Reinforcement Learning Approach,” _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , 2019, pp. 3256–3262.  10.1109/ITSC.2019.8917217.
  * Brittain and Wei [2021] Brittain, M. W., and Wei, P., “One to Any: Distributed Conflict Resolution with Deep Multi-Agent Reinforcement Learning and Long Short-Term Memory,” _AIAA Scitech 2021 Forum_ , 2021.  10.2514/6.2021-1952, URL https://arc.aiaa.org/doi/abs/10.2514/6.2021-1952.
  * Pham et al. [2019] Pham, D.-T., Tran, N. P., Goh, S. K., Alam, S., and Duong, V., “Reinforcement learning for two-aircraft conflict resolution in the presence of uncertainty,” _2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF)_ , IEEE, 2019, pp. 1–6.
  * Wang et al. [2019] Wang, Z., Li, H., Wang, J., and Shen, F., “Deep reinforcement learning based conflict detection and resolution in air traffic control,” _IET Intelligent Transport Systems_ , Vol. 13, No. 6, 2019, pp. 1041–1047.
  * Hong et al. [2014] Hong, Y., Kim, Y., and Lee, K., “Application of complexity map to reduce air traffic complexity in a sector,” _AIAA Guidance, Navigation, and Control Conference_ , 2014, p. 1461.
  * Kochenderfer and Chryssanthacopoulos [2013] Kochenderfer, M. J., and Chryssanthacopoulos, J. P., “Collision avoidance using partially controlled Markov decision processes,” _Agents and Artificial Intelligence_ , Vol. 271, edited by J. Filipe and A. Fred, Springer, 2013, pp. 86–100.  10.1007/978-3-642-29966-7_6.
  * Owen et al. [2019] Owen, M. P., Panken, A., Moss, R., Alvarez, L., and Leeper, C., “ACAS Xu: Integrated collision avoidance and detect and avoid capability for UAS,” _Digital Avionics Systems Conference (DASC)_ , IEEE, 2019, pp. 1–10.
  * Alvarez et al. [2019] Alvarez, L. E., Jessen, I., Owen, M. P., Silbermann, J., and Wood, P., “ACAS sXu: Robust decentralized detect and avoid for small unmanned aircraft systems,” _Digital Avionics Systems Conference (DASC)_ , IEEE, 2019, pp. 1–9.
  * Katz et al. [2022] Katz, S. M., Alvarez, L. E., Owen, M., Wu, S., Brittain, M. W., Das, A., and Kochenderfer, M. J., “Collision Risk and Operational Impact of Speed Change Advisories as Aircraft Collision Avoidance Maneuvers,” _AIAA AVIATION 2022 Forum_ , 2022, p. 3824.  10.2514/6.2022-3824, URL https://arc.aiaa.org/doi/abs/10.2514/6.2022-3824.
  * Julian et al. [2016] Julian, K. D., Lopez, J., Brush, J. S., Owen, M. P., and Kochenderfer, M. J., “Policy compression for aircraft collision avoidance systems,” _2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC)_ , IEEE, 2016, pp. 1–10.
  * Corteguera [2020] Corteguera, O. L., “Airborne collision avoidance with three-dimensional policy,” Ph.D. thesis, Massachusetts Institute of Technology, 2020.
  * Cone et al. [2023] Cone, C., Owen, M., Alvarez, L. E., and Brittain, M. W., “Reward Function Optimization of a Deep Reinforcement Learning Collision Avoidance System,” _AIAA SCITECH 2023 Forum_ , 2023, p. 2155.  10.2514/6.2023-2155, URL https://arc.aiaa.org/doi/abs/10.2514/6.2023-2155.
  * Li et al. [2019] Li, S., Egorov, M., and Kochenderfer, M., “Optimizing collision avoidance in dense airspace using deep reinforcement learning,” _arXiv preprint arXiv:1912.10146_ , 2019.
  * Panoutsakopoulos et al. [2022] Panoutsakopoulos, C., Yuksek, B., Inalhan, G., and Tsourdos, A., “Towards safe deep reinforcement learning for autonomous airborne collision avoidance systems,” _AIAA SCITECH 2022 Forum_ , 2022, p. 2102.
  * Agogino et al. [2024] Agogino, A., Brat, G., He, Y., Hulse, D., Lipkis, R. a., Pressburger, T., Gopinath, D., Lukman, I., Katis, A., Mavridou, A., Pai, G., Pasareanu, C., Perez, I., and Schumann, J., “Recommendations on Evidence and Process for Certification of Learning-enabled Components in Aerospace Systems,” _NASA_ , 2024.
  * Brittain et al. [2022] Brittain, M., Alvarez, L. E., Breeden, K., and Jessen, I., “AAM-Gym: Artificial Intelligence Testbed for Advanced Air Mobility,” _2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC)_ , 2022, pp. 1–10.  10.1109/DASC55683.2022.9925762.
  * Brockman et al. [2016] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W., “OpenAI Gym,” , 2016.  10.48550/ARXIV.1606.01540, URL https://arxiv.org/abs/1606.01540.
  * Brittain et al. [2021] Brittain, M. W., Yang, X., and Wei, P., “Autonomous Separation Assurance with Deep Multi-Agent Reinforcement Learning,” _Journal of Aerospace Information Systems_ , Vol. 18, No. 12, 2021, pp. 890–905.  10.2514/1.I010973, URL https://doi.org/10.2514/1.I010973.
  * Christodoulou [2019] Christodoulou, P., “Soft actor-critic for discrete action settings,” _arXiv preprint arXiv:1910.07207_ , 2019.
  * Brittain and Wei [2022] Brittain, M., and Wei, P., “Scalable Autonomous Separation Assurance With Heterogeneous Multi-Agent Reinforcement Learning,” _IEEE Transactions on Automation Science and Engineering_ , 2022, pp. 1–12.  10.1109/TASE.2022.3151607.
  * Brittain et al. [2024] Brittain, M. W., Alvarez, L. E., and Breeden, K., “Improving autonomous separation assurance through distributed reinforcement learning with attention networks,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 38, 2024, pp. 22857–22863.
  * Ancel et al. [2022] Ancel, E., Young, S. D., Quach, C. C., Haq, R. F., Darafsheh, K., Smalling, K. M., Vazquez, S. L., Dill, E. T., Condotta, R. C., Ethridge, B. E., Teska, L. R., and Johnson, T. A., “Design and Testing of an Approach to Automated In-Flight Safety Risk Management for sUAS Operations,” 2022, p. 3459\.  10.2514/6.2022-3459, URL https://arc.aiaa.org/doi/abs/10.2514/6.2022-3459.
  * Laboratory [2023] Laboratory, O. R. N., “Oak Ridge National Laboratory Population Database,” , 2023.  URL https://landscan.ornl.gov/, accessed 12/5/2023.
  * Commission [2023] Commission, N. Y. C. T. . L., “New York City Trip Record Data,” , August 2023.  URL https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page, accessed 12/5/2023.
  * Alvarez et al. [2021] Alvarez, L. E., Jones, J. C., Bryan, A., and Weinert, A. J., “Demand and Capacity Modeling for Advanced Air Mobility,” 2021, p. 2381.  10.2514/6.2021-2381, URL https://arc.aiaa.org/doi/abs/10.2514/6.2021-2381.
  * Reuther et al. [2018] Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jones, M., Klein, A., Milechin, L., Mullen, J., Prout, A., Rosa, A., Yee, C., and Michaleas, P., “Interactive supercomputing on 40,000 cores for machine learning and data analysis,” _2018 IEEE High Performance extreme Computing Conference (HPEC)_ , IEEE, 2018, pp. 1–6.
  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al., “Human-level control through deep reinforcement learning,” _nature_ , Vol. 518, No. 7540, 2015, pp. 529–533.  10.1038/nature14236.