# Probabilistic Perspectives on Error Minimization in Adversarial
Reinforcement Learning

Roman Belaire
Singapore Management University
Singapore
<EMAIL_ADDRESS>
&Arunesh Sinha
Rutgers University
New Brunswick, NJ
<EMAIL_ADDRESS>
&Pradeep Varakantham
Singapore Management University
Singapore
<EMAIL_ADDRESS>
###### Abstract

Deep Reinforcement Learning (DRL) policies are critically vulnerable to
adversarial noise in observations, posing severe risks in safety-critical
scenarios. For example, a self-driving car receiving manipulated sensory
inputs about traffic signs could lead to catastrophic outcomes. Existing
strategies to fortify RL algorithms against such adversarial perturbations
generally fall into two categories: (a) using regularization methods that
enhance robustness by incorporating adversarial loss terms into the value
objectives, and (b) adopting ”maximin” principles, which focus on maximizing
the minimum value to ensure robustness. While regularization methods reduce
the likelihood of successful attacks, their effectiveness drops significantly
if an attack does succeed. On the other hand, maximin objectives, although
robust, tend to be overly conservative. To address this challenge, we
introduce a novel objective called Adversarial Counterfactual Error (ACoE),
which naturally balances optimizing value and robustness against adversarial
attacks. To optimize ACoE in a scalable manner in model-free settings, we
propose a theoretically justified surrogate objective known as Cumulative-ACoE
(C-ACoE). The core idea of optimizing C-ACoE is utilizing the belief about the
underlying true state given the adversarially perturbed observation. Our
empirical evaluations demonstrate that our method outperforms current state-
of-the-art approaches for addressing adversarial RL problems across all
established benchmarks (MuJoCo, Atari, and Highway) used in the literature.

##  1 Introduction

The susceptibility of Deep Neural Networks (DNNs) to adversarial attacks on
their inputs is a well-documented phenomenon in machine learning [Goodfellow
et al., 2014, Madry et al., 2017]. Consequently, Deep Reinforcement Learning
(DRL) models that rely on neural networks are also vulnerable to input
perturbations, even when the environment remains unchanged [Gleave et al.,
2019, Sun et al., 2020, Pattanaik et al., 2017]. As DRL becomes increasingly
relevant to real-world applications such as self-driving cars, developing
robust policies is of paramount importance [Spielberg et al., 2019, Kiran et
al., 2021]. An example highlighted by [Chen et al., 2018] involves altering a
stop sign both digitally and physically to deceive an object recognition
model, demonstrating the ease and potential dangers of such adversarial
attacks.

Adversarial retraining, which involves adding adversarial perturbations to the
replay buffer during training, effectively enhances the robustness of deep
reinforcement learning (DRL) against known adversaries [Gleave et al., 2019,
Goodfellow et al., 2014, Pattanaik et al., 2017, Sun et al., 2023]. However,
this approach often fails to generalize well to out-of-sample adversaries.
Furthermore, due to issues such as catastrophic forgetting, naive adversarial
training in reinforcement learning can result in unstable training processes
and diminished agent performance [Zhang et al., 2020]. This highlights the
need for algorithms that are not tailored to specific adversarial
perturbations but are generally robust. Instead of developing a policy that is
value-optimal for a wide range of known examples, our goal is to identify and
mitigate behaviors and states that increase unnecessary risk. A widely-
recognized method to achieve robustness is maximin optimization, which seeks
to maximize the minimum reward of a policy [Everett et al., 2020, Liang et
al., 2022]. While this approach can enhance robustness, it often sacrifices
the quality of the unperturbed solution to improve the worst-case scenario.

Another prevalent robustness strategy strengthens value-optimal policies by
incorporating adversarial loss terms. This ensures that actions remain
consistent across similar inputs, thereby reducing the likelihood of
successful adversarial attacks [Oikarinen et al., 2021, Zhang et al., 2020,
Liang et al., 2022]. However, our empirical findings indicate that these
methods still leave policies vulnerable when attacks do succeed. This
vulnerability arises because value-optimizing policies inherently engage in
risky behaviors.

To address these challenges, we introduce a novel objective called Adversarial
Counterfactual Error (ACoE), which aims to calculate the error due to
adversarial perturbations. The key innovation in ACoE, unlike existing
approaches, is recognizing that the true underlying state becomes partially
observable in the presence of adversarial perturbations. Therefore, in our
approach, we predict a belief over the underlying true state and minimize ACoE
given this belief.

Contributions:

  * •

In a significant departure from previous research, we introduce the concept of
Adversarial Counterfactual Error (ACoE), which is defined based on beliefs
about the underlying true state rather than the observable state only. This
shift is necessitated by the partial observability of the underlying state due
to adversarial perturbations.

  * •

We introduce a scalable surrogate for ACoE called Cumulative ACoE (C-ACoE) and
establish its fundamental theoretical properties, which aid in developing
strong solution methods.

  * •

We develop mechanisms to minimize C-ACoE while maximizing expected value by
leveraging established techniques from Deep Reinforcement Learning (e.g., DQN,
PPO).

  * •

Finally, we present comprehensive experimental results on renowned benchmark
problems (MuJoCo, Atari, Highway) to demonstrate the effectiveness of our
approaches compared to leading methods (e.g., RADIAL, RAD, WOCAR) for
adversarial reinforcement learning. While we, like previous studies, validate
our models against potent myopic attacks such as PGD, we distinguish our work
by also demonstrating resilience against macro-strategic attacks specifically
engineered to target a victim policy.

##  2 Related work

Adversarial attacks in RL: Deep RL is vulnerable to attacks on the input,
ranging from methods targeting the underlying DNNs such as an FGSM attack
[Huang et al., 2017, Goodfellow et al., 2014], tailored attacks against the
value function [Kos and Song, 2017, Sun et al., 2020], or adversarial behavior
learned by an opposing policy [Gleave et al., 2019, Everett et al., 2020,
Oikarinen et al., 2021, Zhang et al., 2020]. We compile attacks on RL loosely
into two groups of learned adversarial policies: observation poisonings
[Gleave et al., 2019, Sun et al., 2020, Lin et al., 2017] and direct ego-state
disruptions [Pinto et al., 2017, Rajeswaran et al., 2017]. Each category has
white-box counterparts that leverage the victim’s network gradients to
generate attacks [Goodfellow et al., 2014, Oikarinen et al., 2021, Huang et
al., 2017, Everett et al., 2020]. In this work, we focus on the former group,
observation poisonings, with both white-box and black-box scenarios.

Adversarial training: In this area, adversarial examples are found or
generated and integrated into the set of training inputs [Shafahi et al.,
2019, Ganin et al., 2016, Wong et al., 2020, Madry et al., 2017,
Andriushchenko and Flammarion, 2020, Shafahi et al., 2020]. For a
comprehensive review, we refer readers to [Bai et al., 2021]. In RL, research
efforts have demonstrated the viability of training RL agents against
adversarial examples [Gleave et al., 2019, Bai et al., 2019, Pinto et al.,
2017, Tan et al., 2020, Kamalaruban et al., 2020, Sun et al., 2023]. Naively
training RL agents against known adversaries is a sufficient defense against
known attacks; however, novel or more general adversaries remain effective
[Gleave et al., 2019, Kang et al., 2019]. Therefore, we focus on proactively
robust defense methods instead of reactive methods (reacting to known
adversaries).

Robust regularization: Regularization approaches [Zhang et al., 2020,
Oikarinen et al., 2021, Everett et al., 2020] take vanilla value-optimized
policies and robustify them to minimize the loss due to adversarial
perturbations. These approaches utilize certifiable robustness bounds computed
for neural networks when evaluating adversarial loss and ensure the success
probability of an attack is reduced using these lower bounds. Despite lowering
the likelihood of a successful attack, an attack that does break through is
still effective. Previous works suggests the need to occupy lower-risk states
via robust behaviors, rather than a robust decision classifier only [Belaire
et al., 2024, Liang et al., 2022].

Robust Control: Measuring and optimizing a regret value to improve the
robustness has been studied previously in uncertain Markov Decision Processes
(MDPs)[Ahmed et al., 2013, Rigter et al., 2021]. In RL, [Jin et al., 2018]
established Advantage-Like Regret Minimization (ARM) as a policy gradient
solution for agents robust to partially observable environments. In continuous
time control, Yang et al. [2023] study the composition of robust control
algorithms with a robust predictor of perturbed system dynamics. In contrast,
we form beliefs about true states, recognizing the partial observability
present in the problem and minimize the cumulative adversarial counterfactual
error to ensure a robust policy is computed.

Adversary-agnostic approaches: Unlike adversary-specific robustness training,
the methods we term as “adversary-agnostic” do not train on a perturbed MDP.
While the various forms of adversarial retraining have clear merit, they often
take longer to train (needing to train both victim and adversary policies).
PA-ATLA-PPO [Sun et al., 2023], a SoTA adversarial retraining technique,
reports needing 2 million training frames for MuJoCo-Halfcheetah. For
comparison, both RAD Belaire et al. [2024] and WocaR-PPO [Liang et al., 2022],
SoTA adversary-agnostic methods, require less than 40% of the training frames.
Furthermore, adversary-specific methods may have subpar performance against
novel adversaries, in addition to well-known drawbacks such as catastrophic
forgetting. Zhang et al. [2020] demonstrates how naive adversarial retraining
is a poor solution; it is shown in Belaire et al. [2024] that even advanced
retraining frameworks are not as generally robust.

##  3 Adversarial Counterfactual Error (ACoE)

In this section, we define the ACoE objective for the Adversarial
Reinforcement Learning (RL) problem. Intuitively, ACoE refers to the
difference in the expected value obtained by a defender in the absence of
adversarial perturbations versus in the presence of an adversary. It should be
noted that in the case of adversarial perturbations, the defender only
receives the altered state, and no information that is verified to be
uncorrupted. By minimizing the ACoE objective in conjunction with maximizing
expected value, we aim to derive a policy that provides a good trade-off
between robustness (against adversary perturbations) and effectiveness
(accumulating reward).

Expected value without adversarial perturbations, $V(s)$:

In the case without adversarial perturbations, the defender’s problem is one
of an infinite horizon MDP. Formally, we define the MDP
$\langle\mathcal{S},\mathcal{A},T,R,\gamma\rangle$ where $\mathcal{S}$ is the
state space, $\mathcal{A}$ is the action space,
$T(s^{\prime}\leavevmode\nobreak\ |\leavevmode\nobreak\ s,a)$ is transition
probability, $R(s,a)$ is the immediate reward, and $\gamma$ is the discount
factor. Without loss of generality, we assume $R(s,a)\in[0,1]$. For ease of
presentation, we assume discrete state and actions in the mathematical
sections. The aim in the MDP is to choose actions at every time step
(specified as a policy $\pi$) that maximize the value function $V$. In
infinite horizon MDPs, the optimal policy is memoryless and stationary, i.e. a
function of only the current state. However, to be more general and keep
consistent notation with the case where there is adversarial partially
observable case below, we use $I$ as the current information state, i.e., $I$
is the sequence of observed states and actions up to the present, and the
policy computes the action as a function of $I$, $\pi(I)$. Note that this is
without loss of generality, as the optimal policy in an MDP will simply ignore
the history preceding the current state. Then, the value for a policy $\pi$ is
given by

$\displaystyle V(s)=R(s,\pi(I))+\gamma E_{s^{\prime}\sim
T(\cdot|s,\pi(I))}[V(s^{\prime})]$

Expected value with adversarial perturbations, $U(b)$:

In the case of an adversarial perturbation, the defender only receives an
altered observation, providing only partial information about the underlying
true state (i.e., the true state is near the perturbed state). Formally, we
define the adversary’s policy as a function,
$\nu:\mathcal{S}\rightarrow\Delta(\mathcal{S})$, where $\Delta(\mathcal{S})$
denotes all possible distributions over $\mathcal{S}$; we also abuse notation
slightly to indicate the perturbed random state as $\nu(s)$. We follow the
standard assumption in adversarial learning that the perturbed state is close
to the true underlying state, i.e., $||\nu(s)-s||_{\infty}\leq\epsilon$. This
is an example of a one-sided Partially Observable Stochastic game (POSG)
[Horák et al., 2023] in which the adversary has full observability while the
defender does not observe the underlying state and only observes the perturbed
state. It is well known [Horák et al., 2023] that with a fixed adversarial
perturbation policy (possibly randomized), the defender’s problem reduces to a
Partially Observable Markov Decision Process (POMDP).

A POMDP is a MDP where the state is only partially observed. This partial
observability is captured using an observation space $\mathcal{O}$ and
observation probability $P_{o}(o\leavevmode\nobreak\ |\leavevmode\nobreak\
s^{\prime},a)$ that specifies the probability of observing $o$ given true
state $s^{\prime}$ obtained on taking action $a$. Further, a POMDP is known to
be equivalent to a belief state MDP [Kaelbling et al., 1998] where states are
beliefs over the underlying states in the POMDP. A belief state, $b$ is a
probability distribution over underlying states, s, where $\sum_{s}b(s)=1$. On
taking actions, this belief state changes and is computed by using a standard
Bayesian update:

$b^{\prime}(s^{\prime})=\frac{P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ s^{\prime},a)\sum_{s}T(s^{\prime}\leavevmode\nobreak\
|\leavevmode\nobreak\ s,a)b(s)}{P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ b,a)}\mbox{ where }P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ b,a)=\sum_{s^{\prime}}P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ s^{\prime},a)\sum_{s}T(s^{\prime}\leavevmode\nobreak\
|\leavevmode\nobreak\ s,a)b(s)$

We will employ a short form to represent the above update,
$b^{\prime}=SE(b,o,a)$. As the belief update requires knowledge of the model
(transition function), our initial mathematical analysis is in a model based
framework. An optimal policy in a POMDP can be a function of the belief.
However, it is known that for POMDPs belief $b$ is a sufficient statistic for
information state $I$, so we can consider the more general policy that depends
on $I$, without any loss of generality. We denote by $U$ the value function of
this POMDP for policy $\pi$:

$\displaystyle U(b)=R(b,\pi(I))+\gamma\sum_{o}P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I))U(SE(b,o,\pi(I)))$

The partial observability exhibited in adversarial RL has a particular
structure in which the observation space $\mathcal{O}$ is the same as the
state space $\mathcal{S}$ and the observation probability function,
$P_{o}(o\leavevmode\nobreak\ |\leavevmode\nobreak\ s,a)$ is governed by the
adversary’s perturbation policy. More specifically, in our problem the
observation probability depends only on the true state and not the defender
action, thus, we write $P^{\nu}_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ s)$, but note that $b^{\prime}=SE(b,o,a)$ still depends
on $a$ due to the use of transition $T$. Note that the non-adversarial case
can be considered a special case where the adversary policy is the identity
function $\mathsf{id}$, and then $P^{\mathsf{id}}_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ s)=\mathbb{I}(o=s)$ for indicator function $\mathbb{I}$.
As the observation space $\mathcal{O}=\mathcal{S}$, we will often use the
notation $s_{o}$ to refer to an observation as $s_{o}\in\mathcal{S}$ where the
subscript $o$ is used to denote that this is an observation. In particular,
any distribution over the observation space is a distribution over the state
space.

Adversarial Counterfactual Error, ACoE: We analyze the difference in return
$V-U$ obtained in the non-adversary case (denoted by $V$) and adversary case
(denoted by $U$) using a common policy $\pi$ in each case. We term $V-U$ as
Adversarial Counterfactual Error (ACoE). As the optimal policy depends on
different information structures in these two cases, in order to compare these
cases with the same policy, we have already made the choice generalize the
policy as a function of the information state $I$. We write the value
functions starting in the currently observed belief, which for the non-
adversarial case is the true state itself. For notational ease in the later
sections we will write $s_{o}$ to represent the current observation, which
particularly emphasizes the fact that in our problem observations are
themselves part of the state space. Further, in our particular domain,
$o\in\mathcal{S}$, thus, $P_{o}(\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I))$ specifies a probability distribution over
states. Thus, by renaming variables and dropping the dependence of
observations on actions, we rewrite $\sum_{o}P_{o}(o\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I))U(SE(b,o,\pi(I)))$ as $E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,,\pi(I))}[U(SE(b,s^{\prime}_{o},\pi(I))]$. Then, for both the non-adversary
and adversary scenarios, following standard MDP and POMDP facts, we have a
recursive form as below:

$\displaystyle V(s_{o})$ $\displaystyle=R(s_{o},\pi(I))+\gamma
E_{s^{\prime}_{o}\sim T(\cdot|s_{o},\pi(I))}[V(s^{\prime}_{o})]$
$\displaystyle U(b)$ $\displaystyle=R(b,\pi(I))+\gamma E_{s^{\prime}_{o}\sim
P_{o}(\cdot|b,,\pi(I))}[U(SE(b,s^{\prime}_{o},\pi(I))]$

ACoE is defined as $V(s_{o})-U(b)$.

We also use an additional shorthand notation of
$T_{o}(\cdot,\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\ b,a)$ to denote
the joint probability distribution of $s^{\prime}_{o}$ and $b^{\prime}$
specified by the sampling process: $s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a),b^{\prime}=SE(b,s^{\prime}_{o},a)$. We define the following important
quantity:

######  Definition 3.1 (Cumulative Adversarial Counterfactual Error (C-ACoE)).

Define C-ACoE as

$\displaystyle\delta(s_{o},b)=R(s_{o},\pi(I))-R(b,\pi(I))+\gamma
E_{s^{\prime}_{o},b^{\prime}\sim T_{o}(\cdot,\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I))}[\delta(s^{\prime}_{o},b^{\prime})]$ (1)

######  Theorem 3.2.

Let $K=\max_{s\in\mathcal{S}}{V(s)}$ and assume
$TV(T(\cdot|s_{o},a),P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a))\leq\Xi$ for any observed state $s_{o}$, belief $b$, and action $a$ in
the same time step, then

$\big{|}V(s_{o})-U(b)-\delta(s_{o},b)\big{|}\leq\frac{\gamma K\Xi}{1-\gamma}$

The above result shows that there are two parts to ACoE, the uncontrollable
part with the $TV$ distance captures structural differences in the transition
without attack and transition induced by the attack, while the controllable
part, C-ACoE term $\delta(s_{o},b)$ captures long term return difference due
to the adversarially induced transition. In the appendix, we delve more into
the structural difference in transitions by utilizing Wasserstein distance
instead of Total Variation, TV distance. The above results also suggests that
apart from the inherent structural differences, minimizing C-ACoE
$\delta(s_{o},b)$ can be effective in ensuring that returns in the adversarial
scenario are close to the non-adversarial scenario, which we explore in the
next section.

Since the structural differences in transition are not controllable by the
defender agent, we focus on minimizing the C-ACoE for the defender.
Furthermore, to ensure effectiveness of the policy in accumulating rewards is
high, we minimize C-ACoE while maximizing expected reward.

##  4 Optimizing C-ACoE along with Expected Reward in Adversarial RL

In adversarial RL settings, we do not have the model and hence the transition
dynamics $T$ are not available. Thus, computing $\delta(s_{o},b)$ exactly is
not possible. However, our problem presents a structured scenario where the
observation depends only the current true state and uncertainty is entirely
due to adversarial perturbation. It has been stated in literature and is also
intuitive that adversarial perturbations are effective in causing harm when
they induces a large enough change in defenders action distribution [Oikarinen
et al., 2021, Zhang et al., 2020]. Thus, we propose to derive a belief based
on the observed state $s_{o}$ in conjunction with reasoning about how the
adversary might have forced this observation to arise. We present a few such
belief constructions in the next few sub-sections.

Using the full history of observations and actions (represented as the
information state, $I$) as an input to the policy is computationally expensive
to implement. Prior approaches have used a variety of approximations
[Azizzadenesheli et al., 2018]; we adopt a simple measure [Müller and
Montufar, 2021, Kober et al., 2013] where we restrict solutions to the set of
policies that depend just on the current observation. Next, note that if $b$
depends on $s_{o}$ only, then $\delta(s_{0},b)$ is a function of $s_{o}$ only.
Hence, we redefine the C-ACoE as

$\displaystyle\delta(s_{o})=R(s_{o},\pi(s_{o}))-R(b(s_{o}),\pi(s_{o}))+\gamma
E_{s^{\prime}_{o}\sim\nu(s^{\prime}),s^{\prime}\sim
T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
s,\pi(s_{o}))}[\delta(s^{\prime}_{o}]$ (2)

We note that the underlying true state $s^{\prime}$ is not observed, but
estimating the second term on the RHS above requires only samples of
observation $s^{\prime}_{o}$ which are available from the simulator. In this
form C-ACoE also satisfies the Bellman optimality structure (as stated
formally in the following proposition) and hence allows for use of standard RL
techniques to minimize $\delta(s_{o})$.

######  Proposition 4.1.

Let $\delta^{*}(s_{o})$ be the minimum C-ACoE value from observation $s_{o}$.
Then,

$\delta^{*}(s_{o})=\min_{a}\\{R(s_{o},\pi(s_{o}))-R(b(s_{o}),a)+\gamma
E_{s^{\prime}_{o}\sim\nu(s^{\prime}),s^{\prime}\sim
T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
s,a)}[\delta^{*}(s^{\prime}_{o}]\\}$

Algorithm 1 shows our adaptation of PPO for optimizing $\delta$ while
maximizing $V$. The steps for maximizing $V$ follow standard steps in PPO
leading to the advantage $\hat{A}_{t}$ in line 7. We also compute the C-ACoE-
to-go from the sampled trajectories (line 5) and use it to augment the
objective in line 7 (we need to minimize C-ACoE, hence the negative sign
before $\hat{\delta}_{t}$). Line 9 is a standard PPO step to update the $V$
network and we do so similarly for the $\delta$ network in line 10. We found
that computing an advantage like term for $\delta$ did not improve
performance, thus we used only C-ACoE-to-go. A similar adaptation is also done
for DQN, presented in the appendix. Next, we describe two possible belief
constructions given the observed state $s_{o}$.

1 Initialize policy network weights $\theta_{1}$, value network weights
$\phi_{1}$, and $\delta$-network weights $\psi_{1}$

2 Set robustness-hyperparameter $\lambda$

3 for  _iteration $k\in\\{1,\ldots,M\\}$_ do

4 Collect set of trajectories $\mathcal{D}_{k}$ by running policy
$\pi_{\theta_{k}}$ multiple times for $T$ steps

5 Estimate rewards-to-go $\hat{R}_{t}$ and C-ACoE-to-go $\hat{\delta}_{t}$ at
all time steps $t$ for all trajectory in $\mathcal{D}_{k}$

6 Compute advantage estimates $\hat{A}$ using Generalized Advantage Estimator
[Schulman et al., 2016], based on $\hat{R}_{t}$’s and $V_{\phi_{k}}$

7 Compute C-ACoE Advantage $A_{c,t}=\hat{A}_{t}-\lambda\hat{\delta}_{t}$

8 Update policy parameters to $\theta_{k+1}$ by maximizing the PPO-clipped
[Schulman et al., 2017] form of $A_{c,t}$

9 Update
$\phi_{k+1}=\operatorname*{argmin}_{\phi}\frac{1}{|\mathcal{D}_{k}|T}\sum_{\tau\in
D_{k}}\sum_{t=0}^{T}(V_{\phi}(s_{t})-\hat{R}_{t})^{2}$

10 Update
$\psi_{k+1}=\operatorname*{argmin}_{\psi}\frac{1}{|\mathcal{D}_{k}|T}\sum_{\tau\in
D_{k}}\sum_{t=0}^{T}(\delta_{\psi}(s_{t})-\hat{\delta}_{t})^{2}$

11

Algorithm 1 $\delta$-PPO

Adversary-Aware Belief Estimation (A2B): We aim to assign a belief to states
in neighborhood $N(s_{o})$ of observation, $s_{o}$ where
$N(s_{o})=\\{s\leavevmode\nobreak\ |\leavevmode\nobreak\
||s-s_{o}||\leq\epsilon\\}$. $N(s_{o})$ is restricted to an $\epsilon$ bound
given established adversarial perturbation practices. We know that an
adversarial perturbation from state $s$ to state $s_{o}$ is an effective
attack when the action distribution $\pi(s)$ and $\pi(s_{o})$ are quite
different. Based on this fact, we form a belief:

$b(s)=\frac{e^{D_{KL}(\pi(s)|\pi(s_{o}))}}{\sum_{s^{\prime}\in
N(s_{o})}e^{D_{KL}{(\pi(s^{\prime})|\pi(s_{o}))}}}$

Adversary-Attack-Aware Belief Estimation (A3B): Different from A2B, we assign
scores to states in $N(s_{o})$ based on the KL divergence. These scores depend
on an attack $\nu$; we assign a score $z(s^{\prime})$ to a state
$s^{\prime}\in N(s_{o})$ that is a ratio of the KL divergence of action
distribution at the possibly perturbed state $s^{\prime}$ and $s$ to the KL
divergence of action distribution at $\nu(s)$ and $s$. Then, a belief is
assigned to state $s^{\prime}$ depending on the score $z$ by a softmax
operation:

$b(s)=\frac{e^{z(s)}}{\sum_{s^{\prime}\in N(s_{o})}e^{z(s^{\prime})}}\;\mbox{
where }\;z(s)=\frac{D_{KL}(\pi(s_{o})||\pi(s))}{D_{KL}(\pi(\nu(s))||\pi(s))}$

Figure 1: A3B belief construction. In this visual example, $N(s_{o})$ includes
$s^{\prime}_{2}$, which has a much worse perturbation ($s^{\prime\prime}_{2}$)
within $N(s^{\prime}_{2})$. Our method should discount the possibility that
$\nu(s_{2})=z$, and lessen the score $z(s^{\prime}_{2})$.

For a visual explanation of the above, consider Figure 1. This figure shows
two candidate states $s^{\prime}_{1}$ and $s^{\prime}_{2}$ which could be the
potential true state. The attack from state $s^{\prime}_{1}$ leads to
$s^{\prime\prime}_{1}=\nu(s^{\prime}_{1})$ and from state $s^{\prime}_{2}$
leads to $s^{\prime\prime}_{2}=\nu(s^{\prime}_{1})$. Even though
$s^{\prime\prime}_{2}$ may be close in Euclidean distance to $s_{o}$, it is
possible that
$D_{KL}(\pi(s^{\prime\prime}_{2})||\pi(s^{\prime}_{2}))\leavevmode\nobreak\
>\\!>\leavevmode\nobreak\ D_{KL}(\pi(s_{o})||\pi(s^{\prime}_{2}))$ leading to
a small score $z_{s^{\prime}_{2}}$ for $s^{\prime}_{2}$. This is intuitive, as
an attack from $s^{\prime}_{2}$ will likely not lead to $s_{o}$ because of the
much higher KL divergence for best attack $s^{\prime\prime}_{2}$. Similarly,
the score $z_{s^{\prime}_{1}}$ for $s^{\prime}_{1}$ can be close to 1 due to
$D_{KL}(\pi(s^{\prime\prime}_{1})||\pi(s^{\prime}_{1}))\approx
D_{KL}(\pi(s_{o})||\pi(s^{\prime}_{1}))$, which is intuitive as $s_{o}$
results in same amount of change in action distribution as
$s^{\prime\prime}_{1}$.

One issue to consider above is when the state space is continuous. In such as
scenario we still form a finite set $N(s_{o})$ by uniformly sampling a given
number $n$ (hyperparameter) of samples from the continuous set
$C=\\{s\leavevmode\nobreak\ |\leavevmode\nobreak\ ||s-s_{o}||\leq\epsilon\\}$.
From the definition of $\delta$ (Eq. 2), we use $b$ to estimate $R(b,a)$. Our
true value of this is $R=R(b,a)=\int_{s\in C}R(s,a)p(s)ds$ where the
probability density $p(s)=e^{z(s)}/\int_{s\in C}e^{z(s)}ds$. In contrast, we
sample $n$ states from a uniform distribution $U$ with probability density
given by $u(s)=1/vol(C)$ where $vol$ is the volume of set $C$ and estimate
$\hat{R}=\frac{\sum_{s^{\prime}\in
N(s_{o})}R(s^{\prime},a)e^{z(s^{\prime})}}{\sum_{s^{\prime}\in
N(s_{o})}e^{z(s^{\prime})}}$. We show a result in the appendix that justifies
the estimate $\hat{R}$ by showing that the expected value of this estimate is
close to the true required value $R$.

##  5 Experiments

We provide empirical evidence to show the effectiveness of our proposed
method. In particular, we want to investigate whether A2B and A3B improve over
leading adversarial robustness methods on established baselines, and what
aspects of C-ACoE contribute to a viable defense against strategic
adversaries.

Table 1: Experimental results versus myopic adversaries. Each row shows the mean scores of each RL method against different attacks. Our method is highlighted; the most robust scores are in bold. Methods are evaluated as their corresponding PPO implementations. Method | Unperturbed | MAD | PGD | Unperturbed | MAD | PGD
---|---|---|---|---|---|---
| highway-fast-v0 | merge-v0
PPO | 24.8$\pm$5.42 | 13.63$\pm$19.85 | 15.21$\pm$16.1 | 14.94$\pm$0.01 | 10.2$\pm$0.02 | 10.42$\pm$0.95
CARRL | 24.4$\pm$1.10 | 4.86$\pm$15.4 | 12.43$\pm$3.4 | 12.6$\pm$0.01 | 12.6$\pm$0.01 | 12.02$\pm$0.01
RADIAL | 28.55$\pm$0.01 | 2.42$\pm$1.3 | 14.97$\pm$3.1 | 14.86$\pm$0.01 | 11.29$\pm$0.01 | 11.04$\pm$0.91
WocaR | 21.49$\pm$0.01 | 6.15$\pm$0.3 | 6.19$\pm$0.4 | 14.91$\pm$0.04 | 12.01$\pm$0.28 | 11.71$\pm$0.21
RAD | 21.01$\pm$0.01 | 20.59$\pm$4.1 | 20.02$\pm$0.01 | 13.91$\pm$0.01 | 13.90$\pm$0.01 | 11.72$\pm$0.01
A2B | 24.8$\pm$0.01 | 23.11$\pm$0.01 | 20.8$\pm$12.6 | 14.91$\pm$0.01 | 14.23$\pm$0.8 | 12.92$\pm$0.13
A3B | 23.8$\pm$0.01 | 23.21$\pm$0.01 | 22.61$\pm$14.1 | 14.91$\pm$0.17 | 14.88$\pm$0.17 | 14.89$\pm$0.17

###  5.1 Experiment setup

We evaluate C-ACoE methods on the Atari [Bellemare et al., 2013], MuJoCo
[Todorov et al., 2012], and Highway simulators [Leurent, 2018]. In the Mujoco
and Highway tasks, the agent earns score by traversing distance without
incurring critical collisions. Atari tasks are game-dependent. We use a
standard training setup seen in [Oikarinen et al., 2021, Liang et al., 2022,
Belaire et al., 2024], and detailed in Appendix C.

We compare C-ACoE optimization methods (A2B, A3B) to the following baselines:
PPO [Schulman et al., 2017]; CARRL, a simple but robust minimax method
[Everett et al., 2020]; RADIAL, a leading regularization approach [Oikarinen
et al., 2021]; WocaR, worst-case aware value maximization [Liang et al.,
2022]; and RAD, a SOTA method minimizing a notion of regret [Belaire et al.,
2024]. We test all methods against two greedy attack approaches of reward-
minimizing policy adversaries and gradient attacks. Additionally, we evaluate
against PA-AD [Sun et al., 2023], the leading multi-step strategic attack, on
MuJoCo tasks. We evaluate each method’s PPO implementation, including ours, in
the Highway and Mujoco domains, and DQN implementations in Atari tasks.

Myopic Adversaries: We test the adversarial robustness of each method against
adversaries that we term as “greedy” or myopic, meaning that they compute
worst-case attacks over one time step. Following the preceding lineage of
works, we measure a 10-step PGD attack [Madry et al., 2017] with
$\epsilon=0.1$, and a MAD attack [Zhang et al., 2020] with $\epsilon=0.15$.

Long-Horizon Adversaries: We also assess adversarial robustness of each method
versus more strategic, long-horizon adversaries that compute worst-case
trajectories to deceive an RL agent into following. We evaluate agents against
a Critical Point Attack [Liang et al., 2022], Perfectly Timed Attack [Lin et
al., 2017], and PA-AD [Sun et al., 2023]. We evaluate the adversarial
robustness of the target policies as the depth of strategy increases for the
long-horizon adversaries. In the context of the Critical Point and PA-AD
adversaries, a higher depth of strategy increases the length and number of
trajectories sampled to find the worst-case future outcome, and a stronger
Perfectly Timed attacker has a larger perturbation budget.

Table 2: Experimental results versus myopic adversaries in Atari domains, formatted the same as Table 1. Methods are evaluated as their corresponding DQN implementations. Method | Unperturbed | MAD | PGD | Unperturbed | MAD | PGD
---|---|---|---|---|---|---
| Pong | Freeway
PPO | 21.0$\pm$0 | -20.0$\pm 0.07$ | -19.0$\pm$1.0 | 29 $\pm$ 3.0 | 4 $\pm$ 2.31 | 2$\pm$2.0
CARRL | 13.0 $\pm$1.2 | 11.0$\pm$0.010 | 6.0$\pm$1.2 | 18.5$\pm$0.0 | 19.1 $\pm$1.20 | 15.4$\pm$0.22
RADIAL | 21.0$\pm$0 | 11.0$\pm$2.9 | 21.0$\pm$ 0.01 | 33.2$\pm$0.19 | 29.0$\pm$1.1 | 24.0$\pm$0.10
WocaR | 21.0$\pm$0 | 18.7 $\pm$0.10 | 20.0 $\pm$ 0.21 | 31.2$\pm$0.41 | 19.8$\pm$3.81 | 28.1$\pm$3.24
RAD | 21.0$\pm$0 | 14.0 $\pm$ 0.04 | 14.0 $\pm$ 2.40 | 33.2$\pm$0.18 | 30.0$\pm$0.23 | 27.7$\pm$1.51
A2B | 21.0$\pm$0 | 20.1$\pm$0.04 | 21.0$\pm$0.01 | 33.2$\pm$0.18 | 30.1$\pm$0.43 | 30.8$\pm$1.51
A3B | 21.0$\pm$0 | 20.8$\pm$0.7 | 21.0$\pm$0.01 | 33.2$\pm$0.18 | 31.0$\pm$0.87 | 31.1$\pm$1
Table 3: Experimental results versus myopic adversaries in Mujoco domains, formatted the same as Table 1. Methods are evaluated as their corresponding PPO implementations. Method | Unperturbed | MAD | PGD | Unperturbed | MAD | PGD
---|---|---|---|---|---|---
| Hopper | Walker2d
PPO | 2741 $\pm$ 104 | 970$\pm$19 | 36$\pm$156 | 4621 $\pm$ 12 | 680$\pm$1570 | 730$\pm$262
RADIAL | 3737$\pm$75 | 2401$\pm$13 | 3070$\pm$31 | 5251$\pm$10 | 3895$\pm$128 | 3480$\pm$3.1
WocaR | 3136$\pm$463 | 1510 $\pm$ 519 | 2647 $\pm$310 | 4594$\pm$974 | 3928$\pm$1305 | 3944$\pm$508
RAD | 3473$\pm$23 | 2783$\pm$325 | 3110$\pm$30 | 4743$\pm$78 | 3922$\pm$426 | 4136$\pm$639
A2B | 3710$\pm$11 | 3240$\pm$41 | 3299$\pm$28 | 4760$\pm$61 | 4636$\pm$87 | 4708$\pm$184
A3B | 3766$\pm$23 | 3370$\pm$275 | 3465$\pm$17 | 5341$\pm$60 | 5025$\pm$94 | 5292$\pm$231
| HalfCheetah | Ant
PPO | 5794 $\pm$ 12 | 1491$\pm$20 | -27$\pm$1288 | 5620$\pm$29 | 1288$\pm$491 | 1844$\pm$330
RADIAL | 4724$\pm$76 | 4008$\pm$450 | 3911$\pm$129 | 5841$\pm$34 | 3210$\pm$380 | 3821$\pm$121
WocaR | 5220$\pm$112 | 3530$\pm$458 | 3475$\pm$610 | 5421$\pm$92 | 3520$\pm$155 | 4004$\pm$98
RAD | 4426$\pm$54 | 4240$\pm$4 | 4022$\pm$851 | 4780$\pm$10 | 3647$\pm$32 | 3921$\pm$74
A2B | 5192 $\pm$56 | 4855$\pm$ 120 | 4722$\pm$33 | 5511$\pm$13 | 3824$\pm$218 | 4102$\pm$315
A3B | 5538$\pm$20 | 4986$\pm$41 | 5110$\pm$22 | 5580$\pm$41 | 4071$\pm$242 | 4418$\pm$290

###  5.2 Results

In Tables 1, 2, and 3, we report the mean result over 5 policies initialized
with random seeds, with 50 test episodes each. The variance reported
($\pm\sigma$) is the standard deviation from the mean for each method. The
most robust score is shown in boldface.

Figure 2: Robust agents vs. a PA-AD attacker [Sun et al., 2023], as the
optimality of the attacker policy increases. We find that as the level of
strategy increases from long-horizon attackers, C-ACoE minimization achieves
more robust performance, relative to other methods.  Figure 3: Last 5 frames
of PPO, A3B, and WocaR agents (top to bottom), on MuJoCo-HalfCheetah. PPO
deviates the least from the dashed center line, and has the least balanced
gait. WocaR has arguably the most stable posture when noting the faster front
leg recovery of A3B, but our empirical results suggest optimizing maximum
stability is not always necessary. Full GIFs: tinyurl.com/a3b-gif

Myopic attacks: As seen in Table 1-3, C-ACoE methods A2B and A3B achieve state
of the art robust performance against standard greedy attacker strategies, as
well as nominal performance similar to the best observed value-maximizing
methods such as PPO. We attribute this success to the two parts of ACoE:
framing the adversarial robustness problem as a POMDP and the simultaneous
maximization of value and minimization of ACoE error brings increased
performance over maximin methods and higher robustness overall.

Long-horizon attacks: We also test our methods against attackers with a longer
planning horizon than the commonly-used greedy adversaries. In Figure 2, we
test under the PA-AD policy attack Sun et al. [2023], the state of the art
attacker method. We also include experiments evaluating robust methods against
the Strategically Timed attack [Lin et al., 2017] and the Critical Point
attack[Sun et al., 2020] in the appendix. We find that across domains, C-ACoE
agents maintain robustness even against long-horizon attacks. This is one of
the main advantages of our proposed methods following the C-ACoE-minimizing
philosophy, as the error-robust policies seek stable trajectories rather than
robust single-step action distributions.

Robust Behavior: In Figure 3, we observe qualitative differences between PPO,
A3B, and WocaR. The WocaR agent adopts more stable motion, minimizing the
worst-case, and PPO optimizes for speed, only using the back leg. A3B balances
the two approaches, using both legs in a more full range of motion. Full
videos of the behaviors described in Figure 3 can be viewed from DropBox at
tinyurl.com/a3b-gif, where the extent of robust behavior can be better
observed.

##  6 Discussion and Limitations

We propose a scalable approximation of ACoE, C-ACoE, and demonstrate its
usefulness in proactive adversarial defense, achieving state of the art
robustness against strong observation attacks from both greedy and strategic
adversaries on a variety of benchmarks. More importantly, recognizing the
partially observable nature of the defender agent in adversarial RL problems
and optimizing ACoE can be used to increase the robustness of RL to
adversarial observations, even against stronger or previously unseen
attackers. In this paper, we focused on estimation of belief from single step
perturbed observation. It may be beneficial to estimating belief based on
observations across time steps, even if it would add to computational
complexity of the overall algorithm. We argue this point as all existing
robustness methods are vulnerable to sufficiently long-horizon attackers.
Thus, considering multi-step observations can potentially help deal with long-
horizon attackers. We also note that the efficacy of the belief construct we
use is reliant on the accuracy of using KL Divergence as a notion of attack
strength. We find our measures to be empirically the strongest, compared to
notions such as Euclidean state distance or minimum reward, however, and leave
other more complex measures to future work.

## References

  * Ahmed et al. [2013] Asrar Ahmed, Pradeep Varakantham, Yossiri Adulyasak, and Patrick Jaillet.  Regret based robust solutions for uncertain markov decision processes.  _Advances in neural information processing systems_ , 26, 2013.
  * Andriushchenko and Flammarion [2020] Maksym Andriushchenko and Nicolas Flammarion.  Understanding and improving fast adversarial training.  _Advances in Neural Information Processing Systems_ , 33:16048–16059, 2020.
  * Azizzadenesheli et al. [2018] Kamyar Azizzadenesheli, Yisong Yue, and Animashree Anandkumar.  Policy gradient in partially observable environments: Approximation and convergence.  _arXiv preprint arXiv:1810.07900_ , 2018.
  * Bai et al. [2021] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang.  Recent advances in adversarial training for adversarial robustness.  _arXiv preprint arXiv:2102.01356_ , 2021.
  * Bai et al. [2019] Xueying Bai, Jian Guan, and Hongning Wang.  A model-based reinforcement learning with adversarial training for online recommendation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Belaire et al. [2024] Roman Belaire, Thanh Hong Nguyen, David Lo, and Pradeep Varakantham.  Regret-based defense in adversarial reinforcement learning.  In _Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024_ , pages 2633–2640. ACM, 2024.  doi: 10.5555/3635637.3663250.  URL https://dl.acm.org/doi/10.5555/3635637.3663250.
  * Bellemare et al. [2013] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, jun 2013.  doi: 10.1613/jair.3912.  URL https://doi.org/10.1613%2Fjair.3912.
  * Chen et al. [2018] Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Chau.  Robust physical adversarial attack on faster R-CNN object detector.  _CoRR_ , abs/1804.05810, 2018.  URL http://arxiv.org/abs/1804.05810.
  * Everett et al. [2020] Michael Everett, Björn Lütjens, and Jonathan P. How.  Certified adversarial robustness for deep reinforcement learning.  _CoRR_ , abs/2004.06496, 2020.  URL https://arxiv.org/abs/2004.06496.
  * Ganin et al. [2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gleave et al. [2019] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell.  Adversarial policies: Attacking deep reinforcement learning.  2019\.  doi: 10.48550/ARXIV.1905.10615.  URL https://arxiv.org/abs/1905.10615.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  2014\.  doi: 10.48550/ARXIV.1412.6572.  URL https://arxiv.org/abs/1412.6572.
  * Horák et al. [2023] Karel Horák, Branislav Bošanskỳ, Vojtěch Kovařík, and Christopher Kiekintveld.  Solving zero-sum one-sided partially observable stochastic games.  _Artificial Intelligence_ , 316:103838, 2023.
  * Huang et al. [2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel.  Adversarial attacks on neural network policies.  _arXiv preprint arXiv:1702.02284_ , 2017.  doi: 10.48550/ARXIV.1702.02284.  URL https://arxiv.org/abs/1702.02284.
  * Jin et al. [2018] Peter Jin, Kurt Keutzer, and Sergey Levine.  Regret minimization for partially observable deep reinforcement learning.  pages 2342–2351, 2018.
  * Kaelbling et al. [1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra.  Planning and acting in partially observable stochastic domains.  _Artificial intelligence_ , 101(1-2):99–134, 1998.
  * Kamalaruban et al. [2020] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher.  Robust reinforcement learning via adversarial training with langevin dynamics.  _Advances in Neural Information Processing Systems_ , 33:8127–8138, 2020.
  * Kang et al. [2019] Daniel Kang, Yi Sun, Tom Brown, Dan Hendrycks, and Jacob Steinhardt.  Transfer of adversarial robustness between perturbation types.  _arXiv preprint arXiv:1905.01034_ , 2019.
  * Kiran et al. [2021] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez.  Deep reinforcement learning for autonomous driving: A survey.  _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * Kober et al. [2013] Jens Kober, J Andrew Bagnell, and Jan Peters.  Reinforcement learning in robotics: A survey.  _The International Journal of Robotics Research_ , 32(11):1238–1274, 2013.
  * Kos and Song [2017] Jernej Kos and Dawn Song.  Delving into adversarial attacks on deep policies.  2017\.  doi: 10.48550/ARXIV.1705.06452.  URL https://arxiv.org/abs/1705.06452.
  * Leurent [2018] Edouard Leurent.  An environment for autonomous driving decision-making.  _GitHub repository_ , 2018.
  * Liang et al. [2022] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang.  Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35:22547–22561, 2022.
  * Lin et al. [2017] Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun.  Tactics of adversarial attack on deep reinforcement learning agents.  _CoRR_ , abs/1703.06748, 2017.  URL http://arxiv.org/abs/1703.06748.
  * Madry et al. [2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  2017\.  doi: 10.48550/ARXIV.1706.06083.  URL https://arxiv.org/abs/1706.06083.
  * Müller and Montufar [2021] Johannes Müller and Guido Montufar.  The geometry of memoryless stochastic policy optimization in infinite-horizon pomdps.  In _International Conference on Learning Representations_ , 2021.
  * Oikarinen et al. [2021] Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng.  Robust deep reinforcement learning through adversarial loss.  2021\.  URL https://openreview.net/forum?id=eaAM_bdW0Q.
  * Pattanaik et al. [2017] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary.  Robust deep reinforcement learning with adversarial attacks.  _arXiv preprint arXiv:1712.03632_ , 2017.
  * Pinto et al. [2017] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta.  Robust adversarial reinforcement learning.  In _International Conference on Machine Learning_ , pages 2817–2826. PMLR, 2017.
  * Rajeswaran et al. [2017] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine.  Epopt: Learning robust neural network policies using model ensembles.  2017\.
  * Rigter et al. [2021] Marc Rigter, Bruno Lacerda, and Nick Hawes.  Minimax regret optimisation for robust planning in uncertain markov decision processes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(13):11930–11938, May 2021.  doi: 10.1609/aaai.v35i13.17417.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17417.
  * Schulman et al. [2016] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2016.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.  doi: 10.48550/ARXIV.1707.06347.  URL https://arxiv.org/abs/1707.06347.
  * Shafahi et al. [2019] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Shafahi et al. [2020] Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein.  Universal adversarial training.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 5636–5643, 2020.
  * Spielberg et al. [2019] Steven Spielberg, Aditya Tulsyan, Nathan P Lawrence, Philip D Loewen, and R Bhushan Gopaluni.  Toward self-driving processes: A deep reinforcement learning approach to control.  _AIChE journal_ , 65(10):e16689, 2019.
  * Sun et al. [2020] Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu.  Stealthy and efficient adversarial attacks against deep reinforcement learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(04):5883–5891, Apr. 2020.  doi: 10.1609/aaai.v34i04.6047.  URL https://ojs.aaai.org/index.php/AAAI/article/view/6047.
  * Sun et al. [2023] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang.  Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl.  2023\.
  * Tan et al. [2020] Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al.  Robustifying reinforcement learning agents via action space adversarial training.  In _2020 American control conference (ACC)_ , pages 3959–3964. IEEE, 2020.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.  doi: 10.1109/IROS.2012.6386109.
  * Wong et al. [2020] Eric Wong, Leslie Rice, and J Zico Kolter.  Fast is better than free: Revisiting adversarial training.  _arXiv preprint arXiv:2001.03994_ , 2020.
  * Yang et al. [2023] Jinghan Yang, Hunmin Kim, Wenbin Wan, Naira Hovakimyan, and Yevgeniy Vorobeychik.  Certified robust control under adversarial perturbations.  In _2023 American Control Conference (ACC)_ , pages 4090–4095. IEEE, 2023.
  * Zhang et al. [2020] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh.  Robust deep reinforcement learning against adversarial perturbations on state observations.  2020\.  doi: 10.48550/ARXIV.2003.08938.  URL https://arxiv.org/abs/2003.08938.

##  Appendix A Proofs and Additional Theory Results

###### Proof of Theorem 3.2.

Subtracting $U$ from $V$, and adding and subtracting $\gamma
E_{s^{\prime}_{o}\sim P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})]$ we get

$\displaystyle V(s_{o})-U(b)=$ $\displaystyle\quad
R(s_{o},\pi(I))-R(b,\pi(I))+\gamma E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})-U(b^{\prime})]+$ $\displaystyle\qquad\gamma
E_{s^{\prime}_{o}\sim T(\cdot|s_{o},\pi(I))}[V(s^{\prime}_{o})]-\gamma
E_{s^{\prime}_{o}\sim P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})]$

Note that by definition of $T_{o}$, we have that $E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})-U(b^{\prime})]=E_{s^{\prime}_{o},b^{\prime}\sim
T_{o}(\cdot,\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})-U(b^{\prime})]$

Next, from Holder’s inequality, we get that

$\displaystyle\big{|}E_{s^{\prime}_{o}\sim
T(\cdot|s_{o},\pi(I))}[V(s^{\prime}_{o})]-E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi)}[V(s^{\prime}_{o})]\big{|}\leq\max_{s}\\{V(s)\\}TV(T(\cdot|s_{o},\pi(I),P_{o}(\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I)))$ (3)

Thus, for one side of the inequality above (i.e., using $a\leq b$ from the
shown $|a|\leq b$, the other side is $-b\leq a$)

$\displaystyle V(s_{o})-U(b)\leq$ $\displaystyle\quad
R(s_{o},\pi(I))-R(b,\pi(I))+\gamma E_{s^{\prime}_{o},b^{\prime}\sim
T_{o}(\cdot,\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})-U(b^{\prime}]+\gamma K\Xi$

For notation simplicity, let $R(s,\pi(I))-R(b,\pi(I))=\delta_{R}(s,b)$. We use
$I^{\prime}$ as the updated information state obtained by concatenating $I$
with $\pi(I),s^{\prime}_{o}$. Applying the above recursively, we get

$\displaystyle V(s_{o})-U(b)$
$\displaystyle\quad\leq\delta_{R}(s_{o},b)++\gamma
E_{s^{\prime}_{o},b^{\prime}\sim
T_{o}(\cdot,\cdot|b,\pi(I))}[V(s^{\prime}_{o})-U(b^{\prime})]+\gamma K\Xi$
$\displaystyle\quad\leq\delta_{R}(s_{o},b)+\gamma
E_{s^{\prime}_{o},b^{\prime}\sim
T_{o}(\cdot,\cdot|b,\pi(I))}\big{[}\delta_{R}(s^{\prime}_{o},b^{\prime})+\gamma
E_{s^{\prime\prime}_{o},b^{\prime\prime}\sim
T_{o}(\cdot,\cdot|b^{\prime},\pi(I^{\prime}))}[V(s^{\prime\prime}_{o})-U(b^{\prime\prime})]+\gamma
K\Xi\big{]}+\gamma K\Xi$ $\displaystyle\quad\leq...$ $\displaystyle\quad\leq
E_{(s_{o},b,s^{\prime}_{o},b^{\prime},\ldots)\sim\pi,T,P_{o}}[\delta_{R}(s_{o},b)+\gamma\delta_{R}(s^{\prime}_{o},b^{\prime})+\gamma^{2}\delta_{R}(s^{\prime\prime}_{o},b^{\prime\prime})+...]+\frac{\gamma
K\Xi}{1-\gamma}$

We note that
$E_{(s_{o},b,s^{\prime}_{o},b^{\prime},\ldots)\sim\pi,T,P_{o}}[\delta_{R}(s_{0},b)+\gamma\delta_{R}(s^{\prime}_{o},b^{\prime})+\gamma^{2}\delta_{R}(s^{\prime\prime}_{o},b^{\prime\prime})+...]=\delta(s_{o},b)$,
where

$\delta(s_{o},b)=R(s_{o},\pi(I))-R(b,\pi(I))+\gamma
E_{s^{\prime}_{o},b^{\prime}\sim T_{o}(\cdot,\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ b,\pi(I))}[\delta(s^{\prime}_{o},b^{\prime})]$

Thus,

$\displaystyle V(s_{o})-U(b)\leq\delta(s_{o},b)+\frac{\gamma K\Xi}{1-\gamma}$

By symmetric argument using other side of Eq. 3, we get

$\displaystyle\delta(s_{o},b)-\frac{\gamma K\Xi}{1-\gamma}\leq V(s_{o})-U(b)$

These last two equations led to the statement in the theorem.

∎

The result above uses total variation distance (other work in literature also
do [Zhang et al., 2020]), but, total variation is not as informative a
distance measure as Wasserstein distance. For example, it is easy to see that
$TV(P,Q)=1$ whenever the support of $P$ and $Q$ do not overlap, but it does
not distinguish whether the non-overlapping supports are near or far apart. As
shown in prior work on WGAN , Wasserstein distance provides more fine-grained
distinction. Also, the assumed bound $\Xi$ above hides the effect of the
nature of the underlying transition $T$ on the bound. Hence, we prove the next
result using Wasserstein distance, which reveals these facets of the problem.

######  Theorem A.1.

Assume that (1) $V$ is $L$-Lipschitz and (2) for any
$||s-s^{\prime}||_{\infty}\leq\epsilon$ and any action $a$ we have
$W_{1}(T(\cdot|s,a),T(\cdot|s^{\prime},a)))\leq\xi$. Then,

$\big{|}V(s_{o})-U(b)-\delta(s_{o},b)\big{|}\leq\frac{\gamma
L(\xi+\epsilon)}{1-\gamma}$

###### Proof of Theorem A.1.

The overall proof follows the same structure as Theorem 3.2. The only
difference is in the bound for

$\displaystyle\big{|}E_{s^{\prime}_{o}\sim
T(\cdot|s_{o},\pi(I))}[V(s^{\prime}_{o})]-E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})]\big{|}$ (4)

For a simpler presentation, we use $a,a^{\prime}$ to denote action taken in
current and next time step. As $V/L$ is 1-Lipschitz, by duality of $W_{1}$
Wasserstein distance, we have

$\big{|}E_{s^{\prime}_{o}\sim
T(\cdot|s_{o},a)}[V(s^{\prime}_{o})/L]-E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})/L]\big{|}\leq
W_{1}(T(\cdot|s_{o},a),P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a))$

or multiplying by $L$

$\big{|}E_{s^{\prime}_{o}\sim
T(\cdot|s_{o},a)}[V(s^{\prime}_{o})]-E_{s^{\prime}_{o}\sim
P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,\pi(I))}[V(s^{\prime}_{o})]\big{|}\leq
LW_{1}(T(\cdot|s_{o},a),P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a))$

Next, we bound $W_{1}(T(\cdot|s_{o},a)$. Note that
$P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a))=\sum_{s^{\prime}}P^{\nu}_{o}(\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ s^{\prime})\sum_{s}T(s^{\prime}\leavevmode\nobreak\
|\leavevmode\nobreak\ s,a)b(s)$. First, because the restriction on adversarial
perturbation, we know that if $b(s)>0$ then
$||s-s_{o}||_{\infty}\leq\epsilon$. Then, based on our assumption

$\displaystyle W_{1}(T(\cdot|s_{o},a),T(\cdot|s,a))\leq\xi\mbox{ for any
}s\mbox{ such that }b(s)>0$ (5)

First, note that $W_{1}$ is a convex function of its argument. This can be
seen easily; we show it for the first argument below. Recall that definition
of $W_{1}(\mu,\nu)=\inf_{\gamma\in\Gamma(\mu,\nu)}\int d(x,y)\gamma(dx,dy)$
for couplings (joint distribution) set $\Gamma$ that have marginal as
$\mu,\nu$. Choose $\gamma^{*}_{1}$ as a minimizer in $W_{1}(\mu_{1},\nu)$ and
$\gamma^{*}_{2}$ as a minimizer in $W_{1}(\mu_{2},\nu)$. Let
$\gamma^{*}=\alpha\gamma^{*}_{1}+(1-\alpha)\gamma^{*}_{2}$; it easy to see
that $\gamma^{*}\in\Gamma(\mu.\nu)$. Then,

$\displaystyle W_{1}(\alpha\mu_{1}+(1-\alpha)\mu_{2},\nu)$
$\displaystyle=\inf_{\gamma\in\Gamma(\alpha\mu_{1}+(1-\alpha)\mu_{2},\nu)}\int
d(x,y)d\gamma(x,y)$ $\displaystyle\leq\int d(x,y)d\gamma^{*}(x,y)$
$\displaystyle=\alpha\int d(x,y)d\gamma^{*}_{1}(x,y)+(1-\alpha)\int
d(x,y)d\gamma^{*}_{2}(x,y)$ $\displaystyle=\alpha
W_{1}(\mu_{1},\nu)+(1-\alpha)W_{1}(\mu_{2},\nu)$

Let $T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a)=\sum_{s}T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\ s,a)b(s)$.
Using the above convexity of $W_{1}$, we get that

$\displaystyle
W_{1}(T(\cdot|s_{o},a),T(\cdot|b,a))\leq\sum_{s}b(s)W_{1}(T(\cdot|s_{o},a),T(\cdot|s,a))\leq\xi$
(6)

where the last inequality follows from Eq. 5

Next, we bound $W_{1}(P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a),T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\ b,a))$. First, by
definition of $T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\ b,a)$ we get
that $P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a)=\sum_{s^{\prime}}P^{\nu}_{o}(\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ s^{\prime})T(s^{\prime}\leavevmode\nobreak\
|\leavevmode\nobreak\ b,a)$. Consider the joint distribution $\gamma^{*}$ over
the space $\mathcal{S}\times\mathcal{S}$ given by
$(s^{\prime},s^{\prime}_{o})$ sampled as $s_{o}\sim
T(\cdot|b,a),s^{\prime}_{o}\sim P^{\nu}_{o}(\cdot|s^{\prime})$. It is easy to
check that $\gamma^{*}$ is a coupling, i.e.,
$\gamma^{*}\in\Gamma(P_{o}(\cdot|b,a),T(\cdot|b,a))$. We show this and for
this we drop the dependency on $b,a$ for ease of notation. First,
$\gamma^{*}(A,B)=\int_{A\times
B}d\gamma^{*}(s^{\prime},s^{\prime}_{o})=\int_{A}P^{\nu}_{o}(B|s^{\prime})dT(s^{\prime})$.
Thus, $\gamma^{*}(A,\mathcal{S})=\int_{A}dT(s^{\prime})=T(A)$ and
$\gamma^{*}(\mathcal{S},B)=\int_{\mathcal{S}}P^{\nu}_{o}(B|s^{\prime})dT(s^{\prime})=P_{o}(B)$.
Also, note that $||s^{\prime}-s^{\prime}_{o}||_{\infty}\leq\epsilon$ for $d$
as the infinity norm because of the bound of adversarial perturbation implicit
in $P^{\nu}_{o}$. Then,

$\displaystyle W_{1}(P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a),T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\ b,a))$
$\displaystyle=\inf_{\gamma\in\Gamma(P_{o}(\cdot|b,a),T(\cdot|b,a))}\int||s^{\prime}-s^{\prime}_{o}||_{\infty}d\gamma^{*}(s^{\prime},s^{\prime}_{o})$
$\displaystyle\leq\int||s^{\prime}-s^{\prime}_{o}||_{\infty}d\gamma(s^{\prime},s^{\prime}_{o})$
$\displaystyle\leq\epsilon$ (7)

Combining Eq. 5 and Eq. 7 by triangle inequality we get

$W_{1}(T(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
s_{o},a),P_{o}(\cdot\leavevmode\nobreak\ |\leavevmode\nobreak\
b,a))\leq\xi+\epsilon$

∎

The above results show that some basic structural properties are needed from
the underlying system for bounding ACoE. One is that the value function should
not change by a large amount due to small changes in state and another that
the distribution of the next state should not be very different for two close
by states. Clearly, an adversary can exploit systems that lack these
properties.

###### Proof of Proposition 4.1.

The proof is observed from the fact that C-ACoE can be viewed as an infinite
horizon MDP with observations $s_{o}$ as states, immediate cost as
$R(s_{o},\pi(s_{o}))-R(b(s_{o}),a)$, and transition to next state
$s^{\prime}_{o}$ described by
$s^{\prime}_{o}\sim\nu(s^{\prime}),s^{\prime}\sim T(\cdot\leavevmode\nobreak\
|\leavevmode\nobreak\ s,a)$. ∎

##  Appendix B Adaptation for DQN

1 Initialize network $\delta_{w}$ with random weights $w$ and target network
$\widehat{\delta}_{w^{-}}$ with weights $w^{-}=w$

2 Initialize network $Q_{\theta}$ with random weights $\theta$ and target
network $\widehat{Q}_{\theta^{-}}$ with weights $\theta^{-}=\theta$

3 Initialize replay buffer $B$

4 Set robustness temperature $\lambda$

5 for  _$\text{episode}\in\\{1,\ldots,M\\}$_ do

6 for  _$t=0\to H$_ do

7 With prob. $1-\epsilon$, select
$a^{t}\in\operatorname*{argmax}_{a}Q_{\theta}(s^{t}_{o},a)-\lambda\delta_{w}(s^{t}_{o},a)$,
else select $a^{t}$ at random

8 Sample $k$ states in $N(s_{o})$, compute $b(s)$ for each $s\in N(s_{o})$

9 Compute C-ACoE: $\delta_{R}\\!=\\!R(s^{t}_{o},a^{t})\\!-\sum_{s\in
N(s_{o})}b(s)R(s,a^{t})$

10 Execute action $a_{t}$, get observed state $s^{t+1}_{o}$, store transition
$B=B\cup(s^{t}_{o},s^{t},s^{t+1}_{o},\delta_{R})$

11 Sample mini-batch $M\sim D$;

12 for  _each $(s^{i}_{o},a^{i},s^{i+1}_{o},\delta^{i}_{R})$ in mini-batch
$M$_ do

13 $\text{Set target }y_{i}=\begin{cases}\delta^{i}_{R}\text{, if episode
terminates at step $i+1$}\\\
\delta^{i}_{R}+\gamma\min_{a^{\prime}}{\delta}_{w^{-}}(s^{i+1}_{o},a^{\prime})\text{,
otherwise }\end{cases}$

14 $\text{Set target }q_{i}=\begin{cases}R(s^{t}_{o},a^{t})\text{, if episode
terminates at step $i+1$}\\\
R(s^{i}_{o},a^{i})+\gamma\min_{a^{\prime}}{Q}_{\theta^{-}}(s^{i+1}_{o},a^{\prime})\text{,
otherwise }\end{cases}$

15

16 Perform a gradient descent to update $w$ using loss:
$\sum_{i=1}^{|M|}\big{[}y_{i}-\delta_{w}\big{(}s^{i}_{o},a^{i}\big{)}\big{]}^{2}$

17 Perform a gradient descent to update $\theta$ using loss:
$\sum_{i=1}^{|M|}\big{[}q_{i}-Q_{\theta}\big{(}s^{i}_{o},a^{i}\big{)}\big{]}^{2}$

18

19 Every $K$ steps reset $w^{-}=w$ and $\theta^{-}=\theta$;

Algorithm 2 $\delta$-DQN

##  Appendix C Estimation of Belief for Continuous State Space

######  Lemma C.1.

Assume $z(s)<B$ for some constant $B$. Consider $n$ uniformly random samples
from $C$ stored in $N(s_{o})$. Let $R$ and $\hat{R}$ be as defined above.
Then, $(1/n)\sum_{s^{\prime}\in N(s_{o})}e^{z(s^{\prime})}$ is an unbiased
estimate of $(1/vol(C))\int_{s\in C}e^{z(s)}ds$. There exists $n$ large enough
so that $1+\epsilon>\frac{(1/vol(C))\int_{s\in
C}e^{z(s)}ds}{(1/n)\sum_{s^{\prime}\in N(s_{o})}e^{z(s^{\prime})}}>1-\epsilon$
with probability $1-\delta$ for given small $\epsilon,\delta$. And then,
$R(1+\epsilon)>E[\hat{R}]>R(1-\epsilon)$ with probability $1-\delta$.

###### Proof.

Note that $E_{s^{\prime}\sim U}[e^{z(s^{\prime})}]=(1/vol(C))\int_{s\in
C}e^{z(s)}ds$, which gives us the first unbiasedness result. The second result
comes from a straightforward application of Hoeffding’s concentration
inequality where the bound $B$ is used. Then, we can see that

$\displaystyle E[\hat{R}]$ $\displaystyle=\int_{s_{1}\in
C}\ldots\int_{s_{n}\in
C}\frac{\sum_{i}R(s_{i},a)e^{z(s_{i})}}{\sum_{i}e^{z(s_{i})}}u(s_{1})\ldots
u(s_{n})ds_{1}\ldots ds_{n}$ $\displaystyle=\int_{s_{1}\in
C}\ldots\int_{s_{n}\in C}\frac{\sum_{i}R(s_{i},a)e^{z(s_{i})}}{\int_{s\in
C}e^{z(s)}ds}\frac{\int_{s\in
C}e^{z(s)}ds}{\sum_{i}e^{z(s_{i})}}u(s_{1})\ldots u(s_{n})ds_{1}\ldots ds_{n}$
$\displaystyle\leq\frac{(1+\epsilon)vol(C)}{n}\int_{s_{1}\in
C}\ldots\int_{s_{n}\in C}\frac{\sum_{i}R(s_{i},a)e^{z(s_{i})}}{\int_{s\in
C}e^{z(s)}ds}u(s_{1})\ldots u(s_{n})ds_{1}\ldots ds_{n}$
$\displaystyle=\frac{(1+\epsilon)vol(C)}{n}\sum_{i}\int_{s_{i}\in
C}\frac{R(s_{i},a)e^{z(s_{i})}}{\int_{s\in C}e^{z(s)}ds}u(s_{i})ds_{i}$
$\displaystyle=\frac{(1+\epsilon)vol(C)}{n}\times\frac{n}{vol(C)}\int_{s_{i}\in
C}R(s_{i},a)p(s_{i})ds_{i}$ $\displaystyle=(1+\epsilon)R$

A similar argument holds for the lower bound, thereby, leading to the required
result, ∎

##  Appendix D Additional Experimental Results

Figure 4: Robust agents vs. a Critical Point strategic adversary Sun et al.
[2020] with increasing search sizes.

Figure 5: Robust agents vs. a Strategically Timed Attack adversary [Lin et al., 2017], as the length of perturbation increases. We find that as the level of strategy increases from long-horizon attackers, C-ACoE minimization improves robust performance, relative to other methods.  Table 4: Experimental results versus myopic adversaries. Each row shows the mean scores of each RL method against different attacks. Our method is highlighted; the most robust scores are in bold. Methods are evaluated as their corresponding PPO implementations. Method | Unperturbed | MAD | PGD | Unperturbed | MAD | PGD
---|---|---|---|---|---|---
| highway-fast-v0 | merge-v0
PPO | 24.8$\pm$5.42 | 13.63$\pm$19.85 | 15.21$\pm$16.1 | 14.94$\pm$0.01 | 10.2$\pm$0.02 | 10.42$\pm$0.95
CARRL | 24.4$\pm$1.10 | 4.86$\pm$15.4 | 12.43$\pm$3.4 | 12.6$\pm$0.01 | 12.6$\pm$0.01 | 12.02$\pm$0.01
RADIAL | 28.55$\pm$0.01 | 2.42$\pm$1.3 | 14.97$\pm$3.1 | 14.86$\pm$0.01 | 11.29$\pm$0.01 | 11.04$\pm$0.91
WocaR | 21.49$\pm$0.01 | 6.15$\pm$0.3 | 6.19$\pm$0.4 | 14.91$\pm$0.04 | 12.01$\pm$0.28 | 11.71$\pm$0.21
RAD | 21.01$\pm$0.01 | 20.59$\pm$4.1 | 20.02$\pm$0.01 | 13.91$\pm$0.01 | 13.90$\pm$0.01 | 11.72$\pm$0.01
A2B | 24.8$\pm$0.01 | 23.11$\pm$0.01 | 20.8$\pm$12.6 | 14.91$\pm$0.01 | 14.23$\pm$0.8 | 12.92$\pm$0.13
A3B | 23.8$\pm$0.01 | 23.21$\pm$0.01 | 22.61$\pm$14.1 | 14.91$\pm$0.17 | 14.88$\pm$0.17 | 14.89$\pm$0.17
| roundabout-v0 | intersection-v0
PPO | 10.33$\pm$0.40 | 7.41$\pm$0.69 | 3.92$\pm$1.35 | 9.26$\pm$7.6 | 3.62$\pm$11.63 | 6.75$\pm$12.93
CARRL | 9.75$\pm$0.01 | 9.75$\pm$0.01 | 5.92$\pm$0.12 | 8.0$\pm$0 | 7.5$\pm$0 | 9.0$\pm$0.1
RADIAL | 10.29$\pm$0.01 | 5.33$\pm$0.01 | 8.77$\pm$2.4 | 10.0$\pm$0 | 2.4$\pm$5.1 | 9.61$\pm$0.1
WocaR | 6.75$\pm$2.5 | 6.05$\pm$0.14 | 6.48$\pm$2.7 | 10.0$\pm$0.05 | 9.47$\pm$0.3 | 3.26$\pm$0.4
RAD | 9.22$\pm$0.3 | 8.98$\pm$0.3 | 9.11$\pm$0.3 | 9.85$\pm$1.2 | 9.71$\pm$2.3 | 9.62$\pm$0.1
A2B | 10.5$\pm$0.0 | 10.1$\pm$0.1 | 10.0$\pm$0.5 | 10.0$\pm$0 | 10.0$\pm$0 | 9.88$\pm$0.12
A3B | 10.5$\pm$0.01 | 10.33$\pm$0.01 | 10.18$\pm$2.1 | 10.0$\pm$0 | 9.68$\pm$0 | 9.88$\pm$0.1
Method | Unperturbed | MAD | PGD | Unperturbed | MAD | PGD
| Pong | Freeway
PPO | 21.0$\pm$0 | -20.0$\pm 0.07$ | -19.0$\pm$1.0 | 29 $\pm$ 3.0 | 4 $\pm$ 2.31 | 2$\pm$2.0
CARRL | 13.0 $\pm$1.2 | 11.0$\pm$0.010 | 6.0$\pm$1.2 | 18.5$\pm$0.0 | 19.1 $\pm$1.20 | 15.4$\pm$0.22
RADIAL | 21.0$\pm$0 | 11.0$\pm$2.9 | 21.0$\pm$ 0.01 | 33.2$\pm$0.19 | 29.0$\pm$1.1 | 24.0$\pm$0.10
WocaR | 21.0$\pm$0 | 18.7 $\pm$0.10 | 20.0 $\pm$ 0.21 | 31.2$\pm$0.41 | 19.8$\pm$3.81 | 28.1$\pm$3.24
RAD | 21.0$\pm$0 | 14.0 $\pm$ 0.04 | 14.0 $\pm$ 2.40 | 33.2$\pm$0.18 | 30.0$\pm$0.23 | 27.7$\pm$1.51
A2B | 21.0$\pm$0 | 20.1$\pm$0.04 | 21.0$\pm$0.01 | 33.2$\pm$0.18 | 30.1$\pm$0.43 | 30.8$\pm$1.51
A3B | 21.0$\pm$0 | 20.8$\pm$0.7 | 21.0$\pm$0.01 | 33.2$\pm$0.18 | 31.0$\pm$0.87 | 31.1$\pm$1
| BankHeist | RoadRunner
PPO | 1350$\pm$0.1 | 680$\pm$419 | 0$\pm$116 | 42970$\pm$210 | 18309$\pm$485 | 10003$\pm$521
CARRL | 849$\pm$0 | 830$\pm$32 | 790$\pm$110 | 26510$\pm$20 | 24480$\pm 200$ | 22100$\pm$370
RADIAL | 1349$\pm$0 | 997$\pm$3 | 1130$\pm$6 | 44501$\pm$1360 | 23119$\pm$1100 | 24300$\pm 1315$
WocaR | 1220$\pm$0 | 1207$\pm$39 | 1154$\pm$94 | 44156$\pm 2270$ | 25570$\pm$390 | 12750$\pm$405
RAD | 1340$\pm$0 | 1170$\pm$42 | 1211$\pm$56 | 42900$\pm$1020 | 29090$\pm$440 | 27150$\pm$505
A2B | 1350$\pm$0 | 1230$\pm$42 | 1240$\pm$56 | 44050$\pm$1020 | 38205$\pm$440 | 40015$\pm$505
A3B | 1350$\pm$0 | 1230$\pm$12 | 1250$\pm$30 | 44290$\pm$1250 | 41001$\pm$610 | 42645$\pm$458
| HalfCheetah | Ant
PPO | 5794 $\pm$ 12 | 1491$\pm$20 | -27$\pm$1288 | 5620$\pm$29 | 1288$\pm$491 | 1844$\pm$330
RADIAL | 4724$\pm$76 | 4008$\pm$450 | 3911$\pm$129 | 5841$\pm$34 | 3210$\pm$380 | 3821$\pm$121
WocaR | 5220$\pm$112 | 3530$\pm$458 | 3475$\pm$610 | 5421$\pm$92 | 3520$\pm$155 | 4004$\pm$98
RAD | 4426$\pm$54 | 4240$\pm$4 | 4022$\pm$851 | 4780$\pm$10 | 3647$\pm$32 | 3921$\pm$74
A2B | 5192 $\pm$56 | 4855$\pm$ 120 | 4722$\pm$33 | 5511$\pm$13 | 3824$\pm$218 | 4102$\pm$315
A3B | 5538$\pm$20 | 4986$\pm$41 | 5110$\pm$22 | 5580$\pm$41 | 4071$\pm$242 | 4418$\pm$290

##  Appendix E Training Details and Hyperparameters

###  E.1 Model Architecture

Our DQN and PPO models follow settings common to the current lineage of robust
RL work (SA-MDP, Radial, WocaR, RAD). For C-ACoE estimator functions, we use
two 64x hidden layers with a single linear output, congruent to the CCER
estimator in RAD and Worst-value estimator in WocaR. For Atari image domains,
we use a convolutional layer with an 8x8 kernel, stride of 4 and 32 channels,
a convolutional layer with a 4x4 kernel, stride of 2 and 64 channels, and a
final convolutional layer with a 3x3 kernel, stride of 1 and 64 channels. Each
layer is followed by a ReLU activation, and finally feeds into a fully
connected output.

###  E.2 Hardware

We train our models on an NVIDIA Tesla V100 with 16gb of memory.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: We have made sure that the main claims in the abstract and introduction accurately reflect the contributions and scope of the paper.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: We discuss the limitations in Section 6.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate ”Limitations” section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [Yes]

  14. Justification: All proofs are provided in the appendix.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: Beyond code and models, we provide pseudocode and formal descriptions of our methods to reproduce our algorithms.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: We include code snippets and file replacements to be added to an existing codebase, to reproduce our experimental results. The existing codebase may be found at https://github.com/umd-huang-lab/WocaR-RL. Upon publication, we will attach references to our own codebase in the paper.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: We include details on training and testing parameters in the main body of the paper. Additionally, we use default settings used by other robustness researchers, for ease of comparison.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: We report our variance measures in the results section. We include error in our tabular results, and where possible in our graphical results. Some of our graphical results become illegible with all errors included, however we note our main findings still hold in the absence of this data.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer ”Yes” if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: We compare compute time and rough complexity in the main body of the paper. We include our hardware details in the appendix.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: We follow the established ethics guidelines.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [N/A]

  49. Justification: Broadly, robustness work is more often positively impactful than not. The scope of our contributions, however, is limited to theoretical and empirical improvements, and we do not discuss applications and impact.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: Our contributions are mathematical, though arguably are safeguards themselves.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: All code and models are used with the MIT software license, cited and credited as necessary. Our implementation is an extension of https://github.com/umd-huang-lab/WocaR-RL, which is itself an extension of code used by other previous works Radial, RAD, PA-AD-ATLA, and SA-MDP.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: We include code snippets and comments for the relevant novel algorithms, as well as files and directions to integrate them into existing publicly available codebases to run.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: We do not use any crowdsourced resources in our paper.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: Our work does not involve any aspect requiring IRB approvals.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.