# Reward Shaping for Improved Learning in Real-time Strategy Game Play

John Kliem and Prithviraj Dasgupta  Distributed Intelligent Systems Section,
Information Technology Division
U. S. Naval Research Laboratory
Washington, DC, USA
{john.kliem<EMAIL_ADDRESS>
###### Abstract

We investigate the effect of reward shaping in improving the performance of
reinforcement learning in the context of the real-time strategy, capture-the-
flag game. The game is characterized by sparse rewards that are associated
with infrequently occurring events such as grabbing or capturing the flag, or
tagging the opposing player. We show that appropriately designed reward
shaping functions applied to different game events can significantly improve
the player’s performance and training times of the player’s learning
algorithm. We have validated our reward shaping functions within a simulated
environment for playing a marine capture-the-flag game between two players.
Our experimental results demonstrate that reward shaping can be used as an
effective means to understand the importance of different sub-tasks during
game-play towards winning the game, to encode a secondary objective functions
such as energy efficiency into a player’s game-playing behavior, and, to
improve learning generalizable policies that can perform well against
different skill levels of the opponent.

###### Index Terms:

Reinforcement learning, reward shaping, RTS games, capture-the-flag

##  I Introduction

Deep reinforcement learning (RL) has been shown to be a very successful
technique for enabling software agents to learn to play computer-based games
including complex, real-time strategy (RTS) games [1]. Recent successes of
deep RL include agents that can play single-player Atari games [2] and RTS
games such as Starcraft II, DOTA 2, and Quake III Capture-the-flag (CTF) [3,
4, 5] with human champion level expertise. Learning to play RTS games using
deep RL presents several challenging problems such as sparse rewards, large
state-action spaces, and correctly predicting the opponent’s move so that an
appropriate response move could be selected. Existing techniques to address
these challenges in RTS games mainly utilize opponent modeling techniques.
These include probabilistic tree search algorithms along with human user game-
play information to determine suitable moves that guarantee higher expected
rewards [6] and game-theoretic techniques such as self-play [7] and league
play [3] to explicitly build opponent models and learn winning moves by
training against them. While these techniques have shown promising results for
board and RTS games they require considerable computing resources and learning
times. For example, the Alpha-Star algorithm for playing Starcraft-II trained
$900$ distinct agents on a cloud of $32$ tensor processing units (TPUs) per
agent over a period of $44$ weeks and it required a dataset of $971,000$ game
replays [3] in order to achieve better than human-expert performance. It makes
sense to investigate if less time and resource-intensive techniques could be
used to learn to play RTS games without compromising performance.

To address this challenge, we investigate if reward functions, which are an
inherent part of RL algorithms, could be used to encode the opponent’s
behavior and other game-related learning tasks to learn suitable game-playing
strategies. Our main contribution in this paper focuses around a technique
called reward shaping [8] for successively adapting the reward function during
training of the RL agent guided by the player’s observation of the opponent
and the win/loss states within a game. To the best of our knowledge, our
technique is one of the first detailed studies of reward shaping to improve
learning performance in RTS games. We evaluate our reward shaping technique
where a player learns to play a 1-v-1, marine Capture-the-Flag (CTF) game
called Aquaticus CTF, against two skill levels of opponents. We have validated
our results using the MOOS-IvP simulation environment for marine robots. Our
results show that using reward shaping, a player can successfully learn to
play the game faster, achieve higher scores, and adapt its play suitably to
different levels of opponents, than without reward shaping. From our results,
we also make three novel observations about the impact of reward shaping: (1)
reward shaping can enable the learning agent to identify which sub-tasks are
more important to learn towards the overall task of winning the game, and
appropriately focus the reward shaping function towards those sub-tasks, (2)
reward shaping can be used to provide a suitable means to encode additional
objectives for a player that are not inherent in the game rules, for example,
moving in an energy efficient manner while playing the game, and, (3) reward
shaping enables a player to learn more generalizble policies. The rest of the
paper is structured as follows: in the next section we discuss relevant
literature on reward shaping for RTS games. Section III describes the
Aquaticus CTF game setting and rules, and, the formal framework for the game.
In Sections IV and V we describe the different reward shaping functions
relevant to the Aquaticus CTF game and our experimental validation of the
effects of the reward shaping functions on the player’s score and training
times, and, finally, we conclude.

##  II Related Work

Reward shaping has been proposed as a lightweight yet effective means to focus
an RL agent’s exploration of the environment during training towards regions
that enable it to reach its goal quickly. The main idea in reward shaping is
to update the nominal or sparse reward at a state-action pair as a function of
the suitability of the state-action pair towards reaching the goal [9]. In one
of the earliest works on reward shaping [8], authors proposed this function as
the difference of a potential function between the next and current states,
which was approximated using the negative Manhattan distance to the goal
between the two states. Subsequent researchers have extended this idea by
proposing variations to the potential function using future as well as past
states [10], using distance to nearest states given in expert demonstrations
[11], using a bi-level optimization problem to simultaneously determine
optimal values for the potential function and policy parameter (e.g., weights
of a policy network) of the learning problem [12], and, using a technique
called exploration guided reward shaping [13] where the reward of a state-
action pair is dynamically updated based on the gradient of an objective
function along with a bonus based on state visit counts. Most of these
techniques have been proposed for single agent tasks in a non-game playing
environment. Our work builds on these techniques by applying them to a
competitive, 2-player game. Related to our work, Jaderberg et al. [5] showed
that agents could learn to play CTF while using reward shaping to adaptively
learn to respond to abrupt changes in the game rules or environment. They
proposed reward shaping functions around $12$ consequential game-events such
as grabbing, capturing and retrieving the flag or tagging an opponent. Their
work’s main focus was on learning winning strategies while we focus on the
specific impacts of reward shaping in the CTF game. Another minor difference
is that our RL algorithm uses physically observable features of players like
locations and headings that can be readily mapped to physical robots and
robots’ on-board sensor data, in contrast to using derived or private
information like player scores and pixel maps of the game for learning. We
have used reward shaping as a means to model opponent behavior. The topic of
opponent modeling for learning to play computer games has also been researched
actively in recent years. Early approaches include self-play in repeated games
and its deep learning-based counterpart called fictitious neural self-play
[14]. More recently, He et al. [15] proposed the Deep Reinforcement Opponent
Network (DRON) algorithm that uses opponent behavior traces to update the
player’s DQN parameters directly or via a mixture of experts. In [16, 17],
authors proposed the learning with opponent learning awareness (LOLA)
algorithm that updates the gradient of a player’s policy network while
including both its own and its opponent’s value functions and policy gradient
updates. Opponent modeling as an auxiliary task was proposed in [18] where
players utilized a central policy network with parameter sharing but different
outputs for each player. The technique was mainly validated for smaller, grid-
like environments of games. Also, in contrast to our work, these approaches
directly update the player’s policy network’s gradients or parameters to
account for the opponent’s actions.

##  III Aquaticus Capture-the-Flag Game

Capture the Flag (CTF) is a classic game that is played between two opposing
teams. We consider a marine, $1$-v-$1$ version of the CTF game called Aqaticus
CTF between an attacker (red) and a defender (blue). Figure 1 shows an image
of the Aquaticus CTF playing field. The game is played in a rectangular,
obstacle-free playing field that is divided into two halves, which we will
refer to as a player’s zone. Each player has a base within its zone that
contains the player’s flag. The objective of the attacker is to pick the flag
from the defender’s base (called flag grab) and return with it back to its own
base (called flag capture), without being tagged by the defender during this
process. The defender’s objective is to tag the attacker while the attacker is
inside the defender’s zone. A player gets tagged by its opponent when it gets
within the tag range of its opponent inside the opponent’s zone. Tagging a
player forces the tagged player to return to its base to get un-tagged.
Similarly, if a player goes out-of-bounds (OOB) of the playing area during the
game111A player could go OOB either deliberately (e.g., to avoid collision
with another player) or inadvertently (e.g., due to limitations of its
vehicle’s dynamic capabilities like turn radius, inaccuracies of its on-board
localization system, etc.)., it has to revert to its base before starting to
play again. A player going OOB or getting tagged does not pause the game and
other players inside the playing area continue playing the game. The game
events and corresponding scoring rules for the Aquaticus CTF game are given in
Table I. There is a game arbiter called shoreside that has full observation of
the game, registers the different events as they occur in the game, maintains
players’ scores, and facilitates communication of the game states across
players.

Figure 1: Aquaticus CTF Game Field. TABLE I: Events and points in the Aquaticus CTF game. Event Name | Event | Attacker | Defender
---|---|---|---
| Description | Points | Points
Tag | Defender tags attacker before | $-1$ | $+2$
| grab or attacker goes OOB |  |
Retrieval Tag | Defender tags attacker | $-2$ | $+1$
| after grab |  |
Flag grab | Attacker picks flag | $+1$ | $-1$
| from defender’s base |  |
Flag capture | Attacker returns to its | $+2$ | $-2$
| base with grabbed flag |  |
Defender tag | Defender tagged by attacker | $+2$ | $-2$
| or defender goes OOB |  |

The game goes on for a pre-determined, fixed duration and comprises of one or
more rounds. A round starts with the attacker and defender starting at their
base regions. A round ends when one of the following events occurs:

  1. 1.

The attacker does a successful flag capture.

  2. 2.

The attacker gets tagged by the defender while trying to do a flag grab or
flag capture.

  3. 3.

The game duration ends before either of the aforementioned events occur

Note that by this definition, the duration of rounds, and consequently the
number of rounds in a game is not fixed.

Player Learning Tasks. A player that is learning to play the game has multiple
learning tasks. Each player’s primary task is to learn to defeat its opponent:
the defender learns how to move so that it can tag the attacker quickly before
the attacker grabs or captures the defender’s flag; the attacker learns how to
move so that it can grab and subsequently capture the flag without getting
tagged by the defender. The players are also not directly provided with the
location of the boundaries of the field. A secondary learning task for each
player is to be able to discover these boundaries so that it does not accrue
negative points due to going out-of-bounds (as per Table I), and learn to
remain within the playing area during the game. Finally, we also explore
additional learning tasks for players such as how to determine moves that
reduce energy expenditure while playing.

###  III-A Formal Game Representation

TABLE II: Observation and game parameters in the Aquaticus CTF game. Notation | Description
---|---
$\mathbf{X}^{field}$ | Rectangular playing field of size
| $D\times W$
$\mathbf{L}^{bounds}=\\{L^{upper},$ | Upper, lower, left and right boundaries
$L^{lower},L^{left},L^{right}\\}$ | of playing field
$Z^{att},Z^{def}$ | Attacker and defender zones (halves)
$\mathbf{x^{att}},\theta^{att}$ | Attacker location and heading
$\mathbf{x^{def}},\theta^{def}$ | Defender location and heading
$x^{aflg},x^{dflg}$ | Location of attacker and defender flags
$D^{base}$ | Base radius around player’s flag
$D^{tag}$ | Minimum distance between player and
| opponent locations for tagging
$D^{grab},D^{cap}$ | Min. dist. between player and flag
| or base locations for a flag grab/capture
$D^{warn}$ | Minimum distance between player and
| boundary or opponent for warning
$D^{threat}$ | Minimum distance between player and
| boundary or opponent for threat

The environment of the Aquaticus CTF game is represented by a rectangular,
obstacle-free playing area of width $W$ and depth $D$ bounded by a set of four
straight lines,
$\mathbf{L}^{bounds}=\\{L^{lower},L^{upper},L^{left},L^{right}\\}$. Attacker
and defender zones and bases are denoted by $Z^{att}$ and $Z^{def}$, and,
$B^{att}\subset Z^{att}$ and $B^{def}\subset Z^{def}$, respectively. The
attacker’s and defender’s base regions are denoted by $\mathbf{X}^{abase}$ and
$\mathbf{X}^{dbase}$ and flag locations within respective bases are denoted as
$\mathbf{x}^{aflg}$ and $\mathbf{x}^{dflg}$. The different parameters used in
the Aquaticus CTF game are shown in Table II. In the rest of the paper, for
the sake of brevity, we have given notations and definitions for the defender,
assuming that the corresponding notations and definitions for the attacker are
commensurately defined.

The defender’s state set is given by
$S^{def}=\\{(\mathbf{x^{def}},\theta^{def})\\}$, where
$\mathbf{x^{def}}=(x^{def},y^{def})\in\mathbb{Z}^{2}$ specifies its location
in the playing field.
$\theta^{def}\in\Theta=\\{\theta_{1},\theta_{2},\ldots,\theta_{K}\\}$ is a
discretization of the angular bearing $[-\pi,\pi]$ into $K$ disjoint segments.
The action set of the defender is specified as $A^{def}=\\{(\nu,\theta)\\}$,
where $\nu\in\\{0\cup\Re^{+}\\}$ and $\theta\in\Theta$ are the defender’s
velocity and heading respectively. The attacker’s state and action sets are
defined commensurately and denoted by $S^{att}$ and $A^{att}$. The state space
of the game is represented as the joint defender and attacker state sets,
$S=S^{def}\times S^{att}$. The action space of the game is given by the joint
actions of the defender and attacker, $A=A^{def}\times A^{att}$. The forward
dynamics model or transition function ${\mathbb{T}}:S\times A\times
S\rightarrow[0,1]$ gives a probability distribution for the next game state
given the current game state and current actions taken by the players.
Finally, for legibility, we define a subset of states corresponding to the
point-scoring game events in Table I,
$S^{EV}=\\{S^{tag},S^{ret},S^{oob},S^{grb},S^{cap}\\}$. The formal definition
of these sets in given in the supplementary material.

###  III-B Reinforcement Learning for Aquaticus CTF

We model the Aquaticus CTF game as a Markov game (or stochastic game) [19]
represented as $(S,A,\mathbb{T},R,\gamma)$, where $S,A,\mathbb{T}$ are the
state and action spaces, and transition function of the game as defined above.
$\gamma\in[0,1]$ is a discount factor and $R:S\times A\rightarrow\Re$ is the
players’ joint reward function. $R=(R^{att},R^{def})$ where $R^{def}$ and
$R^{att}$ are the defender’s and attacker’s reward functions respectively.
Each player’s objective in the game is to determine an action sequence for
itself that maximizes its rewards so that it ends up in states that result in
its getting more points than its opponent and winning the game. Because of the
competitive nature of the game, as shown by the players’ points structure in
Table I, higher rewards for one player at a certain state correspond to lower
rewards for its opponent at the same state and vice-versa. This makes each
player’s objective contrary to its opponent’s objective. To calculate a
winning action sequence, each player uses an RL algorithm. A player’s RL
algorithm takes the stochastic game parameters as input and outputs a policy
that prescribes the action for the player at each state in the game. We denote
$\pi^{def}$ and $\pi^{att}$ as the defender and attacker policies,
respectively. $\pi^{def}:S\rightarrow\Delta(A^{def})$, where $\Delta(A^{def})$
is a probability distribution over $A^{def}$. $\pi^{att}$ is defined
correspondingly.

A trajectory in the game is given by a sequence of state-action-reward tuples
over successive time-steps,
$\tau=((s_{0},a_{0},r_{0}),(s_{1},a_{1},r_{1}),...,(s_{T-1},a_{T-1},r_{T-1}),(s_{T},\\{\varnothing\\},r_{T}))$,
where $s_{t}\in S$, $a_{t}=(a^{att}_{t},a^{def}_{t})\in A$,
$s_{t+1}=\mathbb{T}(s_{t},a_{t})$, and $r_{t}\in R$. Here $s_{t}$, $a_{t}$ and
$r_{t}$ represent the game state, the joint action taken by the players and
the joint reward received by them in the $t$-th time-step.
$(a^{att}_{t},a^{def}_{t})=(\pi^{att}(s_{t}),\pi^{def}(s_{t}))$. $T$ denotes
the trajectory length measured in number of time-steps. A trajectory starts
when either the player or its opponent starts from its base to attack or
defend, and terminates when any one of the events in Table I, except ’Flag
grab’ occurs. A game consists of a sequence of trajectories
$(\tau_{1},\tau_{2},\tau_{3}...)$. Note that by the above definition of
trajectory, the length of a trajectory can vary depending on when a trajectory
ending event happens. Also, since a game goes on for a fixed duration, the
number of trajectories in a game could vary depending on the length of each
trajectory in it. For legibility, we denote the defender’s state-action-reward
components in trajectory $\tau$ as the defender trajectory,
$\tau^{def}=(s^{def}_{t},a^{def}_{t},r^{def}_{t}):t=0..T$, and,
correspondingly the attacker trajectory as $\tau^{att}$.

The defender’s return from defender trajectory $\tau^{def}$ is
$G_{\tau^{def}}=\sum_{t=0}^{T}r^{def}_{t}$. The attacker’s return is
correspondingly $G_{\tau^{att}}=\sum_{t=0}^{T}r^{att}_{t}$. The value of a
defender policy $\pi^{def}$ when the attacker uses attacker policy
$\hat{\pi}^{att}$ is given by
$V^{\pi^{def}|\hat{\pi}^{att}}={\mathbb{E}}_{\tau^{def}\sim\pi^{def}}[G_{\tau^{def}}]$.
Note that although attacker actions are not explicitly included in the r.h.s.
of the above equation, it indirectly influences $G_{\tau^{def}}$ as
$(s^{def}_{t+1},s^{att}_{t+1})=\arg\max_{s^{\prime}\in
S}\mathbb{T}(s_{t},(\pi^{def},\hat{\pi}^{att}),s^{\prime})$. Let $\Pi^{def}$
denote the set of all possible defender policies. The optimal defender policy
in response to attacker policy $\hat{\pi}^{att}$ is given by
$\pi^{def^{*}}|\hat{\pi}^{att}=\arg\max_{\pi^{def}\in\Pi^{def}}V^{\pi^{def}|\hat{\pi}^{att}}$.

To evaluate the performance of a defender trajectory
$\tau^{def}\sim\pi^{def}$, we define a trajectory score $J(\tau^{def})$. We
denote the number of tag events in $\tau^{def}$ as
$N^{tag}=\\#(\\{s^{def}_{t}:s^{def}_{t}\in tau^{def}\wedge s^{def}_{t}\in
S^{tag}\\}$, where the function $\\#()$ returns a count of the elements passed
to it. Similarly, we define counters for other game events in the trajectory,
namely, retrieval tags, out-of-bound events, grab and capture events as
$N^{ret},N^{oob},N^{grb}$ and $N^{cap}$ respectively. The trajectory score is
then given by:

$\displaystyle J(\tau^{def})$ $\displaystyle=$ $\displaystyle
c(S^{tag})N^{tag}+c(S^{ret})N^{ret}+c(S^{oob})N^{oob}$ (1)
$\displaystyle+c(S^{grb})N^{grb}+c(S^{cap})\,N^{cap},$

where $c()$ gives the points for the events corresponding to the states from
Table I. The score of defender policy $\pi^{def}$ is given by
$J(\pi^{def})=\mathbb{E}_{\tau^{def}\sim\pi^{def}}[J(\tau^{def})]$.

###  III-C Sparse Reward Functions

A player’s reward function is a central component of its RL algorithm as the
policy output by the RL algorithm is calculated by maximizing the player’s
expected returns. The Aquaticus CTF game, like most RTS games, is
characterized by a sparse reward structure where the players receive non-zero
rewards only at states corresponding to the game events in Table I and zero at
all other states. To model this sparse reward structure, we define a player’s
reward function as consisting of two components - an internal and an external,
as given below:

$\displaystyle R^{def}=R^{def}_{int}(s,a)+R^{def}_{ext}(s)$ $\displaystyle
R^{def}_{int}(s,a)=0,\quad\quad\forall s\in S$ $\displaystyle
R^{def}_{ext}(s)=\begin{cases}C_{ext}c(s),&\text{if }s\in S^{EV}\\\
0,&\text{otherwise}\end{cases}$ (2)

where $R^{def}_{int}$ is an intrinsic reward that is internal to each player,
and $R^{def}_{ext}$ is an external component determined by the game rules. If
the game state $s\in S^{EV}$, corresponding to one of the events in Table I,
$R^{def}_{ext}(s)$ equals to the points for that event in the table, $c(s)$,
scaled up by a constant scaling factor $C_{ext}$, otherwise it is $0$.
$R^{att}$ is correspondingly defined.

##  IV Reward Shaping Around Target and Avoid States

Reward shaping augments the intrinsic component of a player’s sparse reward
function at a state-action pair with a value that represents the suitability
of the state-action towards reaching the goal. Formally, the shaped reward,
$\hat{R}(\cdot)$ is given by:
$\hat{R}(s_{t},a_{t},s_{t+1})=R(s_{t},a_{t},s_{t+1})+\rho(s_{t},a_{t},s_{t+1})$
where $R(\cdot)$ is the sparse reward, and $\rho(\cdot)$ is a reward shaping
function. Following [8], we have used
$\rho(s_{t},a_{t},s_{t+1})=\gamma\phi(s_{t+1})-\phi(s_{t})$, where $\phi(s)$
gives the potential, or informally, the suitability of state $s$ towards
reaching a target state, and $\gamma$ is a discount factor. We consider three
different reward shaping functions222We have used piece-wise linear functions
as approximations of polynomial or exponential functions to speed-up
calculations as the reward shaping function is computed and added to the
rewards at each iteration of a training step. corresponding to principal tasks
to be learned by the players, as described below:

Boundary Reward Shaping. Learning the locations of the boundaries of the
playing field is one of the most difficult tasks for players as the sparse
reward function has an abrupt drop or ‘cliff’ at the boundary from zero to
$C_{ext}*c(s^{oob})$. In the worst case, to learn the location of all the
boundaries, the player would need to traverse from the inside to the outside
of the field across every point on each of the boundary lines during training
A training round also terminates as soon as the player that is learning leaves
the field, and so, the learning player would need to have an enormous number
of training rounds to completely learn the boundary. To ameliorate the
significant training times we associate a gradient with the reward function
near the boundaries, representing a slow drop off instead of a cliff in the
reward function at the boundaries. The boundary reward shaping function
implements a negative or downward gradient towards the boundaries from inside
the field and is given by:

$\rho^{bnd}=\begin{cases}-\kappa_{1}+\kappa_{2}\,d^{bound},&\text{if
}{D^{threat}\leq d^{bound}<D^{warn}}\\\
-\kappa_{3}+\kappa_{4}\,d^{bound},&\text{if }{0\leq d^{bound}<D^{threat}}\\\
0,&\text{otherwise}\end{cases}$ (3)

where $d^{bound}$ is the distance of the player to its closest boundary and
$\kappa_{1},\kappa_{2},\kappa_{3},\kappa_{4}\in\Re^{+}$ are positive
constants. Figure 2(a) illustrates an example of the boundary reward shaping
function.

Tag Reward Shaping. Tagging can be considered as an attractive potential
towards the opponent. With sparse reward for tagging, the learning player
encounters an abrupt increase in reward from zero to $C_{ext}*c(s^{tag})$ when
it gets at a tag state. Relying only on value updates gradually guides the
defender by building an attractive potential towards the tag states, but again
requires training rounds. With a reward shaping function that provides an
attractive gradient towards tag states, we can accelerate the value updates so
that the defender can recognize tag states when it is further away from them
and get attracted to them quickly, resulting in improved tagging performance
by the defender. The tagging reward shaping function we have used implements a
positive or upward gradient towards tag states, as given below:

$\rho^{tag}=\begin{cases}\omega_{1}-\omega_{2}\,d^{opp},&\text{if
}{D^{threat}\leq d^{opp}<D^{warn}}\\\ \omega_{3}-\omega_{4}\,d^{opp},&\text{if
}{D^{tag}\leq d^{opp}<D^{threat}}\\\ 0,&\text{otherwise}\end{cases}$ (4)

where $d^{opp}$ is the distance between a player and its opponent and
$\omega_{1},\omega_{2},\omega_{3},\omega_{4}\in\Re^{+}$ are positive
constants. Figure 2(b) illustrates an example of the tag reward shaping
function.

|
---|---
(a) | (b)
Figure 2: Example graphs for (a) boundary and (b) tag reward shaping
functions.

Energy Reward Shaping. A latent task of the players is to reduce the energy
they expend for moving around while playing the game. In computer games,
moving is usually free, but if we consider games with physical agents like
marine robots, frugal energy expenditure enables the robotic players to
persist longer in the game. Players could reduce their energy expenditure if
they could receive a reward signal that motivates them to follow shorter paths
towards their destination states in the game, or to remain stopped at
strategic locations and move towards their goal only when necessary. Towards
this objective, our energy reward shaping function gives a larger positive
reward for low energy actions like stopping, a slightly positive reward for
medium energy actions like continuing the action from the previous time-step
(e.g., moving at constant speed or remaining stopped), and a negative reward
for higher energy actions like changing speed or making turns, as given below:

$\rho^{enrg}=\begin{cases}\mu_{1},&\text{if }\nu=0\quad\text{(player stopped)
and }a_{t-1}=a_{t}\\\ \mu_{2},&\text{if }a_{t-1}=a_{t}\\\ -\mu_{3},&\text{if
}a_{t-1}\neq a_{t}\\\ \end{cases}$ (5)

where $\mu_{1},\mu_{2},\mu_{3}\in\Re^{+}$ are positive constants.

Reward Shaping for Learning Generalizable Policies. In RTS games, the skill
level of players on the opposing side is not known a priori and an intelligent
player should be able to adjust its playing strategy to win against a diverse
set of opponent skill levels. To achieve this, we investigate if reward
shaping can used to improve learning of policies that generalize well against
different skill levels of opponents. We investigate two training methods with
and without reward shaping: interleaved training and curriculum learning. In
interleaved training, we train a player against a randomly selected opponent
skill level in each training episode. In curriculum learning, a player is
trained sequentially against each opponent skill level. The trained policy
against one opponent skill level is used as the initial policy for training
against the next opponent skill level.

##  V Experimental Results

Game environment. For our experiments we have used the Aquaticus-Gym CTF game
environment that uses MOOS-IvP [20] as the robotic simulation platform. MOOS-
IvP simulates the dynamics of robotic platforms within a marine movement and
provides a message-based communication framework between the robots. The
Aquaticus-Gym CTF layer handles the CTF game rules along with an interface for
RL algorithms. It receives the current game state information from MOOS-IvP
and returns an action (speed, heading) for each player using the learned
policy. Figure 3 shows a schematic of the interaction between the Aquaticus-
Gym CTF and MOOS-IvP components of the software.The parameters in the game are
field width, $W=160\,$m, field depth, $D=80\,$m. This makes each player’s
zone’s size $80\times 80\,$m2. The base radius ($D^{base})$, tag range
($D^{tag}$) and grab range ($D^{grab}$) are $10\,$m each. Game durations
measured in clock time are $40$ minutes during training and $60$ minutes
during evaluation; simulations were run at $4\times$ speed-up. The initial
parameters used in the reward shaping equations, Equations 3\- 5 are
$\kappa_{1}=-0.1875$, $\kappa_{2}=0.0.028125$, $\kappa_{3}=-0.375$,
$\kappa_{4}=0.0125$, $\omega_{1}=0.1875$, $\omega_{2}=0.028125$,
$\omega_{3}=-0.375$, $\omega_{4}=0.075$, $\mu_{1}=0.5$, $\mu_{2}=0.4$, and,
$\mu_{3}=-0.5$.

We experimented with two deep RL algorithms Proximal Policy Optimization (PPO)
[21] and Deep Q-network (DQN) [22] for training a defender agent with
different reward shaping functions. We chose these two algorithm as PPO has
been recently reported to improve learning in RTS games without requiring
excessive hyper-parameter tuning or domain-specific algorithm modifications
[23] while DQN has been one of the most successful algorithms for learning to
play Atari games![2]. We have used the implementation of these algorithms
available in the Stable-Baselines 3 library [24]. In this section, we discuss
the results with PPO as it yielded faster convergence and better game-playing
performance; results with DQN are given in supplementary material. Our PPO
policy network uses a multi-layer perceptron with $2$ layers of $64$
perceptrons each. The training hyper-parameters of the PPO algorithm are:
discount factor, $\gamma=0.99$, learning rate, $\alpha=0.0005$, and buffer
size$=5000$. All our experiments were run on a server with $200$GB RAM, four
$2.7$GHz $12$-core Intel Xeon CPUs each with $19$ MB cache, two Quadro P$5000$
GPUs each with $16$ GB RAM, with Ubuntu $20.10$ as the operating system. Each
PPO training was done for a minimum of $600$K time-steps, corresponding to
roughly $16$ hours of clock time for each training.

Figure 3: Software architecture of the Aquaticus Gym and Moos-IvP (top) used
in our experiments and a screenshot of the Aquaticus CTF game (bottom).

Experimental Setup. To clearly discern the effect of reward shaping, we have
evaluated our reward shaping techniques while training only one side, a
defender agent, against two levels of attackers, easy and hard. Both attackers
use a rule-based play strategy. The easy attacker (Att-E) follows a fixed
trajectory going back and forth between its own base and the defender’s base
and flag. It is agnostic of the defender’s presence or location and does not
avoid getting tagged. The hard attacker (Att-H) uses a potential fields-based
strategy to actively avoid getting tagged by the defender while finding a
trajectory to grab the flag or get back to its zone and base.

|  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) | (f)
Figure 4: (a) - (d) Training times (y-axis) and scores of a defender at the
start and end of score convergence using SR and reward shaping functions TRS,
BRS and BTRS that is trained and evaluted against: (a) Att-E attacker, with
$1X$ gradients of reward functions, (b) Att-E attacker, with $2X$ gradients of
reward functions, (c) Att-H attacker, with $1X$ gradients of reward functions,
(d) Att-H attacker, with $2X$ gradients of reward functions. (e) and (f)
action heat maps of the defender trained against Att-E and Att-H respectively
using SR and reward shaping functions TRS, BRS and BTRS with $1X$ and $2X$
gradients.

###  V-A Experimental Evaluation.

For each of our experiments, we used the sparse reward (SR) as the baseline,
and evaluated the defender agent’s score while using the tag (TRS), boundary
(BRS), and a combination of boundary and tag (BTRS) reward shaping functions.
Learning defender agents were evaluated at intervals of $10,000$ training
steps against Att-E and at intervals of $50,000$ training steps against Att-H.
To understand the effects of the reward shaping techniques, we have evaluated
four hypotheses, as given below:
Hypothesis 1. Reward shaping improves the performance of the defending player
in terms of reward convergence times and improved scores, as compared to using
sparse rewards only.
Hypothesis 2. When an agent has to learn multiple sub-tasks as part of its
overall task, reward shaping for the more difficult tasks results in higher
improvements in the agent’s performance than reward shaping for the less
difficult tasks.
Hypothesis 3. Reward shaping can be used to encode additional objective
functions like energy conservation while ensuring consistently positive
scoring policies when playing the game.
Hypothesis 4. Adjusting gradients of the reward shaping function can guarantee
faster convergence to higher rewards.
Hypothesis 5. Reward shaping improves learning generalizable policies.

Hypothesis 1. Reward shaping improves the performance of the defending player
in terms of reward convergence times without degrading its performance, as
compared to using sparse rewards only. To validate Hypothesis 1, in our first
set of experiments, we trained a defender agent against Att-E and Att-H using
the different rewards functions within the PPO algorithm. The results are
shown in Figure 4 (a)-(b). Each vertical bar in the graphs represents the
number of time steps between the time the defender’s reward starts to converge
and the time when the reward converges for each reward shaping function. The
numbers at the bottom and top of each bar give the defender’s scores at those
times respectively. Against, Att-E, the trend lines of the final reward
convergence times shows a downward gradient indicating that the reward shaping
functions improve the learning times as compared to the SR baseline. The final
score achieved by the defender also remains stable or decreases nominally
against Att-E indicating that the improved learning times do not degrade the
defender’s performance. However, against the harder Att-H, while the learning
times again exhibit a downward trend line, the final score of the defender
decreases more aggressively by about $20\%$ for different the reward shaping
functions. To investigate the degrading scores of the defender against Att-H,
we adjusted the gradients of the reward shaping functions in Equations 3-4 to
lower values ($0.5\times$) and higher values ($2\times$). The corresponding
results with $2\times$ gradients are in Figure 4 (d) and show that the
defenders reward converge times improve without degrading the scores. However,
with $2\times$ gradients, the defender’s learning times and performance
against Att-E get degraded, as seen in Figure 4 (c). This indicates that when
the opponent’s behavior changes, the defender’s reward shaping function’s
gradient needs to be carefully adjusted to continue to yield good performance
in the game.

We ran the same set of experiments with the defender using the DQN algorithm.
The gradient of the BRS function had to be slightly changed for the DQN
algorithm. In general, we found the DQN results to be consistent with the PPO
findings; DQN results are given in detail in the supplementary material.
Overall, our experiments showed that adding reward shaping to the tag task
(especially for Att-H) and to the boundary task (especially for Att-E)
considerably improved the defender’s rewards convergence time without
degrading its peformance, indicating that our results support Hypothesis 1.

Hypothesis 2. When an agent has to learn multiple sub-tasks as part of its
overall task, reward shaping for the more difficult tasks results in higher
improvements in the agent’s performance than reward shaping for the less
difficult tasks. A noteworthy observation in Figures 4(a)-(d) is that against
Att-E, using BRS improves the learning time over using TRS, but against Att-H,
this relationship between the learning times gets reversed. With BRS the
defender gets negative rewards for getting closer to the boundary, and
consequently, learns to play in a more confined area. Against the fixed
trajectory Att-E that moves close to the middle of the field, this culminates
in more tagging opportunities, But against the more dexterous Att-H, playing
in a smaller area away from boundaries leaves more unguarded space for the
attacker to get through. Consequently, against Att-H, targeting the attacker
more aggressively using TRS yields faster learning times for the defender. To
further validate this hypothesis we performed another experiment to make the
tag task more difficult by changing the attacker from easy to hard while
keeping the boundary learning task (size of the field) unchanged. Comparing
Figures 4 (a) and (b), we see that increasing the difficulty of the tagging
sub-task Figure (b) results in agents trained using TRS converge quicker and
achieve higher scores compared to both SR or BRS in Figure (b). This shows
that when the attacker goes from easy to hard (tag task gets harder), the
training time with BRS increases, while the training time using TRS decreases
as compared to SR.

One anomaly we observe within the PPO algorithms is that while combining tag
and boundary reward in the BTRS case against Att-H, we are getting poorer
scores than with TRS or BRS alone. A closer observation of the defender’s
moves showed that this was due to the tag reward interfering with the boundary
reward and causing the defender to lose points by going out-of-bounds often.
Additional experiments showed that for, increasing the boundary reward’s
weight to about $2$ times the tag reward could address this problem with BTRS.

In summary, we see from these experiments that reward shaping for the harder
boundary task gives more improvements than for the easier tag task, supporting
Hypothesis 2 for PPO algorithms. Similar to the evaluation of Hypothesis 1 we
verified these findings with DQN algorithms; the supplementary material has
details of the DQN results.

Hypothesis 3. Reward shaping can be used to encode additional objective
functions like energy conservation without significantly diminishing the
player’s performance in the game. A successful validation of Hypothesis 3
would show that when energy reward shaping is used, the agent chooses actions
with lower energy requirements (e.g., stopping or maintaining speed and
bearing between successive actions) more often than higher energy actions
(e.g., changing speed or bearing). The energy reward shaping function, EFF,
has a limited effect against Att-E as stopping along the attacker’s predefined
path is already the optimal strategy that is converged on by all reward
shaping functions expect 2BTRS, as seen in Figure 4 (e). The significance of
EFF is more evident against Att-H in Figure 4 (f) where for most of the reward
shaping functions the defender rarely uses the low energy stop or constant
speed actions (white square at Speed: $0.0$ and Heading: $0.0$, light squares
at most other headings); BRS and 2BRS prefer to stop more frequently, but they
perform poorly against Att-H as shown in Figures 4(b) and (d). Only when the
defender uses energy reward shaping EFF, the stop action is chosen very often
(dark square at EFF-Speed: $0.0$, light squares at all other headings).
Sacrificing mobility for energy efficiency evidently reduces the defender’s
score by around $50-60\%$ (graph in supplementary material) with EFF as
compared to other reward shaping functions with full mobility. But analysis of
the attacker and defender trajectories in the game show that by using EFF
reward shaping versus an Att-H attacker, the defender can learn to
strategically position itself very near to its flag - the attacker avoids
getting tagged on its approach trajectory to grab the flag, but the defender
is able to tag the attacker on its exit trajectory and retrieve the grabbed
flag while barely moving from its learned stationary position. 333We found
that adjusting the EFF reward function hyper-parameters in Equation 5 can
reduce the defender’s propensity to remain stationary, and, consequently
improve its scores, but at the expense of spending more energy.. Overall, the
defender’s behavior with EFF reward shaping support our claim in Hypothesis 3
that additional objectives of a player like reducing energy consumption during
playing can be controlled via a reward-shaping function without significantly
degrading the player’s performance. Reward shaping with EFF within the DQN
algorithm also supports these claims; details are in the supplementary
material.

Figure 5: Training times for a defender policy playing against Att-E using
$1X$ (solid lines) and $2X$ (dashed lines) gradients with TRS, BRS and BTRS
reward shaping functions. The red and magenta vertical lines show the number
of time-steps at which the defender scores converge with $2X$ and $1X$
gradients respectively.

Hypothesis 4. Hyper-parameter tuning of the reward function can guarantee
faster convergence to higher rewards depending on the opponent’s behavior. To
validate Hypothesis 4, we changed the gradients of TRS and BRS functions
($\kappa_{2},\kappa_{4}$ in Equation 3 and $\omega_{2},\omega_{4}$ in Equation
4) to twice their original gradients. Figure 5 shows that defender scores
against Att-H (dashed lines) for all the reward shaping functions with $2X$
gradient converge to a score between $20-30$ within $400$K training time-steps
(marked with a red vertical line). In contrast, while using $1X$ gradient for
reward shaping functions, defender scores against Att-H (solid lines) take
$550$K training time-steps (solid magenta vertical line) to converge to
comparable scores. This corresponds to a significant improvement of $150$K
training time-steps for $2X$ gradient. Similar experiments of changing the
reward shaping function gradients from $1X$ to $2X$ against Att-E yielded
positive, but smaller improvements of $5$K-$10$K fewer training time-steps
with $2X$ versus $1X$, for all the reward shaping functions. Finally, changing
the gradient from $1X$ to $0.5X$ resulted it poorer performance against both
Att-E and Att-H; it increased the number of training time-steps required for
scores to converge as well as degraded the final converged scores. Overall,
from these experiments, we see that a player’s performance in terms of score
and learning times can be significantly affected by hyper-parameter tuning of
the gradient of the reward shaping function. Also, the optimal hyper-
parameters are a function of the opponent strategy and they have to be re-
tuned for different opponent strategies. Overall, these findings show that our
Hypothesis 4 is supported.

---
Figure 6: Defender training times (y-axis) and scores (shown next to circle) using different reward shaping functions while playing against an unseen attacker strategy (top), and same attacker strategy as the one trained against (bottom). |  |
---|---|---
(a) | (b) | (c)
Figure 7: Training curves for the defender with SR and 2BTRS) with (a)
interleaved training: the attacker (Att-E or Att-H) trained against is picked
randomly at the start of every training episode, (b) curriculum learning (E
$\rightarrow H)$: training first against Att-E up to convergence followed by
training against Att-H, (c) training first against Att-H up to convergence and
then against Att-E.

Hypothesis 5. Reward shaping improves learning generalizable policies. The
main objective of Hypothesis 5 is to evaluate if reward shaping can improve
learning policies that can perform well against different skill levels of
attackers. We performed two sets of experiments to evaluate this hypothesis.
In the first experiment, we evaluated policy transfer, that is, if a defender
policy trained against one attacker type could continue to play effectively if
it encountered the other attacker type during evaluation. The results from the
experiment are given in Figure 6. We see in Figure 6 (top) that when a
defender trained against one attacker strategy (Att-E or Att-H) but played
against a different attacker strategy than the one trained on, it scores
diminished significantly. For comparison, the defender scores while playing
against the same attacker strategy it had been trained with are given in
Figure 6(bottom). The results also show that reward shaping can provide
minimal improvement on these transfer learning tasks.

In order to create more general policies, we looked at two methods,
interleaved and curriculum learning, as discussed in Section IV. For training
in interleaved learning, at each training episode we randomly picked the
attacker type, Att-E or Att-H, the defender would train against. The results
can be seen in Figure 8(a). Without reward shaping the generalized learning
tasks don’t converge within $800$K steps (dashed lines), which is likely
attributed to learning game features other than tagging range or game
boundaries within the game context. On the other hand, generalized learning
using 2BTRS reward shaping results in the convergence of a policy against both
Att-E and Att-H (solid lines), comparable to just training a policy against
one of these strategies. This more efficient training could be attributed to
reward shaping, which allows the agent to learn the boundary game context when
playing against the Att-E strategy and then gain more context on the tagging
range when playing against the Att-H strategy.

For training with curriculum learning, a defender was initially trained
against Att-E for $150$K time-steps and then the Att-E trained policy was
further trained against Att-H up to $800$K time-steps. The training curve for
training the Att-E trained policy against Att-H is shown in Figure 8(b). We
see that using sparse rewards results in learning loss (dashed lines) - the
defender’ score against Att-E decreases as it trains more against Att-H and
the policy does not generalize to play against both Att-E and Att-H. In
contrast, with reward shaping with 2BTRS (solid lines), we see that the
defender’s score improves against Att-H without learning loss against Att-E.
We repeated the curriculum learning experiment while first training the
defender against Att-H for $550$K time-steps followed by training against
Att-E up to $800$K time-steps. The results in Figure 8(c) show that SR faces
learning loss as its score against Att-H decreases (dashed lines), but with
2BTRS reward shaping the defender can generalize to play well against both
strategies. These results show that reward shaping within curriculum learning
leads to a defender policy that can learn to play against a new strategy
without forgetting to play against the initial strategy it had learned to play
against, than with only sparse rewards. In summary, we show that reward
shaping improves learning a generalizable policy that can continue to perform
well if the opponent dynamically changes its strategy during game-play,
supporting Hypothesis 5.

##  VI Conclusions and Future Directions

Our main contribution in this paper showed that reward shaping can be used as
an effective yet computationally frugal means to improve a player’s
performance and training times in a CTF RTS game. Additionally, reward shaping
could identify which sub-tasks have a bigger impact on a player’s performance
and wins in a game, and could also be used to encode more important, secondary
performance objectives such as reduced energy expenditure into a player’s
behavior. In this work, we have hand-crafted different reward-shaping
functions based on common sense and human intuition. An interesting future
problem would be to use a learning algorithm to determine and fine-tune
suitable reward shaping functions along the directions in [25]. Another
related problem is to automatically determine where or for which sub-tasks in
a game do the reward functions have the most impact on a player’s performance.
Since the reward shaping function tuning is a non-trivial task and has to be
done in parallel to the main task of learning to play the game, this would
provide us a recipe for suitably allocating computing resources for reward
shaping tasks. Future work exploring these concepts would be to design a
general policy that leverages a learned hierarchical policy and then compare
the compute time correlates to a pure general learning approach.

Credit Authorship Statement. Dasgupta was responsible for conceptualizing and
supervising the research and writing Sections I-IV of the paper. Kliem was
responsible for developing the software code, running the experiments and
presenting the research results via graphs and other visualizations. Sections
V-VI and the supplementary material in the paper were written jointly by both
authors.

Acknowledgements. This research was supported by the Office of Naval Research
through an Naval Research Laboratory $6.1$ Base Funding project grant titled
’Game Theoretic Machine Learning with Defense Applications’.

## References

  * [1] K. Shao, Z. Tang, Y. Zhu, N. Li, and D. Zhao, “A survey of deep reinforcement learning in video games,” 2019. [Online]. Available: https://arxiv.org/abs/1912.10944
  * [2] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev _et al._ , “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, 2019.
  * [4] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Jozefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. d. O. Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang, “Dota 2 with large scale deep reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1912.06680
  * [5] M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castañeda, C. Beattie, and et al., “Human-level performance in 3d multiplayer games with population-based reinforcement learning,” _Science_ , vol. 364, no. 6443, pp. 859–865, 2019. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.aau6249
  * [6] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * [7] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton _et al._ , “Mastering the game of go without human knowledge,” _nature_ , vol. 550, no. 7676, pp. 354–359, 2017.
  * [8] A. Ng, D. Harada, and S. J. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _ICML_ , 1999\.
  * [9] M. Grzes, “Reward shaping in episodic reinforcement learning,” in _Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, São Paulo, Brazil, May 8-12, 2017_. ACM, 2017, pp. 565–573.
  * [10] E. Wiewiora, G. Cottrell, and C. Elkan, “Principled methods for advising reinforcement learning agents,” in _Proceedings of the Twentieth International Conference on International Conference on Machine Learning_ , ser. ICML’03. AAAI Press, 2003, p. 792–799.
  * [11] T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Nowé, “Reinforcement learning from demonstration through shaping,” in _Proceedings of the 24th International Conference on Artificial Intelligence_ , ser. IJCAI’15. AAAI Press, 2015, p. 3352–3358.
  * [12] Y. Hu, W. Wang, H. Jia, Y. Wang, Y. Chen, J. Hao, F. Wu, and C. Fan, “Learning to utilize shaping rewards: A new approach of reward shaping,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 931–15 941, 2020.
  * [13] R. Devidze, P. Kamalaruban, and A. Singla, “Exploration-guided reward shaping for reinforcement learning under sparse rewards,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 5829–5842, 2022.
  * [14] J. Heinrich and D. Silver, “Deep reinforcement learning from self-play in imperfect-information games,” 2016. [Online]. Available: https://arxiv.org/abs/1603.01121
  * [15] H. He, J. Boyd-Graber, K. Kwok, and H. Daumé III, “Opponent modeling in deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1804–1813.
  * [16] J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch, “Learning with opponent-learning awareness,” in _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018_ , 2018, pp. 122–130.
  * [17] T. Willi, A. H. Letcher, J. Treutlein, and J. Foerster, “Cola: consistent learning with opponent-learning awareness,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 23 804–23 831.
  * [18] P. Hernandez-Leal, B. Kartal, and M. E. Taylor, “Agent modeling as auxiliary task for deep reinforcement learning,” in _Proceedings of the Fifteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2019, October 8-12, 2019, Atlanta, Georgia, USA_ , G. Smith and L. Lelis, Eds. AAAI Press, 2019, pp. 31–37. [Online]. Available: https://ojs.aaai.org/index.php/AIIDE/article/view/5221
  * [19] L. S. Shapley, “Stochastic games,” _Proceedings of the national academy of sciences_ , vol. 39, no. 10, pp. 1095–1100, 1953.
  * [20] M. R. Benjamin, H. Schmidt, P. M. Newman, and J. J. Leonard, “Autonomy for unmanned marine vehicles with moos-ivp,” _Marine Robot Autonomy_ , pp. 47–90, 2013.
  * [21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [22] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [23] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 611–24 624, 2022.
  * [24] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.
  * [25] D. Mguni, J. Wang, T. Jafferjee, N. Perez-Nieves, W. Song, Y. Yang, F. Tong, H. Chen, J. Zhu, Y. Du _et al._ , “Learning to shape rewards using a game of switching controls,” _arXiv preprint arXiv:2103.09159_ , 2021.

## Appendix

|  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) | (f)
|  |
(g) | (h) | (i)
Figure 8: Training times for different gradients of the TRS, BRS and BTRS reward shaping functions (a-d) Att-H and (e-h) Att-E. Figure (i) bar graph representing the final scores achieved at the end of training 200k timesteps against Att-E and 800k timesteps against Att-H. Notation | Description
---|---
$\theta_{p}$ | Player’s heading
$\\{(d_{o_{j}},\hat{\theta}_{o_{j}},\theta_{o_{j}})\\}$ | Distance, angle and heading to opponent $j$
$\\{(d_{p_{j}},\hat{\theta}_{p_{j}},\theta_{p_{j}})\\}$ | Distance, angle and heading to teammate $j$
($d_{flag,o},\hat{\theta}_{flag,o})$ | Distance, angle to opponent’s flag
($d_{flag,p},\hat{\theta}_{flag,p})$ | Distance and angle to player’s (self) flag
$\mathbf{D}_{bounds}=\\{d_{upper},d_{lower},$ | Distances to upper, lower, left and right
$d_{left},d_{right}\\}$ | (lines) boundaries of playing field
TABLE III: Feature space derived from observations used for training the RL
algorithm of a player in the Aquaticus CTF Game.

### Event State Sets

We define the following state sets corresponding to game events where players
score or lose points:

  * •

Tag set is the set of states at which players can tag each other, before an
attacker has grabbed the defender’s flag. Formally, it is defined as
$S^{tag}=\\{s:s=(s^{def},s^{att})\in
S\wedge|\mathbf{x}^{def}-\mathbf{x}^{att}|\leq D^{tag}\\}$ (before attacker
grabs flag). A defender tag happens if both players are in the defender’s zone
in a tag state, otherwise it is an attacker tag.

  * •

Retrieval Tag Set is similar to a tag set, except it happens only after the
attacker has grabbed the defender’s flag. Formally, it is defined as
$S^{ret}=\\{s:s=(s^{def},s^{att})\in
S\wedge|\mathbf{x}^{def}-\mathbf{x}^{att}|\leq D^{tag}\\}$ (after attacker
grabs flag). The same conditions for defender and attacker tag as in the tag
set apply to retrieval tag states as well.

  * •

Grab Set is the set of states at which an attacker grabs the defender’s flag
from the defender’s base. Formally, $S^{grb}=\\{s:s=(s^{def},s^{att})\in
S\wedge|\mathbf{x}^{att}-\mathbf{x}^{dflg}|\leq D^{grab}\\}$.

  * •

Capture Set is the set of states at which an attacker captures the defender’s
flag by bringing it back to its own base. Formally,
$S^{grb}=\\{s:s=(s^{def},s^{att})\in
S\wedge|\mathbf{x}^{att}-\mathbf{x}^{abase}|\leq D^{cap}\\}$.

  * •

Out-of-Bounds (OOB) Set is the set of states at which a player is outside the
playing field. Formally, the defender OOB set is defined as
$S^{oob,def}=\\{s:s=(s^{def},s^{att})\in
S\wedge\mathbf{x}^{def}\notin\mathbf{X}^{field}\\}$. The attacker OOB set
$S^{oob,att}$ is defined commensurately.

The game event set $S^{EV}=\\{S^{tag},S^{ret},S^{grb},S^{cap},S^{oob}\\}$
gives a collection of the above state sets.

###  VI-A Attacker and Defender Sparse Reward Functions

The attacker and defender sparse reward functions specify the external reward
component of their respective reward functions given in Equation 2. They are
obtained by multiplying the points obtained at game event states $S^{EV}$ by a
constant scaling factor $C_{ext}$. The different attacker and defender sparse
reward functions for $C_{ext}=50$ (used in our experiments) are given below:

$R^{tag,def}=\begin{cases}100,&\text{if
}{|\mathbf{x}^{att}-\mathbf{x}^{def}|\leq D^{tag}}\text{ and}\\\
&\mathbf{x}^{att},\mathbf{x}^{def}\in Z^{def},\quad\text{(before grab)}\\\
0,&\text{otherwise}\end{cases}$

$R^{ret,def}=\begin{cases}50,&\text{if
}{|\mathbf{x}^{att}-\mathbf{x}^{def}|\leq D^{tag}}\text{ and}\\\
&\mathbf{x}^{att},\mathbf{x}^{def}\in Z^{def},\quad\text{(after grab)}\\\
0,&\text{otherwise}\end{cases}$

$R^{grab,att}=\begin{cases}50,&\text{if
}{|\mathbf{x}^{att}-\mathbf{x}^{dflg}|\leq D^{grab}}\\\
0,&\text{otherwise}\end{cases}$

$R^{cap,att}=\begin{cases}100,&\text{if
}{|\mathbf{x}^{att}-\mathbf{x}^{abase}|\leq D^{cap}},\quad\text{(after
grab)}\\\ &\text{ where }\mathbf{x}^{abase}\in\mathbf{X}^{abase}\\\
0,&\text{otherwise}\end{cases}$

$R^{oob,def}=\begin{cases}-100,&\text{if player out-of-bounds}\\\
0,&\text{otherwise}\end{cases}$

###  VI-B Deep Q-Network and Related Experiments

|
---|---
(a) | (b)
Figure 9: Example graphs for (a) boundary and (b) tag reward shaping
functions.

DQN Experimental Settings. The same game settings used for the DQN results are
the same as the PPO results except where explicitly stated. The parameters
used in the reward shaping equations, Equations 3\- 5 are $\kappa_{1}=-0.75$,
$\kappa_{2}=0.0375$, $\kappa_{3}=-1.5$, $\kappa_{4}=0.1125$,
$\omega_{1}=0.75$, $\omega_{2}=0.025$, $\omega_{3}=1.75$, $\omega_{4}=0.075$,
$\mu_{1}=0.5$, $\mu_{2}=0.4$, and, $\mu_{3}=-0.5$.

We have used Stable-Baselines 2 library [24] for the DQN algorithm
implementation. The policy network of the DQN has a multi-layer perceptron
with $2$ layers of $64$ perceptrons each. DQN training hyper-parameters are:
discount factor, $\gamma=0.99$, learning rate, $\alpha=0.0005$, and buffer
size$=5000$. Each DQN training was done for a minimum of $600$K time-steps,
corresponding to roughly $16$ hours of clock time for each training.

DQN Hypothesis 1. To validate Hypothesis 1, we trained a defender against an
easy attacker while using sparse rewards only (SR), and reward shaping
functions for tag (TRS), boundary (BRS), and a combination of boundary and tag
(BTRS). Our results are summarized in Table IV. Each sub-table gives the
attacker’s performance for one of these reward function settings. The
performance of the attacker in terms of its main events, grab and capture, and
of the defender, in terms of its main events, tagging the attacker, retrieving
the flag from the attacker and getting tagged are recorded at intervals of
$50,000$ training steps. When the defender plays optimally, the attacker
should get $0$ grabs and captures, while the defender should get most possible
tags, $0$ retrieval tags (as it tags the attacker before attacker grabs flag),
and $0$ defender tags (never get tagged by attacker or leave the playing
field). Highlighted rows in Table IV show the best performance for each of our
reward function settings. We see that with sparse rewards (leftmost sub-table)
the defender gets its best result at $400$K training steps when it gets $10$
tags ($6$ before attacker’s flag grab, $4$ before attacker’s flag capture) and
never gets tagged itself. When we augment sparse rewards with tag reward
shaping (second sub-table), the defender’s rewards convergence time reduces by
$50\%$ from $400$K to $200$K training steps, although its performance in terms
of tags remains comparable to sparse rewards, albeit a bit lower in terms of
defender getting tagged $4$ times due to leaving the playing field. A higher
improvement over sparse reward comes with adding boundary reward shaping
(third sub-table). It improves the reward convergence time by $75\%$ over
sparse reward while increasing the number of before-grab tags to $9$. Finally,
the best result comes with combining boundary and tag rewards - the reward
convergence time improves by $87.5\%$ and performance improves to $11$ tags,
all before the attacker is able to grab the flag even once, without the
defender itself ever getting tagged. Overall, we see that adding reward
shaping to the tag task and, more importantly, to the boundary task
considerably improves the rewards convergence time and game-play quality of
the defender. These results indicate that Hypothesis 1 is supported.

DQN Hypothesis 2. For validating Hypothesis 2, note that in Table IV learning
the boundary task is more difficult and more important for the defender.
Consequently, shaping the boundary reward gives a bigger improvement in
performance than shaping tag reward. For further validation of Hypothesis 2 we
did another experiment to make the tag task more difficult by changing the
attacker from easy to hard while keeping the boundary learning task (size of
the field) unchanged. In Table V, we report the defender’s scores calculated
using Equation 1 for this experiment. The rows of the sub-tables correspond to
defenders trained against an easy attacker (Def-E) or hard attacker (Def-H),
while the columns correspond to easy and hard attackers (Att-E and Att-H) that
the defenders in the rows are evaluated to play against. The scores in Table V
show that when the attacker goes from easy to hard (tag task gets harder), the
performance of defender Def-E diminishes significantly. Also, as shown in the
bottom sub-table (Def-H vs. Att-H), the defender is able to recuperate its
performance to a score of $20$ at $100$K training steps indicating that the
boundary task is a more difficult learning task even with the hard attacker
and shaping its reward gives the most performance improvement. In summary, we
see from these experiments that reward shaping for the harder boundary task
gives more improvements than for the easier tag task, supporting Hypothesis 2.
One anomaly we observe is that while combining tag and boundary reward in the
BTRS case against Att-H, we are getting poorer scores than with reward shaping
for tag (TRS) or boundary (BRS) alone. A closer observation of the defender’s
moves showed that this was due to the tag reward interfering with the boundary
reward and causing the defender to lose points by going out-of-bounds often.
Additional experiments showed that for the BTRS case, increasing the boundary
reward’s weight to about $8$ times the tag reward could address this issue,
improving the Def-H defender’s score to $8$ in $150$K training steps against
Att-H.

DQN Hypothesis 3. Our Hypothesis 3 relates to evaluating if energy reward
shaping could reduce the energy expenditure in terms of a player’s movements
or actions. The lowest energy consuming action is the stop action (SPD:
$0.0$), intermediate energy is required for maintaining the same speed and
heading without accelerating or decelerating (HDG: $0.0$), while higher
energies are required for steering or changing the heading of the player. A
successful validation of the hypothesis would show that when energy reward
shaping is used, the agent chooses actions with lower energy requirements more
often than higher energy actions. In Figure 10, the left vertical axis shows
the different actions available to the defender, the horizontal axis shows the
different reward shaping functions with and without energy reward shaping, and
the plot shows the frequency with which different actions are picked by the
defender while using different reward shaping functions, in terms of a heat-
map. We see that using only sparse rewards (SR) the player rarely uses the low
energy stop or constant speed actions (dark square at SR-SPD: 0), but when
energy reward shaping is added (SR+ERS), the stop action is chosen very often
(bright square at (SR+ERS)-SPD: 0). Similarly, comparing boundary and
boundary-tag reward shaping with and without energy reward shaping (SR+BRS vs.
SR+BERS and SR+BTRS vs. SR+BTERS), we see that adding energy reward shaping
the player is learning the energy conservation sub-task and picking the stop
and constant speed actions more frequently over other, more energy consuming
actions. These comparisons support our claim in Hypothesis 3 that additional
objectives of a player like reducing energy consumption during playing can be
controlled via a reward shaping function.

DQN Hypothesis 4. Hypothesis 4’s objective is to evaluate the affect of the
reward shaping function’s shape, determined by the function’s gradient, on the
reward convergence times and player performance. We changed the gradients of
the tag and boundary reward shaping functions ($\kappa_{2},\kappa_{4}$ in
Equation 3 and $\omega_{2},\omega_{4}$ in Equation 4) to three times their
original gradients. We expect $3\times$ gradient on the reward shaping
functions to yield lower training times. For visualizing the reward shaping
function gradient’s effect, we plot position heat maps that show the frequency
with which the defender visits different locations inside the playing field
for different reward shaping functions, in Figure 11. We see that when tag
reward gradient goes from $1\times$ to $3\times$ (top row sub-plots in Figure
11), the defender starts behaving more aggressively - it ignores the boundary
sub-task and prioritizes the tag sub-task, resulting in it following the
attacker’s path (light blue trail in center of top right sub-plot), stopping
just outside its zone border (bright yellow spot) to tag the attacker every
time, as soon as it enters into the defender’s zone. However, this behavior
also effects more action changes, and, consequently, more energy expenditure
by the defender, as evidenced by comparing the SR+TRS and SR+3TRS action heat
maps in Figure 10. Changing the gradient of the boundary reward from $1\times$
to $3\times$ (bottom row sub-plots in Figure 11) has the effect of contracting
the area in which the defender moves around during playing. This is manifested
by the lighter zone getting more concentrated near the defender’s flag. The
defender also chooses the stop action more frequently because the increased
boundary gradient makes it learn that if it does the stop action within its
base and near its flag more often, it can avoid entering or getting closer to
areas near the boundary with negative rewards. Overall, we see that while the
player’s behavior can be modified significantly by changing the gradient of
the reward shaping function, our Hypothesis 4, increasing reward function
gradient ensures faster convergence times and higher rewards, is not fully
supported. In some instances, the $3\times$ gradient resulted in a higher
score than their non-increased counterpart, but the higher score came with
significant increases in the training time. Increasing the gradient also
resulted in poorer generality and performs worse against Att-H than with
$1\times$ gradient. These findings point to the direction that reward shaping
functions associated with different tasks interact with others in a complex,
non-linear manner and need to be understood and designed carefully.

TABLE IV: Performance of different reward shaping functions with defender trained against Att-E and evaluated against Att-E. First two columns and latter three columns of each sub-table show attacker’s and defender’s successful events resp. | Att-E | Def-E-SR | Att-E | Def-E-SR+TRS | Att-E | Def-E-SR+BRS | Att-E | Def-E-SR+BTRS
---|---|---|---|---|---|---|---|---
Steps | Grb | Cap | Tag | Ret | Def | Grb | Cap | Tag | Ret | Def | Grb | Cap | Tag | Ret | Def | Grb | Cap | Tag | Ret | Def
|  |  |  | Tag | Tag |  |  |  | Tag | Tag |  |  |  | Tag | Tag |  |  |  | Tag | Tag
50k | 8 | 0 | 2 | 8 | 19 | 6 | 3 | 4 | 3 | 13 | 10 | 3 | 0 | 7 | 16 | 0 | 0 | 11 | 0 | 0
100k | 9 | 9 | 1 | 0 | 8 | 3 | 3 | 7 | 0 | 14 | 1 | 0 | 9 | 1 | 0 | 4 | 1 | 7 | 3 | 0
150k | 5 | 1 | 7 | 4 | 18 | 7 | 5 | 4 | 2 | 10 | 6 | 0 | 4 | 6 | 2 | 7 | 0 | 3 | 7 | 0
200k | 8 | 5 | 2 | 3 | 10 | 2 | 2 | 10 | 0 | 4 | 2 | 0 | 8 | 2 | 2 | 6 | 1 | 4 | 5 | 2
250k | 9 | 7 | 1 | 1 | 12 | 9 | 7 | 1 | 0 | 16 | 5 | 0 | 5 | 5 | 0 | 4 | 1 | 7 | 2 | 5
300k | 10 | 8 | 0 | 2 | 16 | 3 | 2 | 9 | 0 | 9 | 8 | 6 | 2 | 2 | 2 | 9 | 4 | 1 | 5 | 6
350k | 9 | 8 | 1 | 1 | 12 | 3 | 3 | 7 | 0 | 9 | 6 | 0 | 4 | 6 | 0 | 6 | 1 | 4 | 5 | 0
400k | 4 | 0 | 6 | 4 | 0 | 7 | 5 | 3 | 2 | 10 | 10 | 8 | 0 | 1 | 11 | 5 | 2 | 5 | 3 | 8
TABLE V: Scores of Defender trained with different reward shaping functions against Att-E (top half) and Att-H (bottom half) and evaluated against Att-E (left sub-table) and Att-H (right sub-table) |  | vs. Att-E | vs. Att-H
---|---|---|---
| Steps | SR | TRS | BRS | BTRS | SR | TRS | BRS | BTRS
Def-E | 50k | -15 | -14 | -7 | 22 | -31 | -29 | -55 | -20
100k | -33 | -9 | 18 | 11 | -30 | -28 | -25 | -22
150k | -7 | -17 | 6 | 6 | -21 | -17 | -28 | -27
200k | -21 | 10 | 14 | 3 | -25 | -22 | -12 | 4
250k | -32 | -37 | 10 | 5 | -32 | -16 | -20 | -17
300k | -40 | 2 | -16 | -16 | -47 | -14 | -10 | -11
350k | -34 | -4 | 8 | 5 | -41 | -1 | -18 | -24
400k | 12 | -19 | -36 | -4 | -23 | -7 | -38 | -24
| Steps | SR | TRS | BRS | BTRS | SR | TRS | BRS | BTRS
Def-H | 50k | -5 | -18 | 6 | -19 | -32 | -14 | 0 | -30
100k | -15 | -3 | 13 | -11 | -34 | -5 | 20 | -3
150k | -45 | 7 | 20 | 22 | -47 | 1 | -11 | -8
200k | -21 | 1 | 22 | -3 | -30 | -1 | -27 | -10
250k | -25 | -11 | -14 | -3 | -36 | -6 | -3 | -5
300k | -46 | 18 | 0 | 14 | -42 | -11 | -20 | -14
350k | -13 | -49 | 3 | 16 | -29 | -16 | -22 | -8
400k | -30 | 3 | -20 | -26 | -36 | -17 | -4 | -30
Figure 10: Action heat map showing actions taken by defender playing against Att-E with: (SR) sparse reward, (BTRS) bound and tag reward, (3BRS) boundary reward with $3\times$gradient, (3TRS) tag reward shaping with $3\times$gradient, (ERS) efficiency reward, (BERS) boundary with efficiency reward, (BTERS) boundary, tag with efficiency reward. |
---|---
SR + TRS ($10,200$K) | SR + $3$TRS ($26,400$K)
|
SR + BRS ($18,100$K) | SR + $3$BRS ($24,250$K)
Figure 11: Position heat map showing locations in the playing field that are
visited by defender playing against Att-E with tag-shaped reward with
$1\times$ and $3\times$ gradient (top row) and boundary-shaped reward with
$1\times$ and $3\times$ gradient (bottom row).


Determination of Proton Parton Distribution
Functions using ATLAS Data

Zhiqing Zhang, on behalf of the ATLAS Collaboration111Copyright 2021 CERN for
the benefit of the ATLAS Collaboration. Reproduction of this article or parts
of it is allowed as specified in the CC-BY-4.0 license.

E-Mail<EMAIL_ADDRESS>

Laboratoire de Physique des 2 infinis Irène Joliot-Curie $-$ IJCLab,
Université Paris-Saclay, CNRS/IN2P3, 91405 Orsay, France

Presented at the Low-$x$ Workshop, Elba Island, Italy, September 27–October 1
2021

###### Abstract

Using differential cross-section data of inclusive $W$ and $Z/\gamma^{\ast}$
boson production and of their production in association with jets, and
differential cross-section data of top-quark pair production in the lepton +
jets and dilepton channels in proton–proton collisions at the CERN Large
Hadron Collider at center-of-mass energies of $\sqrt{s}=7$ and 8 TeV, together
with deep inelastic scattering data from electron–proton collisions at the
HERA collider, several determinations of parton distribution functions of the
proton have been performed by the ATLAS experiment. The results are presented
here. The emphasis is given to the determination of the strange sea quark
distribution and its evolution over time.

##  1 Introduction

Parton distribution functions (PDFs) of the proton cannot be predicted from
first principle and they used to be determined in deep inelastic scattering
(DIS) experiments using point-like beams of charged or neutral leptons to
probe the proton or other nucleon targets, in earlier fixed target mode and
later collision mode at HERA. Thanks to the factorization theorem, the PDFs
determined from one process can be used as prediction for other processes.

At hadron colliders such as the Large Hadron Collider (LHC), the uncertainty
of the PDFs is now the dominant uncertainty source for precision measurements,
one example is the recent determination of the $W$ boson mass 1701.07240  by
ATLAS detector . It also becomes a limiting factor for searches beyond the
Standard Model. It is therefore important to improve our knowledge on PDFs by
using relevant measurements at the LHC.

Since 2012, several analyses at next-to-next-to-leading order (NNLO) accuracy
in quantum chromodynamics (QCD) have been performed by the ATLAS experiment
using Drell-Yan or top-quark pair cross-section data as outlined in Table 1.
In the following sections, these QCD analyses and their corresponding results
will be briefly described and shown.

Table 1: Overview of ATLAS QCD analyses and the used data sets including the
neutral current (NC) and charged current (CC) data from HERA-I and HERA-I+II,
and various ATLAS measurements of the Drell-Yan processes $W\to\ell\nu$ and
$Z(\gamma^{\ast})\to\ell\ell$ and of top-quark pair production at different
center-of-mass energies.

Data sets | epWZ12 1203.4051  | epWZ16 1612.03016  | epWZtop18 ATL-PHYS-PUB-2018-017  | epWZVjet20 2101.05095
---|---|---|---|---
HERA-I NC, CC 0911.0884  | ✓ |  |  |
HERA-I+II NC, CC 1506.06042  |  | ✓ | ✓ | ✓
ATLAS $W$, $Z$, 7 TeV 35 pb-1 1109.5141  | ✓ |  |  |
ATLAS $W$, $Z/\gamma^{\ast}$, 7 TeV 4.2 fb-1 1612.03016  |  | ✓ | ✓ | ✓
ATLAS $t\bar{t}$ $(\ell+$ jets, dilepton$)$ 8 TeV 1511.04716 ; 1607.07281  |  |  | ✓ |
ATLAS $W/Z+$ jets 8 TeV 1711.03296 ; 1907.06728  |  |  |  | ✓

##  2 Analysis ATLAS-epWZ12

In 2012, a NNLO QCD analysis, ATLAS-epWZ12 1203.4051 , was performed by the
ATLAS experiment to assess the strange quark distribution, using the first
differential cross-section measurements of inclusive $W^{\pm}$ and $Z$ boson
production based on a $pp$ collision data sample corresponding to an
integrated luminosity of 35 pb-1 recorded in 2010 at a center-of-mass energy
of $\sqrt{s}=7$ TeV 1109.5141 , together with the combined $e^{\pm}p$ neutral
current (NC) and charged current (CC) cross-section measurements of H1 and
ZEUS from HERA-I 0911.0884 .

The HERA measurements were included since they are the primary source for
constraining PDFs with their large kinematic coverage for $Q^{2}$, the
absolute four-momentum transfer squared, from near 1 GeV2 to above $10^{4}$
GeV2 and of Bjorken-$x$ from $\sim 0.6$ down to $10^{-4}$. However, they do
have limitations. They cannot distinguish quark flavor between the down-type
sea quarks, $\bar{d}$ and $\bar{s}$. Also the measurement precision at high
$x$ and large $Q^{2}$ are statistically limited.

The ATLAS data, measured as functions of the $W$ decay lepton pseudorapidity,
$\eta_{\ell}$, and of the $Z$ boson rapidity, $y_{Z}$, with a typical
precision of $(2-3)\%$, access a kinematic range prescribed by the boson
masses, $M_{W,Z}$, and the proton beam energy of $E_{p}=3.5$ TeV,
corresponding to $Q^{2}\simeq M^{2}_{W,Z}$ and an $x$ range $0.001\lesssim
x\lesssim 0.1$, with a mean $x=M_{Z}/2E_{p}=0.013$ for the $Z$ boson. The data
provide new constraints on the strange quark distribution at high scale,
$Q^{2}\sim M^{2}_{W,Z}$, which imply constraints at low $Q^{2}$ through
perturbative QCD evolution.

The QCD analysis used the HERAFitter framework HERAFitter . The light quark
coefficient functions were calculated to NNLO as implemented in QCDNUM
1005.1481 . The contributions of heavy quarks are calculated in the general-
mass variable-flavor-number scheme of Refs. 9709442 ; 0601245 . The
electroweak (EW) parameters and corrections relevant for the $W$ and $Z$ boson
production processes were determined following the procedure described in Ref.
1109.5141  and cross-checked between the FEWZ 0312266  and the DYNNLO
0903.2120  programs. The fit package used APPLGRID code 0911.2985  interfaced
to the MCFM program 1007.3492  for fast calculation of the differential $W$
and $Z$ boson cross sections at next-to-leading order (NLO) and a $K$-factor
technique to correct from NLO to NNLO predictions. The data were compared to
the theory using the $\chi^{2}$ function defined in Ref. 0904.0929 .

The quark and gluon distributions at an initial scale of $Q^{2}_{0}=1.9$ GeV2,
chosen such that it is below the charm mass threshold $m_{c}$, were
parameterized by generic forms:

$\displaystyle xq_{i}(x)=A_{i}x^{B_{i}}(1-x)^{C_{i}}P_{i}(x)\,,$
$\displaystyle
xg(x)=A_{g}x^{B_{g}}(1-x)^{C_{g}}P_{g}(x)-A^{\prime}_{g}x^{B^{\prime}_{g}}(1-x)^{C^{\prime}_{g}}\,,$

where $P_{i,g}$ denotes polynomials in powers of $x$. The fitted quark
distributions were the valence quark distributions $(xu_{v},xd_{v})$ and the
light quark sea distributions $(x\bar{u},x\bar{d},x\bar{s})$. The parameters
$A_{u_{v}}$ and $A_{d_{v}}$ were fixed using the quark counting rule, $A_{g}$
using the momentum sum rule, and $C^{\prime}_{g}$ was set to 25 to suppress
negative contributions at high $x$. The normalization and slope parameters,
$A$ and $B$, of $\bar{u}$ and $\bar{d}$ were set equal such that
$x\bar{u}=x\bar{d}$ at $x\to 0$. For the strange quark distribution results
presented here, it was parametrized with $P_{\bar{s}}=1$ and
$B_{\bar{s}}=B_{\bar{d}}$. Terms were added in the polynomial expansion
$P_{i}(x)$ only if required by the data with a significant decrease in
$\chi^{2}$ of the fit results, following the procedure described in Ref.
0911.0884 . This led to one additional term, $P_{u_{v}}=1+E_{u_{v}}x^{2}$, for
the $u$ valence quark distribution, giving in total 15 free parameters for the
fit. The parton distributions at other $Q^{2}$ values are obtained using the
evolution equations.

The fit resulted in a good overall $\chi^{2}$ value per number of degree of
freedom of 538.4/565 and determined the ratio of the strange quark
distribution over the down quark one at $Q^{2}_{0}$ and $x=0.023$ (the $x$
value corresponds to 0.013 at $Q^{2}=M^{2}_{Z}$) to be:

$r_{s}=\frac{s+\bar{s}}{2\bar{d}}=1.00\pm 0.20(\textrm{exp})\pm
0.07(\textrm{mod})^{+0.10}_{-0.15}(\textrm{par})^{+0.06}_{-0.07}(\alpha_{s})\pm
0.08(\textrm{th})\,,$

where uncertainties are experimental, model, parametrization, $\alpha_{s}$ and
theoretical. The determination, consistent with the prediction that the light
quark sea at low $x$ is flavor symmetric, was compared in Figure 1 with
predictions obtained from four global PDF determinations. The ABKM09 0908.2766
and MSTW08 0901.0002  determinations gave a value around 0.5 and the NNPDF2.1
1107.2652  determination was lower at about 0.25. On the other hand, the CT10
(NLO) 1007.2241  determination gave a large ratio consistent with the ATLAS
determination.

Figure 1: Relative strange-to-down sea quark ratio for $Q^{2}_{0}=1.9$ GeV2
and $x=0.023$ comparing the determination from ATLAS (shown in the vertical
line with bands for experimental and total uncertainties) with predictions
from different PDF sets (shown in closed symbols with horizontal error bars).
The plot is taken from Ref. 1203.4051 .

##  3 Analysis ATLAS-epWZ16

The ATLAS-epWZ16 analysis 1612.03016  is very similar to ATLAS-epWZ12. The
main differences are that the HERA-I $ep$ data were replaced with the final NC
and CC DIS HERA-I+II data 1506.06042 . The ATLAS Drell-Yan measurements used
1612.03016  were based on the full 7 TeV data sample of 4.6 fb-1 with improved
precision. In addition, the measurement of the $Z/\gamma^{\ast}$ process was
extended in three mass ranges of $46<m_{\ell\ell}<66$ GeV,
$66<m_{\ell\ell}<116$ GeV and $116<m_{\ell\ell}<150$ GeV for the central
rapidity region of $|y_{\ell\ell}|<2.5$ and in two mass ranges
$66<m_{\ell\ell}<116$ GeV and $116<m_{\ell\ell}<150$ GeV in the forward
rapidity region up to 3.6. The NNLO QCD fit with the same PDF parameterization
as the ATLAS-epWZ12 fit was performed using the xFitter program xFitter .

The strange quark ratio $r_{s}$ was determined with improved precision at
$Q^{2}_{0}$ and $x=0.023$:

$r_{s}=1.19\pm
0.07(\textrm{exp})^{+0.13}_{-0.14}(\textrm{mod}+\textrm{par}+\textrm{th})\,.$

The comparison with predictions from other global PDF sets ABM12 1310.3059 ,
NNPDF3.0 1410.8849 , MMHT14 1412.3989 , CT14 1506.07443 , as well as from
ATLAS-epWZ12 is shown in Figure 2 (left). In addition, the ratio of strange
quarks over the sum of up and down sea quarks
$R_{s}=(s+\bar{s})/(\bar{u}+\bar{d})$ as a function of $x$ has also been
obtained (Figure 2 (right)), though the uncertainty in particular the
parametrization uncertainty was large.

Figure 2: Left: Relative strange-to-down sea quark ratio $r_{s}$ for
$Q^{2}_{0}=1.9$ GeV2 and $x=0.023$ comparing the determination from ATLAS
(shown in the vertical line with bands for experimental data, QCD fit and
theoretical uncertainties) with predictions from different NNLO PDF sets
(shown in closed symbols with horizontal error bars) as well as previous ATLAS
results (shown in open square). Right: relative strange quark ratio over the
sum of up and down sea quarks as a function of Bjorken-$x$. The plots are
taken from Ref. 1612.03016 .

##  4 Analysis ATLAS-epWZtop18

Another NNLO QCD analysis, ATLAS-epWZtop18 ATL-PHYS-PUB-2018-017 , included,
in addition to those data used in the ATLAS-epWZ16 fit, top-quark pair
production data measured in the lepton + jets 1511.04716  and dilepton
1607.07281  channels at 8 TeV corresponding to an integrated luminosity of
20.2 fb-1. The top-quark pair production data are complementary to the other
data sets in their PDF constraining power since they are expected to be
sensitive to the high-$x$ gluon distribution 1611.08609 . The corresponding
NNLO prediction 1704.08551  were supplied in the form of FastNLO grids
1208.3641  for the data in the lepton + jets channel. For the dilepton
channel, APPLGRID was interfaced to MCFM to produce NLO grids and a $K$-factor
technique was used to correct from NLO to NNLO predictions.

The setup of the fit differs from that of ATLAS-epWZ16 in three aspects, apart
from the addition of top-quark pair production data. Firstly, the low mass
off-peak $Z/\gamma^{\ast}$ boson data was not used because it became clear
that the lower $x$ region probed by the data was subject to further QCD
1710.05935  corrections which were not readily calculable, this had negligible
impact on the fits. Secondly, the minimum $Q^{2}$ selection on the HERA data
was raised from 7.5 GeV2 to 10 GeV2, this larger cut was already considered as
one of the model variations of the ATLS-epWZ16 fit and was used as standard in
the new analysis to avoid the region of the HERA data which may be subject to
$\ln(1/x)$-resummation effects 1710.05935  that were not accounted for in the
prediction. Thirdly, the ATLAS-epWZtop18 fit had one additional term,
$P_{\bar{d}}=1+D_{\bar{d}}x$, for the down sea quark distribution than those
of ATLAS-epWZ16 where it was considered as a parameterization variation.

The main result of the fit is presented in Figure 3, showing the impact of the
top-quark pair production data on the gluon distribution; reducing its
uncertainty in particular at high $x$ and making it softer and harder at
medium $x$ and high $x$, respectively.

Figure 3: Gluon distribution as a function of Bjorken-$x$ from fitting HERA
data and ATLAS $W$ and $Z/\gamma^{\ast}$ boson data plus the $t\bar{t}$ lepton
+ jet $p^{t}_{\textrm{T}}$ and $m_{t\bar{t}}$ data and the $t\bar{t}$ dilepton
$y_{t\bar{t}}$ data compared to the fit to HERA and ATLAS $W$ and
$Z/\gamma^{\ast}$ boson data alone. The plot is taken from Ref. ATL-PHYS-
PUB-2018-017 .

##  5 Analysis ATLAS-epWZVjet20

The analysis ATLAS-epWZVjet20 2101.05095  included new ATLAS data on the
production of a vector ($W$ or $Z$) boson in association with at least on jet
1711.03296 ; 1907.06728 . The measurements were based on the same data sample
as for the top-quark pair production at 8 TeV. They provided a novel source of
input to PDF determination that is sensitive to partons at higher $x$ and
$Q^{2}$ than can be accessed by $W$ and $Z$ boson data alone. These data are
thus complementary to the inclusive $W/Z$ boson measurements.

Predictions for $W+$ jets and $Z+$ jets production 1511.08692  were obtained
similarly to the $W$ and $Z$ predictions to NLO in QCD and leading order in EW
couplings by using the APPLGRID code interfaced to the MCFM program. NNLO
(NLO) corrections in QCD (EW) were implemented as $K$-factors.

The setup of the ATLAS-epWZVjet20 fit differs from that of ATLAS-epWZ16 in
that there was one more term, $P_{g}=1+D_{g}x$, for the gluon distribution and
a tighter selection of $Q^{2}>10$ GeV2 instead of 7.5 GeV2 on the DIS data.
The resulting down and strange sea quark distributions of the fit and $R_{s}$
at the starting scale $Q^{2}_{0}$ are shown in Figure 4 in comparison with the
results of the ATLAS-epWZ20 fit when the $V+$ jets data were not included. The
inclusion of the $V+$ jets data made the $x\bar{d}$ distribution notably
higher in the range of $x\gtrsim 0.02$ and the $x\bar{s}$ distribution lower
in the same range, with significantly smaller experimental and
parameterization uncertainties at high $x$. The difference between the two
$R_{s}$ determinations at high $x$ is essentially covered by the large
uncertainty dominated by the parameterization uncertainty in the ATLAS-epWZ20
fit. The new $R_{s}$ determination at $x=0.023$ was again compared with the
predictions from ABMP16 1701.05838 , CT18 and CT18A 1912.10053 , MMHT14
1412.3989 , NNPDF3.1 and NNPDF3.1_strange 1706.00428  and the ATLAS
determinations ATLAS-epWZ20 and ATLAS-epWZ16, the latter two being different
mainly in their PDF parameterizations. Better agreement was observed with the
CT18A PDF set, which included both the data used in the CT18 fit and the ATLAS
7 TeV data, although tension remains with the NNPDF3.1_strange PDF set, which
also used this data.

The large change at high $x$ was investigated by performing a $\chi^{2}$ scan
of the $C_{\bar{d}}$ parameter which controls the high-$x$ behavior of the
$x\bar{d}$ distribution. The $\chi^{2}$ variation subtracting the minimum
value is presented in Figure 5. The other data sets showed double minima in
$\chi^{2}$ with a preferred solution for a $C_{\bar{d}}$ value around 10
corresponding to a soft $x\bar{d}$ distribution at high $x$. The inclusion of
the $V+$ jet data allowed to resolve the ambiguity and resulted in a
$C_{\bar{d}}$ value around 1.6 and thus a harder $x\bar{d}$ distribution at
high $x$. Since the sum of $x\bar{d}+x\bar{s}$ is well constrained by e.g. the
DIS data, the harder $x\bar{d}$ distribution gives thus the softer $x\bar{s}$
distribution and smaller $R_{s}$ at high $x$.

Figure 4: Top-left: Down sea quark distribution as a function of Bjorken-$x$
when including (blue) or without including (green) the $V+$ jets data in the
fits. Top-right: similar plot for strange sea quark distribution. Bottom-left:
similar plot for relative strange quark ratio over the sum of up and down sea
quarks $R_{s}$ as a function of Bjorken-$x$. Bottom-right: $R_{s}$
determination in comparison with other predictions for $Q^{2}_{0}=1.9$ GeV2
and $x=0.023$. The plots are taken from Ref. 2101.05095 . Figure 5: Variation
of $\chi^{2}$ subtracting the minimum value when including (blue) or without
including (green) the $V+$ jets data in the fits as a function of parameter
$C_{\bar{d}}$ . The plot is taken from Ref. 2101.05095 .

##  6 Summary and prospect

The results of several NNLO QCD analyses performed using the Drell-Yan
production and top-quark pair production cross sections measured by ATLAS with
$pp$ collision data at the LHC taken at center-of-mass energies of 7 and 8
TeV, together with the neutral and charged current deep inelastic scattering
cross sections from $ep$ collisions at HERA, have been presented. The analyses
show the constraining power of the ATLAS measurements on parton distribution
functions of the proton, complementary to the HERA data. In particular, the
strange quark distribution at low $x$ is found unsuppressed contrary to what
was assumed and obtained in the other PDF analyses. The shape of the strange
quark ratio over the other light sea quarks at high $x$ has been better
understood thanks to the measurement of vector-boson production in association
with at least one jet.

Another valuable constraint for the strange quark distribution at the LHC
concerns the associated production of $W$ bosons and charm quarks since it
probes the strange quark content of the proton directly through the leading
order processes $g+\bar{s}\to W^{+}+\bar{c}$ and $g+s\to W^{-}+c$. Earlier
analyses, not presented here, performed NLO fits using the measurements of
ATLAS at 7 TeV 1402.6263  and of CMS at 7 TeV 1310.1138  and at 13 TeV
1811.10021 . With the recent available NNLO predictions 2011.01011 , there is
a good prospect in including the $W+c$ data in the future NNLO analyses,
though the interpretation of the data is sensitive to the modelling of
$c$-quark fragmentation.

## References

  * (1) ATLAS Collaboration, Eur. Phys. J. C 78 (2018) 110 [erratum: Eur. Phys. J. C 78 (2018) 898] doi:10.1140/epjc/s10052-017-5475-4 [arXiv:1701.07240 [hep-ex]].
  * (2) ATLAS Collaboration, JINST 3 (2008) S08003 doi:10.1088/1748-0221/3/08/S08003.
  * (3) ATLAS Collaboration, Phys. Rev. Lett. 109 (2012) 012001 doi:10.1103/PhysRevLett.109.012001 [arXiv:1203.4051 [hep-ex]].
  * (4) ATLAS Collaborartion, Eur. Phys. J. C 77 (2017) 367 doi:10.1140/epjc/s10052-017-4911-9 [arXiv:1612.03016 [hep-ex]].
  * (5) ATLAS Collaboration, ATL-PHYS-PUB-2018-017, url: https://cds.cern.ch/record/2633819.
  * (6) ATLAS Collaboration, JHEP 07 (2021) 223 doi:10.1007/JHEP07(2021)223 [arXiv:2101.05095 [hep-ex]].
  * (7) H1 and ZEUS Collaborations, JHEP 01 (2010) 109 doi:10.1007/JHEP01(2010)109 [arXiv:0911.0884 [hep-ex]].
  * (8) H1 and ZEUS Collaborations, Eur. Phys. J. C 75 (2015) 580 doi:10.1140/epjc/s10052-015-3710-4 [arXiv:1506.06042 [hep-ex]].
  * (9) ATLAS Collaboration, Phys. Rev. D 85 (2012) 072004 doi:10.1103/PhysRevD.85.072004 [arXiv:1109.5141 [hep-ex]].
  * (10) ATLAS Collaboration, Eur. Phys. J. C 76 (2016) 538 doi:10.1140/epjc/s10052-016-4366-4 [arXiv:1511.04716 [hep-ex]].
  * (11) ATLAS Collaboration, Phys. Rev. D 94 (2016) 092003 doi:10.1103/PhysRevD.94.092003 [arXiv:1607.07281 [hep-ex]].
  * (12) ATLAS Collaboration, JHEP 05 (2018) 077 [erratum: JHEP 10 (2020) 048] doi:10.1007/JHEP05(2018)077 [arXiv:1711.03296 [hep-ex]].
  * (13) ATLAS Collaboration, Eur. Phys. J. C 79 (2019) 847 doi:10.1140/epjc/s10052-019-7321-3 [arXiv:1907.06728 [hep-ex]].
  * (14) HERAFitter, url: http://projects.hepforge.org/herafitter.
  * (15) M. Botje, Comput. Phys. Commun. 182 (2011) 490 doi:10.1016/j.cpc.2010.10.020 [arXiv:1005.1481 [hep-ph]].
  * (16) R. S. Thorne and R. G. Roberts, Phys. Rev. D 57 (1998) 6871 doi:10.1103/PhysRevD.57.6871 [arXiv:hep-ph/9709442 [hep-ph]].
  * (17) R. S. Thorne, Phys. Rev. D 73 (2006) 054019 doi:10.1103/PhysRevD.73.054019 [arXiv:hep-ph/0601245 [hep-ph]].
  * (18) C. Anastasiou, L. J. Dixon, K. Melnikov and F. Petriello, Phys. Rev. D 69 (2004) 094008 doi:10.1103/PhysRevD.69.094008 [arXiv:hep-ph/0312266 [hep-ph]].
  * (19) S. Catani, L. Cieri, G. Ferrera, D. de Florian and M. Grazzini, Phys. Rev. Lett. 103 (2009) 082001 doi:10.1103/PhysRevLett.103.082001 [arXiv:0903.2120 [hep-ph]].
  * (20) T. Carli et al., Eur. Phys. J. C 66 (2010) 503 doi:10.1140/epjc/s10052-010-1255-0 [arXiv:0911.2985 [hep-ph]].
  * (21) J. M. Campbell and R. K. Ellis, Nucl. Phys. B Proc. Suppl. 205-206 (2010) 10 doi:10.1016/j.nuclphysbps.2010.08.011 [arXiv:1007.3492 [hep-ph]].
  * (22) H1 Collaboration, Eur. Phys. J. C 63 (2009) 625 doi:10.1140/epjc/s10052-009-1128-6 [arXiv:0904.0929 [hep-ex]].
  * (23) S. Alekhin, J. Blumlein, S. Klein and S. Moch, Phys. Rev. D 81 (2010) 014032 doi:10.1103/PhysRevD.81.014032 [arXiv:0908.2766 [hep-ph]].
  * (24) A. D. Martin, W. J. Stirling, R. S. Thorne and G. Watt, Eur. Phys. J. C 63 (2009)189 doi:10.1140/epjc/s10052-009-1072-5 [arXiv:0901.0002 [hep-ph]].
  * (25) R. D. Ball et al. [NNPDF], Nucl. Phys. B 855 (2012) 153 doi:10.1016/j.nuclphysb.2011.09.024 [arXiv:1107.2652 [hep-ph]].
  * (26) H. L. Lai et al., Phys. Rev. D 82 (2010) 074024 doi:10.1103/PhysRevD.82.074024 [arXiv:1007.2241 [hep-ph]].
  * (27) xFitter program, url: https://www.xfitter.org/xFitter/.
  * (28) S. Alekhin, J. Blumlein and S. Moch, Phys. Rev. D 89 (2014) 054028 doi:10.1103/PhysRevD.89.054028 [arXiv:1310.3059 [hep-ph]].
  * (29) R. D. Ball et al. [NNPDF], JHEP 04 (2015) 040 doi:10.1007/JHEP04(2015)040 [arXiv:1410.8849 [hep-ph]].
  * (30) L. A. Harland-Lang, A. D. Martin, P. Motylinski and R. S. Thorne, Eur. Phys. J. C 75 (2015) 204 doi:10.1140/epjc/s10052-015-3397-6 [arXiv:1412.3989 [hep-ph]].
  * (31) S. Dulat et al., Phys. Rev. D 93 (2016) 033006 doi:10.1103/PhysRevD.93.033006 [arXiv:1506.07443 [hep-ph]].
  * (32) M. Czakon, N. P. Hartland, A. Mitov, E. R. Nocera and J. Rojo, JHEP 04 (2017) 044 doi:10.1007/JHEP04(2017)044 [arXiv:1611.08609 [hep-ph]].
  * (33) M. Czakon, D. Heymes and A. Mitov, [arXiv:1704.08551 [hep-ph]].
  * (34) D. Britzger et al. [fastNLO], doi:10.3204/DESY-PROC-2012-02/165 [arXiv:1208.3641 [hep-ph]].
  * (35) R. D. Ball, V. Bertone, M. Bonvini, S. Marzani, J. Rojo and L. Rottoli, Eur. Phys. J. C 78 (2018) 321 doi:10.1140/epjc/s10052-018-5774-4 [arXiv:1710.05935 [hep-ph]].
  * (36) A. Gehrmann-De Ridder, T. Gehrmann, E. W. N. Glover, A. Huss and T. A. Morgan, Phys. Rev. Lett. 117 (2016) 022001 doi:10.1103/PhysRevLett.117.022001 [arXiv:1507.02850 [hep-ph]].
  * (37) S. Kallweit, J. M. Lindert, P. Maierhofer, S. Pozzorini and M. Schönherr, JHEP 04 (2016) 021 doi:10.1007/JHEP04(2016)021 [arXiv:1511.08692 [hep-ph]].
  * (38) S. Alekhin, J. Blümlein, S. Moch and R. Placakyte, Phys. Rev. D 96 (2017) 014011 doi:10.1103/PhysRevD.96.014011 [arXiv:1701.05838 [hep-ph]].
  * (39) T. J. Hou et al., Phys. Rev. D 103 (2021) 014013 doi:10.1103/PhysRevD.103.014013 [arXiv:1912.10053 [hep-ph]].
  * (40) R. D. Ball et al. [NNPDF], Eur. Phys. J. C 77 (2017) 663 doi:10.1140/epjc/s10052-017-5199-5 [arXiv:1706.00428 [hep-ph]].
  * (41) ATLAS Collaboration, JHEP 05 (2014) 068 doi:10.1007/JHEP05(2014)068 [arXiv:1402.6263 [hep-ex]].
  * (42) CMS Collaboration, JHEP 02 (2014) 013 doi:10.1007/JHEP02(2014)013 [arXiv:1310.1138 [hep-ex]].
  * (43) CMS Collaboration, Eur. Phys. J. C 79 (2019) 269 doi:10.1140/epjc/s10052-019-6752-1 [arXiv:1811.10021 [hep-ex]].
  * (44) M. Czakon, A. Mitov, M. Pellen and R. Poncelet, JHEP 06 (2021) 100 doi:10.1007/JHEP06(2021)100 [arXiv:2011.01011 [hep-ph]].