realistic values, and the agents now track one nearby car (which includes
other agents, making the configuration applicable to multi-agent settings).
Since the agents now know the pose of its nearest car obstacle, we only use
one ring to detect and avoid walls. In addition, we now have 16 parked cars in
the environment, which is approximately $50\%$ of the maximum. By default, we
share the agents’ goals and velocities in multi-agent experiments.

####  5.2.2 Setup

After tuning PPO’s hyperparameters using grid-searching on the suggested
ranges provided by ML-Agents [11] and harnessing the hyperparameters provided
by Thunyapoo et al. [26] as a baseline, we obtain a set of stable
hyperparameters for IPPO in our higher granularity environment, listed in
Table 5.1. We also find that a total of 16 million steps are required to
converge IPPO to global optima with 7 agents, and a smaller total of 10
million are required with a single agent. Training for such a number of time-
steps takes approximately one day of compute time with two CPUs and 6
gigabytes of RAM on the department’s CPU cluster. As explained in Section
4.3.1, we reserve $80\%$ of those episodes for training, and the last $20\%$
for evaluation. In our subsequent applications of IPPO, we measure the
performance of our models using the various metrics recorded as explained in
Section 4.3.2, since manual analysis of the agents’ cumulative reward is less
informative and time-consuming. Some key metrics used are the agents’ park,
crash and halt rates, which are the ratio of their episodes in which they
successfully parked, crashed, or halted, respectively (here, the agent
‘halted’ if they neither parked nor crashed).

Parameter Name | Parameter Value
---|---
Batch Size | 32
Buffer Size | 4096
Learning Rate | 0.0001
Beta | 0.002
Epsilon | 0.25
Lambda | 0.925
# Epoch | 10
| # Hidden Units
---
(Neural Network)
256
| # Layers
---
(Neural Network)
3
Gamma | 0.999
Time Horizon | 128
Table 5.1: PPO hyperparameters.

One particular hyperparameter with a large effect on the quality our learned
models is the discount factor, $\gamma$. When training with 7 agents, we find
that their crash rate varies between $4.41\%$ and $2.04\%$ when $\gamma$
varies between $0.92$ and $0.9998$, and the crash rate decreases the larger
the value of $\gamma$. Figure 5.4 show how the agents’ crash rate varies with
$\gamma$, where the bottom graph shows the larger values of $\gamma$ in more
detail. Since $\gamma=0.999$ yields a low crash rate and is outside the region
of marginal instability beyond $0.999$, we use $\gamma=0.999$ in our other
experiments. We tuned the the other hyperparameters in a similar way, seeking
values that yield low crash rates.

Figure 5.4: Crash rate when varying the discount factor $\gamma$ with 7
agents.

####  5.2.3 Density and Dimensionality

Having obtained a stable set of hyperparameters for IPPO in our environment,
we vary some of the environment’s key parameters to identify trends relating
to its density and dimensionality. In this section, we describe the results
from our experiments of applying IPPO and varying the number of agents, the
number of parked cars, and the number of tracked cars ($n_{track}$) in our
environment, since such parameters are critical to the environment’s density
and dimensionality.

##### Number of Agents

When varying the number of agents from $1$ to $7$ and training for an equal
number of $14$ million steps shared among the agents (with triple redundancy),
we find that the agents’ park rate decreases from $99.3\%$ with 2 agents to
$97.4\%$ with $7$ agents, and a significantly lower $30.9\%$ with $1$ agent.
Figure 5.5 shows a graph of the agents’ park rate when varying the number of
agents from $2$ to $7$ (with the single-agent outlier excluded). Thus, $2$
agents yields the optimal park rate in our environment, and increases beyond
$2$ decrease the park rate by a small amount, however decreases below $2$
decrease the park rate by a prohibitively large amount.

Figure 5.5: Park rate when varying the number of agents.

While the non-stationarity of our multi-agent setting makes the decreasing
park rate beyond $2$ agents expected, the significant increase in park rate
between $1$ and $2$ agents is an unexpected result. We believe such an
increase is a result of an increase in the density of the environment with $2$
agents, since there are more states in which the agent has to avoid a car
(since agents have to avoid other agents as well as parked cars). Hence, the
‘training data’ (i.e. the situations that arise in the environment) is denser
with more agents, giving the agents more situations to learn from. We arrive
at such a conclusion by manually inspecting the cumulative reward graphs for
the models, observing that the agents get stuck in halting local optima for a
long time during training (in which the agents learn to halt rather than to
park or crash), and the agents escape such local optima more quickly when
there are more agents present in the environment.

Since our experiments reveal a relatively high parking rate with $7$ agents,
our other multi-agent experiments use $7$ agents, due to the attractive
density they provide to our training data (as shown in Figure 4.2).

##### Number of Parked Cars

Another parameter varying the density of our environment is the number of
parked cars within. With more parked cars, the agents’ parking spaces are
increasingly likely to be surrounded by more parked cars, making it more
difficult for them to successfully park without colliding. Hence, we desire a
large number of parked cars in our environment, such that the agents learn to
park in challenging scenarios.

To assess the effect of the number of parked cars on the agents’ learned
models, we vary the number of parked cars between $0$ (one) and $35$ (all)
with $1$ agent, and between $0$ (none) and $29$ (all) with $7$ agents. We
train for $6$ million steps with $1$ agent, and $16$ million steps among the
$7$ agents, with triple redundancy. Figure 5.6 and Figure 5.7 show how the
agents’ park and halt rates vary with the number of parked cars in the
environment, for the experiments with $1$ and $7$ agents respectively.

Figure 5.6: Park rate when varying the number of parked cars, with $1$ agent.
Figure 5.7: Park rate when varying the number of parked cars, with $7$ agents.

It can be seen from Figure 5.6 that with a single agent in the environment,
they’re unable to learn to successfully park when there are more than
approximately 16 parked cars in the environment, since their park rate drops
to $0\%$. Instead, they learn to halt, getting stuck in local optima, after
manual analysis of their cumulative reward graphs. Intuitively, this happens
because when the agent begins training it does not yet know how to reach its
parking space, and when its parking space is more likely to be surrounded by
parked cars, it’s more likely to collide with a neighbouring parked car (as
more exist), making it less likely to reach its parking space during training.
Since the agent reaches its parking space less often, it’s less able to
reinforce the behaviour of reaching it. This result shows that there is a
trade-off that must be made between the density of our environment and the
convergence of our models, where too high a density results in poor
convergence, and vice-versa.

On the contrary, Figure 5.6 shows that when training with a large number of
agents, the agents learn to park successfully even with a large number of
parked cars. Hence, we have another result showing multiple agents aiding
training. The reason we obtain higher parking rates here is again due to
density, since the agents are able to learn to avoid cars also from each-other
- not only with the parked cars in the single-agent case.

Thus, in light of our findings, we use $16$ parked cars in our environment for
other experiments (populating approximately half of the parking spaces with
parked cars). Such a value is large enough to arise sufficiently difficult
parking scenarios, yet it’s small enough to yield a high parking rate
irrespective of the number of agents.

##### Number of Tracked Cars

Since the agents use the localised pose of its nearby cars to avoid them, the
number of nearby cars they’re tracking ($n_{track}$) may vary their park rate.
Hence, we measure the agents’ park rate with $n_{track}$ varying between $0$
and $4$, from no knowledge of nearby cars to full knowledge of $4$ nearby
cars. Figure 5.8 shows the resulting park rate for different values of
$n_{track}$, where we trained $7$ agents for $16$ million steps and averaged
the results over $3$ repetitions.

Figure 5.8: Park rate when varying the number of tracked cars.

It can be seen from the graph in Figure 5.8 that $n_{track}=1$ is optimal, and
that tracking more than $1$ agent reduces the park rate. Hence, the agents’
optimal learned policy greedily avoids the closest car to it, and the
inclusion of the information of other cars only hinders the agents’ learned
policy. Such a result is unexpected, because it’s natural to assume that
giving the agent more information will enable it to make better decisions.
However it appears that the information of more than one nearby car is not
utilised, and it increases the dimensionality of the state-space without much
benefit.

In addition, it can be observed from the graph that the park rate is much
larger when tracking agents than not tracking them. Hence, there are a
significant number of scenarios that arise in our environment where the agents
require the localised pose of nearby cars to avoid them, which validates our
environment’s construction since the agents are unable to yield a high park
rate by following a trivial policy within.

##### Shared State

In addition to tracking nearby agents, agents may share their goals and
velocities with each other to enable better path planning, as explained in
Section 3.2.4. Hence, we assess the agents’ improvement in park rate when
sharing their goals and velocities, to quantify the benefit of such shared
state in our MDP.

We train 7 agents for $16$ million steps in our environment under four
configurations:

  * •

Without shared goals or velocities;

  * •

With shared goals but without shared velocities;

  * •

With shared velocities but without shared goals;

  * •

With both shared velocities and shared goals.

In each configuration, we measured the agents’ average park rate, with a
redundancy of 6 repititions. Figure 5.9 shows the agents’ average park rate in
each of the four configurations.

Figure 5.9: Park rate when varying the shared information between the agents.

It can be seen from the graph in Figure 5.9 that the agents have a higher park
rate with shared state, and the highest park rate occurs when the agents share
only their goals, increasing the park rate from $95.2\%$ with no shared state
to $98.1\%$ with only shared goals, a $2.9\%$ increase. Interestingly, when
the agents share both their velocity and their goals, the park rate decreases
from when only one of them are shared. This may be due to sharing both
increasing the dimensionality of the MDP too much, with the combination of the
two data being less useful than one datum alone. However, since sharing both
doesn’t decrease the park rate by much, and the agents have access to more
information which may be useful as the complexity of the MDP grows, we share
both as a baseline.

####  5.2.4 Safety

Having assessed the agents’ park rate in our environment with varying density
and dimensionality, we next seek to assess the safety of their driving, since
it’s also important how the agents reach their goals. In this section, we
attempt two techniques to improve the safety of the agents’ driving. Our first
technique is to punish the agents proportional to their forward and angular
velocities to increase the smoothness of their driving, as explained in
Section 3.2.3. Secondly, we train the agents with larger hitboxes so they
learn to keep a threshold distance away from the other cars, as explained in
and Section 4.1.3.

##### Smoothness

To measure the effect of constraining the agents’ forward velocities ($v$) and
angular velocities ($\omega$) such that they drive smoothly, we reward each
agent $\frac{-k}{\tau}|v||\omega|$ at each time-step, varying the punishment
sum $k\geq 0$, with larger values enforcing the punishment more heavily. We
measure the agents’ average forward and angular velocities, as well as their
crash rates, to determine the effect of enforcing the punishment.

After training $7$ agents for $16$ million steps with varying values of $k$,
we obtain the results shown in Figure 5.10 and Figure 5.11. In Figure 5.10, we
see that as the punishment is enforced, both the agents’ forward and angular
velocities decrease (we normalise their values in the graph). However, the
angular velocities decrease to an asymptote much more quickly than the forward
velocities. Hence, the agents conform to driving more smoothly when enforcing
the punishment.

Figure 5.10: Average forward and angular velocity when varying the magnitude
of the smoothness punishment sum. Figure 5.11: Crash rate when varying the
magnitude of the smoothness punishment sum.

However, as can be seen in Figure 5.11, the agents’ crash rate increases when
the punishment is enforced. This is expected, because the punishment limits
the agents’ choice of their rotations, making them avoid sharp turns, in-tern
reducing their ability to escape dangerous scenarios. Hence, we do not get an
increase in driving safety for free - it comes at the expense of an increased
crash rate. Thus, a compromise must be made between driving smoothness and
park rate in our models.

##### Larger Hitboxes During Training

To attempt to further increase the safety of the agents in our environment, we
increase the cars’ hitboxes during training, such that the agents learn to
keep a threshold distance away from them. We measure the agents’ average
distance to their closest car and their crash rates to determine the effect of
increasing the cars’ hitboxes during training.

After training $7$ agents for $16$ million steps with varying scale factors of
the cars’ hitboxes in the environment (with double redundancy), we obtain the
results shown in Figure 5.12 and Figure 5.13.

Figure 5.12: Average distance to the agents’ nearest car when varying the
scale factor of the cars’ hitboxes during training. Figure 5.13: Park rate
when varying the scale factor of the cars’ hitboxes during training.

We can see from Figure 5.12 that increasing the scale of the cars’ hitboxes
during training increases the agents’ average distance to their closest car,
as desired. However, Figure 5.13 shows that such an increase in safety doesn’t
come for free, with the park rate decreasing as the scales increase, due to
the environment having a higher density during training with larger hitboxes.
Hence, similar to the smoothness punishment, a compromise must be made between
the distance the agents leave with the cars and the park rate in our models.

###  5.3 PPO with Dynamic Goals

Having assessed the behaviour of the agents in our environment with fixed
goals under various configurations, we next assess their behaviours with
dynamic goals. In this section, we describe the results of applying IPPO in
our environment with dynamic goals, quantifying properties relating to its
dimensionality as well as competitive and collaborative behaviours that arise
in the environment.

####  5.3.1 Environment Configuration

As explained in Section 4.3.4, our environment with dynamic goals uses the
same environment parameters as our environment for IPPO with fixed goals in
Section 5.2, but with some additional modifications listed in Table C.10. By
default, the agents track only $1$ parking space (i.e. $n_{space}=1$),
$r_{\delta g}(e\rightarrow e)=-0.002$ to discourage exploration, and
$r_{\delta g}(p\rightarrow p^{\prime})=-0.05$ to make the agents commit to a
goal. We also enforce Equation 3.6 so $r_{\delta g}(p\rightarrow e)$ is always
equal to $r_{\delta g}(p\rightarrow p^{\prime})$, hence when we tune
$r_{\delta g}(p\rightarrow p^{\prime})$ we also tune $r_{\delta
g}(p\rightarrow e)$.

####  5.3.2 Density and Dimensionality

As in our environment with fixed goals, we vary the environment’s parameters
relating to density and dimensionality, and describe the results of applying
IPPO with the environment in its different configurations. We first vary the
density of the environment by varying the number of agents, then we vary the
MDP’s dimensionality by varying the number of tracked cars ($n_{track}$) and
parking spaces ($n_{space}$).

##### Number of Agents

When varying the number of agents from $1$ to $7$ and training for $14$
million steps among the agents (and averaging the results over $3$
repetitions), the agents’ park rate decreases from $99.9\%$ with $1$ agent to
$99.5\%$ with $7$ agents, as shown in Figure 5.14. Hence, we have much higher
park rates in our environment with dynamic goals than with fixed goals, and
this is likely due to the fact that the agent can always choose the closest
parking space to them as their goal, which reduces the distance the agents
have to travel and thus reduces the chance of a collision. In addition, now
the optimal density of the environment contains $1$ agent rather than $2$ in
the environment with fixed goals. As with fixed goals, our other experiments
in this section use $7$ agents due to the attractive density they provide to
the environment.

Figure 5.14: Park rate when varying the number of agents in the environment
with dynamic goals.

##### Number of Tracked Cars and Parking Spaces

Since $n_{space}$ is now also a parameter relating to the dimensionality of
our MDP (along with $n_{track}$), we vary both $n_{space}$ and $n_{track}$ to
identify the optimal values of both. We train $7$ agents for $16$ million
steps, and vary both $n_{space}$ and $n_{track}$ between $1$ and $4$
respectively (obtaining $4*4=16$ total combinations), assessing the agents’
park rate for each combination (with triple redundancy).

Figure 5.15: Park rate when varying the number of tracked cars and parking
spaces.

Figure 5.15 shows our results, showing that the agents’ park rate is highest
when both $n_{track}=n_{space}=1$, and the park rate decreases when either
$n_{track}$ or $n_{space}$ are increased. Hence, similar to the environment
with fixed goals, the agents do not benefit from the extra dimensionality of
tracking more than one car or parking space, and have the highest park rate
when acting greedily and only observing their closest car and parking space.
Hence, we use $n_{track}=n_{space}=1$ as a benchmark in our other experiments.
It should be noticed that $n_{space}=1$ does not prevent the agents from
changing their goal parking space in our MDP, since they’re still able to
explore and find a new one.

####  5.3.3 Competition

Since competition is an important aspect of the agents’ behaviour, in this
section we assess competitive behaviours exhibited by the agents in our
environment with dynamic goals.

##### Method

One parameter controlling the amount of competition exhibited by the agents in
our environment is the magnitude of the change goal punishment, $r_{\delta
g}(p\rightarrow p^{\prime})$, since larger punishments cause the agents to be
more reluctant to change their goal parking spaces, causing them to pursue
their goal parking space in a more competitive manner. Hence, we vary
$r_{\delta g}(p\rightarrow p^{\prime})$ and measure various metrics relating
to the agents’ goal seeking behaviour. We measure the following metrics:

  * •

The ratio of the time-steps in which the agents are exploring. This metric
enables us to quantify the ‘decisiveness’ of the agents, since a high
exploration ratio means the agents are less willing to dedicate to a goal
parking space, exhibiting indecisive behaviour.

  * •

The ratio of the time-steps in which the agents move towards their goal
parking space when they have a goal parking space, and the ratio of time-steps
in which they move towards their tracked parking space when they’re exploring.
These metrics enable us to quantify the agents’ accuracy towards their goal
parking space, both when they have a goal parking space and also when they do
not. In the case that they do not have a goal parking space but have have a
high accuracy towards their tracked parking space, the agent is moving towards
an uncommunicated goal (since to the other agents it appears the agent is
exploring) which can considered selfish as the agent is pursuing its goal
without telling the other agents.

  * •

The average number of stopped goals ($p\rightarrow e$ goal transitions) per
episode. This metric also enables us to measure the decisiveness of the
agents, since a high number of stopped goals per episode means the agents are
willing to change their mind quickly and stop pursuing a goal, which can be
considered indecisive (and also uncompetitive) behaviour.

  * •

The average number of lost goals (times when the agent is no longer able to
track their goal parking space) per episode. This metric enables us to measure
the accuracy of their agents towards their goal, as well as the likelihood
that their chosen goals are likely to not be taken by other agents, since the
agent may lose its goal if it moves away from it or if another agent takes it.
Hence, a low number of lost goals indicates the agent is picking good goals
and that they travel accurately towards them.

##### Results

With our measured metrics defined, we train $7$ agents for $16$ million
episodes with $r_{\delta g}(p\rightarrow p^{\prime})$ varying between $0$ and
$-0.3$. Figure 5.16 shows the results of the ratio of the agents’ exploration
and the ratio of their movement towards their goal while both having a goal
and exploring, and Figure 5.17 shows the results of the average number of
stopped and lost goals.

Figure 5.16: Ratio of exploration and movement towards parking spaces when
varying the change goal punishment. Figure 5.17: Number of lost and stopped
goals when varying the change goal punishment.

Reading the graphs from right to left, it can be seen from Figure 5.16 that as
the change goal punishment increases beyond approximately $-0.8$, the agents
explore more and travel towards an uncommunicated goal more often. In
addition, from Figure 5.17 it can be seen that the agents lose and change
their goals less often. As a result, their exhibited behaviour becomes
increasingly selfish, since they spend more time exploring and travelling
towards an uncommunicated goal. The agents’ goal parking space selections also
become better because as $|r_{\delta g}(p\rightarrow p^{\prime})|$ increases,
we see the number of lost goals decrease significantly but the ratio of
movement towards chosen goals remain very close to $1$, so the reason for the
reduction must be due to their goals being taken by other agents less often.

This is an interesting result, because the learned behaviour may have instead
been that the agents explore less and pursue their goal more competitively,
however our models exhibit another kind of competitive behaviour whereby the
agents are unlikely to choose their goal until they’re certain it’s a good
one.

In our experiments we also find that a too small value of $|r_{\delta
g}(p\rightarrow p^{\prime})|$ causes the agents to get stuck in a halting
local optima, with a very low park rate, and that $r_{\delta g}(p\rightarrow
p^{\prime})\leq-0.005$ is required for convergence to a model with a high park
rate. However, within that threshold, increases in the punishment do not cause
a clear trend to the park rate. As a result, in light of our results, it’s
important that a sufficiently large value of $|r_{\delta g}(p\rightarrow
p^{\prime})|$ is chosen so the agents learn to park successfully, but the
value should not be too large that the agents become too selfish.

####  5.3.4 Collaboration

In addition to competition being an important aspect of the agents’ behaviour,
collaboration is a dually important aspect. Hence, in this section, we assess
the collaborative behaviours exhibited by the agents in our environment with
dynamic goals, classifying their collaborative behaviour with our giving way
framework defined in Section 3.2.6.

##### Method

To assess collaborative behaviour among the agents, we measure their
conformity to the way give-way contexts, where conformity to a context is
defined in Equation 3.11. Likewise, we enforce collaborative behaviour among
the agents by enforcing different give-way contexts with different strengths,
specifically by punishing the agent on a $p\rightarrow p$ goal transition when
$G_{\downarrow}(s^{*})$ is true (i.e. the agent is pursuing a bad goal, as
determined by $\Delta_{C}$ for the context $C$ being enforced).

Hence, we vary the bad goal punishment across all of the give-way schemes as
defined in Section 3.2.6, and measure the agents’ conformity to to each
context when each give-way scheme is used. Specifically, we use the $(L,L,S)$,
$(L,L,A)$, $(G,L,S)$, $(G,L,A)$, $(G,G,S)$ and $(G,G,A)$ give-way schemes, and
measure the agents’ conformity to the $(L,S)$, $(L,A)$, $(G^{+},S)$ and
$(G^{+},A)$ give-way contexts. We enforce every scheme for completion, and we
measure the $G^{+}$ contexts instead of $G$ contexts because $G^{+}$ contexts
are fully disjoint from the $L$ contexts, preventing the obtained conformities
from potentially being misleading (conformity to $G$ contexts may occur as a
result of conformity to $L$ contexts, but the agents may not actually be
conforming to the $G$ contexts, but rather only their $L$ subset).

In addition to measuring the agents’ conformity to the give-way contexts, we
also measure the same metrics as in our competition experiments (Section
3.2.6) to measure competitive behaviour that may also arise.

##### Results

With our method defined, we train $7$ agents for $16$ million episodes under
each give-way scheme, where in each we vary the bad goal punishment between
$0$ and $-0.2$, and we measure the conformity to the give-way schemes in each
configuration. Figure 5.18 shows the results of the agents’ conformity to each
give-way context (y axis) under each give-way scheme (each individual graph)
as the bad goal reward varies (x axis). Hence, when reading the graphs from
right to left, the give-way schemes are more greatly enforced. For example,
the top-left graph shows the conformities to the give-way contexts as the
$(L,L,S)$ give-way scheme is enforced (from right to left), and the bottom-
right graph shows their conformities as the $(G,G,A)$ give-way scheme is
enforced.

Figure 5.18: Conformity to each give-way context under each give-way scheme
when varying the bad goal reward.

While we obtain the expected result of the agents conforming to all of the
contexts that are a subset of each give-way scheme being enforced, Figure 5.18
entails other interesting results.

One mysterious result is that the $(G^{+},S)$ context is being conformed to in
the $(L,L,A)$ scheme (since its conformity is noticeably larger than in the
$(L,L,S)$ scheme). However, the agents do not have access to global
information in the $(L,L,A)$ scheme, making the conformity to the non-local
context non-trivial, since the agents do not have access to the global
information required to conform to it. The author believes such a result is
due to to the collective behaviour of the agents under the $(L,L,A)$ scheme,
and the author conjectures that such ‘leaking’ conformity may be due to the
fact that $(L,A)\not\subseteq(G,S)$ and $(G,S)\not\subseteq(L,A)$ in Figure
3.11. However, the reason for such leaking conformity is an open question, and
is future work (Section 6.2).

In addition, the addition of global information to the $(L,L,A)$ scheme does
not reduce the conformity to $(G^{+},S)$, despite the agents having sufficient
information to determine the scenarios under the $(G^{+},S)$ context in the
$(G,L,A)$ scheme. Hence, it appears that schemes enforcing giving way to other
agents with any goal causes more general behaviour than just conforming to the
contexts being enforced. This is further supported by the increase in
conformity to $(L,S)$ contexts in the $(L,L,A)$ scheme rather than the
$(L,L,S)$ scheme, despite the $(L,L,S)$ scheme specifically enforcing giving
way in the context. This may be due to there being more situations to give way
under the any goal contexts than the same goal contexts, hence when being
enforced the agent receives more punishments and thus has more experiences to
reinforce the giving way behaviour, resulting in generally more collaborative
behaviour.

Since the $(G,G,A)$ scheme enforces giving way in the $(G,A)$ context which is
a superset of all of the other contexts, enforcing the $(G,G,A)$ enables us to
compare the conformity of every context when they are all simultaneously
enforced. As mentioned in Section 3.2.6, we can thus devise an alternate
hierarchy of strength among the contexts, where those that are less likely to
be conformed to can be considered ‘stronger’ than those that are not. Hence,
from the $(G,G,A)$ scheme in Figure 5.18 we see that
$(G^{+},A)\succ(L,A)\succ(G^{+},S)\succ(L,S)$ (where $C_{1}\succ C_{2}$
denotes that $C_{1}$ is ‘stronger’ than $C_{2}$). It’s important to note here
that $(L,A)\succ(G^{+},S)$, indicating that $(L,A)\succ(G,S)$ and providing
one potential answer to the question of which context between $(L,A)$ and
$(G,S)$ is stronger in Section 3.2.6. Hence, in terms of conformity, the
agents are more likely to conform to same goal contexts than any goal
contexts, irrespective of whether the context is non-local or not.

Finally, in all of the graphs one can observe that the agents do not conform
any give-way context when the bad goal punishment is equal to $0$. Hence,
without enforcing giving way in particular contexts, the agents do not learn
to give way to eachother in our environment. While interesting, this result
isn’t very surprising, because in IPPO each agent is independent and thus
self-interested.

##### Competition

In addition to collaborative behaviours being exhibited by the agents under
the give-way schemes, they also exhibit competitive behaviours. Specifically,
under the $(G,G,A)$ scheme, we observe that the agents learn to exhibit
increasingly selfish behaviour as the bad goal punishment increases, as shown
by the graphs in Figure 5.19 and Figure 5.20, using the same reasoning as in
Section 5.3.3. Hence, aspects of both collaborative and competitive behaviours
are exhibited when we enforce the give-way schemes, but unfortunately
collaboration does not arise in our models without selfishness.

Figure 5.19: Ratio of exploration and movement towards parking spaces when
varying the bad goal punishment in the $(G,G,A)$ give-way scheme. Figure 5.20:
Number of lost and stopped goals when varying the bad goal punishment in the
$(G,G,A)$ give-way scheme.

##  Chapter 6 Conclusions

In conclusion, our project is a success, as it fully achieves its objectives
defined in Section 1.2. In this chapter, we summarise our project’s
achievements in relation to its objectives, discuss some difficulties and
lessons learned, and discuss any of the project’s limitations and future work
that may follow from it. In what follows, we refer to the objectives by their
indexes in Section 1.2.

###  6.1 Summary

To summarise our project, we designed and and implemented flexible
abstractions of a car parking environment suitable for the application of
Q-Learning and IPPO, achieving Objective 1. In our abstraction, the agents may
have fixed or dynamic goals, and several properties of the environment can be
varied such as its density and dimensionality.

With the environment implemented, we implement and apply Q-Learning to our
environment, discovering several limitations in the obstacle avoidance
behaviours that can be learned, achieving Objective 2.

With the limitations of Q-Learning in our environment known, we apply IPPO to
our environment, harnessing the superior scalability of deep RL methods. We
implement an automated grid-searching and model analysis suite to train and
evaluate models at scale, enabling the achievement of Objectives 2, 3 and 4.

After tuning PPO’s hyperparameters in our environment with fixed goals and
obstacles present, we learn models using IPPO parking up to $7$ agents with a
very high success rate of at least $98.1\%$, and $2$ agents with a success
rate of at least $99.3\%$, achieving Objective 2. Such an achievement
significantly outperforms Thunyapoo et al. [26]’s maximal $96.15\%$ success
rate with single agents in their comparable environment, even with the non-
stationarity present from multiple agents in our environment. The main
difference between our studies is that we share the localised pose information
between the cars for collision avoidance, whereas they use several lasers
around their agent to detect nearby obstacles, hence we see a significant
advantage with our information-sharing approach. Various parameters of the
environment are also varied to obtain results relating to its density and
dimensionality, shared state (communication) among the agents, as well as the
agents’ safety within, achieving Objectives 2 and 3.

Having successfully applied IPPO to our environment with fixed goals, we apply
IPPO to our environment with dynamic goals, obtaining a very high success rate
of at least $99.5\%$ with 7 agents and obstacles present, achieving Objective
2. By varying the environment’s parameters, we obtain several results relating
to competitive and collaborative behaviours exhibited the agents in our
environment, achieving Objective 4. We find that competition arises in the
form of selfishness, and that collaboration only arises when explicitly
enforced, in the form of giving way to other agents. In addition, we find that
collaboration does not arise in our models without selfishness, requiring a
trade-off between the two behaviours, and that enforcing giving way in certain
contexts causes the agents to give way in other unexpected contexts.

###  6.2 Limitations and Future Work

Our project has several limitations, leading to a variety of potential future
work.

##### Realism

Several of the limitations of our project relate to the design of the MDP and
its realism. First, the basic motion model employed by the environment’s
dynamics may not be fully realistic, since there exist more realistic models,
such as Ackermann-type steering [31] and the physics implemented in the Gazebo
simulator [13]. Thus, future work could potentially enhance the motion models
of the agents. In addition, we find from our experiments that the agents do
not utilise their full velocity range much, often travelling forwards with
full velocity. Such behaviour doesn’t align with realistic driving, which
often utilises a range of velocities and alternates between forwards and
reversing motion while parking for fine-grained motion. Hence, future work may
also investigate improving the velocity ranges utilised by the agents. Despite
the increase in realism such an extension would provide, the specifics of how
the agents move themselves from position A to position B in our environment
may not be crucial to the investigation of the agents’ higher level behaviours
relating to goal selection and planning, thus such an extension may not change
our higher-level results much.

Another limitation with respect to the realism of our environment is that it
lacks road rules and the mechanisms to enforce them. In realistic car parking
scenarios, the cars would have to conform to road rules, which may potentially
modify the agents’ selection of their goals, potentially yielding different
results with respect to goal selection and path planning. Thus, extending the
MDP to support dynamic road rules and investigating the effect of such road
rules on the agents’ goal planning is another area of future work of the
project. However, the pursuit of such future work may be negated if one
assumes that road rules may be relaxed when implementing our model in
practice, hence the pursuit of such work again depends on the objectives.

On a related vein of constraining the agents, additional future work may also
entail varying the field of view of the agents in our environment, reducing
the range in which they can communicate. Such reductions may entail
interesting results with respect to the horizons of the agents’ goal planning
behaviour. Since our environment supports a dynamic field of view via the
_obsNearbyCarsDiameter parameter, such work wouldn’t require much additional
implementation.

Finally, our MDP is fully deterministic. Future work may entail relaxing that
assumption by introducing stochasticity into the environment’s dynamics, and
investigating its effect.

##### Safety

There are several limitations relating to the safety of the agents’ learned
models. First, we assume that each car has the same size hitbox, which may be
dangerous for cars with larger hitboxes. Hence, future work may entail making
the hitboxes of the cars dynamic and encoding hitbox information into the
agents’ state. In addition, the agents’ obstacle avoidance behaviour is
extremely limited with respect to non-car obstacles. Thus, if non-car
obstacles are deemed important in our environment, then future work may entail
additionally improving obstacle avoidance for non-car obstacles.

Since the agents may make mistakes, additional future work may entail enabling
real drivers to override the behaviour, for example if they get dangerously
close to a car. So far we have investigated techniques to improve the agents’
driving safety by making them keep a threshold distance from other cars,
however our MDP does not provide the tools for manual intervention of the
agents’ behaviours, which may be a useful extension.

##### Infrastructure

Our suite of tools for training models may also be improved. Currently, models
can only be trained for a maximum duration as determined by the maximum job
length of the department’s compute cluster (two days), limiting the training
of larger models that take longer to train. Training time may also be reduced
by splitting the models into different re-usable parts, so re-learning e.g.
locomotion and basic collision avoidance behaviour isn’t necessary every time
a model is trained. Such re-use may come with the additional benefit of
successfully training in environments with many parked cars present,
increasing the difficulty of the scenarios the agents are exposed to during
training, and potentially increasing the safety of the learned models.

Since our environment has became fairly complex with lots of different
parameters, future work may also wish to use more sophisticated techniques to
optimise the environment’s parameters (rather than the basic technique of
grid-searching used in our study). In particular, with respect to optimising
rewards, several techniques in the literature may be utilised. One such
technique is the use of genetic algorithms, as proposed by Sehgal et al. [23]
(2019). Another is the optimisation of dynamic rewards which change as
training progresses, as proposed by Pasunuru et al. [19] (2020). Finally,
Pasunuru and Bansal [18] (2018) propose a novel technique of splitting the
loss function optimised in deep RL into several individual ones for each
reward, and optimising each loss function simultaneously.

##### Collaboration

Finally, there is a variety of potential future work in resolving the
questions that arised from our results relating to the agents’ collaborative
behaviours in Section 5.3.4. In particular, we desire a rigorous explanation
of the ‘leaking’ conformity to the $(G^{+},S)$ give-way context under the
$(L,L,A)$ scheme, providing an answer to our conjecture proposed in Section
5.3.4. One may also wish to extend the give-way contexts to harness additional
information available in our MDP, such as angular information, to increase the
use of our MDP’s information. Alternatively, deep learning may be exploited to
identify new give-way contexts without them being user-defined.

As a large extension, it may be interesting to investigate whether other MARL
algorithms learn to conform to give-way contexts without them being explicitly
enforced, unlike IPPO as discovered in our study. Some candidate algorithms
for such investigations may be MAPPO [30] due to its close ties to IPPO but
the agents are not fully independent, and MA-POCA [11]: a true multi-agent
algorithm invented and implemented in ML-Agents, which may be harnessed
without significant additional implementation.

## References

  * Abadi et al. [2015] Abadi, Martín & Agarwal, Ashish & Barham, Paul & Brevdo, Eugene & Chen, Zhifeng & Citro, Craig & Corrado, Greg S. & Davis, Andy & Dean, Jeffrey & Devin, Matthieu & Ghemawat, Sanjay & Goodfellow, Ian & Harp, Andrew & Irving, Geoffrey & Isard, Michael & Jia, Yangqing & Jozefowicz, Rafal & Kaiser, Lukasz & Kudlur, Manjunath & Levenberg, Josh & Mané, Dandelion & Monga, Rajat & Moore, Sherry & Murray, Derek & Olah, Chris & Schuster, Mike & Shlens, Jonathon & Steiner, Benoit & Sutskever, Ilya & Talwar, Kunal & Tucker, Paul & Vanhoucke, Vincent & Vasudevan, Vijay & Viégas, Fernanda & Vinyals, Oriol & Warden, Pete & Wattenberg, Martin & Wicke, Martin & Yu, Yuan & Zheng, Xiaoqiang.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Busoniu et al. [2008] Busoniu, Lucian & Babuska, Robert & De Schutter, Bart.  A comprehensive survey of multiagent reinforcement learning.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , 38(2):156–172, 2008.  doi: 10.1109/TSMCC.2007.913919.
  * Choi et al. [2021] Choi, Jaewan & Lee, Geonhee & Lee, Chibum.  Reinforcement learning-based dynamic obstacle avoidance and integration of path planning.  _Intelligent Service Robotics_ , 14:1–15, 11 2021.  doi: 10.1007/s11370-021-00387-2.
  * Chu et al. [2019] Chu, Tianshu & Wang, Jie & Codecà, Lara & Li, Zhaojian.  Multi-agent deep reinforcement learning for large-scale traffic signal control.  _CoRR_ , abs/1903.04527, 2019.  URL http://arxiv.org/abs/1903.04527.
  * de Witt et al. [2020] de Witt, Christian Schroeder & Gupta, Tarun & Makoviichuk, Denys & Makoviychuk, Viktor & Torr, Philip H. S. & Sun, Mingfei & Whiteson, Shimon.  Is independent learning all you need in the starcraft multi-agent challenge?, 2020.  URL https://arxiv.org/abs/2011.09533.
  * Fujimoto et al. [2018] Fujimoto, Scott & Meger, David & Precup, Doina.  Off-policy deep reinforcement learning without exploration, 2018.  URL https://arxiv.org/abs/1812.02900.
  * Garg et al. [2018] Garg, Deepeka & Chli, Maria & Vogiatzis, George.  Deep reinforcement learning for autonomous traffic light control.  In _2018 3rd IEEE International Conference on Intelligent Transportation Engineering (ICITE)_ , pages 214–218, 2018.  doi: 10.1109/ICITE.2018.8492537.
  * Gronauer and Dieopold [2021] Gronauer, Sven & Dieopold, Klaus.  Multi-agent deep reinforcement learning: a survey.  _Artificial Intelligence Review_ , pages 1–49, 04 2021.  doi: 10.1007/s10462-021-09996-w.
  * Harris et al. [2020] Harris, Charles R. & Millman, K. Jarrod & van der Walt, Stéfan J. & Gommers, Ralf & Virtanen, Pauli & Cournapeau, David & Wieser, Eric & Taylor, Julian & Berg, Sebastian & Smith, Nathaniel J. & Kern, Robert & Picus, Matti & Hoyer, Stephan & van Kerkwijk, Marten H. & Brett, Matthew & Haldane, Allan & del Río, Jaime Fernández & Wiebe, Mark & Peterson, Pearu & Gérard-Marchant, Pierre & Sheppard, Kevin & Reddy, Tyler & Weckesser, Warren & Abbasi, Hameer & Gohlke, Christoph & Oliphant, Travis E.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, September 2020\.  doi: 10.1038/s41586-020-2649-2.  URL https://doi.org/10.1038/s41586-020-2649-2.
  * Hunter [2007] Hunter, J. D.  Matplotlib: A 2d graphics environment.  _Computing in Science & Engineering_, 9(3):90–95, 2007.  doi: 10.1109/MCSE.2007.55.
  * Juliani et al. [2018] Juliani, Arthur & Berges, Vincent-Pierre & Teng, Ervin & Cohen, Andrew & Harper, Jonathan & Elion, Chris & Goy, Chris & Gao, Yuan & Henry, Hunter & Mattar, Marwan & Lange, Danny.  Unity: A general platform for intelligent agents, 2018.  URL https://arxiv.org/abs/1809.02627.
  * Kingma and Ba [2014] Kingma, Diederik P. & Ba, Jimmy.  Adam: A method for stochastic optimization, 2014.  URL https://arxiv.org/abs/1412.6980.
  * Koenig and Howard [2004] Koenig, N. & Howard, A.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In _2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566)_ , volume 3, pages 2149–2154 vol.3, 2004.  doi: 10.1109/IROS.2004.1389727.
  * Krajzewicz et al. [2002] Krajzewicz, Daniel & Hertkorn, Georg & Feld, Christian & Wagner, Peter.  Sumo (simulation of urban mobility); an open-source traffic simulation.  pages 183–187, 01 2002.  ISBN 90-77039-09-0.
  * Kurniawati [2021] Kurniawati, Hanna.  Partially observable markov decision processes (pomdps) and robotics.  _CoRR_ , abs/2107.07599, 2021.  URL https://arxiv.org/abs/2107.07599.
  * Lai [2018] Lai, Leonardo.  Autoparking.  https://github.com/leoll2/Autoparking, 2018.
  * Mnih et al. [2013] Mnih, Volodymyr & Kavukcuoglu, Koray & Silver, David & Graves, Alex & Antonoglou, Ioannis & Wierstra, Daan & Riedmiller, Martin A.  Playing atari with deep reinforcement learning.  _CoRR_ , abs/1312.5602, 2013.  URL http://arxiv.org/abs/1312.5602.
  * Pasunuru and Bansal [2018] Pasunuru, Ramakanth & Bansal, Mohit.  Multi-reward reinforced summarization with saliency and entailment, 2018\.  URL https://arxiv.org/abs/1804.06451.
  * Pasunuru et al. [2020] Pasunuru, Ramakanth & Guo, Han & Bansal, Mohit.  Dorb: Dynamically optimizing multiple rewards with bandits, 2020.  URL https://arxiv.org/abs/2011.07635.
  * Paszke et al. [2019] Paszke, Adam & Gross, Sam & Massa, Francisco & Lerer, Adam & Bradbury, James & Chanan, Gregory & Killeen, Trevor & Lin, Zeming & Gimelshein, Natalia & Antiga, Luca & Desmaison, Alban & Kopf, Andreas & Yang, Edward & DeVito, Zachary & Raison, Martin & Tejani, Alykhan & Chilamkurthy, Sasank & Steiner, Benoit & Fang, Lu & Bai, Junjie & Chintala, Soumith.  Pytorch: An imperative style, high-performance deep learning library.  In Wallach, H. & Larochelle, H. & Beygelzimer, A. & d'Alché-Buc, F. & Fox, E. & Garnett, R., editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Schulman et al. [2015] Schulman, John & Levine, Sergey & Moritz, Philipp & Jordan, Michael I. & Abbeel, Pieter.  Trust region policy optimization, 2015.  URL https://arxiv.org/abs/1502.05477.
  * Schulman et al. [2017] Schulman, John & Wolski, Filip & Dhariwal, Prafulla & Radford, Alec & Klimov, Oleg.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.  URL http://arxiv.org/abs/1707.06347.
  * Sehgal et al. [2019] Sehgal, Adarsh & La, Hung Manh & Louis, Sushil J. & Nguyen, Hai.  Deep reinforcement learning using genetic algorithm for parameter optimization, 2019.  URL https://arxiv.org/abs/1905.04100.
  * Sutton and Barto [2018] Sutton, Richard S. & Barto, Andrew G.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA, 2018.  ISBN 0262039249.
  * Thrun et al. [2005] Thrun, Sebastian & Burgard, Wolfram & Fox, Dieter.  _Probabilistic Robotics (Intelligent Robotics and Autonomous Agents)_.  The MIT Press, 2005.  ISBN 0262201623.
  * Thunyapoo et al. [2020] Thunyapoo, Baramee & Ratchadakorntham, Chatree & Siricharoen, Punnarai & Susutti, Wittawin.  Self-parking car simulation using reinforcement learning approach for moderate complexity parking scenario.  In _2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)_ , pages 576–579, 2020.  doi: 10.1109/ECTI-CON49241.2020.9158298.
  * Watkins and Dayan [1992] Watkins, Christopher J. C. H. & Dayan, Peter.  Q-learning.  _Machine Learning_ , 8(3):279–292, May 1992.  ISSN 1573-0565.  doi: 10.1007/BF00992698.  URL https://doi.org/10.1007/BF00992698.
  * Ying [2019] Ying, Xue.  An overview of overfitting and its solutions.  _Journal of Physics: Conference Series_ , 1168:022022, feb 2019.  doi: 10.1088/1742-6596/1168/2/022022.  URL https://doi.org/10.1088/1742-6596/1168/2/022022.
  * Yoo et al. [2003] Yoo, Andy B. & Jette, Morris A. & Grondona, Mark.  Slurm: Simple linux utility for resource management.  In Feitelson, Dror & Rudolph, Larry & Schwiegelshohn, Uwe, editors, _Job Scheduling Strategies for Parallel Processing_ , pages 44–60, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.  ISBN 978-3-540-39727-4.
  * Yu et al. [2021] Yu, Chao & Velu, Akash & Vinitsky, Eugene & Wang, Yu & Bayen, Alexandre M. & Wu, Yi.  The surprising effectiveness of MAPPO in cooperative, multi-agent games.  _CoRR_ , abs/2103.01955, 2021.  URL https://arxiv.org/abs/2103.01955.
  * Zhao et al. [2013] Zhao, Jing-Shan & Liu, Zhi-Jing & Dai, Jian.  Design of an ackermann type steering mechanism.  _Journal of Mechanical Engineering Science_ , 227, 11 2013.  doi: 10.1177/0954406213475980.

##  Appendix A Spatial Symmetry of Local Object Pose Encoding

This appendix shows how our encoding of the pose of nearby objects as
described in Section 3.2.2 preserves $d$ and $\theta$ after both cars change
their position by the same constants $a$ and $b$ on either axis, and preserves
$\delta\theta$ after both cars change their rotations by the same constant
$c$.

$d$ is preserved since if the cars are at a different position only differing
by a constant $a$ and $b$ on either axis, the new positions of the green and
grey car are $(x+a,y+b)$ and $(x^{\prime}+a,y^{\prime}+b)$ respectively, thus
the L2 distance between them is:

$\sqrt{(x^{\prime}+a-(x+a))^{2}+(y^{\prime}+b-(y+b))^{2})}=\sqrt{(x^{\prime}-x)^{2}+(y^{\prime}-y)^{2}}$

, which is the same as the L2 distance between the original positions of the
cars, yielding the same value of $d$. $\theta$ is also preserved, since the
global angle between them is

$\arctan(\frac{(y^{\prime}+b)-(y+b)}{(x^{\prime}+a)-(x+a)})=\arctan(\frac{y^{\prime}-y}{x^{\prime}-x})$

, which is equal to $\theta$ after taking the angle relative to the green
car’s global rotation.

Likewise, $\delta\theta$ is preserved, since if both cars both rotate by a
constant $c$, and their global rotations are $\theta_{1}$ and $\theta_{2}$
respectively, then their new global rotations are $\theta_{1}+c$ and
$\theta_{2}+c$ respectively, thus the difference in their global rotation is
$(\theta_{1}+c)-(\theta_{2}+c)=\theta_{1}-\theta_{2}=\delta\theta$, preserving
$\delta\theta$.

##  Appendix B Deadlock Free Evolution Proof

This appendix informally proves Theorem 2.

###### Proof.

First, observe that the position of each agent is randomly distributed, since
they’re spawned in random positions and travel towards their randomly chosen
parking space to park in it.

Next, observe that every parked car in the environment is eventually moved,
because we move the furthest parked car from all of the agents in the
environment. The positions of the agents in our environment are randomly
distributed, thus each parked car has a chance to be the furthest parked car
from all of the agents.

When a parked car is moved, it is moved into an agent’s parking space when the
agent successfully parked in it. Thus, there is a chance a parked car is moved
to a parking space that is outside the cluster, because the agent may have a
goal parking space outside of the cluster, since we are guaranteed that such a
space exists.

Thus, every parked car in the cluster has a chance to be moved to a parking
space that is outside the cluster, eventually breaking it. ∎

##  Appendix C Implementation Tables

This appendix contains the tables for Chapter 4.

Parameter | Type |  | Default
---
Value
Description
_positionGranularity | int | 1 | $G_{p}$
_velocityGranularity | int | 1 | $G_{v}$
_thetaGranularity | int | 8 | $G_{\theta}$
_maxVelocityMagnitude | int | 1 | ${v_{max}}^{+}$
_minVelocityMagnitude | int | 1 | ${v_{max}}^{-}$
_maxDeltaVMagnitude | int | 1 | $a_{max}$
_maxDeltaThetaMagnitude | int | 1 | $\omega_{max}$
_numAgents | int | 1 | Number of agents in the environment.
_normalizeObs | bool | false | Whether the state values are normalised to the range $[0,1]$.
_debugObs | bool | false | Whether to print the values of the observations to the console.
_visualiseNearbyCars | bool | false | Show lines of the agents’ tracked cars and parking spaces.
_numParkedCars | int | 0 | Number of parked cars.
_obsDist | bool | false | Whether to encode the agents’ distance to their goal in the state.
_obsAngle | bool | true | Whether to encode the agents’ angle to their goal in the state.
_obsRings | bool | false | Whether the agents have rings around them.
_ringMaxNumObjTrack | int | 0 | Maximum number of objects each ring can count.
ringDiams | int[] | [] |  | Diameters of the rings (in world units).
---
Also can be populated via _rd<...> below.
_rd<...> | int | 0 |  | Diameters of each ring in separate _rd<...> parameters
---
(with different <...> suffixes).
ringDiams is populated with these values at run-time.
_obsGoalDeltaPose | bool | false |  | Whether to encode the agents’ difference in global rotation
---
in the state.
_ringNumPrevObs | int | 0 | Number of historical ring states to encode in the state.
_ringOnlyWall | bool | false | Whether the rings should only detect walls.
_obsNearbyCars | bool | false |  | Whether to encode the state of tracked cars
---
(if false, $n_{track}=0$).
_obsNearbyCarsCount | int | 0 | $n_{track}$
_obsNearbyCarsDiameter | int | 0 | Maximum distance nearby cars can be tracked (in world units).
spawnCloseRatio | float | 0 | Ratio in which an agent is spawned near its goal parking space.
_spawnCloseDist | int | 10 |  | Maximum distance (in world units) to spawn the agent away
---
from its goal parking space when it’s to be spawned near it.
carSpawnMinDistance | int | 9 |  | Minimum distance (in MDP units) to an agent in which the
---
agent can be legally spawned.
_maxSteps | int | 85 | Maximum number of steps in an agent’s episode.
_obsNearbyCarsGoal | bool | false | Whether to share the goals of tracked cars.
_obsNearbyCarsVelocity | bool | false | Whether to share the velocities of tracked cars.
spawnCrashRatio | float | 0 | Ratio in which an agent is spawned to crash with another agent.
spawnCrashTargetAgentMinDist | float | 10 |  | Minimum distance (in MDP units) an agent must be to its goal
---
for it to be picked as a candidate to try and spawn a crash with.

Table C.1: Environment parameters.

Parameter | Type |  | Default
---
Value
Description
_dynamicGoals | bool | false | Whether the agents have dynamic goals.
_obsNearbyParkingSpotsCount | int | 0 | $n_{park}$
rewDeltaGoalContinueExp | float | 0 | $r_{\delta g}(e\rightarrow e)$
rewDeltaGoalStopExp | float | 0 | $r_{\delta g}(e\rightarrow p)$
rewDeltaGoalContinueGoal | float | 0 | $r_{\delta g}(p\rightarrow p)$
rewDeltaGoalDiffGoal | float | 0 | $r_{\delta g}(p\rightarrow p^{\prime})$
_rewDeltaGoalStopGoal | float | 0 |  |  $r_{\delta g}(p\rightarrow e)$.
---
If equal to -1, we compute it via eq. 3.6.
| rewDeltaGoal
---
ContinueGoalBetterOtherAgent
float | 0 |  |  $r_{\delta g}(p\rightarrow p)$ when the
---
current goal is a bad goal.
rewReachGoal | float | 10 | Reward for reaching the goal parking space.
rewCrash | float | 10 | Magnitude of the punishment for crashing with an obstacle.
rewTimeSum | float | 0.5 | Magnitude of the sum of the time-based punishment.
rewReverseSum | float | 0 | Sum of the reversing punishment.
rewDistSum | float | 1 | Magnitude of the sum of the move towards goal reward.
_obsParkingSpotClosestAgent | bool | false |  | Whether to encode global information of the agent’s
---
current goal with respect to all agents (minimum distance
across all agents to the goal).
_obsParkingSpotClosestGoalAgent | bool | false |  | Whether to encode global information of the agent’s
---
current goal with respect to only agents with the same goal
(minimum distance to the goal
across all agents with the same goal).
_punishBetterOtherAgentLocal | bool | false | Whether to enforce the local give-way scheme (global if false).
_punishBetterOtherGoalAgent | bool | false |  | Whether to enforce the same goal give-way scheme
---
(any goal scheme if false).
_numStepsTrain | int | 0 | Total number of steps in the training phase.
carScaleTrain | float | 1 | Multiplier of the scale of the cars during training.
rewFinalVelocitySum | float | 0 |  | Maximum punishment for the agent’s velocity upon parking.
---
We multiply the fraction between the min and max of the
magnitude of the agent’s velocity upon parking by this
value to get the punishment.
rewDeltaThetaSum | float | 0 |  | Sum of the magnitude of the smoothness punishment
---
to apply to the agent at each time-step.
_rewDeltaThetaVelMult | bool | false |  | Whether the magnitude of the agent’s velocity should
---
be included in the smoothness punishment at each time-step.
If not, only the agent’s angular velocity is considered.

Table C.2: Environment parameters (continued).

CSV Metric ID | Description |  | TensorBoard
---
Metric Path
| Computation
---
Method
Max Steps |  | Number of steps
---
the agents were
evaluated over.
Via metadata. | Metadata.
Final Mean Reward |  | Mean cumulative reward
---
received by an
agent in its episode.
| Environment
---
/Cumulative Reward
mean_metric_period
| Final Mean
---
Episode Length
| Mean number of steps
---
in each of the
agents’ episodes.
| Environment
---
/Episode Length
mean_metric_period
c | Crash rate. |  | Metrics
---
/Total Crashes
per_eps
p | Parking rate. |  | Metrics
---
/Total Reached Goal
per_eps
h |  | Halt rate
---
(halting is reaching
$\tau$ time-steps and
not crashing or parking).
| Metrics
---
/Total Halted
per_eps
Num Episodes |  | Number of episodes
---
the agents were
evaluated over.
| Metrics
---
/Num Episodes
| last_minus_start
---
_metricperiod
Total Crashes Car |  | Number of crashes
---
with cars.
| Metrics
---
/Total Crashes Car
| last_minus_start
---
_metricperiod
Total Crashes Wall |  | Number of crashes
---
with walls.
| Metrics
---
/Total Crashes Wall
| last_minus_start
---
_metricperiod
Total Crashes StaticCar |  | Number of crashes
---
with parked cars.
| Metrics
---
/Total Crashes StaticCar
| last_minus_start
---
_metricperiod
Num Lost Goal PerEps |  | Average number of
---
lost goals per episode.
| Metrics
---
/Num Lost Goal
per_eps
Num Stop Explore PerEps |  | Average number of
---
stop explores per
episode.
| Metrics
---
/Num Stop Explore
per_eps
Num Stop Goal PerEps |  | Average number of
---
stopped goals per
episode.
| Metrics
---
/Num Stop Goal
per_eps
Num Change Goal PerEps |  | Average number of
---
changed goals per
episode.
| Metrics
---
/Num Change Goal
per_eps

Table C.3: analysis.py metrics.

CSV Metric ID | Description |  | TensorBoard
---
Metric Path
| Computation
---
Method
GaveWayLocalAnyGoal |  | Conformity to the
---
$(L,A)$ give-way context.
| Via
---
context_conformance
context_conformance
GaveWayGlobalAnyGoal |  | Conformity to the
---
$(G,A)$ give-way context.
| Via
---
context_conformance
context_conformance
GaveWayLocalSameGoal |  | Conformity to the
---
$(L,S)$ give-way context.
| Via
---
context_conformance
context_conformance
GaveWayGlobalSameGoal |  | Conformity to the
---
$(G,S)$ give-way context.
| Via
---
context_conformance
context_conformance
GaveWayNonLocalSameGoal |  | Conformity to the
---
$(G^{+},S)$ give-way context.
| Via
---
context_conformance
context_conformance
GaveWayNonLocalAnyGoal |  | Conformity to the
---
$(G^{+},A)$ give-way context.
| Via
---
context_conformance
context_conformance
DeltaThetaAvg |  | Average value of
---
$\omega$ (angular velocity).
| Metrics
---
/DeltaThetaAvg
mean_metric_period
VelocityAvg |  | Average value of
---
$v_{c}$ (velocity).
| Metrics
---
/VelocityAvg
mean_metric_period
NearestCarDistAvg |  | Average distance
---
of the nearest car
to an agent.
| Metrics
---
/NearestCarDistAvg
mean_metric_period
RatioExplorePerEps |  | Average ratio of
---
time-steps in which
the agents explore
per episode.
| Metrics
---
/RatioExplorePerEps
mean_metric_period
RatioGoalPerEps | $1-\texttt{RatioExplorePerEps}$ |  | Metrics
---
/RatioGoalPerEps
mean_metric_period
RatioMoveTowardsPerEps |  | Average ratio of
---
time-steps in which
the agents move
towards their goal
parking space
per episode.
| Metrics
---
/RatioMoveTowardsPerEps
mean_metric_period
RatioMoveTowardsExploringPerEps |  | Average ratio of
---
time-steps in which
the agents move
towards their nearest
parking space while
exploring per episode.
| Metrics
---
/RatioMoveTowards
ExploringPerEps
mean_metric_period
Park Velocity |  | Average velocity of
---
the agents upon
parking.
| Metrics
---
/Park Velocity
mean_metric_period

Table C.4: analysis.py metrics (continued). | Argument
---
Index
| Argument Description
---
1 | Path to the directory of models.
2 |  | Number of steps in the training period.
---
Metrics aren’t collected in this period -
we only collect metrics after this period
(since after we’re in the metric period).
3 |  | Path to the output CSV file
---
containing the metrics
4 |  | Expected number of models in the
---
directory of models. If the number of
models doesn’t equal this value then
the script prematurely terminates since
the models haven’t finished learning yet.
5 |  | Paths to parameters in the
---
base PPO YAML configuration file
we wish to also record as a metric.
Table C.5: analysis.py arguments. Argument |  | Argument Description
---
\--alpha | Q-Learning learning rate.
\--gamma | Q-Learning discount factor.
\--epsilon | Q-Learning epsilon greedy factor.
\--episodes | Number of episodes to train.
\--episodeseval |  | Number of episodes to evaluate the agent
---
($\epsilon=0$).
\--dump |  | Interval of number of episodes to
---
print the current results to
standard output.
\--build | Directory of the Unity environment build.
\--output |  | Directory to output the learned model
---
and metrics.
\--workerid |  | Unique training ID, if running the program
---
in parallel.
\--alphamin |  | Minimum value of the learning rate.
---
Decays linearly to the value originally
from \--alpha over the number of training
episodes specified by \--episodes.
\--epsmin |  | Minimum value of epsilon.
---
Decays linearly to the value originally
from \--epsilon over the number of training
episodes specified by \--episodes.
\--envconfig |  | JSON string containing the
---
EnvironmentConfig.
\--numagents | Number of agents.
Table C.6: iql_train.py arguments. Argument |  | Argument Description
---
\--modelpath |  | Path to the pickled model file
---
(Q-Table) from training.
\--episodes | Number of episodes to run.
\--envconfig |  | JSON string containing the
---
EnvironmentConfig.
\--numagents | Number of agents.
\--build | Directory of the Unity environment build.
Table C.7: iql_run.py arguments. | Argument
---
Index
Argument Description
First $n$ |  | Paths to the $n$ pickled metrics files
---
(cumulative rewards for each episode for all agents)
for the $n$ runs.
Each path is suffixed by &<legend> where
<legend> is the legend for the line for the model.
$n+1$ |  | Size of the window to take the rolling average over
---
in the cumulative reward graph.
$n+2$ |  | Number of episodes used for training, used to plot the
---
red dashed line indicating when the agents entered
their evaluation zone.
Table C.8: plot.py arguments. Parameter | Value
---|---
_positionGranularity | 4
_velocityGranularity | 4
_thetaGranularity | 24
_maxVelocityMagnitude | 4
_minVelocityMagnitude | 2
_maxDeltaVMagnitude | 2
_minDeltaVMagnitude | 2
_maxDeltaThetaMagnitude | 3
_normalizeObs | true
_numParkedCars | 16
_obsDist | true
_obsRings | true
_ringMaxNumObjTrack | 1
ringDiams | [11]
_rd0 | 11
_obsGoalDeltaPose | true
_ringOnlyWall | true
_obsNearbyCars | true
_obsNearbyCarsCount | 1
_obsNearbyCarsDiameter |  | 300
---
(no constrained
FOV)
spawnCloseRatio | 0.2
carSpawnMinDistance | 210
_maxSteps | 200
_obsNearbyCarsGoal | true
_obsNearbyCarsVelocity | true
spawnCrashRatio | 0.2
spawnCrashTargetAgentMinDist | 210
rewTimeSum | 0.2
rewReachGoal | 1
rewCrash | 1
rewReverseSum | 0.1
rewDistSum | 0.15
rewDeltaThetaSum | 0.05
Table C.9: Values in the EnvironmentConfig for PPO experiments with fixed goals. Parameter | Value
---|---
_dynamicGoals | true
_obsNearbyParkingSpotsCount | 1
rewDeltaGoalContinueExp | -0.002
rewDeltaGoalDiffGoal | -0.05
_rewDeltaGoalStopGoal |  | -1
---
(compute via
eq. 3.6)
Table C.10: Values in the EnvironmentConfig for PPO experiments with dynamic goals. Argument Index | Description
---|---
1 |  mlagents-learn command.
2 |  | Base PPO YAML configuration file to base the
---
grid-search upon, containing the base PPO
hyperparameters and environment parameters.
We inject the grid-searched parameters into
copies of it.
3 |  | Base .slurm job file to base the grid-search upon.
---
We inject the different commands in each job into copies
of it corresponding to each parameter combination.
4 |  | Base port. Each run uses the ports sequentially
---
greater than the base port.
5 |  | Base run id, which is the prefix of the run ID
---
for each generated run.
6 to last |  | The parameters in the PPO YAML configuration
---
to grid-search over, and their respective ranges.
Each argument is of the form: <yaml key>=<values>,
where <yaml key> is a key in the YAML file,
and <values> is of the form <type>[$v_{1},...,v_{n}$]
where type is one of f(loat), i(nt), s(tring) or b(ool),
and $v_{i}$ are the values, all of type <type>.
Table C.11: gridsearch_ppo.py arguments.