# Loss and Reward Weighing for increased learning in Distributed Reinforcement
Learning

Martin Holen  Centre for Artificial Intelligence Research  Per-Arne Andersen
Centre for Artificial Intelligence Research  Kristian Muri Knausgård  Top
Research Centre Mechatronics  Morten Goodwin  Centre for Artificial
Intelligence Research

###### Abstract

This paper introduces two learning schemes for distributed agents in
Reinforcement Learning (RL) environments, namely Reward-Weighted (R-Weighted)
and Loss-Weighted (L-Weighted) gradient merger. The R/L weighted methods
replace standard practices for training multiple agents, such as summing or
averaging the gradients. The core of our methods is to scale the gradient of
each actor based on how high the reward (for R-Weighted) or the loss (for
L-Weighted) is compared to the other actors. During training, each agent
operates in differently initialized versions of the same environment, which
gives different gradients from different actors. In essence, the R-Weights and
L-Weights of each agent inform the other agents of its potential, which again
reports which environment should be prioritized for learning. This approach of
distributed learning is possible because environments that yield higher
rewards, or low losses, have more critical information than environments that
yield lower rewards or higher losses. We empirically demonstrate that the
R-Weighted methods work superior to the state-of-the-art in multiple RL
environments.

##  1 Introduction

Distributed Machine Learning (DML) is often used to speed up the training of
Neural Networks. Within DML, Federated Machine Learning (FML), aims to make
heterogeneous data, speed up training, and increase the robustness of the
learned model [7]. Handler dette paperet om Federated learning?

DML techniques focus on averaging neural network (NN) parameters after
multiple local updates or averaging the gradients on each update. This allows
learning from numerous environments with the same NN setup in many
interactions in the same environment [4]. Federated Averaging (FedAvg) is one
of the established methods, which updates each agent multiple times locally,
and then weights the parameters of each agent based on the amount of data used
for the updates [7].

When performing DML or FML, among the standard methods are averaging the
parameters after multiple local updates or averaging the gradients on each
update [7]. DML allows for learning from multiple environments with the same
NN or from various interactions in the same environment. Distributed and
federated machine learning leads to quicker NN training [4].

Among the topics covered by the federated implementations is the Byzantine
generals’ problem [2, 3]. The training may be hindered if there is a byzantine
agent or agents.

One common use-case for distributed machine learning is to speed up training
for complex RL environments, like autonomous vehicles, and cooperative games
[18, 9]. Training in complex RL environments can be achieved either for
single-agent or multi-agent implementations. Typical single agent
implementations include Proximal Policy Optimization (PPO) and Deep-Q Networks
(DQN), used for many tasks such as videogames like DOTA2, Atari, and StarCraft
[9, 8, 16]. Multi-Agent (MA) implementations include QMix and Value
Decomposition Networks (VDN). MA works for many tasks such as Multi Particle
Environments (MPE), Starcraft 2’s marines, stalkers and zealots, Switch, Fetch
and Checkers [15, 11, 14] and more examples.

Our implementation aims to find a novel technique to calculate the gradients
over multiple agents, using experiences from each agent. The gradients from
each agent are weighted based on their reward (R-weighted) or loss
(L-weighted). The weights are directly proportional to the reward implying a
higher reward leads to a larger weight. From reward weighting, we expect the
model to go faster towards actions that give higher rewards. For loss-based
weights, the higher losses mean that the agents predicted more incorrectly the
reward they received. By training agents for a longer time with loss-based
weights, they should be able to forecast rewards more accurately, resulting in
higher rewards in the end.

When creating weights for the gradients of each agent, we add a small number
to each agent’s weights (1/k) to ensure each agent’s gradients have some
effect on the NN weights. These hyperparameters make the weights closer to the
FedAvg and summed gradient algorithms, as each agent has a minimum weight that
can be placed relatively high. Using the logic behind the implementation,
high-reward agents take precedence over lower-reward agents, resulting in
faster convergence than either summing or averaging gradients. This logic
gives an edge to this implementation over current state-of-the-art methods.

To help explain the method; think of a scenario where there is a small game of
a set of doors. If the majority of the doors give a low reward (1.5), and
every n-th door gives a higher reward (4.5), but one last door gives an
extremely high reward (1000.0). If we implement the standard methods for
distributing training, we can use n agents with the same understanding;
namely, every nth door gives a high reward. Even if there is, say, 1 out of
128 agents choosing the door with the largest reward, their gradient will be
overshadowed by the other agents. To fix this, we scale the gradients based on
this reward so that even if 1 out of 128 agents chooses the last door, it will
have more importance than the other agents. Scaling the gradients aims to
increase the speed at which actions that give a higher reward are chosen. As
each of the 127 other agents could be good at predicting rewards of 1.5 and
4.5, these changes to the gradient could make the training take too long.

There are some scenarios in which most agents get negative rewards. If most
agents get negative rewards, then the agent could get good at predicting such
rewards but struggle with predicting higher rewards. The loss aims at
predicting all rewards, though when including a group of agents whose
gradients are summed or averaged, the gradient of agents which showed
improvement in the reward might be overshadowed.

Our implementation is not Multi-Agent Reinforcement Learning (MARL), as we use
identical agents, each interacting in its own environment [15]. We also do not
consider the byzantine generals’ problem, assuming that all of the agents are
good actors and the communication is spotless.

##  2 Background

Recent advancements in RL have led to significant progress in single and
multi-agent implementations, including work such as DQN, PPO, DDPG, VDN, and
QMIX, amongst others. Along with the advancements in RL, some new RL tasks
requiring enormous training have arisen, such as autonomous driving. One
common way of solving them for training in very complex environments is
through distributed RL (DistRL). DistRL allows for training to be done across
a large number of computers, with each agent learning something from their
interactions with the environment [6], which can be implemented for both SARL
and MARL using multiple agents. Among the MARL tasks, new algorithms such as
QMIX and its variants have been introduced [15, 11], and for SARL tasks, the
PPO algorithm, along with others, attempts to increase the speed of
convergence [13]. However, one problem seems constant: the majority of updates
are the same importance, only being weighted by the scale of the loss. To fix
this issue, we introduce a weighting scheme that makes larger rewards more
important, such that all the other agents’ updates do not overshadow a larger
loss.

Proximal Policy Optimization (PPO) is a set of policy gradient methods that
repeatedly samples data from the environment and then optimizes a surrogate
objective function [13]. PPO works by calculating a probability ratio as shown
in eq:1, multiplying it by an advantage function $\hat{A}$, then taking the
minimum of their product and a clipped ratio times the advantage function, as
shown in eq:2. The PPO algorithms are based on Trust Region Policy
Optimization (TRPO), minimizing large updates which can cause the algorithm to
make a destructively large policy update [13, 12].

$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}$
(1)

$L^{CLIP}(\theta)=\hat{E}_{t}[min(r_{t}\hat{A}_{t},clip(r_{t},1-\epsilon,1+\epsilon)\hat{A}_{t})]$
(2)

As well as single-agent RL algorithms, we have multi-agent RL algorithms, and
these optimize the agents for the collective reward.

QMIX is a multi-agent reinforcement learning algorithm that performs training
in a central location and executes decentralized. QMIX works by having
individual agents perform actions when they update their network, taking each
agent’s predicted Q value as an input to a mixing network; The mixing network
then predicts the global Q value $Q_{tot}$. QMIX ensures that the argmax of
$Q_{tot}$ is the same as the argmax $Q_{i}$ for each agent i. Each agent only
knows what they can see and will not receive information from the other
agents, instead keeping such information to the updating of the network [11].

Federated learning is a group of algorithms focusing on distributing training
while addressing problems such as statistical challenges, communication
efficiency, client selection and scheduling, security concerns, privacy
concerns, and service pricing. One such statistical challenge is making the
data as heterogeneous as possible, meaning that agents with more different
data are more likely to be selected for training. For communication
efficiency, the idea is that the less communication which needs to be done,
the better; as such, it is common to use only some agents for the training
instead of selecting a subset of them. This selection phase needs to select
all the agents eventually and not only learn from individual agents who may
have more powerful hardware. When the agents learn, it is essential to ensure
that no one can read the data during transmission, as this could lead to a
hacker gaining access to one or more of the servers and not transmitting
sensitive data between machines [17]. Performing Federated learning often
requires a parameter server, which can take in weights from the different
agents to average them and send back the new weights to each agent.
Distributed learning is similar to federated learning, focusing on
distributing the training. However, it may not consider the other factors
crucial to the federated learning algorithms [10]. The standard methods for
distributed learning are to average or sum the gradients of the agents, as in
eq:4 and 3.

####  2.0.1 Baseline Sum

One of the standard algorithms takes the gradients and sums them $\nabla
g_{sum}$.

$\nabla g_{sum}=\sum_{i=0}^{k}\nabla g_{i}$ (3)

####  2.0.2 Baseline Avg

One state-of-the-art algorithm is to average the gradients $\nabla g_{avg}$.
Which takes the gradients, sums the gradients of the agents, and divides the
sum by the number of agents.

$\nabla g_{avg}=\nabla g_{sum}/k$ (4)

As for implementations using these techniques for machine learning, there are
a couple, namely FedAvg and FedProx [7, 5]. FedAvg trains multiple times on
the data and then sends the parameters (p) to the central agent for
averaging[7].

$p_{i}=\sum_{i=0}^{k}optim(\nabla g_{i})$ (5)

$p_{avg}=\frac{1}{n}\sum_{i=0}^{n}p_{i}$ (6)

FedProx is an improvement on FedAvg, allowing for semi-finished updates on
some agents, and still updating in the central location. Semi-finished updates
are essential as some agents may only be able to perform n-k passes on the
data. At the same time, another may be able to finish all n passes on the data
due to different hardware [5].

##  3 Methods

This section discusses the different implementations we tested. We will
discuss the implementation and what was included here, then move over to what
the different algorithms did. The variations include weighting based on either
the reward or the loss, averaging and summing the gradients, and taking the
average and summed gradients plus each agent’s gradient to create a different
gradient for each agent.

###  3.1 State of the art

Amongst the baseline methods, we have the summed gradients (Baseline Sum) and
the averaged gradients (Baseline Avg).

####  3.1.1 Baseline-sum

One of the standard algorithms takes the gradients and sums them $\nabla
g_{sum}$.

$\nabla g_{sum}=\sum_{i=0}^{k}\nabla g_{i}$ (7)

####  3.1.2 Sync-SGD

One state-of-the-art algorithm is to average the gradients $\nabla
g_{avg}$[1].

$\nabla g_{avg}=\nabla g_{sum}/k$ (8)

###  3.2 Our Algorithms

####  3.2.1 R-Weighted gradient

The rationale of a reward-weighted gradient was to learn more from scenarios
that gave a higher reward. We take all rewards ($r$) and find the smallest
one. Given the lowest reward, we add this as an offset ($o$) to each agent’s
reward.

$\\\ o=\min(r)\\\ $ (9)

This means all values are above $0$, which is required for the weighting.

$r_{i}=r_{i}+o$ (10)

We then calculate the total reward ($r_{tot}$) among all of the agents.

$r_{tot}=\sum_{i=0}^{k}r_{i}$ (11)

To calculate the weight $w_{i}$, we take the reward of the specific agent and
divide it by the total reward; we also add a small number $(1/h)$ to it. The
number h is a hyperparameter set close to the number of agents, meaning every
agent gradient will be weighted at a minimum of $(1/h)$. $(1/h)$ is set to a
small number based on the number of agents to set a lower end on the weight
used for calculating the gradient. This means that each agent’s gradient will
at least matter 1/h, while the good gradients could have a weight of 0.9.

$w_{i}=r_{i}/r_{tot}+(1/h)$ (12)

Using this weight, we scale the gradient $\nabla g_{i}$ for each agent with
their weight. The sum of these weights is 2.

$\nabla g=\sum_{i=0}^{k}\nabla g_{i}w_{i}$ (13)

####  3.2.2 L-Weighted gradient

A somewhat different approach that is more similar to prioritized experience
replay is that of weighting based on loss. For the loss-weighted gradient
implementation, we calculate the total loss.

$l_{total}=\sum_{i=0}^{k}l_{i}$ (14)

Using the loss of each agent and dividing it by the total loss gives us the
weight. We also add a small number $h$ to it; this number means every agent’s
gradients are weighted at a minimum of $(1/h)$.

$w_{i}=\frac{l_{i}}{l_{total}}+(1/h)$ (15)

This weight is used to calculate the gradient of each agent.

$\nabla g=\sum_{i=0}^{k}\nabla g_{i}w_{i}$ (16)

The calculated gradients result in the scenarios in which the model predicted
more incorrectly to be the gradients used for most of the update. The update
will focus on predictions from which it gets a larger loss.

####  3.2.3 Actor-avg

Letting the agents update based on their own experience. Allows each agent to
sample differently. Using the average gradients of all agents means that each
agent needs to have the same starting point but with a different environment.
Given that they start with the same weights, different experiences will make
the agents diverge somewhat. To compensate for this divergence, we add the
average gradient to it. This way, the agents aim at reaching the global
optimum and will act differently given that their environments have been
different from the start of the training. The rationale for the actor plus
average algorithm was to create agents or agents which started with the same
hyperparameters but diverged slightly in each iteration. When they diverged,
they should be close enough to each other to converge on a singular point. The
average part of this algorithm is to make the agents’ gradient more
significant in the updates. The main point here is that each actor searches
the parameter space and can push the other agents in a specific direction.
With the idea of the agents converging on a global optimum.

$\nabla g_{agent(i)}=\nabla g_{avg}+\nabla g_{i}$ (17)

####  3.2.4 Actor-sum

A similar approach is Actor-sum, which takes an agents gradient and adds all
the agents’ summed gradients. The rationale of the algorithm is similar to
Actor-avg, though the individual agents’ gradient has a significantly smaller
significance. With the actor’s gradient having a smaller effect on the update
for each agent, this means that the parameter space is searched to a smaller
degree than that of ActorPlusAvg; meaning that the algorithm should be more
stable while still aiming at a faster convergence towards a global optimum.

$\nabla g_{agent(i)}=\nabla g_{sum}+\nabla g_{i}$ (18)

###  3.3 Neural Network

For testing we used three differently sized NNs, a small neural network, a
medium sized network and a large network. The number of parameters vary from
environment to environment, but the small network is approximately 9 000
parameters; the medium sized network is then approximately 45 000 parameters;
lastly the large network is nearly 750 000 parameters.

The smallest network has an input layer ($X$) which is the state size of the
agent, a hidden layer, and an output layer ($Y$) which corresponds to the
action space of the agent. Its hidden layer uses 64 hidden units, meaning
there are $N*64*2$ where N is the number of input elements in $X$ weights and
biases connecting the input and hidden layer, next there are $M*64*2$
parameters from the hidden layer to the output layer where $M$ is the number
of outputs in $Y$.

While the medium network has a bit more layers, with an input layer, 4 hidden
layers and the output layer. As for the largest network has an input layer, 6
hidden layers and the output layer.

##  4 Results and discussion

Given the methods introduced, we tested the efficacy of each of the algorithms
by averaging the rewards over ten runs for each of the three neural networks
of different sizes, reducing the randomness due to the initial neural network
state.

###  4.1 Experimental Setup

Testing the different algorithms required a few different systems. To
distribute the training, we used ray, in which we used a parameter server and
worker agents. For the NN we used torch to train the algorithms, while we used
gym for the RL environments. For the gym environments, we selected
environments similar to [3], changing out the Half-Cheetah for BipedalWalker,
as it is easier to install and is verified to be easily distributable on the
same machine. Each worker trained 4000 timesteps, gathering experience replay,
then backpropagating it, averaging the rewards and losses over these episodes
before sending them back. This is to fill the experience replay buffer before
training, as to include more variety of memories. Each environment required
less than 1000 timesteps before terminating, meaning we ran through a minimum
of 4 episodes before backpropagating. The experiments were run with the NN
models, at least 10 times for each model before averaging them for the graphs.
This lead to a significant decrease in the randomness, which could affect the
performance of the algorithms. The anonymized code is available at
111https://anonymous.4open.science/r/Federated-learning-5CEC/README.md.

The resulting rewards show the methods we introduced for the small NN, the
different sizes, and the difference when using softmax weights.

Figure 1: Average rewards for PPO during training Cartpole Figure 2: Average
rewards for PPO during training BipedalWalker Figure 3: Average rewards for
PPO during training LunarLander Figure 4: Shows the last 50 episodes in the
LunarLander environment with the small NN Figure 5: Shows the last 50 episodes
in the LunarLander environment with the medium NN Figure 6: Shows the last 50
episodes in the LunarLander environment with the large NN Figure 7: Shows the
difference between our weights for R/L-Weighted implementation versus
softmaxed weights

We have an effective implementation: the R-Weighted gradients. R-Weighted
gradient could get issues with the environments being the same, for example, a
self-driving environment in which the road is the same. If the environment is
the same or very similar, then overfitting is very likely. If the agent only
has to turn left to get a high reward, it will always predict the left to be
the best action. An environment where the vehicle has to turn right would then
get a low reward. The L-Weighted gradient has issues with the fact that
information from lower loss experiences may be more valuable, and if the low
loss experiences are not trained on, then it will not learn what they are
offering. This means that both algorithms have a similar issue in that the
weights used to scale the gradients cannot go too low; to counteract this, we
included a hyperparameter that adds a number to the weight. This weight is
supposed to stabilize the gradients, though it also requires testing to find
the correct value as it is a hyperparameter. Testing the different values for
this hyperparameter is to find the more stable, well-performing parameter.
When testing with a 1/16 value, we see that the performance is significantly
worse than when testing with 1/8 in fig:7; this makes sense as the training
used eight agents; since eight agents were used for training, each agent at
least has the same weight as it would during the Baseline-Avg, but could have
a significantly higher value if its reward or loss is high, dependent on the
algorithm used for training.

For our results, the graphs show us the general trend of these algorithms,
with our R-Weighted and L-Weighted algorithms functioning similarly to that of
the Baseline algorithms, as shown in fig:3, 1 and 2. Our Actor-sum and Actor-
avg algorithms do not perform well however, this could be due to the weights
of each agent diverging from the starting point, thereby making the gradients
of one agent useless to the other agents; we therefore only included it in
fig:1. Another exciting find is that FedAvg does worse the more complex the
environment is; this may be due to FedAvg being designed for supervised
learning and being tested on MNIST and Cifar. The reason for this could be
that the difference in the gradients is too large, leading to the gradients
pulling in vastly different directions, or the worse gradients simply telling
the algorithm what not to do when there is a lot more information in what
gives a higher reward.

Results are pretty different when changing the size of the NN, though, as we
can see the difference between each NN in fig:455. The performance differences
for the size of the NN can have multiple reasons, one of which is overfitting.
When training while using the R-Weighted algorithm, there is a risk of it
overfitting towards specific scenarios. If the environment being trained in is
simple, and NN is large, it is likely that having a large reward from one
agent only happens a few times. With the NN only optimizing a couple of high
reward values, the training is more likely to skew towards overfitting those
values initially and the model trying to generalize later. When the model
tries to generalize later, it will take longer to reach the optimal value,
slowing compared to the baseline algorithms.

To more easily understand our results, we can show their average reward
(avgRew) and the average reward from when the first algorithm moving average
reached a threshold (avgEndRew), a reward of 100 for Lunarlander and 200 for
BipedalWalker. Using these scores, we can see which algorithm gets faster
toward the optimum and which does best when converging. If an algorithm gets
up to a higher score faster, we see its avgRew being higher. If an algorithm
gets a higher avgEndRew, then the algorithm gets consistently high rewards in
different randomly initiated environments after converging.

For the BipedalWalker environment, we see that the R-Weighted algorithm gets a
higher avgRew and avgEndRew as shown in tab:3. While for the LunarLander
environments, we find R-Weighted getting the highest rewards, shown in tab:2.

Looking at the episode number at which the algorithms reach the threshold of
either 80, 400 or 200 depending on the gym environment, as shown in tab:4, we
find that for the CartPole environment, our R-Weighted underperforms the
Baseline-sum, as shown in tab:1; When we look at the BipedalWalker and
LunarLander, however, we find that it is significantly better, especially in
the more complex BipedalWalker environment.

The different environments have different complexities, as the LunarLander
environment takes approximately 150 episodes for each agent to converge, and
the BipedalWalker environment takes approximately 1200 episodes for each agent
to converge. This could imply that R-Weighted algorithm speeds up the learning
and R-Weighted functions better for more complex environments.

To show the performance in a different light, we created a set of tables
showing the results. These tables include avgRew, meaning the total average
reward for the environment, avgEndRew, which gives the average reward after
the first algorithm reaches the threshold of the environment. These thresholds
are taken from the gym environment websites and shown in tab 4 in parethesis.
All values are normalized on baseline-sum, and whenever its value is negative,
we add the lowest reward to all algorithms before calculating the performance.
In tab 4 we have some values, which are the length of the training with a + at
the end; this is because they did not reach the threshold in time. When
listing the point at which the algorithm reached the threshold, we use the
moving average with a value of 0.9 here.

Algorithm | avgRew | avgEndRew | percentAvg(end)
---|---|---|---
R-Weighted | 264.47 | 446.24 | 97.69%(99.22)
L-Weighted | 265.21 | 445.99 | 97.87%(99.16)
Baseline-sum | 270.98 | 449.74 | 100%(100)
Baseline-avg | 261.71 | 446.25 | 96.57%(99.22)
FedAvg | 244.70 | 423.02 | 90.30%(94.05)
ActorSum | -289.98 | -285.84 | 0.0%(0.0)
ActorAvg | -346.20 | -346.70 | 0.0%(0.0)
Table 1: CartPole, all values significantly lower than the baseline, which are negative when it is highly possitive is set to 0.0 Algorithm | avgRew | avgEndRew | percentAvg(end)
---|---|---|---
R-Weighted | -52.69 | 84.55 | 101.59%(107.85)
L-Weighted | -55.32 | 81.22 | 98.25%(103.61)
Baseline-sum | -53.94 | 78.39 | 100%(100)
Baseline-avg | -132.84 | -36.45 | 0.0%(0.0)
FedAvg | -121.93 | -30.28 | 13.82%(0.0)
Table 2: LunarLander, all values significantly lower than the baseline, which are negative when it is highly possitive is set to 0.0 Algorithm | avgRew | avgEndRew | percentAvg(end)
---|---|---|---
R-Weighted | 156.95 | 219.49 | 106.32%(105.65)
L-Weighted | 136.62 | 190.55 | 92.54%(91.72)
Baseline-sum | 147.62 | 207.75 | 100%(100)
Baseline-avg | 121.67 | 180.70 | 82.42%(86.97)
FedAvg | -104.71 | -113.42 | 0.0%(0.0)
Table 3: BipedalWalker, all values significantly lower than the baseline, which are negative when it is highly possitive is set to 0.0 Algorithm | LL | CP | BW | percent
---|---|---|---|---
| (80) | (400) | (200) | (LL,CP,BW)
R-Weighted | 143 | 104 | 690 | (104.8, 97.1,
|  |  |  | 120.1)
L-Weighted | 146 | 103 | 1000 | (102.7, 98.0,
|  |  |  | 82.9)
Baseline-sum | 150+ | 101 | 829 | (100, 100,
|  |  |  | 100)
Baseline-avg | 150+ | 103 | 1137 | (100, 98.0,
|  |  |  | 72.9)
FedAvg | 150+ | 113 | 1200+ | (0.0, 89.3,
|  |  |  | 0.0)
Table 4: Shows the episode number at which the moving average score reached
the threshold, this threshold is shown in the parethesis.
LL refers to LunarLander, CP refers to CartPole and BW to BipedalWalker
150+ or 1200+ means they did not reach the number

Our door scenario may make intuitive sense, though the results show a
different story than what was predicted. This is due to the complexity of the
environment, and a more complex implementation of the environment may lead to
different results.

###  4.2 Future Work

Future work includes expanding the environments to test these algorithms on,
trying more complex environments such as the atari suite and the CarRacing
Environment. Testing how the large NN works on very complex environments would
also give us better input if it is simply the complexity of the environment
that gives the worse performance. Checking out memory-based models, such as
having shared memory amongst the agents, while the agents start with different
weights and update on the shared memory themselves.

##  5 Conclusion

We introduced a new distributed algorithm for backpropagation, including
weights for each agent. The weights of each agent’s gradient indicated how
high the reward and loss were compared to the total loss. The new algorithms
were computationally simple and should not add much to the backpropagation
method. R-Weighted allowed us to solve the environment up to 20.14% faster,
improving the training speed.

With the R-Weighted method, it is able to utilize a higher reward to start
converging faster in simpler environments. The L-Weighted method does very
similarly to the Baseline-Sum over all the environments though rarely
outperforming it in any metric. Comparing multiple actors to a single one, we
see a significant increase in the reward it achieves.

We conclude that R-Weighted works better than the baseline algorithm in
complex environments, improving convergence speed more the smaller the network
is. The R-Weighted algorithm allows for faster training in very complex
environments with very small and fast NN.

## References

  * [1] Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. Revisiting Distributed Synchronous SGD.  ICLR (4 2016).
  * [2] Chen, Y., Zhang, X., Zhang, K., Wang, M., and Zhu, X. Byzantine-Robust Online and Offline Distributed Reinforcement Learning.
  * [3] Fan, F. X., Ma, Y., Dai, Z., Jing, W., Tan, C., Kian, B., and Low, H. Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee.  Advances in Neural Information Processing Systems 34 (2021), 1007–1021.
  * [4] Hall, L. O., Bowyer, K. W., Kegelmeyer, W. P., Moore, T. E., and Chao, C.-M. Distributed Learning on Very Large Data Sets.  Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. (2000).
  * [5] Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated Optimization in Heterogeneous Networks.
  * [6] Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., and Stoica, I. RLlib: Abstractions for Distributed Reinforcement Learning, 7 2018.
  * [7] McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication-Efficient Learning of Deep Networks from Decentralized Data.  Proceedings of Machine Learning Research 54 (4 2017), 1273–1282.
  * [8] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with Deep Reinforcement Learning.  NIPS Deep Learning Workshop (2013).
  * [9] OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dȩbiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., Pinto, H. P. d. O., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. Dota 2 with Large Scale Deep Reinforcement Learning.
  * [10] Peteiro-Barral, D., and Guijarro-Berdiñas, B. A survey of methods for distributed machine learning.  Progress in Artificial Intelligence 2012 2:1 2, 1 (11 2012), 1–11.
  * [11] Rashid, T., Samvelyan, M., Schroeder De Witt, C., Farquhar, G., Foerster, J., and Whiteson, S. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning.  Proceedings of the 35th International Conference on Machine Learning, PMLR 80 (2018), 4295–4304.
  * [12] Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust Region Policy Optimization, 6 2015.
  * [13] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Openai, O. K. Proximal Policy Optimization Algorithms.
  * [14] Sunehag DeepMind, P., Lever DeepMind, G., Gruslys DeepMind, A., Marian Czarnecki DeepMind, W., Zambaldi DeepMind, V., Jaderberg DeepMind, M., Lanctot DeepMind, M., Sonnerat DeepMind, N., Leibo DeepMind, J. Z., Tuyls, K., and Graepel DeepMind, T. (Value Decomposition) Value-Decomposition Networks For Cooperative Multi-Agent Learning.  AAMAS (2018).
  * [15] Terry, J. K. B., Benjamin Grammel, Nathaniel Jayakumar, Mario Hari, Ananth Sullivan, Ryan Santos, Luis Perez, Rodrigo Horsch, Caroline Dieffendahl, Clemens Williams, and Niall L Lokesh, Yashas Ravi, P. PettingZoo: Gym for Multi-Agent Reinforcement Learning.  Advances in Neural Information Processing Systems 34 (2021), 15032–15043.
  * [16] Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gulcehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in StarCraft II using multi-agent reinforcement learning.  Nature 2019 575:7782 575, 7782 (10 2019), 350–354.
  * [17] Wahab, O. A., Mourad, A., Otrok, H., and Taleb, T. Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems.  IEEE Communications Surveys and Tutorials 23, 2 (4 2021), 1342–1397.
  * [18] Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F., Gilpin, L., Khandelwal, P., Kompella, V., Lin, H. C., MacAlpine, P., Oller, D., Seno, T., Sherstan, C., Thomure, M. D., Aghabozorgi, H., Barrett, L., Douglas, R., Whitehead, D., Dürr, P., Stone, P., Spranger, M., and Kitano, H. Outracing champion Gran Turismo drivers with deep reinforcement learning.  Nature 2022 602:7896 602, 7896 (2 2022), 223–228.