Evolving Multi-Objective Neural Network Controllers for Robot Swarms
Many swarm robotics tasks consist of multiple conflicting objectives. This
research proposes a multi-objective evolutionary neural network approach to
developing controllers for swarms of robots. The swarm robot controllers are
trained in a low-fidelity Python simulator and then tested in a high-fidelity
simulated environment using Webots. Simulations are then conducted to test the
scalability of the evolved multi-objective robot controllers to environments
with a larger number of robots. The results presented demonstrate that the
proposed approach can effectively control each of the robots. The robot swarm
exhibits different behaviours as the weighting for each objective is adjusted.
The results also confirm that multi-objective neural network controllers
evolved in a low-fidelity simulator can be transferred to high-fidelity
simulated environments and that the controllers can scale to environments with
a larger number of robots without further retraining needed.
Karl Mason, Sabine Hauert
# Evolving Multi-Objective Neural Network Controllers for Robot Swarms

###### Abstract

Many swarm robotics tasks consist of multiple conflicting objectives. This research proposes a multi-objective evolutionary neural network approach to developing controllers for swarms of robots. The swarm robot controllers are trained in a low-fidelity Python simulator and then tested in a high-fidelity simulated environment using Webots. Simulations are then conducted to test the scalability of the evolved multi-objective robot controllers to environments with a larger number of robots. The results presented demonstrate that the proposed approach can effectively control each of the robots. The robot swarm exhibits different behaviours as the weighting for each objective is adjusted. The results also confirm that multi-objective neural network controllers evolved in a low-fidelity simulator can be transferred to high-fidelity simulated environments and that the controllers can scale to environments with a larger number of robots without further retraining needed.

Swarm Robotics Evolutionary Robotics Neural Networks Natural Evolution Strategies Evolutionary Algorithms Multi-Objective

## 1 Introduction

Many robotics tasks consist of multiple objectives. For example, manufacturing robots must accomplish tasks quickly and accurately [3]. These are conflicting objectives. Similarly, multiple objectives are also present in swarm robotics tasks. Minimizing both movement time and energy costs is a primary example of this [14].

Current approaches to addressing these multi-objective robotic tasks often involve applying a multi-objective optimisation algorithm to optimise the actions of the robot for different objectives, e.g. for path planning [17, 18]. These approaches work well for determining the optimum paths under multiple objectives for a specific environment configuration. If the environment changes, e.g. if more robots are added to the environment, these optimisation based techniques must be reapplied to determine the new set of optimal paths under the new conditions. This requires additional computational time.

An alternative solution to this multi-objective robotics problem is presented in this paper using Neuroevolution, or evolutionary neural networks [19]. Neuroevolution involves applying evolutionary algorithms to train the parameters of neural networks to solve a machine learning task, e.g. play Atari games [16], energy forecasting [15], CPU utilization prediction [15] and economic dispatch [14]. A multi-objective evolutionary neural network approach is proposed for developing swarm robot controllers. This proposed approach to developing multi-objective swarm robot controllers has the benefit of evolving a control policy that can still effectively control the robots even if the environmental conditions change and will not require the computational cost associated with retraining the robot controller.

The research presented in this paper makes the following contributions:
1. To propose a Natural Evolution Strategies based evolutionary multi-object neural network approach for swarm robotics.
2. To investigate the transferability of multi-objective neural network controllers trained in a low-fidelity simulator to a high-fidelity simulator.
3. To determine if controllers trained in an environment with a small number of robots, can scale to larger robot swarms.

The structure of the paper is as follows. Sections 2 will give an overview of the background literature relating to this research. Section 3 will outline the experimental methods, including simulator design, application of proposed approach and experiments conducted. Section 4 will present the results of the simulations conducted. Finally, Section 6 will present a conclusion for the research described in the paper.

## 2 Background

### Swarm Robotics

Robot swarms typically have characteristics including: acting autonomously within their environment, homogeneity across robots, and utilizing local information only [23]. Robot swarms have many practical applications [23], e.g. search and rescue [1] and warehouse operations [16].

There have been a number of publications that have explored the application of machine learning to robot swarms, e.g. in 2020 Tolstaya et al. applied a graph neural network to learn to control robots in a swarm [15]. A 2021 paper by Dorigo et al. mentions the application of machine learning to robot swarms as one of the future directions of research in swarm robotics [17].

Research published recently in the literature has recognised that the task of robot path planning often consists of multiple objectives [14], e.g. to minimize the time taken to locate a target, to minimise chance of collisions, to maximize energy efficiency, etc. Each of these objectives are important and should be considered when determining the behaviour of the robot. This motivates the research presented in this paper to develop robot controllers that can take preferences for each objective as input to influence the behaviour of robot swarms.

Multiple studies have been published in the literature that apply evolutionary methods to swarm robotics. Birattari et al. provide a manifesto for automatic off-line design in the area of robot swarms [18]. This paper discusses existing applications of evolutionary neural networks to robot controllers. Floreano et al. provide a comprehensive account of applications of evolutionary computing to robotics [19]. Evolutionary methods have been successfully applied in swarm robotics tasks. Hauert et al. applied evolutionary neural networks to control individuals in a swarm of simulated Micro Air Vehicles (MAVs) [15]. Evolutionary methods have also been applied to swarm robotics by evolving behaviour trees [11], Jones et al.(2018). Recent studies have applied MAP-Elites for swarm robotics [10]. These studies demonstrate the effectiveness of evolutionary methods for swarm robotics tasks. The research outlined in the paper builds on these previous studies by extending evolutionary swarm robot controllers to multi-objective tasks using Natural Evolution Strategies.

There have been a number of studies published in the literature that explore multi-objective control in robot swarms. Mai et al. developed a Multi-Objective collective search strategy for robot swarms based on the Particle Swarm Optimisation algorithm [16]. Miao et al. proposed a Multi-objective region reaching controller for a swarm of robots [16]. These studies demonstrate the utility of multi-objective control and path planning for robot swarms. These studies do not consider the use of multi-objective neural network controllers for robot swarm control, as this research paper presents.

### Evolutionary Neural Networks

Neural networks are machine learning models that take inspiration from the brain. The field of Evolutionary Neural Networks (or Neuroevolution) utilize evolutionary algorithms and principals to train the parameters of neural networks [16]. Target network outputs are not required when evolving neural networks, only a fitness function. Neuroevolution has been shown to be a competitive approach when compared to reinforcement learning algorithms [23], Mason and Grijalva(2021).

There are many Neuroevolution algorithms that evolve the weights and architecture of the neural network, e.g. NeuroEvolution of Augmenting Topologies (NEAT) [23] and hyperNEAT [15]. Many studies implement methods such as evolutionary strategies to evolve only the weights
of the network [Chen et al.(2019), Pourchot et al.(2018), Mason and Grijalva(2021)]. Covariance Matrix Adaption Evolutionary Strategy (CMA-ES) [Hansen et al.(2003)] and Natural Evolution Strategies (NES) [Wierstra et al.(2014)] are two well known evolutionary strategies. This research will use a variant of NES called Exponential Natural Evolution Strategies (xNES) [Glasmachers et al.(2010)] when evolving MO-NNs. It was selected as it is an effective optimisation algorithm.

Algorithm 1 describes the xNES algorithm. The algorithm samples \(\lambda\) normally distributed solutions \(z_{i}\). These are used to calculate \(\lambda\) solutions \(x_{i}=\mu+\sigma\textbf{B}z_{i}\), based on the center of the search distribution \(\mu\), the normalized covariance factor \(\textbf{B}=\textbf{A}/\sigma\), and \(\sigma\) is the scalar step size. When the algorithm begins, **A** is initialized as the identity matrix and \(\sqrt[4]{|det(\textbf{A})|}\), where \(d\) is the number of dimensions.

After solutions are sampled, the gradients of the objective function are calculated with respect to \(\delta,M,\sigma\) and **B**. Where M is a \(d\times d\) exponential map used to represent the covariance matrix **C**, where **C = AA\({}^{T}\)** and \(\delta\) is the change in **C**.

## 3 Experimental Methods

### Evolving Multi-Objective Neural Networks

This research consists of evolving neural networks for a multi-objective swarm robotics task. The pseudocode in Algorithm 2 illustrates the MO-NN training process.

``` Initialize \(\textbf{A}\), \(\textbf{B}\), \(\textbf{C}\), \(\textbf{
located at the front, back and on each side. Each robot can rotate \(\pm 45\deg\) and can only move forward up to a maximum velocity of \(2m/sec\). This simulator design is based on the DOTS swarm robot testbed [Jones et al.(2022)].

A high-fidelity simulator was also developed using Webots [Webots([n. d.]), Michel(2004)]. This was used to test the evolved controllers. The parameters of the Webots simulator are the same as the Python simulator. Figure 1 illustrates the simulated robot and arena.

Both simulators model non-holonomic robot drive. The low-fidelity simulator does not simulate any physics. Collision detection is implemented in the low-fidelity simulator to prevent a robot from colliding with another robot or the environment boundary. Physics is simulated in the high-fidelity Webots simulator. Collisions are not detected/prevented in the high-fidelity simulator.

Figure 1: Simulated robot (a) and arena (b).
### Multi-Objective Neural Network Robot Controllers

The implementation of the neural network controller as a robot controller is illustrated in Figure 2. The robot senses its environment using 4 rangefinder sensors. These measure the distance between the sensor and the nearest object. These values are normalized and passed to the network as input. In addition to sensor measurements, two additional inputs are passed into the network representing the weightings assigned to each of the two objectives \([w_{1},w_{2}]\). The network then does a forward pass and gives two outputs, i.e. robot commands. These commands are the rotation angle \([-45\deg,45\deg]\) and the forward velocity \([0m/sec,2m/sec]\). The robot will move to a new position based on these commands.

Figure 3 illustrates the training process of the MO-NNs for the robot swarm. Homogeneity is a key property of robot swarms, therefore the same network parameters are assigned to each NN controller for all robots in the swarm during each simulation.

The quality of each set of NN parameters \(x_{i}\) is measured using the fitness function outlined in Equation 1.

\[networkFit(NN(x_{i}))=\sum_{\Delta w=0}^{1}\sum_{t=1}^{t_{Max}}currentFit_{w, t}(NN(x_{i})) \tag{1}\]

where \(\Delta w\) is the change in objective weighting applied to \(w_{1}\) and \(w_{2}\). These are updated as \(w_{1}=1-\Delta w\) and \(w_{2}=\Delta w\)\(\therefore w_{1}+w_{2}=1\)\(\forall\)\(\Delta w\). The fitness at the current time-step for the current objective weights is calculated using Equation 2.

\[currentFit_{w,t}(NN(x_{i}))= w_{1}\times-1\times Obj1_{w,t}(NN(x_{i})) \tag{2}\] \[+w_{2}\times Obj2_{w,t}(NN(x_{i}))\]

where \(w_{1}\) is the weighting of objective 1 (\(Obj1\)) and \(w_{2}\) is the weighting of objective 2 (\(Obj2\)). Objective 1 is to minimize the distance between robots and the origin (center of the arena), calculated using Equation 3. Objective 2 is to maximize the velocity of the robots, calculated using Equation 4. These two objectives were chosen as they are in direct conflict with one another. It should be noted here that \(Obj1\) is multiplied by \(-1\) as the overall optimisation problem is framed as a maximization problem.

\[Obj1_{w,t}(NN(x_{i}))=\sum_{r=1}^{numRobots}[|position_{x,r}|+|position_{y,r}|] \tag{3}\]

Figure 2: Neural network controller.
where \(position_{x,r}\) and \(position_{y,r}\) represent the x and y positions of robot r, respectively.

\[Obj2_{w,t}(NN(x_{i}))=\sum_{r=1}^{numRobots}[|velocity_{x,r}|+|velocity_{y,r}|] \tag{4}\]

where \(velocity_{x,r}\) and \(velocity_{y,r}\) represent the x and y velocities of robot r, respectively.

During training, each set of NN parameters \(x_{i}\) is evaluated for 30 seconds of simulated time (\(t_{Max}=30sec\)) for each increment of \(\Delta w\). The simulator time step is 1 second. The value of \(\Delta w\) is incremented between 0 and 1 in increments of 0.5. This is to ensure that each network is evaluated based on its ability to control the robot such that each objective is optimised with maximum weighting, i.e. when \(w_{1}=1,w_{2}=0\) and \(w_{1}=0,w_{2}=1\), and also its ability to control the robot such that each objective is weighted equally, i.e. when \(w_{1}=0.5,w_{2}=0.5\). This number of \(\Delta w\) increments was selected to minimize training time. A smaller \(\Delta w\) increment can be implemented during training but would increase training time. Note, a smaller \(\Delta w\) increment can be applied when evaluating the MO-NN, irrespective of the increment size during training.

The NN architecture implemented in this research consisted of 6 input nodes (4 sensor inputs and 2 objective weights), 1 hidden layer with 5 nodes, and an outputs layer with 2 nodes (1 for rotation, 1 for forward movement). The network was evolved over 20,000 evaluations.

### Experimental Setup

1. **Evolving Neural Network Controller**. This experiment consists of evolving the multi-objective neural network controller in the low-fidelity python robot simulator. The MO-NN is evolved in the low-fidelity python simulator with the Objective (obj) preferences: MO-NN 1: Obj 1 - Maximize velocity. Obj 2 - Minimize distance to origin (center of arena).
2. **Deployment to High-Fidelity Webots Simulator** The next experiment is to test the performance of the best performing network evolved in the low-fidelity python simulator by deploying the network to control robots in a more realistic high-fidelity Webots simulator.

Figure 3: Evolving neural network swarm robot controller.
3. **Evaluating Evolved Controller on Larger Swarm Sizes** The final experiment is to determine the scalability of the evolved MO-NN to a larger number of robots, specifically 5 and 10 robots.

## 4 Results

### Evolving Neural Network Controller in Low-Fidelity Python Simulator

The evolved MO-NNs were capable of controlling the robot swarm in the desired manner in the low-fidelity python simulator. Figure 4 illustrate the trajectories for the evolved MO-NN.

The trajectories in Figure 4 refer to the training objectives: Objective 1 - Maximize velocity and Objective 2 - Minimize distance to origin (center of arena). It is clear from Figure 4 that the robot trajectories are significantly different when a high weighting is given to maximizing velocity compared to when a high weighting is given to minimizing distance to the origin. When a high weighting is given to maximizing velocity, the robots move in large circles around the arena. When a high weighting is given to minimizing distance to the origin, the robots move to the center of the arena and stay there. When an equal weighting is given to both objectives, the robots traverse the arena close to the center in smaller circles.

Figure 5 illustrates how the each of the objective function evaluations of the behaviour of the evolved neural network change as the objective weighting changes. When a maximum weighting is given to minimizing the distance to the origin objective, both the distance and velocity objective scores are lowest. Conversely when maximizing velocity, the distance to the origin also increases as the robots are moving with higher velocity around their environment. This graph illustrates how different behaviour can be observed from the swarm of robots using a single neural network by simply modifying the weighting for each objective function. No retraining is required for different objectives or different robots. This is the key advantage of the proposed MO-NN approach.

### Testing Evolved Controllers in High-Fidelity Webots Simulator

After training in a low-fidelity Python simulator, the evolved MO-NN robot controller was then deployed to control robots implemented in the high-fidelity Webots simulator. The motivation for doing this was to test whether the behaviours of the evolved controllers have the potential to translate to simulators with more realistic physics without the need to adapt the motor commands.

Figure 4: Low-fidelity simulator robot trajectories: Maximize velocity objective (left). Equal objective preference (middle). Minimize distance to origin (right).

Figure 5: Pareto Front for evolving multi-objective neural network controller.
Two simulations were conducted. In the first simulation, the robots' objective preference was to minimize distance to the origin, i.e., all robots were passed an objective weighting of 1 for the minimize distance to the origin objective, and 0 for the maximize velocity objective. In the second simulation, the robots' objective preference was to maximize velocity. It was observed that the trained MO-NNs exhibited similar behaviour when tested in the Webots environment.

When minimizing distance to the origin, the robots moved much slower. The robots gradually made their way to the center of the arena and do not adjust their position much thereafter. Each robot continually rotates in order to sense more of its environment.

When maximizing velocity, the robots move continuously around the arena and do not stop at the center. There is a greater risk of collisions when moving with higher velocity. In order to reduce the severity of the collisions, the maximum wheel rotational speed was reduced from 80 radians/sec to 60 radians/sec. Without this speed reduction, robots collided with one another, overturned and remained immobilized. Note, this velocity clamping was applied in both simulations reported.

### Larger Robot Swarms in High-Fidelity Webots Simulator

The next set of simulations conducted was to establish if the evolved MO-NN can scale to larger swarm sizes without retraining. The motivation for this was to test if the controllers evolved in an environment with fewer robots are robust enough to scale to environments with more robots and therefore greater chance of collisions. In order to test this, simulations were conducted for robot swarm sizes of 5 and 10 robots, using the evolved MO-NN trained in a swarm size of 3.

It was observed that the robot swarm gather at the center of the arena after 60 seconds in all robot swarm sizes when minimizing distance to the origin. Different behaviour was observed when maximizing velocity. After 60 seconds, the robots are dispersed around the arena as they are traversing the arena with higher velocity.

Figure 6 illustrates the spread in the distance to the origin and the velocity averaged over 10 robots at each second in the 10 robot swarm for 60 seconds. This graph clearly illustrates how the robots travel with significantly higher velocity when the weighting on the velocity objective is at its maximum, compared to when a maximum weighting is applied to the distance to origin objective. This difference is statistically significant when compared using the two tailed Wilcoxon signed rank test, with significance level \(\alpha=1\%\). Similarly, when comparing the distance to the origin at each second, it can be seen that the distance is significantly lower when maximizing the weighting for the distance objective compared to maximizing the velocity objective, with a significance level \(\alpha=1\%\).

Figure 6: Distribution of distance to origin (left) and velocity (right) when varying objective weighting.
Figure 7 presents a heatmap of the positions of all robots in the 10 robot swarm over 60 seconds when maximizing velocity (a) and minimizing distance to the origin (b). The figure on the left illustrates how the robots are more dispersed throughout the map when maximizing velocity. When minimizing distance to the origin, the robot positions are concentrated heavily in the center of the map. This is as expected.

Figure 8 presents the robot velocity (left) and distance to origin (right) over 60 seconds of simulation time when maximizing the weighting to each objective. Under both objective preferences, the average velocity of the swarm is slow initially. When minimizing the distance to the origin, there is a large spike in velocity at time step 8. This corresponds in a significant reduction in the average distance to the origin. There are multiple smaller spikes in average velocity which further reduce the swarm distance to the origin. When maximizing velocity, there is a similar large increase in velocity early in the simulation. The average velocity of the robots then stabilize with some oscillations thereafter.

## 5 Conclusion

This research proposed an evolutionary multi-objective (MO) neural network (NN) for robot swarm control. The MO-NN was evolved using a low-fidelity Python simulator in an environment with 3 robots. The controller was then tested in a high-fidelity simulated environment developed using Webots. The MO-NN controller was then evaluated for a larger numbers of robots.

The primary findings of this research are:

Figure 8: 10 robot swarm average velocity (left) and distance to origin (right) over 60 seconds.

Figure 7: Position heatmaps of 10 robots over 60 seconds when maximizing velocity (a) and minimizing distance to origin (b).