# marl-jax: Multi-agent Reinforcement Leaning Framework

Kinal Mehta1<EMAIL_ADDRESS>
Anuj Mahajan2<EMAIL_ADDRESS>
Pawan Kumar1<EMAIL_ADDRESS>
1International Institute of Information Technology Hyderabad, India
2University of Oxford, Oxford, UK

###### Abstract

Recent advances in Reinforcement Learning (RL) have led to many exciting
applications. These advancements have been driven by improvements in both
algorithms and engineering, which have resulted in faster training of RL
agents. We present marl-jax, a multi-agent reinforcement learning software
package for training and evaluating social generalization of the agents. The
package is designed for training a population of agents in multi-agent
environments and evaluating their ability to generalize to diverse background
agents. It is built on top of DeepMind’s JAX ecosystem Babuschkin et al.
(2020) and leverages the RL ecosystem developed by DeepMind. Our framework
marl-jax is capable of working in cooperative and competitive, simultaneous-
acting environments with multiple agents. The package offers an intuitive and
user-friendly command-line interface for training a population and evaluating
its generalization capabilities. In conclusion, marl-jax provides a valuable
resource for researchers interested in exploring social generalization in the
context of MARL. The open-source code for marl-jax is available at:
https://github.com/kinalmehta/marl-jax

Keywords: Multi-agent Reinforcement Learning, Zero-Shot Generalization,
General Sum Games

##  1 Introduction

Multi-agent reinforcement learning (MARL) is an important framework for
training autonomous agents that operate in dynamic environments with multiple
learning agents. Many potential real-world applications require the trained
agents to cooperate with humans or agents not seen during training. That is,
they should be able to zero-shot generalize to novel social partners. Most of
the existing MARL frameworks Samvelyan et al. (2019); Papoudakis et al.
(2021); Sarkar et al. (2022); Zhou et al. (2023); Hu et al. (2022) are either
designed for cooperative MARL research or naively extend existing single-agent
RL frameworks to work with multiple agents.

On the contrary, marl-jax is designed specifically for multi-agent research
and facilitate the training and assessment of the generalization capacities of
multi-agent reinforcement learning (MARL) algorithms when facing new social
partners. We utilize the functionalities of JAX Bradbury et al. (2018)
including autograd, vectorization through _vmap_ , parallel processing through
_pmap_ , and compilation through _jit_ , resulting in highly optimized
training for multiple agents.

##  2 Related Works

The RL community has developed several frameworks targeting various aspects
such as implementation simplicity, ease of adaptation and scaling deep RL
agents. In marl-jax, we focus on ease of experimentation and adaption for
training a population of agents in multi-agent environments.

A number of libraries that concentrate on single-agent reinforcement learning
have been created, such as stable-baselines3 Raffin et al. (2021), dopamine
Castro et al. (2018), acme Hoffman et al. (2022), RLlib Liang et al. (2018),
and CleanRL Huang et al. (2022). These libraries prioritize features like
modularity by providing useful abstractions, ease of use by requiring minimal
code to get started, distributed training and ease of comprehension and
reproducibility. Other libraries such as Reverb Cassirer et al. (2021), rlax
Babuschkin et al. (2020), and launchpad Yang et al. (2021) concentrate on
specific components of an RL system.

For multi-agent reinforcement learning, several libraries have been developed,
including PyMARL Samvelyan et al. (2019), epymarl Papoudakis et al. (2021),
RLlib Liang et al. (2018), Mava Pretorius et al. (2021), and PantheonRL Sarkar
et al. (2022). RLlib and PantheonRL enhance existing single-agent RL
algorithms to enable multi-agent training, while Mava, PyMARL, and epymarl are
specifically designed for MARL but only support cooperative environments.

The advancements of Reinforcement Learning (RL) algorithms have been greatly
influenced by libraries providing a range of environments. Single agent RL has
been aided by libraries such as OpenAI Gym Brockman et al. (2016) and dm-env
Muldal et al. (2019), which established the framework for environment
interactions. Multi-agent RL has been supported by SMAC Samvelyan et al.
(2019) and PettingZoo Terry et al. (2021). Recently, DeepMind has contributed
to the field by open-sourcing MeltingPot Agapiou et al. (2022), a library for
evaluating multi-agent generalization to new social partners at scale.
Similarly, efforts for measuring generalization in cooperative multi-agent
settings Mahajan et al. (2022) are being supported by libraries like Ellis et
al. (2022).

Several recent works Hoffman et al. (2022); Huang et al. (2022); Babuschkin et
al. (2020); Pretorius et al. (2021) have begun utilizing JAX Bradbury et al.
(2018) due to its various benefits. These benefits include auto-vectorization,
just-in-time compilation, and easy multi-GPU scaling.

##  3 marl-jax

Inspired by Acme Hoffman et al. (2022), we share a lot of design philosophies
with it. Reverb Cassirer et al. (2021) is used as a data-store server for the
replay buffer. Launchpad Yang et al. (2021) is used for distributed computing.
We use JAX Bradbury et al. (2018) as the numerical computation backend for
neural networks. We use _dm-env_ API as our environment interaction API and
extend it for multi-agent environments.

###  3.1 System Architecture

We implement four different training architectures

####  3.1.1 Single Threaded

Figure 1: Single-threaded RL training pipeline

Figure 1 illustrates the sequential flow of operations in a single-threaded RL
training process. At each step, the agent receives an observation from the
environment, selects an action based on its policy, interacts with the
environment, and receives a reward. The observation, action, reward, and next
observation are collected to form a batch and then used to update agent’s
policy and value function using gradient descent. This process continues
iteratively until the desired convergence or a specified number of iterations
is reached. Algorithm 1 describes the pseudocode for training a policy in
singe-threaded manner.

Algorithm 1 Single-threaded RL training

1: repeat

2: params = initialize params for the policy and value function

3: sequence = [ ]

4: timestep = environment.reset()

5: repeat

6: actions = predict_actions(params, timestep.observation)

7: new_timestep = environment.step(actions)

8: sequence.append((timestep, actions))

9: timestep = new_timestep

10: until not timestep.last() or len(sequence) $<$ MAX_BATCH_SIZE

11: params, logs = sgd_step(params, sequence)

12: logger.write(logs)

13: until actor_steps $<$ MAX_STEPS

####  3.1.2 Synchronous Distributed

Figure 2: RL training with vectorized environment

The synchronous distributed architecture builds upon the single-threaded
architecture by utilizing multiple environment instances running in parallel
processes to collect a batch of experiences simultaneously. Each environment
synchronously interacts with a common policy, generating sequences of states,
actions, rewards, and next observations. This batch of sequences is used to
update the policy and value function weights through gradient descent. By
leveraging parallelization, the synchronous distributed architecture enables
more efficient data collection and faster updates, leading to accelerated
training and improved convergence in reinforcement learning. Fig.2 illustrates
how synchronous parallelization is achieved by running each environment
instance in a separate process. The pseudocode is similar to that described in
algorithm1, with the main difference being that each interaction with the
environment results in a batch of data collected from environments across all
processes.

####  3.1.3 IMPALA-style Asynchronous Distributed

Figure 3: Asynchronous distributed RL training pipeline

In the IMPALA-style asynchronous distributed training architecture, multiple
actors run in parallel and interact with their respective environments
asynchronously. Each actor collects trajectories of experience by executing
its own copy of the current policy. The three main components running in
parallel as separate processes are described below

  * •

Environment Loop: The environment loop process interacts with the environment
using the available policy and adds the collected experience to the replay
buffer. Multiple parallel environment loop processes are run, each with its
own copy of the environment and policy parameters. We use CPU inference for
action selection on each process. To keep the policy parameters in sync with
the learner process, the parameters are periodically fetched from the learner
process. The action selection step is optimized using _vamp_ auto-
vectorization to select the action for all agents in the environment.
Algorithm 2 shows the pseudocode for this process.

  * •

Learner: The actual policy learning happens in this process. The learner
fetches experience from the replay buffer and performs the optimization step
on policy and value function parameters. We use _pmap_ to auto-scale the
optimization step to multiple GPUs and _vmap_ based auto-vectorization to
perform the optimization step for all agents in parallel. Algorithm 3 shows
the pseudocode for this process.

  * •

Replay Buffer: A separate process with reverb Cassirer et al. (2021) server is
used as a replay buffer. All the actors add experience to this server, and the
learner process samples experience from the server to optimize for policy and
value function parameters.

Figure 3 illustrates the how data flows between different components enabling
asynchronous experience collection and training of the RL agent.

Algorithm 2 Asynchronous Environment loop

1: repeat

2: params = fetch latest params from learner

3: episode = [ ]

4: timestep = environment.reset()

5: repeat

6: actions = predict_actions(params, timestep.observation)

7: new_timestep = environment.step(actions)

8: episode.append((timestep, actions))

9: timestep = new_timestep

10: until not timestep.last()

11: replay_buffer.write(episode)

12: logger.write(logs)

13: until actor_steps $<$ MAX_STEPS

Algorithm 3 Asynchronous Learner

1: repeat

2: batch = replay_buffer.sample()

3: new_params, logs = sgd_step(params, batch)

4: logger.write(logs)

5: until actor_steps $<$ MAX_STEPS

####  3.1.4 Sebulba: Asynchronous Distributed with Inference Server

Figure 4: Asynchronous distributed with Inference Server

Inspired by Sebulba architecture from Podracer Hessel et al. (2021) and seed-
rl Espeholt et al. (2019), this architecture uses a common inference server in
the asynchronous distributed architecture. When a common inference server is
used in the asynchronous distributed architecture for RL training, the
architecture is further enhanced to centralize the inference process. In this
setup, multiple actors interact with their respective environments
asynchronously, collecting trajectories of experience as before. However,
instead of each actor performing its own inference, they send their collected
experiences to a common inference server which usually has access to hardware-
accelerator such as GPU.

By utilizing a common inference server, several advantages can be achieved.
First, it reduces the computational load on the individual actors, as they no
longer need to perform their own inference. This enables the actors to focus
on data collection, resulting in more efficient and faster interaction with
the environment. Second, the use of a shared policy network ensures that all
actors are making decisions based on the same set of parameters. This improves
the consistency and stability of the training process, as it prevents any
discrepancies that may arise from differences in local copies of the policy
network.

Figure 4 shows the block-diagram and data-flow in asynchronous architecture
with inference server. The pseudocode for learner will be exactly the same as
the Algorithm 3 and for actor it is same as Algorithm 2 where the
_predict_actions_ functions will be a call to the inference server instead of
local policy network.

###  3.2 Supported Environments

We support two multi-agent environment suits, which consist of simultaneous
acting homogeneous agents.

####  3.2.1 Overcooked

The Overcooked environment Carroll et al. (2019) is a popular benchmark in the
field of Multi-Agent Reinforcement Learning (MARL) that simulates a
cooperative cooking scenario based on the popular game _Overcooked_. It
provides a challenging and interactive environment where multiple agents
collaborate to prepare dishes in a virtual kitchen.

In Overcooked, the goal is to efficiently work together as a team to prepare
and serve a variety of meals. The agents control different characters within
the kitchen and must coordinate their actions to complete tasks such as
chopping ingredients, cooking, and delivering finished dishes to customers.
Collaboration and coordination are essential to maximize efficiency and
achieve high scores.

The environment features various elements that add complexity to the task. For
example, the kitchen layout may include obstacles that require agents to
navigate around, limited resources like cutting boards and stoves that need to
be shared, and time-sensitive customer orders that must be fulfilled promptly.
Additionally, agents need to strategize and communicate effectively to
optimize their actions and avoid potential bottlenecks or collisions.

Overcooked is designed to test the ability of MARL algorithms to solve
cooperative tasks in dynamic and complex environments. It challenges agents to
exhibit skills such as coordination, planning, communication, and adaptive
decision-making.

####  3.2.2 Melting Pot

Melting Pot Agapiou et al. (2022) suite designed with the objective of
evaluating generalization to novel situations and coplayers. The Melting Pot
2.0 suite consists of $50$ different environments and over $256$ unique test
scenarios to evaluate the trained population of agents on broad range of
topics such as social dilemmas, task partioning, resource sharing, etc

Melting Pot evaluation methodology is captured by the following equation:

Substrate + Background Population = Scenario

  * •

Substrate: The term ”substrate” refers to the static or physical aspects of
the environment in a simulation. It encompasses elements such as the layout of
the map, the placement of objects, the rules governing their movement, and the
physics involved. In essence, the substrate defines the stationary or
unchanging components of the environment’s dynamics. It sets the foundation
and structure upon which other dynamic elements and interactions can take
place. By defining the substrate, the simulation establishes the framework for
how the environment behaves and provides a stable backdrop against which other
entities and events can unfold.

  * •

Background Population: The term ”background population” refers to a group of
simulated entities within a simulation that have their own agency or ability
to take actions and make decisions. In other words, these entities are not
passive or static; they actively participate in the simulation and contribute
to its dynamics. They can interact with other entities, respond to stimuli or
events, and potentially influence the overall behavior and outcomes of the
simulation. The background population adds an element of realism and
complexity to the simulation, making it more dynamic and reflective of real-
world scenarios.

  * •

Scenario: In the context of simulation or modeling, a scenario is created by
combining the substrate and the background population. The substrate refers to
the static or physical part of the environment, such as the layout, objects,
and physics rules. On the other hand, the background population consists of
simulated entities with agency, meaning they can take actions and make
decisions within the simulation.

By integrating the substrate and the background population, a scenario is
formed that represents a specific setting or situation within the simulation.
The substrate provides the foundation, defining the physical attributes and
constraints of the environment. This includes factors like the terrain,
structures, objects, and their spatial arrangement. The substrate sets the
stage for interactions and events to occur.

The background population adds a dynamic aspect to the scenario. These
simulated entities have their own behaviors, goals, and decision-making
processes. They can interact with each other, respond to stimuli or events in
the environment, and potentially influence the overall dynamics of the
scenario. The actions and interactions of the background population create a
realistic and evolving simulation environment.

Together, the substrate and background population create a scenario that
encapsulates a particular context or situation within the simulation. This
scenario can be designed to simulate real-world scenarios, test hypotheses,
study the behavior of complex systems, or provide a platform for
experimentation and analysis. By carefully defining the substrate and
background population, researchers and practitioners can create meaningful and
informative scenarios that capture the intricacies of the system being
studied.

Figure 5: marl-jax supports two major environment suits, Meltingpot Agapiou et
al. (2022) and Overcooked Carroll et al. (2019)

###  3.3 Algorithms Implemented

We currently support two major algorithms

  * •

Actor-Critic Baseline: A standard actor-critic based independent learning
algorithm using V-trace Espeholt et al. (2018) for off-policy corrections.

  * •

OPRE: Options as Responses Vezhnevets et al. (2020) follows actor-critic based
learning but its objective is specifically designed to generalize to novel
partners. It is used as one of the baseline in MeltingPot Agapiou et al.
(2022). We are the first to provide an open-source implementation of OPRE.

###  3.4 Utilities

We provide two major utilities 1) train.py and 2) evaluate.py

  * •

train.py: The entry point for training a population of agents in the given
environment

  * •

evaluate.py: Used to evaluate the generalization performance on with various
partner agents

  * •

evaluation_results.py: Aggregates the evaluation results by averaging across
multiple seeds and presents a table.

##  4 Results

| OPRE | IMPALA
---|---|---
Substrate | 0.00 | 0.00
Scenario 0 | 4.91 | -7.10
Scenario 1 | 3.79 | -2.65
Scenario 2 | 10.52 | 5.97
Scenario 3 | 13.52 | 11.52
Scenario 4 | 6.60 | 0.66
Table 1: Evaluation on Running with Scissors in the matrix Figure 6: Training Plot on Running with Scissors in the matrix | IMPALA | OPRE
---|---|---
Substrate | 106.85 | 38.18
Scenario 0 | 131.00 | 59.71
Scenario 1 | 176.54 | 114.69
Scenario 2 | 79.58 | 27.97
Scenario 3 | 62.80 | 41.76
Scenario 4 | 48.63 | 38.75
Scenario 5 | 65.82 | 47.66
Scenario 6 | 101.83 | 40.34
Scenario 7 | 83.33 | 49.82
Scenario 8 | 77.75 | 32.59
Scenario 9 | 78.41 | 74.62
Table 2: Evaluation on Prisoners Dilemma in the Matrix Figure 7: Training Plot
on Prisoners Dilemma in the Matrix

We evaluate our implementation in two environments to assess its performance
and generalization capabilities across different types of multi-agent
scenarios. The first environment, Meltingpot, encompasses a wide range of game
types, including cooperative, competitive, and general-sum games. In
Meltingpot, agents interact with each other to achieve various objectives,
which can involve cooperation, competition, or a combination of both. This
environment allows us to examine how well our implementation handles different
types of interactions and strategies, evaluating its performance in
cooperative, competitive, and general-sum settings.

The second environment, Overcooked, focuses specifically on cooperative multi-
agent scenarios. In Overcooked, agents work together in a shared kitchen to
prepare meals and serve customers. The emphasis in this environment is on
effective coordination, communication, and cooperation among the agents to
maximize efficiency and customer satisfaction. By evaluating our
implementation in Overcooked, we can specifically assess its performance and
effectiveness in cooperative multi-agent settings, where collaboration and
teamwork are crucial for success.

By evaluating our implementation in both Meltingpot and Overcooked, we gain a
comprehensive understanding of its performance in a range of multi-agent
scenarios. This evaluation enables us to analyze how well our approach adapts
to different types of interactions, strategies, and objectives, and provides
valuable insights into its strengths and limitations. The findings from these
evaluations contribute to advancing our understanding of multi-agent
reinforcement learning and inform further research and development in this
field.

###  4.1 MeltingPot

We conduct evaluations on four distinct environments from the Meltingpot-v1
and Meltingpot-v2 domains.

In Meltingpot-v1, we evaluate our approach on two environments:

  * •

Running with Scissors in the Matrix: The training progress is visualized in
Figure 6, and the evaluation scores across different episodes are provided in
Table 1.

  * •

Prisoners’ Dilemma in the Matrix: The training progress is depicted in Figure
7, and the evaluation scores for various scenarios are presented in Table 2.

In Meltingpot-v2, we assess our approach on two additional environments:

  * •

Daycare: The training progress is shown in Figure 8, and the evaluation scores
for different scenarios are summarized in Table 3.

  * •

Externality Mushrooms Dense: The training progress is illustrated in Figure 9,
and the evaluation scores for various scenarios are provided in Table 4.

These evaluations allow us to analyze the performance of our approach across
different environments and scenarios within the Meltingpot framework. The
training plots provide insights into the learning progress, while the
evaluation scores offer quantitative measures of the agent’s performance in
various scenarios.

| IMPALA | OPRE
---|---|---
Substrate | 65.94 | 67.83
Scenario 0 | 0.89 | 0.33
Scenario 1 | 109.11 | 126.00
Scenario 2 | 0.22 | 0.00
Scenario 3 | 154.56 | 171.33
Table 3: Evaluation on Daycare Figure 8: Training Plot on Daycare | IMPALA
---|---
Scenario 0 | 547.70
Scenario 1 | 13.21
Scenario 2 | 293.02
Scenario 3 | 38.04
Substrate | 91.56
Table 4: Evaluation on Externality Mushrooms Figure 9: Training Plot on
Externality Mushrooms

###  4.2 Overcooked

We evaluate the performance of the algorithms on the _Cramped Room_
environment from the Overcooked domain. The training progress of the
algorithms is visualized in Figure 10, providing insights into their learning
dynamics and convergence behavior.

Figure 10: Training Plot on Cramped Room

##  5 Conclusion and Future Works

In this paper, we introduced marl-jax, a highly optimized package for training
and evaluation of the generalization of a population of agents to novel
partners. The package provides an easy-to-use utility to train and evaluate
the trained agents. It also provides an open-source implementation of ORPE
Vezhnevets et al. (2020), a MARL algorithm designed for generalization. This
package is targeted for researchers working on generalization in MARL and
reduces the entry barrier for new researchers in MARL generalization. As any
software package, marl-jax is under continuous development and, in future,
aims to implement other population learning algorithms Team et al. (2021).

## References

  * Agapiou et al. (2022) John P. Agapiou, Alexander Sasha Vezhnevets, Edgar A. Duéñez-Guzmán, Jayd Matyas, Yiran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, D. J. Strouse, Michael B. Johanson, Sukhdeep Singh, Julia Haas, Igor Mordatch, Dean Mobbs, and Joel Z. Leibo.  Melting Pot 2.0, 2022.
  * Babuschkin et al. (2020) Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Luyu Wang, Wojciech Stokowiec, and Fabio Viola.  The DeepMind JAX Ecosystem, 2020.
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * Carroll et al. (2019) Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan.  On the Utility of Learning about Humans for Human-AI Coordination.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Cassirer et al. (2021) Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss.  Reverb: A Framework For Experience Replay, 2021.
  * Castro et al. (2018) Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.  Dopamine: A Research Framework for Deep Reinforcement Learning.  2018\.
  * Ellis et al. (2022) Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Foerster, and Shimon Whiteson.  Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.  _arXiv preprint arXiv:2212.07489_ , 2022.
  * Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures.  In _Proceedings of the 35th International Conference on Machine Learning_. PMLR, 2018.
  * Espeholt et al. (2019) Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski.  Seed rl: Scalable and efficient deep-rl with accelerated central inference.  2019\.
  * Hessel et al. (2021) Matteo Hessel, Manuel Kroiss, Aidan Clark, Iurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt.  Podracer architectures for scalable reinforcement learning.  2021\.
  * Hoffman et al. (2022) Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas.  Acme: A Research Framework for Distributed Reinforcement Learning, 2022.
  * Hu et al. (2022) Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Zhihui Li, Xiaodan Liang, Xiaojun Chang, and Yaodong Yang.  Marllib: A scalable multi-agent reinforcement learning library.  _arXiv preprint arXiv:2210.13708_ , 2022.
  * Huang et al. (2022) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G. M. Araújo.  CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms.  _Journal of Machine Learning Research_ , 23(274), 2022.  ISSN 1533-7928.
  * Liang et al. (2018) Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica.  RLlib: Abstractions for Distributed Reinforcement Learning.  In _Proceedings of the 35th International Conference on Machine Learning_. PMLR, 2018.
  * Mahajan et al. (2022) Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rocktäschel, and Shimon Whiteson.  Generalization in cooperative multi-agent systems.  _arXiv preprint arXiv:2202.00104_ , 2022.
  * Muldal et al. (2019) Alistair Muldal, Yotam Doron, John Aslanides, Tim Harley, Tom Ward, and Siqi Liu.  dm_env: A python interface for reinforcement learning environments, 2019\.
  * Papoudakis et al. (2021) Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht.  Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS)_ , 2021.
  * Pretorius et al. (2021) Arnu Pretorius, Kale-ab Tessera, Andries P. Smit, Claude Formanek, St John Grimbly, Kevin Eloff, Siphelele Danisa, Lawrence Francis, Jonathan Shock, Herman Kamper, Willie Brink, Herman Engelbrecht, Alexandre Laterre, and Karim Beguir.  Mava: A research framework for distributed multi-agent reinforcement learning, 2021.
  * Raffin et al. (2021) Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.  Stable-Baselines3: Reliable Reinforcement Learning Implementations.  _Journal of Machine Learning Research_ , 22(268), 2021.  ISSN 1533-7928.
  * Samvelyan et al. (2019) Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson.  The StarCraft Multi-Agent Challenge.  _CoRR_ , abs/1902.04043, 2019.
  * Sarkar et al. (2022) Bidipta Sarkar, Aditi Talati, Andy Shih, and Dorsa Sadigh.  PantheonRL: A MARL Library for Dynamic Training Interactions.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(11), 2022.  ISSN 2374-3468.
  * Team et al. (2021) Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki.  Open-ended learning leads to generally capable agents, 2021.
  * Terry et al. (2021) J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al.  Pettingzoo: Gym for multi-agent reinforcement learning.  _Advances in Neural Information Processing Systems_ , 2021.
  * Vezhnevets et al. (2020) Alexander Sasha Vezhnevets, Yuhuai Wu, Remi Leblond, and Joel Z. Leibo.  Options as responses: Grounding behavioural hierarchies in multi-agent RL, 2020.  URL http://arxiv.org/abs/1906.01470.
  * Yang et al. (2021) Fan Yang, Gabriel Barth-Maron, Piotr Stańczyk, Matthew Hoffman, Siqi Liu, Manuel Kroiss, Aedan Pope, and Alban Rrustemi.  Launchpad: A Programming Model for Distributed Machine Learning Research, 2021.
  * Zhou et al. (2023) Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Yong Yu, Jun Wang, and Weinan Zhang.  Malib: A parallel framework for population-based multi-agent reinforcement learning.  _Journal of Machine Learning Research_ , 24(150):1–12, 2023.  URL http://jmlr.org/papers/v24/22-0169.html.


11institutetext: University of Vermont, Burlington VT 05401, USA
11email<EMAIL_ADDRESS>
# No-brainer: Morphological Computation driven Adaptive Behavior in Soft
Robots

Alican Mertan 11 0000-0002-5947-8397 Nick Cheney 11 0000-0002-7140-2213

###### Abstract

It is prevalent in contemporary AI and robotics to separately postulate a
brain modeled by neural networks and employ it to learn intelligent and
adaptive behavior. While this method has worked very well for many types of
tasks, it isn’t the only type of intelligence that exists in nature. In this
work, we study the ways in which intelligent behavior can be created without a
separate and explicit brain for robot control, but rather solely as a result
of the computation occurring within the physical body of a robot.
Specifically, we show that adaptive and complex behavior can be created in
voxel-based virtual soft robots by using simple reactive materials that
actively change the shape of the robot, and thus its behavior, under different
environmental cues. We demonstrate a proof of concept for the idea of closed-
loop morphological computation, and show that in our implementation, it
enables behavior mimicking logic gates, enabling us to demonstrate how such
behaviors may be combined to build up more complex collective behaviors.

###### Keywords:

Soft robotics Adaptive behavior

##  1 Introduction and Background

Recent advances in artificial intelligence and machine learning have benefited
greatly from the rise of modern deep learning systems, ultimately aimed at
artificial general intelligence [22]. The coming-of-age of these artificial
neural network systems includes a long history of bio-inspiration, dating back
to Mcculloch and Pitts [26]. Yet the processes behind biological intelligence
reach far beyond systems and processes confined to the brain of living
organisms.

Our bias toward attributing intelligent behavior to the mind is far from new.
Descartes’ mind-body-dualism dates back to the 1600s [13]. However, Bongard
warns that “thinking about thinking is misleading” [4]. Moravec’s paradox, the
observation that reasoning takes less computational resources than sensory-
motor skills (contrary to the expectations of experts), is an example of how
intuition could fail in introspection on thinking [30]. Examples of complex,
perhaps intelligent-seeming, behavior stemming from non-neural processes are
abound in both engineered and natural systems. Perhaps the most notable
example of a system built to mimic a complex behavior in simple and purely
mechanical form is the passive walking robot, which gracefully walks down an
incline plane simply due to the well-tuned structure of its joints and limbs
interacting with the physics of its environment [27]. In natural systems, the
growth and shape change towards stimuli in plants serve as an obvious example
of non-neural behavior [20, 38].

More generally, these phenomena – of the body’s complex and “intelligent”
behavior, with or without a brain – are widely studied within the fields of
embodied cognition [39] and morphological computation [34]. The literature
argues that the brain is not the only source of adaptive and complex behavior,
and that the body “shapes the way we think” [33]. Even for humans, who
naturally consider their brains as the source of their intelligent behavior,
there are strong indications that the body plays an important role in
intelligence, for example, natural language, something we consider one of the
fundamental products of our intelligence, is deeply connected to our bodies
[21, 35].

Prior works have demonstrated the concept of morphological computation in
physical bodies, designing or evolving a body plan which executes a given
behavior with as little neural or control intelligence as possible [1, 9, 11,
12, 16, 18, 32], but these body plans tend to focus on finding an effective
morphology for a single desired behavior. Conversely, very simple adaptive
behaviors in neural circuits demonstrate minimally cognitive systems that
adapt the behavior of a fixed body plan to environmental stimuli via
extraordinary simple neural circuitry [2, 7]. Bridging the gap between the
two, prior work on brain-body co-optimization attempt to simultaneously evolve
simple controllers and evolving robot bodies [17, 5, 19, 23, 36] or develop
approaches to rapidly adapt a controller to an ever-evolving body plan [10,
28, 29]. But, to the best of our knowledge, prior work thus far has failed to
demonstrate an example of adaptive real-time behavior stemming entirely from
morphological computation (i.e. with no controller or neural circuitry).

Inspired by the diverse forms of complex behaviors in nature, here we study
ways in which adaptive behavior can be created in an artificial agent without
a separately postulated brain. Particularly, we evolve voxel-based soft robots
[3] for the task of locomotion to study how morphology could be a source of
adaptive behavior. Four stimuli-responsive materials change shape in response
to environmental cues, resulting in a gross morphological change to the robot.
Robots are optimized via an evolutionary algorithm to produce body-plans that
adaptively change direction on-the-fly in response to combinations of these
environmentally-driven morphology changes (Fig. 1), resulting in closed-loop
robot behaviors with no controller or neural circuitry (Figs. 3, 4, 2), that
can be combined to create complex morphological logic gates (Fig. 5).

##  2 Methodology

Simulation: We run our experiments on the EvoGym simulator [3]. It is a 2D
soft robot simulator where robots are represented by voxels that can actuate
by changing their areas ($a\in[0.6\times r,1.6\times r]$ where $r$ is the
resting area). Designing robots’ bodies means placing voxels with varying
materials in a grid layout and specifying connections with their neighbors. In
our experiments, we assume all neighboring voxels are connected to simplify
the design space.

(a) Sensor type 1 (stimulus 1)

(b) Sensor type 2 (stimulus 1)

(c) Sensor type 3 (stimulus 2)

(d) Sensor type 4 (stimulus 2)

(e) Example morphologies under different stimuli patterns. The behavior of
sensory voxels molds the morphology of the robot, potentially changing its
behavior.

Figure 1: Images from the simulator environment [3] that shows the behavior of
sensory voxels. Voxels in (a) and (b) respond to stimulus 1 and voxels in (c)
and (d) respond to stimulus 2. Voxels in (a) and (c) shrink to their minimum
width in response to the absence of their respective stimulus (left), and
expand to their maximum width in response to the presence of their respective
stimulus (right). Voxels in (b) and (d) respond the opposite way. These
materials enable robots (e) that actively undergo shape change to produce new
behaviors according to cues from some sensory stimuli.

Environment: We use EvoGym’s Python API to define a custom environment with
two stimuli. These stimuli work in a binary fashion – they are either present
or absent in the environment. For our robots to sense these stimuli and adapt
their behavior, we hand-design ”sensory voxels”. We consider two environmental
stimuli, each with one voxel type that expands in response to that stimuli and
one that contracts in the presence of that stimuli – for a total of 4 sensory
voxel types (Fig. 1; green, red, magenta, blue). We also hand-designed two
active voxels (orange and teal) that horizontally expand and contract to
provide energy for locomotion following a sinusoidal signal with a 180° phase
offset, similar to the designs in [11], but do not provide any neural or
informational connectivity between them to organize a coordinated controller
by means other than physical forces traveling through the robot’s body.
Lastly, we provide evolution with two passive voxels, soft (gray) and rigid
(black).

Evolutionary algorithms [14] are the most common approach to optimize robot
body plans in the above-cited prior work, as they work by evaluating the
empiric and holistic phenotype/behavioral effect of random mutations of a
robot’s artificial genome, selecting the most fit robots to survive and
reproduce – and thus do not require a differentiable model of the robot’s
morphology for credit assignment of the robot’s fitness to individual voxels.
Here we employ the Map-Elites algorithm [31] to evolve an archive of diverse
robots. Specifically, the archive consists of seven feature dimensions. Three
of these dimensions are based on morphological features – the number of active
voxels, sensory voxels, and total number of voxels. Following the common
practice [11], we limit the morphology space with a $H\times W$ bounding box.
Limiting the total number of voxels in this way allows us to discretize the
three morphological dimensions of the archive into four bins each. The next
four dimensions are based on the behavior of the robot, the direction of
locomotion in the x-axis, under different stimuli patterns. The two binary
stimuli result in four patterns, and a robot can move in either the positive x
or the negative x direction, resulting in 16 possible behaviors in total. By
utilizing QD algorithms and defining the dimensions of the archive in the
described way, we allow evolution to search for all possible behaviors with
varying morphological structures in a single run, and keep the best-performing
robot for each unique combination of morphological-behavioral traits, via the
MAP-Elites algorithm.

Evaluation and selection: Individuals are evaluated under 5 simulated
conditions. First, they are evaluated for 40 actuation cycles (i.e. 40 periods
of the sinusoidal actuation signal) where at each 10th actuation cycle the
environmental stimuli change as follows: (Stimulus 1, Stimulus 2) = {(Absent,
Absent), (Absent, Present), (Present, Absent), (Present, Present)}. In the
next 4 simulations, individuals are simulated under each of these 4 fixed
stimuli patterns for 10 actuation cycles each. We use multi-objective
selection to select robots that maximize total displacement in the 40-cycle
variable stimuli simulation and maximize the minimum displacement across the 4
single stimuli 10-cycle simulations. Inspired from [25], an offspring replaces
an individual from the archive, only if it dominates the individual, i.e. is
better than the individual in both fitness values. In order to also select for
robots with dynamically stable gaits that are not overly dependent on initial
conditions, we disregard all robots that move in different directions for the
same stimuli pattern in combined vs. single stimuli simulations.

Genome: Similar to [11, 12], robot morphologies are represented by
compositional pattern-producing networks (CPPNs) [37]. CPPN is a directed
acyclic graph, where each node represents a math function. Here we use the
sine, absolute value, square, and square-root as activation functions within
each node. The CPPN takes the normalized coordinates of a voxel as well as its
distance to the center of the bounding box as an input and outputs a scalar
value for each possible voxel type, including the empty voxel. The type
indicated by the max output value is assigned to the queried location. During
evolution, offspring are created through mutation by the addition/removal of
nodes/links, change of activation functions at nodes, and change of weights in
edges.

Parameters: We experiment with three different bounding boxes,
$(H,W)\in\\{(5,5),(7,7),(10,10)\\}$, and repeat each experiment 10 times.
Evolution was run for 3000 generations where each generation 16 individuals
were chosen as parents from the archive uniformly randomly to create 16
offspring through mutation. Code to repeat our experiments is available at
github.com/mertan-a/no-brainer.

##  3 Responding to Binary Stimuli

Figure 2: The spacetime diagram of a run champion (left) and its gait under
different stimuli patterns as snapshots from the simulation (right). In the
spacetime diagram, the solid black line in the middle keeps track of the
robot’s center of mass over time. The two accompanying lines show the current
stimuli pattern over time. The robot moves toward the positive x-direction
when one stimulus is present, otherwise, it moves in the opposite direction.
The different gait behaviors are achieved by the morphological changes caused
by sensory voxels – an example of morphological computation and how the body
could be the source of adaptive behavior. See more robots in action at our
repository.

Firstly we investigate whether purely morphological changes can result in
behavior that adapts to environmental stimuli. Over the 4 stimuli patterns
presented to the robot, our MAP-Elites archive collects different combinations
of behavioral patterns that different morphologies produce (labeled for
whether a robot moves (L)eft or (R)ight for each of the 4 stimuli). Fig. 2
visualizes the behavior of a robot exhibiting LRRL behavior as a spacetime
diagram to investigate how the morphology gives rise to adaptive behavior. The
solid line tracks the position of the robot’s center of mass during
simulation. The two lines accompanying the solid line display the stimuli
pattern at each time step. The dotted horizontal line shows the moments when
the stimuli pattern changes. The robot’s morphology and its gait under
different stimuli can be seen on the right. The robot moves to the right only
if one of the two stimuli is present (the middle two settings). In the case
where both stimuli are present or absent (the first and last setting), the
robot moves to the left. This behavior is achieved with the help of sensory
voxels – the change in the sensory voxels alters the robot’s shape and thus
changes its gait. The robot exhibits an adaptive behavior without the help of
a dedicated brain. Reactive materials are enough to unlock morphological
computation.

Figure 3: The spacetime diagrams of all run champions from independent runs,
plotted together. We omit the behaviors where the robot doesn’t respond to any
stimulus (LLLL and RRRR behaviors). Time is normalized to reflect actuation
cycles and position is normalized to reflect the robot’s body length. The
numbers in the bottom right corner show the average distance traveled by run
champions.

Fig. 3 displays spacetime diagrams of all run champions successfully
exhibiting different adaptive behaviors. Each run champion is drawn in a
different color and the average normalized space traversed by run champions in
each plot can be seen in the bottom right corner. We see that given adequate
environmentally-sensitive building blocks to work with, evolution can find
morphologies that exhibit various adaptive behaviors – responding to stimuli
in different ways via shape change alone (without a learned and/or stimuli-
aware controller).

Figure 4: Performances of run champions from 10 independent runs with a
$10\times 10$ bounding box. Symmetric robot behaviors are grouped by the
pattern of their movement direction across stimuli (e.g. LRLR and RLRL are
both represented as XYXY). Colors represent the number of stimuli a robot’s
morphology adapts to. Each data point is plotted and mean values are marked
with an ’x’. Behaviors are separated with empty columns based on statistically
significant performance differences (at $P<0.001$) – each distribution in a
group is statistically significantly better compared to each other
distribution belonging to any group to their right. Bounding boxes of size
$5\times 5$ and $7\times 7$ are qualitatively consistent.

To analyze the effect of different behavior patterns, we group the run
champions from independent runs based on the directions of their movement
trajectories across the 4 stimuli. For this analysis, we group behaviors that
are symmetric in the x-axis (e.g. LLLL and RRRR are grouped as XXXX, LLRR and
RRLL are grouped as XXYY, etc.) as the challenge of achieving symmetric
behaviors is equal. As can be seen in Fig. 4, robots ignoring both stimuli and
moving in the same direction throughout (behavior XXXX; mustard color) are
able to locomote further compared to robots that exhibit a sensory-dependent
behavior. Similarly, robots that respond only to a single stimulus (behavior
patterns XXYY and XYXY; light blue color) perform better compared to robots
adapting their behavior following both stimuli (green). As expected, there is
no statistically significant difference between following the first stimulus
(XXYY) or the second (XYXY). Adapting behavior only when one of the two
stimuli is present but not both (behavior XYYX; right-most green) is harder
compared to all other behaviors that use both stimuli (other green).
Interestingly, this behavior (XYYX) is analogous to XOR and XNOR binary
boolean functions, which are the only ones that are not linearly separable.
All comparisons are statistically significant at the level of $P<0.001$ and
hold for all experimented bounding boxes.

##  4 Making a Swarm for More Complex Behavior

Figure 5: Top row High-level schematic of the swarm (left) and its inner
wiring (right). Lines connecting to robots from the left represent stimuli and
the ones on the right represent outputs. We assume that the behavior of one
robot could determine the stimulus of another. Bottom row The behavior of each
robot with respect to their immediate stimuli (left) and the behavior of the
starred robot with respect to swarm stimuli (right). While each robot
successfully performs the behavior that it evolved for with respect to its
immediate stimuli, examination of the selected robot with respect to swarm
stimuli shows complex behavior that would otherwise require memory to exhibit.

The above section demonstrated the evolution of robots exhibiting adaptive
behavior in response to the two stimuli. To the best of our knowledge, this is
the first example of a closed-loop fully-morphological behavior in evolved
robots. The simplicity of the chosen behavior (more right or left) allows us
to clearly see how shape change due to the stimuli-sensing materials affects
the robots’ behavior. But this also invites a thought experiment: how could
this proof of concept scale to create complex “intelligent” behaviors from
this simple framework?

In this section, we study how these robots can be grouped to create a swarm
exhibiting more complex behavior. As a proof of concept, we use a subset of
previously evolved robots and hand-design the swarm to show a certain
behavior.

We start with an observation. Given the binary nature of the stimuli and the
possible behaviors of the robot (moving in a positive or negative direction on
the x-axis), the robots could be considered implementing logic gates from the
perspective of an outside observer. Above, we evolved robots implementing all
possible boolean functions by evolving an archive of robots exhibiting all
possible behaviors (Fig. 3). For instance, if we assume that a robot moving in
the positive x direction on the x-axis is outputting a 1, robots exhibiting
the RRRL behavior can be considered as implementing the NOT-AND boolean
function.

We can organize robots, by assuming that the behavior of one robot could
determine the stimulus of another robot. While this is arbitrary given our
manually and abstractly designed stimuli, it’s not a great stretch of the
imagination to envision some analogous physical or chemical sensory apparatus
that would allow voxels to sense the relative change in distance of a
neighboring robot similar to how animals do (e.g. it became “louder” or its
“smell” became stronger).

To demonstrate a more complex function derived from these morphological logic
gates, we manually design a swarm implementing a D-type latch111While we hand-
design the swarm as a proof of concept, one could evolve the design of the
swarm towards achieving any target behavior as well.. Fig. 5 top left, shows
the schematic of the swarm from an outside observer’s perspective. The swarm
consists of a single robot with behavior RRLL (leftmost robot, implements AND
gate), and four robots with behavior RRRL (implement NAND gate). The swarm has
two swarm stimuli, observed by three of the robots. The rest of the stimuli
that robots observe are determined by others’ behavior. The inner wiring of
the swarm can be seen in Fig. 5 top right, where wires on the left side of the
robots represent stimuli and the wires on the right side of the robots
represent the output of the robot (which determines the stimulus of another
robot). If a robot’s center of mass has moved in the positive x direction in
the last actuation cycle it is considered to be outputting 1 and turns the
stimulus on for another robot and vice versa.

We plot the behavior of the swarm as a spacetime diagram in Fig. 5 bottom row.
The behavior of each individual and their immediate stimuli can be seen on the
left. As expected, robots successfully perform the behaviors that are evolved
for, seamlessly responding to changing stimuli patterns. However, if we
observe the behavior of the starred robot (top-right robot) with respect to
swarm stimuli (as opposed to the robot’s immediate stimuli) in Fig. 5 bottom
right, we see that the robot is moving following the swarm stimulus 1, only
when the swarm stimulus 2 is present, otherwise the robot keeps moving in the
direction of last observed swarm stimulus 1. Given the right frame of
reference (observing the robot’s behavior with respect to the swarm stimuli)
and certain assumptions (behavior of one robot determining the stimulus of
another), we show that robots exhibit complex adaptive behavior, without any
separately postulated brain, that would typically require memory to achieve.
This achievement is, certainly, not a surprise, given the equivalence of
robots’ behavior with boolean functions. Once we have robots that can
implement the NOT-AND function, we could build a swarm that would perform any
computable behavior. We leave the design of such swarms as an exercise for the
reader.

##  5 Discussion

The above results demonstrate “intelligent” morphological computation that
adapts the robot’s behavior via stimuli-educed shape change rather than by a
dedicated control or brain-like mechanism. While we feel that this proof of
concept is an important, and hopefully entertaining, demonstration, we fully
acknowledge that the most complex and adaptive behaviors in natural systems
are not fully morphological, but are an interplay of embodied cognition
connecting the body and the brain.

Recent literature had demonstrated the challenges of brain-body co
optimization and noted fragile co-adaptation to be a major challenge of
evolving these systems [8, 29], further suggesting that increasing the
adaptability and robustness of the brain and body to modification in the other
could result in less fragile and thus more effective brain-body systems [10,
28]. We hope that by creating more intelligent and adaptive bodies, this work
helps to open doors to transfer additional computation to the morphology and
enable less fragile co-adaptation.

Lastly, we want to connect this research to recent ideas in artificial and
natural evolution. Specifically, it has been argued that a key feature of
natural evolution is that it works with material that is active. Cells that
already have an array of functionalities come together to create multi-
cellular structures, tissues, organs, and systems, creating a “multiscale
competence architecture” where each scale exploits and regulates the already-
existing functionalities of the scale below to achieve its goals [24]. This is
considered to be an important feature of natural evolution and
understanding/exploiting this in artificial evolution could have substantial
effects [6, 15]. Inspired by these ideas, here we show a first pass at
demonstrating that simple materials each with straightforward behaviors can be
optimized in their emergent design to result in a complex and adaptive system.
In future work, we hope to examine how we can consider such a system as a
substrate for further evolution (potentially in combination with separately
postulated brain models).

##  6 Conclusion

We investigate ways in which complex, adaptive behavior can be created in
virtual voxel-based soft robots without the use of a separately postulated
brain. We hand-design simple muscle (that continuously expands and contracts)
and sensory (that expands/shrinks in the presence or absence of a binary
stimulus) voxel materials and show that evolution can create designs that
exhibit closed-loop adaptive behavior, responding to two binary stimuli.
Moreover, a swarm of such robots can exhibit even more complex behavior. In
future work, we hope to study more ways of creating adaptive and complex
behavior.

###  6.0.1 Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grants No. 2008413 and 2239691. Computations were performed on the
Vermont Advanced Computing Core supported in part by NSF Award No. 1827314.

## References

  * [1] Auerbach, J.E., Bongard, J.C.: Evolving cppns to grow three-dimensional physical structures. In: Proceedings of the 12th annual conference on Genetic and evolutionary computation. pp. 627–634 (2010)
  * [2] Beer, R.D.: Toward the evolution of dynamical neural networks for minimally cognitive behavior. In: From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. vol. 4, p. 421. MIT Press (1996)
  * [3] Bhatia, J., Jackson, H., Tian, Y., Xu, J., Matusik, W.: Evolution gym: A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems 34, 2201–2214 (2021)
  * [4] Bongard, J.: Evolutionary robotics lectures, https://www.youtube.com/playlist?list=PLAuiGdPEdw0iyApypcLk˙xBKjLchQM-Mg
  * [5] Bongard, J.C., Pfeifer, R.: Evolving complete agents using artificial ontogeny. In: Morpho-functional machines: the new species: designing embodied intelligence. pp. 237–258. Springer (2003)
  * [6] Bongard, J., Levin, M.: There’s Plenty of Room Right Here: Biological Systems as Evolved, Overloaded, Multi-Scale Machines. Biomimetics 8(1), 110 (Mar 2023). https://doi.org/10.3390/biomimetics8010110
  * [7] Braitenberg, V.: Vehicles: Experiments in synthetic psychology. MIT press (1986)
  * [8] Cheney, N., Bongard, J., Sunspiral, V., Lipson, H.: On the Difficulty of Co-Optimizing Morphology and Control in Evolved Virtual Creatures. In: Proceedings of the Artificial Life Conference 2016. pp. 226–233. MIT Press, Cancun, Mexico (2016). https://doi.org/10.7551/978-0-262-33936-0-ch042
  * [9] Cheney, N., Bongard, J., Lipson, H.: Evolving soft robots in tight spaces. In: Proceedings of the 2015 annual conference on Genetic and Evolutionary Computation. pp. 935–942 (2015)
  * [10] Cheney, N., Bongard, J., SunSpiral, V., Lipson, H.: Scalable co-optimization of morphology and control in embodied machines. Journal of The Royal Society Interface 15(143), 20170937 (Jun 2018). https://doi.org/10.1098/rsif.2017.0937
  * [11] Cheney, N., MacCurdy, R., Clune, J., Lipson, H.: Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding. In: Proceedings of the 15th annual conference on Genetic and evolutionary computation. pp. 167–174 (2013)
  * [12] Corucci, F., Cheney, N., Lipson, H., Laschi, C., Bongard, J.: Material properties affect evolutions ability to exploit morphological computation in growing soft-bodied creatures. In: Proceedings of the Artificial Life Conference 2016. pp. 234–241. MIT Press, Cancun, Mexico (2016). https://doi.org/10.7551/978-0-262-33936-0-ch043
  * [13] Descartes, 1596-1650, R.: Discourse on method ; and, Meditations on first philosophy. Third edition. Indianapolis : Hackett Pub. Co., [1993] ©1993 (1993), https://search.library.wisc.edu/catalog/999718190702121
  * [14] Eiben, A.E., Smith, J.E.: Introduction to evolutionary computing. Springer (2015)
  * [15] Hartl, B., Risi, S.L., Levin, M.: Evolutionary Implications of Multi-Scale Intelligence (Apr 2024). https://doi.org/10.31219/osf.io/sp9kf
  * [16] Hauser, H., Ijspeert, A.J., Füchslin, R.M., Pfeifer, R., Maass, W.: Towards a theoretical foundation for morphological computation with compliant bodies. Biological cybernetics 105, 355–370 (2011)
  * [17] Hornby, G.S., Pollack, J.B.: Body-brain co-evolution using l-systems as a generative encoding. In: Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation. pp. 868–875 (2001)
  * [18] Joachimczak, M., Suzuki, R., Arita, T.: Artificial metamorphosis: Evolutionary design of transforming, soft-bodied robots. Artificial life 22(3), 271–298 (2016)
  * [19] Joachimczak, M., Wróbel, B.: Co-evolution of morphology and control of soft-bodied multicellular animats. In: Proceedings of the 14th annual conference on Genetic and evolutionary computation. pp. 561–568 (2012)
  * [20] Karban, R.: Plant behaviour and communication. Ecology letters 11(7), 727–739 (2008)
  * [21] Lakoff, G., Johnson, M.: Metaphors we live by. University of Chicago press (2008)
  * [22] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * [23] Lehman, J., Stanley, K.O.: Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th annual conference on Genetic and evolutionary computation. pp. 211–218 (2011)
  * [24] Levin, M.: Darwin’s agential materials: evolutionary implications of multiscale competency in developmental biology. Cellular and Molecular Life Sciences 80(6), 142 (2023)
  * [25] Macé, V., Boige, R., Chalumeau, F., Pierrot, T., Richard, G., Perrin-Gilbert, N.: The quality-diversity transformer: Generating behavior-conditioned trajectories with decision transformers. In: Proceedings of the Genetic and Evolutionary Computation Conference. pp. 1221–1229 (2023)
  * [26] McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics 5, 115–133 (1943)
  * [27] McGeer, T.: Passive dynamic walking. The international journal of robotics research 9(2), 62–82 (1990)
  * [28] Mertan, A., Cheney, N.: Modular controllers facilitate the co-optimization of morphology and control in soft robots. In: Proceedings of the Genetic and Evolutionary Computation Conference. pp. 174–183 (2023)
  * [29] Mertan, A., Cheney, N.: Investigating premature convergence in co-optimization of morphology and control in evolved virtual soft robots. In: European Conference on Genetic Programming (Part of EvoStar). pp. 38–55. Springer (2024)
  * [30] Moravec, H.: Mind children: The future of robot and human intelligence. Harvard University Press (1988)
  * [31] Mouret, J.B., Clune, J.: Illuminating search spaces by mapping elites. arXiv:1504.04909 [cs, q-bio] (Apr 2015), http://arxiv.org/abs/1504.04909
  * [32] Paul, C.: Morphological computation: A basis for the analysis of morphology and control requirements. Robotics and Autonomous Systems 54(8), 619–630 (2006)
  * [33] Pfeifer, R., Bongard, J.: How the body shapes the way we think: a new view of intelligence (2006)
  * [34] Pfeifer, R., Gómez, G.: Morphological computation–connecting brain, body, and environment. Creating brain-like intelligence: From basic principles to complex intelligent systems pp. 66–83 (2009)
  * [35] Pulvermüller, F., Fadiga, L.: Active perception: sensorimotor circuits as a cortical basis for language. Nature reviews neuroscience 11(5), 351–360 (2010)
  * [36] Sims, K.: Evolving virtual creatures. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 699–706 (2023)
  * [37] Stanley, K.O.: Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines 8(2), 131–162 (Jun 2007). https://doi.org/10.1007/s10710-007-9028-8
  * [38] Trewavas, A.: Aspects of plant intelligence. Annals of botany 92(1), 1–20 (2003)
  * [39] Wilson, M.: Six views of embodied cognition. Psychonomic bulletin & review 9, 625–636 (2002)


line, meaning that now $\ell(u,v)=0$ describes points of $\ell$, and
$\ell(u,v)$ for a general point gives the negatively signed distance of the
point from $\ell$ i.e. positive signs to the left, and negative signs to the
right of $\ell$.

Now minimizing over all points of $K$ corresponds to finding the distance of
$K$ from $\ell$. Further, if $(u_{0},v_{0})\in\partial K$ is one such minimal
distance point, then the line $L$ passing through $(u_{0},v_{0})$ and parallel
to $\ell$ is just a supporting line to $K$.

That means in particular that in case we set $\xi:=a_{k}$, then the prescribed
slope matches the slope of the supporting line drawn to $\rho_{k}^{*}$, whence
the above constructed line $L$ will match $L_{k}$. As said, it is a supporting
line to $K$ through $\rho_{k}^{*}$, which also means that all points of $K$,
in particular all points on the curve ${\mathcal{C}}_{1}$, will have to lie to
the left from it. The distance to the left from $\ell$ is now signed
positively, and ${\rho}_{k}^{*}$ provides the least such distance as compared
to all points of $K$; in particular it provides a smaller (not larger) value
than minimizing over all points of ${\mathcal{C}}_{1}$, and also it provides
the minimal value among all points $\rho_{n}^{*}$. So in particular the
distance $\ell(u_{k},v_{k})$ is minimal among all $(u_{n},v_{n})$:
equivalently,
$(1-u_{k})a_{k}+v_{k}=\min_{n\in{\mathbb{N}}}(1-u_{n})a_{k}+v_{n}$. We
therefore find that the slope $a_{k}$ of the supporting line through the
extreme point ${\rho}_{k}^{*}$ defines a value $x_{k}:=e^{a_{k}}$ such that
$\omega(\rho_{k};x_{k})=(1-\beta_{k})\log
x_{k}+\log\gamma_{k}=(1-u_{k})a_{k}+v_{k}=\min_{n\in{\mathbb{N}}}\omega(\rho_{n};x_{k})=\omega(x_{k})$.

Further, we can read the meaning of the other distance minimization: for any
point $(u,v)\in{\mathcal{C}}_{1}$, we have
$\frac{1}{\sqrt{1+\xi^{2}}}\omega(\rho_{k};x_{k})=\ell({\rho}_{k}^{*})\leq\ell(u,v)=\frac{1}{\sqrt{1+\xi^{2}}}\left((1-u)a_{k}+v\right)$.
Multiplying by $\sqrt{1+\xi^{2}}$ and taking minimum on the right hand side
furnishes
$\omega(x_{k})=\omega(\rho_{k},x_{k})\leq\min_{(u,v)\in{\mathcal{C}}_{1}}(1-u)a_{k}+v=\min_{t\geq
1}(1-\eta(t))\log x_{k}+\log t=\omega_{\eta}(x_{k})$.

So we get that setting $x_{k}:=e^{\xi}=e^{a_{k}}$, it holds
$\omega_{\eta}(x_{k})\geq\omega(\rho_{k};x)=\omega(x_{k})$. This is not a
general equation, it is conditional to the property that ${\rho}_{k}^{*}$ is
an extreme point with the supporting line through it having slope $\xi:=\log
x_{k}=a_{k}$, but it holds for arbitrary values of $x=x_{k}$ to which there is
a supporting line through a point ${\rho}_{k}^{*}$. As said, we already know
that there is an infinitude of ${\rho}_{k}^{*}$ being extreme points of the
convex hull $K$, therefore there is an infinite sequence of $\xi_{k}=\log
x_{k}$ values for which the lines with slopes $\xi_{k}$ and passing through
${\rho}_{k}^{*}$ provide supporting lines to $K$. With that sequence of
$x_{k}$ we therefore have that
$\omega(x_{k})=\omega(\rho_{k},x_{k})\leq\omega_{\eta}(x_{k})$. It is clear
that these slopes are tending to infinity, therefore also
$x_{k}=e^{\xi_{k}}\to\infty$.

Here we can apply Theorem 11 (with $x:=x_{k}$), more precisely its proof,
where in the first step we have got (67). We obtain values
$\widetilde{x_{k}}\in[x_{k}^{1-{\varepsilon}},x_{k}]$, for which
$|{\Delta}(\widetilde{x_{k}})|\geq\widetilde{x_{k}}\exp(-(1+{\varepsilon})\omega(x_{k}))\geq\widetilde{x_{k}}\exp(-(1+{\varepsilon})\omega_{\eta}(x_{k}))$,
the last step being clarified above.

Note that $\omega_{\eta}$ satisfies similar properties as $\omega$–see Lemma
10. That means that we can argue as above in the end of the proof of Theorem
11, finally deriving
$|{\Delta}(\widetilde{x_{k}})|\geq\exp(-\frac{1+{\varepsilon}}{1-{\varepsilon}}\omega_{\eta}(\widetilde{x_{k}}))$.
∎

##  9\. Concluding remarks

In the paper and in the previous parts of the series we obtained several
results on the order of magnitude of the error function ${\Delta}(x)$ in the
Beurling PNT under Axiom A In particular, we emphasized the role of the
Riemann-von Mangoldt formula, which seems to suggest reliable information on
the order of magnitude of the error function even if it is a divergent, hardly
manageable series, whose "interference difficulty" was truthfully pointed out
by Littlewood. Even if this series is divergent–so that in concrete analysis
only truncated versions, i.e. partial sums can be used–and even if it is not a
regular say Taylor series, its characteristics seem to have many resemblance
to well-behaved entire functions of say finite order, such as its order of
growth being comparable to the size of its largest term, or to the total sum
of the absolute values of terms. Therefore, in our opinion, reasonable
conjectures can be made by analogy to power series, and the challenge lies in
extracting the conjectured information from the otherwise complicated series
and indeed prove what seems to be suggested by it.

We need to mention another direction here, which seems to originate from a
paper by Knapowsky [31] in the classical case. Namely, from assuming a known
$\zeta$-zero, he obtained a lower estimate for the mean value $D(x)$, too.
Also this was improved to very sharp, essentially optimal forms by Pintz [43]
[44] [41], made suitable also for handling dependence on zero-free regions
(and not only on one pre-set zero) similar to our topic here. That enabled
Pintz to give an essentially full picture of the essentially equivalent order
of magnitude of the functions ${\Delta}(x)$, its average $ßD(x)$, its maximum
$S(x)$, and the zero-distribution dependent functions $W(x)$ and $Z(x)$, see
[41].

However, we cannot fully extend Pintz’ result on the lower estimation of the
average $D(x)$ to the Beurling case. The main reason is that Pintz used
Vinogradov estimates heavily, while we found surpassing that part of the
argument very hard, essentially not working. Nevertheless, already here we
employed certain finer calculus and estimates at some steps, than were merely
necessary for the proofs of our statements right here, in the good hope that
these finer estimates will be of good use in further investigations–in
particular concerning $D(x)$.

##  10\. Acknowledgement

We would like to thank János Pintz, Frederik Broucke, Gregory Debruyne and the
anonymous referee for many useful comments and advices, which were benefitted
throughout our work and its subsequent revision. Also we would like to thank
Ghent University for the hospitality and the motivating professional
environment, which was greatly stimulating during our visit in May 2022. Also
the financial support of the BOF grant, project no. 01J04017 is thankfully
acknowledged. During the fifteen years of writing this paper the author was
supported by the Hungarian Research, Development and Innovation Office, Grant
# T-72731, T-049301, K-61908, K-119528 and K-132097.

##  Literature

  * [1] Al-Maamori, F.; Hilberdink, T. W. An example in Beurling’s theory of generalised primes. _Acta Arith._ 168 (2015), no. 4, 383–396.
  * [2] Aramaki, Junichi, On an extension of the Ikehara Tauberian theorem, _Pacific J. Math._ 133 (1988), 13–30.
  * [3] T. Bekehermes, Allgemeine Dirichletreihen und Primzahlverteilung in arithmetische Halbgruppen, Dissertation zur Erlagung des Grades eines Doktors, Technischen Universität Clausthal, 2003.
  * [4] A. Beurling, Analyse de la loi asymptotique de la distribution des nombres premiers généralisés I. _Acta Math._ 68 (1937), 255–291.
  * [5] F. Broucke, Note on a conjecture of Bateman and Diamond concerning the abstract PNT with Malliavin-type remainder, _Monatsh. Math._ 196 (2021), no. 3, 457–470.
  * [6] F. Broucke and G. Debruyne, On zero-density estimates and the PNT in short intervals for Beurling generalized numbers. To appear in _Acta Arith._. See also at https://arxiv.org/abs/2211.08716.
  * [7] F. Broucke, J. Vindas, A new generalized prime random approximation procedure and some of its applications, ArXiv preprint no. arxiv:arXiv:2102.08478, see at https://arxiv.org/abs/arXiv:2102.08478v1.
  * [8] F. Broucke, G. Debruyne  and J. Vindas, Beurling integers with RH and large oscillation. _Adv. Math._ 370 (2020), Article no. 107240, 38 pp.
  * [9] G. Debruyne  and J. Vindas, On PNT equivalences for Beurling numbers. _Monatsh. Math._ 184 (2017), no. 3, 401–424.
  * [10] G. Debruyne  and J. Vindas, On general prime number theorems with remainder. (English summary) _Generalized functions and Fourier analysis_ , 79–94, in: Oper. Theory Adv. Appl., 260, Adv. Partial Differ. Equ. (Basel), Birkhäuser/Springer, Cham, 2017.
  * [11] G. Debruyne  and J. Vindas, On Diamond’s $L^{1}$ criterion for asymptotic density of Beurling generalized integers. _Michigan Math. J._ 68 (2019), no. 1, 211–223.
  * [12] G. Debruyne, H. Diamond  and J. Vindas, $M(x)=o(x)$ estimates for Beurling numbers. _J. Théor. Nombres Bordeaux_ 30 (2018), no. 2, 469–483.
  * [13] Debruyne, G.; Schlage-Puchta, J.-C.; Vindas, J., Some examples in the theory of Beurling’s generalized prime numbers. _Acta Arith._ 176 (2016), no. 2, 101–129.
  * [14] H. G. Diamond, H. L. Montgomery  and U. Vorhauer, Beurling primes with large oscillation, _Math. Ann._ , 334 (2006) no. 1, 1–36.
  * [15] H. G. Diamond, When do Beurling generalized integers have a density?, _J. Reine Angew. Math._ 295 (1977), p. 22–39.
  * [16] Diamond, H. G.; Zhang, Wen-Bin, Optimality of Chebyshev bounds for Beurling generalized numbers. _Acta Arith._ 160 (2013), no. 3, 259–275.
  * [17] Diamond, H. G.; Zhang, Wen-Bin, Chebyshev bounds for Beurling numbers. _Acta Arith._ 160 (2013), no. 2, 143–157.
  * [18] Diamond, H. G.; Zhang, Wen-Bin, Prime number theorem equivalences and non-equivalences. _Mathematika_ 63 (2017), no. 3, 852–862.
  * [19] Diamond, H. G.; Zhang, Wen-Bin, _Beurling generalized numbers_ , Mathematical Surveys and Monographs, 213. American Mathematical Society, Providence, RI, 2016. xi+244 pp.
  * [20] Hilberdink, T. W. Power moments and value distribution of functions. (English summary) Trans. Amer. Math. Soc. 371 (2019), no. 1, 1–31.
  * [21] Hilberdink, T. W. Generalised prime systems with periodic integer counting function. _Acta Arith._ 152 (2012), no. 3, 217–241.
  * [22] Hilberdink, T. W. $\Omega$-results for Beurling’s zeta function and lower bounds for the generalised Dirichlet divisor problem. _J. Number Theory_ 130 (2010), no. 3, 707–715.
  * [23] Hilberdink, T. W.; Lapidus, M. L. Beurling zeta functions, generalised primes, and fractal membranes. _Acta Appl. Math._ 94 (2006), no. 1, 21–48.
  * [24] Hilberdink, T. W. Well-behaved Beurling primes and integers. _J. Number Theory_ 112 (2005), no. 2, 332–344.
  * [25] A. E. Ingham, The distribution of prime numbers, Cambridge University Press, 1932.
  * [26] A. E. Ingham, A note on the distribution of primes. _Acta Arith._ 1 (1936), 201–211.
  * [27] D. Jonhnston, Improving bounds on prime counting functions by partial verification of the Riemann hypothesis, _Ramanujan J._ 59 (2022), no. 4, 1307–1321.
  * [28] Kahane, J.-P., Le rôle des algebres $A$ de Wiener, $A^{\infty}$ de Beurling et $H^{1}$ de Sobolev dans la théorie des nombres premiers généralisés, _Ann. Inst. Fourier_ (Grenoble) 48(3) (1998) 611–648.
  * [29] Kahane, J. P., Un théoreme de Littlewood pour les nombres premiers de Beurling. (French) [A Littlewood theorem for Beurling primes] _Bull. London Math. Soc._ 31 (1999), no. 4, 424–430.
  * [30] Kahane, J.-P., Conditions pour que les entiers de Beurling aient une densité. _J. Théor. Nombres Bordeaux_ 29 (2017), no. 2, 681–692.
  * [31] S. Knapowski On the mean values of certain functions in prime number theory, Acta Math. Acad. Sci. Hungar. 10 (1959), 375–390.
  * [32] J. Knopfmacher, Abstract analytic number theory, North Holland & Elsevier, Amsterdam–Oxford & New York, 1975. (Second edition: Dover Books on Advanced Mathematics. Dover Publications, Inc., New York, 1990. xii+336 pp.)
  * [33] J. E. Littlewood, Mathematical notes (12). An inequality for a sum of cosines, J. London Math. Soc. 12 (1937), 217–222.
  * [34] J. E. Littlewood, Sur la distribution des nombres premiers, C. R. Acad. Sci. Paris 158 (1914) 1869–1872.
  * [35] H. L. Montgomery, Topics in multiplicative number theory, Lecture Notes in Mathemtics, 227, Springer, 1971\.
  * [36] M. J. Mossinghof, T. S. Trudgian, Improving bounds on prime counting functions by partial verification of the Riemann hypothesis, _J. Number Theory_ 157 (2015), 329–349.
  * [37] Neamah, A. A.; Hilberdink, T. W. The average order of the Möbius function for Beurling primes. _Int. J. Number Theory_ 16 (2020), no. 5, 1005–1011.
  * [38] J. Pintz, On the remainder term of the prime number formula. I. On a problem of Littlewood, Acta Arith. 36 (1980), 341–365.
  * [39] J. Pintz, On the remainder term of the prime number formula. II. On a theorem of Ingham, Acta Arith. 37 (1980), 209–220.
  * [40] J. Pintz, Elementary methods in the theory of $L$-functions IX. Density theorems Acta Arith. XLIX (1980), 387-394.
  * [41] J. Pintz Distribution of the zeroes of the Riemann zeta function and oscillations of the error term in the asymptotic law of the distribution of prime numbers. (Russian) _Tr. Mat. Inst. Steklova_ 296 (2017), Analiticheskaya i Kombinatornaya Teoriya Chisel, 207–219. English version published in _Proc. Steklov Inst. Math._ 296 (2017), no. 1, 198–210.
  * [42] J. Pintz, Some new density theorems for Dirichlet L-functions. _Number theory week 2017, Banach Center Publ._ , 118, 231–244. Polish Acad. Sci. Inst. Math., Warsaw, 2019. See also at https://arxiv.org/abs/1804.05552.
  * [43] Pintz, J., Oscillatory properties of the remainder term of the prime number formula. _in: Studies in pure mathematics_ , 551–560, Birkhäuser, Basel, 1983.
  * [44] Pintz, J., On the remainder term of the prime number formula and the zeroes of Riemann’s zeta-function. _in: Number theory, Noordwijkerhout 1983 (Noordwijkerhout, 1983)_ , 186–197, _Lecture Notes in Mathematics_ , 1068, Springer, Berlin, 1984.
  * [45] Pintz, J., On the mean value of arithmetic error terms, _Math. Pann._ , 28 (NS 2) (2022) no. 1, 58–64.
  * [46] D. J. Platt and T. S. Trudgian. The error term in the prime number theorem. _Math. Comp._ 90 (2021), no. 328, 871–881.
  * [47] Platt, D., Trudgian, T.: The Riemann hypothesis is true up to $3\cdot 10^{12}$. _Bull. Lond. Math. Soc._ 53 (3) (2021), 792–797.
  * [48] Sz. Gy. Révész, Irregularities in the distribution of prime ideals. I, Studia Sci. Math. Hungar. 18 (1983), 57–67.
  * [49] Sz. Gy. Révész, Irregularities in the distribution of prime ideals. II, Studia Sci. Math. Hungar. 18 (1983), 343–369.
  * [50] Sz. Gy. Révész, On a theorem of Phragmèn, Complex analysis and applications ’85, (Proceedings of the conference held in Varna, Bulgaria, 1985), Publ. House Bulgar. Acad. Sci. Sofia, 1986, p. 556–568.
  * [51] Sz. Gy. Révész, Effective oscillation theorems for a general class of real-valued remainder terms, Acta Arith. XLIX (1988), 482-505.
  * [52] Sz. Gy. Révész, On Beurling’s prime number theorem, Periodica Math. Hungar. 28 (1994), no. 3, 195–210.
  * [53] Sz. Gy. Révész, On some extremal problems of Landau. _Serdica Math. J._ 33 (2007), no. 1, 125–162.
  * [54] Sz. Gy. Révész, A Riemann-von Mangoldt-type formula for the distribution of Beurling primes. _Math. Pann._ 27 (N.S. 1) (2021) no. 2, 204–232.
  * [55] Sz. Gy. Révész, Density theorems for the Beurling zeta function. _Mathematika_ , 68 (2022), 1045–1072.
  * [56] Sz. Gy. Révész, Oscillation of the remainder term in the prime number theorem of Beurling, "caused by a given $\zeta$-zero". _Int. Math. Res. Notices_ , https://doi.org/10.1093/imrn/rnac274. See also at https://arxiv.org/abs/arXiv:2012.09045.
  * [57] Sz. Gy. Révész, The Carlson-type zero-density theorem for the Beurling zeta function. ArXiv preprint no. arXiv:2209.01689 see at https://arxiv.org/abs/arXiv:2209.01689.
  * [58] R. T. Rockafellar, _Convex Analysis_ , Princeton University Press, Princeton, 1970.
  * [59] J.-C. Schlage-Puchta, Oscillations of the error term in the prime number theorem. _Acta Math. Acad. Sci. Hungar._ 156 (2018), no. 2, 303–308.
  * [60] J.-C. Schlage-Puchta and J. Vindas, The prime number theorem for Beurling’s generalized numbers. New cases, _Acta Arith._ 133(3) (2012) 293–324.
  * [61] E. Schmidt, Über die Anzahl der Primzahlen unter gegebener Grenze, Math. Ann. 57 (1903), 195–204.
  * [62] W. Staś, Über eine Anwendung der Methode von Turán auf die Theorie des Restgliedes im Primidealsatz, Acta Arith. 5 (1959), 179–195.
  * [63] W. Staś, Über die Abschätzung des Restgliedes im Primzahlsatz, Acta Arith. 5 (1959), 427–434.
  * [64] W. Staś, Über die Umkehrung eines Satzes von Ingham, Acta Arith. 6 (1961), 435–446.
  * [65] W. Staś, K. Wiertelak, Some estimates in the theory of functions represented by Dirichlet’s series, Funct. Approx. Comment. Math. 1 (1974), 107–111.
  * [66] W. Staś, K. Wiertelak, A comparison of certain remainders connected with prime ideals in ideal classes mod $f$, Funct. Approx. Comment. Math. 4 (1976), 99–107.
  * [67] Sós, V. T. and Turán, P., On some new theorems in the theory of diophantine approximations, Acta Math. Acad. Sci. Hungar. 6 (1955), 241–255.
  * [68] T. S. Trudgian. Updating the error term in the prime number theorem. _Ramanujan J._ , 39(2), 225–234 (2016).
  * [69] P. Turán, On a new method of analysis and its applications, Wiley-Interscience, New York, 1984.
  * [70] P. Turán, On the remainder term of the prime-number formula, I, Acta Math. Acad. Sci. Hungar. 1 (1950), 48–63.
  * [71] P. Turán, On the remainder-term of the prime number formula. II, Acta Math. Acad. Sci. Hungar. 1 (1950), 155–160.
  * [72] la Vallée Poussin, Ch.-J. de , Sur la function $\zeta(s)$ de Riemann … , _Memoaire couronnés … de Belgique_ , 59 No 1 (1899), 1–74.
  * [73] J. Vindas, Chebyshev estimates for Beurling’s generalized prime numbers. I, _J. Number Theory_ 132 (2012) 2371–2376.
  * [74] Vindas, J., Chebyshev upper estimates for Beurling’s generalized prime numbers. _Bull. Belg. Math. Soc. Simon Stevin_ 20 (2013), no. 1, 175–180.
  * [75] Zhang, Wen-Bin, Exact Wiener-Ikehara theorems. _Acta Arith._ 187 (2019), no. 4, 357–380.
  * [76] Zhang, Wen-Bin, Extensions of Beurling’s prime number theorem. _Int. J. Number Theory_ 11 (2015), no. 5, 1589–1616.
  * [77] Zhang, Wen-Bin, A proof of a conjecture of Bateman and Diamond on Beurling generalized primes. _Monatsh. Math._ 176 (2015), no. 4, 637–656.
  * [78] Zhang, Wen-Bin, Beurling primes with RH and Beurling primes with large oscillation. _Math. Ann._ 337 (2007), no. 3, 671–704.
  * [79] Wen-Bin Zhang, Chebyshev type estimates for Beurling generalized prime numbers. II. _Trans. Amer. Math. Soc._ 337 (1993), no. 2, 651–675.