# SwarmRL: Building the Future of Smart Active Systems

Samuel Tovey222These authors contributed equally, Christoph Lohrmann222These
authors contributed equally, Tobias Merkt &David Zimmer, Konstantin Nikolaou,
Simon Koppenhöfer &Anna Bushmakina, Jonas Scheunemann, Christian Holm
Institute for Computational Physics
University of Stuttgart
70569, Stuttgart, Germany
{stovey, clohrmann<EMAIL_ADDRESS>

###### Abstract

This work introduces SwarmRL, a Python package designed to study intelligent
active particles. SwarmRL provides an easy-to-use interface for developing
models to control microscopic colloids using classical control and deep
reinforcement learning approaches. These models may be deployed in simulations
or real-world environments under a common framework. We explain the structure
of the software and its key features and demonstrate how it can be used to
accelerate research. With SwarmRL, we aim to streamline research into micro-
robotic control while bridging the gap between experimental and simulation-
driven sciences. SwarmRL is available open-source on GitHub at
https://github.com/SwarmRL/SwarmRL.

_K_ eywords Active Matter, Deep Reinforcement Learning, HPC, Micro-robotics

##  1 Introduction

Mastering control of microrobots at the microscopic scale has the potential
for insights and the development of new technologies capable of changing the
world. Whether it be an improved understanding of bacterial navigation
strategies or direct control over microscopic robots, numerous fields,
including construction [Hsu et al., 2016], plant pollination and ecosystem
defense [Stefanec et al., 2022], and search and rescue [Murphy et al., 2008],
will be enhanced by advancements in micro-scale control research. One field in
particular that will see significant changes is medicine, where it is expected
that micro robotic agents will be capable of advanced treatment strategies
including targeted cancer therapies [Schmidt et al., 2020], assisted and drug
delivery [Nelson and Pané, 2023, Felfoul et al., 2016, Hosseinidoust et al.,
2016, Zhuang et al., 2015], assisted fertilisation [Medina-Sánchez et al.,
2016, Khalil et al., 2014], amongst many others [Nelson et al., 2010, Li et
al., 2022]. The two most significant challenges in realising this future are
developing the tools and materials to build such capable microscopic agents
and programming them to achieve complex tasks with minimal observed input.
These agents should not only be capable of performing complex actions such as
swimming, pushing, and perhaps, communications, but they should do so without
constant input from a supervisor. The first of these challenges has been, and
is being, extensively investigated. Currently, there are many approaches for
designing mobile micro robots [Dabbagh et al., 2022]. Initial approaches
included the light-driven Janus particles [Su et al., 2019] and coils [Wang et
al., 2019] and magnetically driven devices taking on numerous shapes [Shen et
al., 2023, Dhatt-Gauthier et al., 2023]. Recent approaches have also included
using living cells to produce motion on this microscopic scale, the Xenobots
being a prime example [Kriegman et al., 2020, Blackiston et al., 2021].
However, the control of these agents, once built and deployed, is a
complicated problem. Due to the size of these agents, directly installing
computational processing power onto them is, thus far, intractable. Therefore,
simplistic algorithms that aim to encode reactionary behaviour into the agents
are often constructed, something that would require limited on-board
processing. Examples of this include the use of microrobots to perform
chemotaxis [Hartl et al., 2021] by changing their shapes upon exposure to
varying fields, object manipulation [Kim et al., 2016] through the use of an
external magnetic field, and reproducing swarming behaviour by applying simple
rotations and translations to individual agents by a set of rules triggered by
changes in local environment [Bäuerle et al., 2020, Lavergne et al., 2019].
More recent approaches have utilised reinforcement learning approaches to
further push in the direction of learned control strategies. Qin et al. [2023]
utilised a Q-learning algorithm to learn swimming strategies in gated
microswimmers, Borra et al. [2022] used an actor-critic framework to learn
competitive capture-evasion strategies from hydrodynamic cues, and Muiños-
Landin et al. [2021] investigated actor-critic learned navigation strategies
under the influence of stochastic environments. In all cases, demonstrating
under what conditions microrobots can perform complex tasks in various
environments brings us a step closer to realising the full potential of this
technology.

It is clear that research in the direction of micro-robotic control is both
promising and excitingly multi-disciplinary; however, due to the nature of the
technology required, whether it be complex experimental equipment, deep
knowledge of state-of-the-art machine learning algorithms and their
implementation, or the expertise to construct physically realistic
simulations, the entry barrier can be high for specialists who focus on only
one of these areas. For this reason, we have developed the SwarmRL Python
package to combine each control workflow element under a common framework and
accelerate algorithm development and deployment. SwarmRL enables researchers
to apply control strategies to particles in simulation or experiments. These
strategies can be built from sets of rules that are rigidly programmed and
utilise the environment of the particles, or they can be trained using the
actor-critic reinforcement learning approach, offering complete flexibility in
how the agents are controlled, what actions they are capable of, and what
tasks they must perform. Beyond the customizability, wherever possible, it is
built on top of the JAX Bradbury et al. [2018] ecosystem to maximise
performance and is suited for deployment on large distributed compute
clusters. This paper aims to introduce the SwarmRL software and demonstrate
how it can be applied to perform frontier research. We begin with an overview
of the theory underpinning the software, explicitly discussing microscale
active matter, simulating it in a physically realistic manner, and the
reinforcement learning we have built to control it. Afterwards, the
architecture of the software is described, detailing how each piece of the
software fits together and what can be done with them. Finally, we discuss
some of the unique features of SwarmRL, including performance and
visualisation capabilities, before concluding with an outlook of the project.
With SwarmRL, we hope to enable scientists from diverse backgrounds to
contribute to and develop the field of micro-robotics research and realise the
potential of this technology.

##  2 Theory

SwarmRL harnesses tools from a range of fields, the largest of which are
active matter simulation and reinforcement learning. In order to understand
and better utilize the software infrastructure, it is important to also have
an understanding of the theory it implements. In this section, we introduce
the important theoretical concepts on which SwarmRL is built, before exploring
the architecture of the package.

###  2.1 Active Matter

#### Biological and artificial active systems

Active matter refers to systems that consume energy from their surroundings
and are thus internally driven out of equilibrium [Romanczuk et al., 2012].
All biological and human-made machines fall under this term. However, in the
context of this work, we focus on the branch of active systems that consists
of small active particles that use energy to perform persistent, directed
motion. This much more narrow definition covers plenty of examples in biology
and engineering. Many bacteria like Escherichia coli [Wadhwa and Berg, 2022],
archaea like Thermoplasma volcanium [Jarrell et al., 2021] and small
eukaryotes like Chlamydomonas reinhardtii [Silflow and Lefebvre, 2001] use
molecular motors and organelles like archaella, flagella or pili to self-
propel. Artificial microswimmers can be made from colloidal particles that
use, e.g., catalytic reactions [Howse et al., 2007], self-generated
temperature gradients [Jiang et al., 2010] or magnetic fields [Mandal et al.,
2018] to generate propulsion. In some more rare cases, the active particles
can self-propel and self-steer, i.e., use torque to actively change their
direction. For bacteria, tumbling [Berg and Brown, 1972, Darnton et al., 2007]
is a well-known mechanism for change of direction. Some artificial
microswimmers can also be steered by, e.g., change in laser focus [Bäuerle et
al., 2020] or magnetic field direction [Carlsen et al., 2014]. In all these
examples, the active particle is of micrometer size and suspended in a liquid,
making translational and rotational Brownian motion a non-negligible factor
that competes with the deterministic active motion.

#### Simulation

Active matter systems with self-propelled small particles are usually modelled
within the framework of stochastic dynamics of the individual constituents. We
use the two- or three dimensional overdamped Langevin equations

$\dot{\mathbf{r}}_{i}(t)=\gamma_{t}^{-1}\left[F^{\text{act}}_{i}(t)\mathbf{\hat{e}}_{i}(t)+\mathbf{F}_{i}(\mathbf{r}_{i},\quantity{\mathbf{r}_{j}})\right]+\sqrt{2k_{\text{B}}T\gamma_{t}^{-1}}\bm{\xi}^{t}_{i}(t),$
(1)
$\dot{\leavevmode\hbox{\set@color$\mathbf{\hat{e}}$}}_{i}(t)=\left[\gamma_{r}^{-1}\left[M^{\text{act}}_{i}(t)\mathbf{\hat{n}}_{i}(t)+\mathbf{M}(\mathbf{r}_{i},\quantity{\mathbf{r}_{j}})\right]+\sqrt{2k_{\text{B}}T\gamma_{r}^{-1}}\bm{\xi}^{r}_{i}(t)\right]\crossproduct\mathbf{\hat{e}}_{i}(t),$
(2)

where $\mathbf{r}_{i}$ denotes the position of particle $i$, $\gamma_{t(r)}$
the translational (rotational) friction coefficient, $F^{\text{act}}$ the
active driving force that models self-propulsion, $\mathbf{\hat{e}}$ the unit
vector representing the direction of the particle,
$\mathbf{F}_{i}(\mathbf{r}_{i},\quantity{\mathbf{r}_{j}})$ a conservative
force from interactions between particle $i$ and its environment as well as
from interactions with other particles $j$, $k_{\text{B}}$ the Boltzmann
constant, $T$ the absolute temperature, $\bm{\xi}^{t,(r)}$ the translational
(rotational) noise with $\expectationvalue{\bm{\xi}^{t,(r)}_{i}(t)}=0$ and
$\expectationvalue{\bm{\xi}^{t,(r)}_{i}(t)\otimes\bm{\xi}^{t,(r)}_{j}(t^{\prime})}=\delta_{ij}\delta(t-t^{\prime})\mathbf{1}$,
where $\expectationvalue{\cdot}$ denotes an ensemble average (expectation
value) and $\mathbf{1}$ the unity matrix, $M^{\text{act}}$ an active steering
torque, $\mathbf{\hat{n}}$ a unit vector perpendicular to $\mathbf{\hat{e}}$
that sets the steering direction and
$\mathbf{M}_{i}(\mathbf{r}_{i},\quantity{\mathbf{r}_{j}})$ an interaction
torque analogous to $\mathbf{F}_{i}$. We stress the time dependence of
$F^{\text{act}}$ and $M^{\text{act}}$ because SwarmRL is designed to focus on
control and decision-making on the level of single particles. Collective
behavior and task fulfillment are to be achieved by the particles’ actions and
what they can control rather than by external fields acting on all particles.

###  2.2 Reinforcement Learning

As a sub-field of ML, RL is concerned with how intelligent agents can take
actions in an environment to maximize a cumulative reward. It allows these
agents to learn optimal protocols to achieve a given task or a set of tasks
without explicitly telling them how. This is oftentimes useful when a task is
too complex for a simple algorithm or control system. It can also be used when
the focus of the investigation is the emergence of a strategy for a problem.
RL is, however, data-inefficient and is thus used mostly for problems where
data exists abundantly and where the action space is relatively small such as
autonomous driving [Kiran et al., 2021], video games [Mnih et al., 2015], and
robotics [Ibarz et al., 2021]. It reached great popularity in 2016 when
Deepmind’s AlphaGo won against the top Go player Lee Sedol [Silver et al.,
2016]. When using it for specific tasks, however, RL can achieve remarkable
results. In 2022, it solved a 50-year-old open question in mathematics how to
most efficiently use matrix multiplication [Fawzi et al., 2022]; this
discovery likely improves the efficiency of machine learning significantly due
to its widespread use of matrix multiplication. It is further increasingly
used in Physics research for sophisticated tasks such as quantum error
correction [Zeng et al., 2023] and the magnetic control of tokamak plasmas
[Degrave et al., 2022]. This section provides an introduction to RL and
explains the training approach currently implemented in SwarmRL, namely, deep
Actor-Critic (AC).

####  2.2.1 Actor-Critic Reinforcement Learning

The RL training paradigm is concerned with iteratively improving an agent’s
decision-making such that it can achieve a pre-defined task as efficiently as
possible. In practice, this is performed by placing agents into an environment
with which they can interact by performing actions, $a_{t}$. To decide on such
an action, the agent receives a state description, $s_{t}$, that describes the
environment with sufficient information. The model used to decide on the
action is referred to as the policy, $\pi_{\theta}:s_{t}\rightarrow a_{t}$,
often taken to be a neural network with parameters, $\theta$ that is then
referred to as an actor. These actors typically produce a probability
distribution, which can then be sampled to select an action given the state.
After an action is taken, a new state is produced along with a reward,
quantifying the agent’s progress towards achieving its task, a process
outlined in Figure 1.

Figure 1: Graphical overview of the RL workflow. The agent selects an action,
$a_{t}$, based in its current state, $s_{t}$, and acts within the environment.
This action yields a new state, $s_{t+1}$ and an associated reward, $r_{t+1}$.

Fundamentally, the goal of training in RL is to find the optimal policy that,
throughout a trajectory, $\tau$, maximises a cost function, $J(\pi_{\theta})$,
i.e.,

$\pi^{*}=\operatorname*{arg\,max}_{\pi}J(\pi_{\theta})=\operatorname*{arg\,max}_{\pi}\int_{\tau}P\left(\tau|\pi_{\theta}\right)\cdot
R(\tau)=\operatorname*{arg\,max}_{\pi}\left\langle
R(\tau)\right\rangle_{\tau\sim\pi_{\theta}},$ (3)

where $R(\tau)$ is the cumulative reward computed from a trajectory,
$P(\tau|\pi_{\theta})$ is the probability of a trajectory given the current
policy, and the angled brackets denote an average over many trajectories
sampled under the $\pi_{\theta}$ policy. This is achieved by encouraging the
policy to increase the probability of selecting actions that returned good
rewards and discouraging those that didn’t. In practice, training is done
throughout discrete-time episodes where state, policy probabilities, and
reward data are collected and used to update the policy via the gradient
ascent algorithm

$\theta^{{}^{\prime}}=\theta+\eta\cdot\nabla_{\theta}J(\pi_{\theta}),$ (4)

with learning rate $\eta$, and

$\nabla_{\theta}J=\left\langle\sum\limits_{t}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\cdot
r_{t}\right\rangle_{\tau\sim\pi_{\theta}}$ (5)

with time step, $t$, and instantaneous reward, $r_{t}$. Typically, to ensure
convergence in the long-term rewards and to encourage near-term over long-term
progress, an expected returns function, $G_{t}$, will be used in the form

$G_{t}=\sum\limits_{t^{{}^{\prime}}=t}^{T}\gamma^{t^{{}^{\prime}}-t}r_{t^{{}^{\prime}}},$
(6)

where $T$ is the final time step of the episode, and $\gamma$ is a discount
factor set as a hyperparameter during training influencing the importance of
late-time rewards. While this approach can be sufficient in training agents,
it can break down in stochastic environments and high-dimensional spaces
[Nachum et al., 2017]. To address this, Sutton and Barto [2018] showed that
introducing a trainable baseline or value function, $V^{\pi}$ to compute an
advantage function

$A_{t}^{\pi}=G_{t}-V^{\pi}_{t},$ (7)

can result in the stabilization of the expected returns and improved
convergence time in the training, resulting in the reformulation of Equation 5
to

$\nabla_{\theta}J=\left\langle\sum\limits_{t}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\cdot
A^{\pi}_{t}\right\rangle_{\tau\sim\pi_{\theta}}.$ (8)

When this value function is implemented as a neural network, it is often
referred to as a critic, thus actor-critic reinforcement learning. The
critic’s role is to determine the value of the state in which the agent
currently finds itself. While several definitions exist, in SwarmRL, the value
is defined as the theoretical returns possible if one starts in a state,
$s_{t}$ and follows policy $\pi_{\theta}$. It improves this prediction by
training on the true expected returns, Equation 6, computed during episodes
through a desired regression algorithm. Therefore, in Equation 8, the
advantage function guides training by encouraging selected actions that are
better than or equal to those expected from the value function and
discouraging those considered below what is theoretically possible. For this
reason, AC RL training is often considered adversarial in nature, with the
actor learning to beat the critic and the critic, in turn, learning the
correct value of the encountered states. The training procedure outlined here
is referred to as Vanilla Policy Gradient due to it being the simplest form of
actor-critic training. More complex methods extend on the method discussed
here, such as trust region policy optimisation [Schulman et al., 2017a] and
proximal policy optimisation [Schulman et al., 2017b], the latter of which is
also implemented in SwarmRL.

####  2.2.2 Multi-Agent Reinforcement Learning

As SwarmRL specializes in controlling microscopic agents, there will be many
occasions in which multiple agents must be simultaneously controlled and
trained. This is addressed by multi-agent reinforcement learning (MARL) [Zhang
et al., 2021]. In MARL, $N$ autonomous agents interact with the environment
and each other. This extension brings several new aspects to the problem, such
as cooperation and competition among agents and various information
structures. In the following, we will address these aspects briefly.

##### Cooperative, Competitive and Mixed Setting

In a fully cooperative setting, all agents share a common reward, i.e.,
$r^{1}=r^{2}=...=r^{N}=r$ or a team-average reward where
$r=\frac{1}{N}\sum_{i\in N}r_{i}$ [Zhang et al., 2018]. The latter is a
generalized version of the former. Different reward functions for each agent
allow more heterogeneity among agents [Zhang et al., 2021]. A fully
competitive setting in MARL is typically modelled as a zero-sum Markov game
with $\sum_{i\in N}r^{i}=0$ [Littman, 1994]. Most literature focuses on two
players where one agent’s reward is the other’s loss, something leveraged in,
for example, predator-prey style games. A mixed setting imposes no
restrictions on the goal and the relationship among agents [Hu and Wellman,
2003].

##### Information Structures

Another critical challenge in MARL is the question who knows what during
training and execution. This involves both the decision-making process as well
as the optimization process. Three different forms of information structure
are generally accepted. The first is the centralized setting with a so-called
central controller aggregating joint observations, actions, and rewards and
allowing for centralized learning. The execution, on the other hand, can
either be centralized or decentralized. A centralized executed policy produces
a joint action for all agents, while decentralized produces individually
executed policies. The latter is the popular learning scheme of centralized-
learning-decentralized-execution [Oliehoek et al., 2016, Hansen et al., 2004].
On the other side, there is the fully decentralized setting in which each
agent acts based only on local observation. This setting results in the so-
called independent learning scheme [Tan, 1993]. A combination of both is a
decentralized setting with networked agents [Zhang et al., 2021]. This allows
agents to share local information with their neighbours. The choice of setting
depends on the problem at hand.

##  3 Architecture

SwarmRL has been designed to lower the entry barrier for scientists to access
state-of-the-art reinforcement learning algorithms efficiently. It also offers
a wide degree of customization and choice of algorithms that can be used
during experiments to maximise the experienced user’s flexibility. To achieve
both of these goals, SwarmRL is constructed using a modular, object oriented
programming structure where almost all pipeline components are written as
implementations of an abstract parent that defines the interface through which
the components communicate. The Python programming language has been used
throughout as it is most common amongst scientific disciplines, however,
whenever possible, libraries have been used to improve performance through
vectorized maps (vmaps), multi-threading, GPU use, and Just In Time (JIT)
compiling [Bradbury et al., 2018]. A coarse-grained flowchart of SwarmRL is
presented in Figure 2 where each major component of the architecture is shown
with its corresponding connections. The structure of SwarmRL is built around
the interaction between the Engine, whether that be simulation or real
experiment, and the Agents, the algorithms used to control the particles in
the simulation. Connecting the Agents to the Engine is the Force Function.
Force Functions are either constructed directly, or through a Trainer in the
cases where trainable Agents are in use. Finally, the Agent holds Tasks,
Actions, and Observables, each of which provide information and functionality
to the algorithm being used. The rest of this section is dedicated to
explaining each important component of SwarmRL, where they fit into the
different capabilities of the software, and how they can be adapted or
extended for individual implementations. It should be noted that, to reduce
space and improve readability, all comments, type hints, and methods not
directly related to the discussion have been removed from code samples.

Figure 2: Overview of the SwarmRL software architecture. Light green boxes on
the right-hand side of the figure represent modules with default settings that
can be adjusted by users but do not need to be, dark green boxes indicate
settings that need to be included in a system definition, blue boxes
correspond to those directly handling colloid intelligence and properties, the
engine is in orange, and grey represents classes that talk to the engine.

###  3.1 Engine

The Engine class of SwarmRL is one of its most important components. It
represents the environment of the Agents and propagates the state according to
the given actions. SwarmRL ships with an engine connected to the ESPResSo
simulation software [Weik et al., 2019] to solve Equations (1) and (2)
numerically. Through ESPResSo, we also offer the underdamped Langevin equation
and a coupling to explicit hydrodynamics with lattice Boltzmann as the solver
of the Navier-Stokes equations. SwarmRL also ships with rudimentary examples
for connecting it directly to an experiment. Direct experimental connections
are often kept private out of respect to the institutions that host the
experiments. However, most implementations thus far use the TCP communications
protocol to send actions and receive state information to hardware. Future
work on SwarmRL aims to unify this communication procedure using the ROS
ecosystem [Macenski et al., 2022]. In cases where users must connect their own
engines, be it simulation or experiment, they inherit from the Engine parent
class. They must implement the two methods outline in Code Sample
LABEL:lst:engine.

⬇

1class Engine:

2

3 def integrate(

4 self, n_slices, force_model,

5 ):

6 raise NotImplementedError

7

8 def get_particle_data(self):

9 raise NotImplementedError

Code Sample 1: SwarmRL Engine parent class.

The integrate method is used to step forward either in simulation or in an
experiment. It takes as arguments the force model, which holds the control
algorithms for the particles in the experiment, see section 3.5, and number of
slices to be looped over, that is, the number of times the force model is
called for new actions to be applied to the particles. In a simulation, the
time in-between successive time slices is filled with integrator steps,
applying the chosen action at each time-step. In contrast, in experiment, this
would be applying the actions for a fixed period between slices. When
instantiated, these parameters, time-step or time between actions, are given
to the specific engine. The second method that must be implemented is
get_particle_data. This method must collect information about the particles in
the system and give it back to the user uniformly so that all parts of SwarmRL
can work with them. In its current state, SwarmRL defines colloids with
properties outlined in Code Sample LABEL:lst:colloid.

⬇

1class Colloid:

2 pos

3 director

4 id

5 velocity

6 type

Code Sample 2: SwarmRL Colloid dataclass.

Here, all information relevant to the models is assigned to a colloid class
and used to predict new actions and to compute rewards.

###  3.2 Agents

SwarmRL was built to service a variety of simulations and RL studies. A large
component in enabling this, is the Agent class, which manages how agents in
the simulations are controlled, what they see, and what they are tasked with
doing. The Agent collects the components introduced in the previous sections
and represents the full set of sensing, control, and objectives of the
colloids in question. This information can then be used by SwarmRL to
effectively deploy these agents in the simulation environment. Code samples
LABEL:lst:ac-agent and LABEL:lst:class-agent demonstrate the construction of
two different agents which could both be added to a single simulation within
SwarmRL.

⬇

1ac_agent = srl.agents.ActorCriticAgent(

2 agent_type=0,

3 network=srl.networks.FlaxNetwork(...),

4 task=srl.tasks.RotateRod(),

5 observable=srl.observables.VisionCones(),

6 actions=action_dict

7)

Code Sample 3: Actor-Critic Agent

⬇

1ac_agent = srl.agents.Lymburn(

2 agent_type=1,

3 homing_location=[500., 500., 500.]

4)

Code Sample 4: Classical Agent

In the above examples, the actor-critic reinforcement learning approach
controls one group of particles. This means their policy can be trained and
adapted to new environments. The other example uses a so-called
ClassicalAgent, in particular, one based on the swarming agents of [Lymburn et
al., 2021].

###  3.3 Classical Control

SwarmRL, allows users to control the actions of the particles in the system
using both RL and classical algorithms. Classical algorithms, in this work,
can be implemented without training and typically follow a defined set of
rules. Such sets of rules have often been studied in the literature to
reproduce behaviour in microswimmers [Bäuerle et al., 2020, Lavergne et al.,
2019]. Classical agents are implemented directly into the Agent child class on
the software side. However, they must follow the same structure, namely, at
each time slice, the information of all particles in the system is passed into
the class and in turn, it must return an action for each particle. In its
current state, SwarmRL offers classical algorithms for the published models of
Lymburn et al. [2021], Bäuerle et al. [2020], and Lavergne et al. [2019].
Future work aims at including Model Predictive Control (MPC) algorithms into
the SwarmRL ecosystem.

###  3.4 Machine Learning Control

The machine learning side of agent control is driven by multi-agent
reinforcement learning. However, due to the complex nature of this approach,
several design decision have been made to balance flexibility with limiting
complexity. The main workhorse of the machine learning driven control is the
Network class, which requires two methods to be implemented, compute_action
which computes the action of the agent/s under study, and __call__, which
returns action probabilities and, in the case of actor-critic approaches, also
the predicted state values.

#### Architecture

SwarmRL currently utilizes the Flax neural network library [Heek et al., 2023]
exclusively. In order to provide an interface for complex architectures,
SwarmRL requires only a single Flax module to be passed to the FlaxNetwork
class. This module must return both the action probabilities, and the
predicted value function in case of actor-critic training. This approach
allows implementing very complex neural network architectures such as graph
models, with the same API interface as simple dense actors and critics. An
example of two commonly used approaches is listed in Code Samples
LABEL:lst:disjoint and LABEL:lst:joint.

⬇

1class DisjointActorCritic:

2

3 def __call__(self, x):

4 # Perform the actor forward pass.

5 actor_values = Dense(12)(x)

6 actor_values = relu(actor_values)

7 actor_output = Dense(4)(actor_values)

8

9 # Perform the critic forward pass.

10 critic_values = Dense(12)(x)

11 critic_values = relu(critic_values)

12 critic_output = Dense(1)(critic_values)

13

14 return actor_output, critic_output

Code Sample 5: Disjoint actor-critic

⬇

1class CombinedActorCritic:

2

3 def __call__(self, x):

4 # Perform the shared layer computation.

5 shared_layer = Dense(12)(x)

6 shared_layer = relu(shared_layer)

7

8 # Compute the actor output.

9 actor_output = Dense(4)(shared_layer)

10

11 # Compute the critic output.

12 critic_output = Dense(1)(shared_layer)

13

14 return actor_output, critic_output

Code Sample 6: Combined actor-critic

These code blocks show a disjoint and combined actor critic approach bundled
into the same Flax network structure. In this way, many complex neural network
architectures can be constructed under a single module including graph or
transformer modules.

Apart from the architecture, several components are essential to training AC
models: the policy update algorithm, the approach used in the expectation
computation, and the strategy for sampling actions from the learned
distribution.

Policy Update | Vanilla Gradient Update [Sutton et al., 1999]
---|---
Proximal Policy Optimization [Schulman et al., 2017b]
Returns | Expected Returns
Generalized Advantage Estimation [Schulman et al., 2018]
Sampling Strategy | Categorical Distribution
Gumbel Trick [Gumbel, 1954]
Table 1: Overview of RL algorithm options available in SwarmRL

#### Actions

Actions in SwarmRL are dictated by dataclasses which can be read by an engine
and turned into forces, torques, or other values in a simulation or
experiment. At the time of this report, SwarmRL has been focused on micro-
robotics with standard direction control, therefore, the class is built from a
simple set of attributes outlined in Code Sample LABEL:lst:action.

⬇

1class Action:

2 force = 0.0

3 torque= np.zeros((3,))

4 new_direction = None

Code Sample 7: SwarmRL Action dataclass.

Force is applied along the agent’s director axis, torque describes the agent’s
rotation, and new_direction provides a sometimes simpler approach to a torque
calculation. From this template, we create an action by specifying the class’s
force, torque, and new direction attributes and add them to a dictionary to be
passed to an agent. For example, if the agent should be capable of translation
forwards, translation backwards, clockwise, and counterclockwise rotation, the
set of actions can be passed to the Agent as in Code Sample
LABEL:lst:actionex.

⬇

1agent_action_dictionary = {

2 "Forwards": Action(force=10.0),

3 "Backwards": Action(force=-10.0),

4 "CCW": Action(torque=np.array(0., 0., 10.))

5 "CW": Action(torque=np.array(0., 0., -10.))

6}

Code Sample 8: Example Action dictionary.

###  3.5 Force Function

The interaction model in SwarmRL is responsible for consuming data from the
simulation and sending back computed actions for each of the agents. SwarmRL
can control agents using machine learning and standard control theory
approaches. The parent class for the models requires only a single function to
be called by the Engine: calc_action, which takes a list of agent objects as
input and returns a list of actions. This is the only occasion in which the
environment communicates with the control algorithms and the communication
must adhere to this very ’narrow’ interface. This design choice enables novice
SwarmRL users to switch out environments without adapting their RL setup and
experienced users to easily implement new environments according to their
needs.

Agent information is passed through as a dictionary when creating a model,
demonstrated in Code Sample LABEL:lst:model.

⬇

1interaction_model = ForceFunction(

2 agents={

3 "0": ActorCriticAgent,

4 "1": ClassicalAgent

5 }

6)

Code Sample 9: Example ForceFunction instantiation for a classical and actor-
critic RL agent.

In the case above, the experiment has at least two types of particles. One is
controlled using an actor-critic reinforcement learning approach while the
other takes actions computed by a classical, non-trainable algorithm. In this
way, SwarmRL users have complete control over the agents in their simulations
and experiments.

###  3.6 Tasks

As the goal of SwarmRL is to be used for the study of micro-scale robotic or
biological systems to learn about their behaviour and control, it is often
important to measure how well the system is doing. In RL, this is often
referred to as the reward which is maximised during training. For classical
algorithms, a task can also be used as a measure of how well the approach is
doing or passed into the model as feedback. In SwarmRL, system performance
measurement is handled by the Task class. The parent class of the task is
built up of several methods that need to be implemented in the case of a
custom task. The reduced class architecture is displayed in Code Sample
LABEL:lst:task.

⬇

1class Task:

2 @property

3 def kill_switch(self):

4 pass

5 def __call__(self, colloids):

6 raise NotImplementedError("Implemented in child class.")

Code Sample 10: SwarmRL Task parent class.

The above class contains two fundamental components, the kill switch and the
call method. As its name suggests, the kill switch is used to end the current
run. If this property is set to True, SwarmRL will stop the current running
engine and either restart it, or just finish. This is used, for example, in
episodic training, where the training should stop and the engine should reset
after an amount of time or when a criterion is reached. However, the kill
switch can also be used for more critical cases, such as in an experiment when
it appears that the agents would perform damaging actions. The second
component, the call function, is used to compute a value describing the
current state of this task, ideally, large positive values mean the task is
being achieved, whereas negative values mean it is actively not being
achieved. For example, when training RL agents to rotate an object, the call
method would compute the rotation speed and return some positive value if it
was increasing or large, and a negative value if it were decreasing.

At times, we want agents to perform more than just one task, in these cases,
two options exist. The first is to write a custom task that provides feedback
on two different objects such as how fast something is rotating and how far it
has been pushed to a desired location. However, these tasks often quickly
become convoluted and difficult to reuse. Therefore, SwarmRL provides the
Multitasking class which takes a list of single tasks as an argument and
applies them to the engine. This approach is more modular and allows for
faster testing of strategies.

###  3.7 Observables

A critical component of any RL algorithm and an important part of classical
agent control approaches is their state description. Rather than passing all
state information (e.g., all positions and directions of the agents) to the
decision making model, the observable usually selects from or condenses the
information considerably. This mimics the limited sensing capabilities that
agents might have. For example, agents might only be able to retrieve
information about other particles within a finite sensing radius or might not
be able to tell the direction of their neighbours. Investigating the amount of
information needed for an agent to still be able to perform its task is very
relevant for the design of cost-efficient micro agents. In SwarmRL, the
description of the environment as sensed by the agents is computed by
Observables. All observables implemented in SwarmRL inherit from a simple
parent class outlined in Code Sample LABEL:lst:observable.

⬇

1class Observable:

2 def compute_observable(self, colloids):

3 raise NotImplementedError("Implemented in child class.")

Code Sample 11: SwarmRL Observable parent class.

The class requires implementing one method, compute_observable, which takes a
list of particle objects as input and returns the state description vector for
each of them. This state vector is then passed on to the models by the force
function. As with the Task class, we often want to construct agents capable of
sensing the world in multiple ways. To achieve this, we also provide a
MultiSensing class which takes a list of observables and computes a single
output.

###  3.8 Intrinsic Reward

In RL agents can be rewarded intrinsically to either encourage the exploration
behavior of an agent or to assist in building its knowledge about the
environment [Pathak et al., 2017]. As the name suggests, such a method is
intrinsic to the agent and is therefore implemented as a separate sub-module
IntrinsicReward which the agent can be equipped with.

#### Random Network Distillation

Random network distillation (RND) was introduced in 2018 by Burda et al.
[2018] as an efficient method for exploring the state space of reinforcement
learning agents. It has since been studied more broadly for its application
and data selection for ML applications [Finkbeiner et al., 2023] and its
implications on understanding data requirements for neural networks [Tovey et
al., 2023].
SwarmRL provides RNDReward, a module implementing RND, defining an intrinsic
reward for agents, encouraging them to explore unseen regions of their state
space. Going beyond the original version from Burda et al. [2018], SwarmRL
provides two implementations of RND, differing in their capability to memorize
previous states. For both versions of RND, a default setup is provided in
swarmrl/intrinsic_reward/rnd_configs.py. Utilizing such configuration the
implementation of RND is shown in Code Sample LABEL:lst:intrinsic_reward_rnd.

⬇

1rnd_config = RNDConfig(...)

2

3ac_agent = srl.agents.ActorCriticAgent(

4 ...,

5 intrinsic_reward=RNDReward(rnd_config)

6)

Code Sample 12: Random Network Distillation as Intrinsic Reward

The implementation of the RND backend is based on the ZnNL python package
[Nikolaou and Tovey, 2021], offering a flexible but simple interface for
complex training algorithms for neural networks.

###  3.9 Exploration Policy

In addition to the sampling strategy which can select actions from its
distributions that do not have the largest probability, SwarmRL also provides
direct access to exploration policies to further guide the agents into new,
unexplored regions of their environment.

#### Random Exploration

The simplest form of exploration policy is the RandomExploration class, which
an exploration probability can parameterize, $\zeta$ and decay rate,
$\epsilon$. The exploration rate dictates how likely it is for an agent to
take an exploration action over a policy action, and the decay rate slowly
decreases this probability as the training evolves by

$\zeta^{{}^{\prime}}=\exp^{-\epsilon\frac{t}{T}}\zeta,$ (9)

where $t$ is simulation time, and $T$ is the time for a single training
episode.

###  3.10 Trainer

The trainer is module in SwarmRL responsible for handling the updates of the
models and their interaction with the environment while developing a policy.
In the simplest case, the trainer deploys the models in the environment,
collects the rewards over an episode, and performs a network update, as
outlined in Code Sample LABEL:lst:cont-trainers. However, in many cases more
is demanded during RL training and even how rewards are collected can change
depending on the problem. Therefore, in SwarmRL, we have implemented several
variations of the Trainer.

#### Continuous vs Episodic

A user must first choose whether to perform episodic or continuous training.
In episodic training, agents are deployed in the environment for a fixed
number of steps or until some condition is met. After this time, a reward is
computed based on their final state or trajectory during the episode. This
reward is then used to update the networks before the environment is reset and
the process is started again. In this approach, the episodes should be long
enough to allow the agents to achieve their tasks but short enough to maximize
efficiency. A similar but alternative approach is semi-episodic training,
wherein the agents are updated continuously during the training but after some
time, or again, once a condition is met, the environment is reset and the
training continues. These options are available through the EpisodicTrainer
packaged into SwarmRL. The EpisodicTrainer, displayed in Code Samples
LABEL:lst:cont-trainers and LABEL:lst:ep-trainers, allows users to define how
often the environment should be reset as a function of RL-episodes, or to pass
a task which can force an environment reset.

⬇

1rl_trainer = srl.trainers.ContinuousTrainer(

2 [protocol_1, protocol_2],

3 loss,

4)

5rl_trainer.perform_rl_training(

6 system_runner=srl.Engine(...),

7 n_episodes=5000,

8 episode_length=50,

9)

Code Sample 13: Continuous Training

⬇

1rl_trainer = srl.trainers.EpisodicTrainer(

2 [protocol_1, protocol_2],

3 loss,

4 )

5rl_trainer.perform_rl_training(

6 get_engine=engine_fn,

7 n_episodes=10,

8 reset_frequency=30, # > 1 is semi-episodic

9 episode_length=100,

10)

Code Sample 14: Episodic Training

Continuous training is a paradigm in which agents are deployed into an
environment and are updated after a fixed amount of time without resetting the
environment. This online training is useful for situations where training must
occur in the real world or where the time it will take to achieve a task is
not well known. SwarmRL offers the ContinuousTrainer class to perform this
kind of training. It should be noted that if a Task is provided to the
continuous trainer capable of resetting the environment, the training will
end. This can also be helpful in cases where an end to training can be
strictly identified.

##  4 Performance

Reinforcement learning typically does not require expensive neural networks,
and classical control algorithms are limited in complexity by the number of
particles they control. However, speed becomes critical when considering the
number of iterations they have to go through during training and deployment.
Therefore, SwarmRL has been built with a performance-first mentality,
leveraging the JAX ecosystem [Heek et al., 2023] heavily. JAX is a Python
library that, among other things, re-implements the NumPy library [Harris et
al., 2020] with multi-threading, GPU deployment, and XLA-enhanced JIT compile
capabilities. Most variables in SwarmRL are written using JAX arrays or their
PyTree counterparts. PyTrees extend the functionality JAX has for NumPy
objects to data structures like classes. Therefore, most of the class objects
including the Colloid class can be placed onto a GPU or parallelized over.
Furthermore, large parts of code are JIT compiled to ensure optimal
performance.

##  5 Software Practices

In order to ensure the longevity and continued usability of the SwarmRL
software, we adhere to several software practices. SwarmRL is thoroughly
tested with unit, integration, and functional tests in place. Overall, we have
87 % coverage on the code with all classes having at least one unit test. The
code follows the NumPy doc-string format and is extensively documented which
is hosted at https://swarmrl.github.io/SwarmRL.ai/. Before new code is added
to the project, all tests must pass and new tests should be included. The
project is hosted on GitHub where all these standards are enforced
automatically. Furthermore, we require at least one review from a code-owner,
that is, a member of the core SwarmRL team, to accept new changes to the
software.

##  6 Visualization

Visualization is an essential part of the scientific process, particularly in
simulations where one is searching for emergent strategy. It is not always
clear from plots alone if the agents have learned what they should do. For
this reason, SwarmRL interfaces closely with the ZnVis [Tovey, 2021]
visualization library. ZnVis, by design, handles the output data from SwarmRL
and is developed alongside the project. The package can use custom mesh files
for the agents, export rendered videos, and capture still-frame renderings
using the Mitsuba engine [Jakob et al., 2022]. Figure 3 illustrates the
visualization capabilities of the package used on SwarmRL experiments.

Figure 3: Example renderings using SwarmRL and the ZnVis visualization engine.
(left) A snapshot from a reservoir computing experiment using a reinforcement
learning controlled swarm. (center) Colloids are trained to perform chemotaxis
in the same fashion as bacteria. Here an open-source bacteria 3d model file
has been used inside of SwarmRL to convey realism. The mesh was created by
Sketchfab artist andrewfrueh, whose explicit permission was requested and
granted before being used in this publication. The model is licensed under
creative commons 4.0. (right) Demonstration of colloids rotating a rod,
controlled using an RL algorithm.

##  7 Conclusion

We have introduced SwarmRL, a Python package aimed at accelerating research
into control and understanding of microscopic active particles, whether
biological or artificial. SwarmRL allows researchers to implement state-of-the
art control algorithms driven by reinforcement learning or classical policies,
in both simulated and experiment settings. As a Python package, SwarmRL has a
familiar interface for many scientists, and is simple enough to use with
limited background in software. Beyond its functionality, we have built
SwarmRL to be performant, capable of running on many cores, GPUs, and scalable
to HPC clusters. Our current focus is extending SwarmRL to handle more control
algorithms, e.g., MPC, Q-learning, additional experiment implementations, and
additional engines. Overall, our goal is that SwarmRL allow scientists to
spend more time creating new ideas, building new technology, and changing the
field, rather than implementing and staying up to date with state of-of-the
art machine learning methods.

##  8 Acknowledgements

C.H and S.T acknowledge financial support from the German Funding Agency
(Deutsche Forschungsgemeinschaft DFG) under Germany’s Excellence Strategy EXC
2075-390740016. C.H and S.T acknowledge financial support from the German
Funding Agency (Deutsche Forschungsgemeinschaft DFG) under the Priority
Program SPP 2363, “Utilization and Development of Machine Learning for
Molecular Applications - Molecular Machine Learning” Project No. 497249646.
C.H and C.L acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) under Project Number 327154368-SFB 1313. The
authors acknowledge support by the state of Baden-Württemberg through bwHPC
and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG.
The authors acknowledge support from the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) Compute Cluster grant no. 492175459. S.T would
like to acknowledge Annalena Daniels for her detailed review and comments on
the manuscript.

## References

  * Bäuerle et al. [2020] T. Bäuerle, R. C. Löffler, and C. Bechinger.  Formation of stable and responsive collective states in suspensions of active colloids.  _Nature Communications_ , 11(1):2547, May 2020\.  ISSN 2041-1723.  doi: 10.1038/s41467-020-16161-4.  URL https://doi.org/10.1038/s41467-020-16161-4.
  * Berg and Brown [1972] H. C. Berg and D. A. Brown.  Chemotaxis in escherichia coli analysed by three-dimensional tracking.  _Nature_ , 239(5374):500–504, 1972.
  * Blackiston et al. [2021] D. Blackiston, E. Lederer, S. Kriegman, S. Garnier, J. Bongard, and M. Levin.  A cellular platform for the development of synthetic living machines.  _Science Robotics_ , 6(52):eabf1571, 2021.  doi: 10.1126/scirobotics.abf1571.  URL https://www.science.org/doi/abs/10.1126/scirobotics.abf1571.
  * Borra et al. [2022] F. Borra, L. Biferale, M. Cencini, and A. Celani.  Reinforcement learning for pursuit and evasion of microswimmers at low reynolds number.  _Phys. Rev. Fluids_ , 7:023103, Feb 2022.  doi: 10.1103/PhysRevFluids.7.023103.  URL https://link.aps.org/doi/10.1103/PhysRevFluids.7.023103.
  * Bradbury et al. [2018] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Burda et al. [2018] Y. Burda, H. Edwards, A. Storkey, and O. Klimov.  Exploration by random network distillation, 2018.
  * Carlsen et al. [2014] R. W. Carlsen, M. R. Edwards, J. Zhuang, C. Pacoret, and M. Sitti.  Magnetic steering control of multi-cellular bio-hybrid microswimmers.  _Lab on a Chip_ , 14(19):3850–3859, 2014.
  * Dabbagh et al. [2022] S. R. Dabbagh, M. R. Sarabi, M. T. Birtek, S. Seyfi, M. Sitti, and S. Tasoglu.  3d-printed microrobots from design to translation.  _Nature Communications_ , 13(1):5875, Oct 2022\.  ISSN 2041-1723.  doi: 10.1038/s41467-022-33409-3.  URL https://doi.org/10.1038/s41467-022-33409-3.
  * Darnton et al. [2007] N. C. Darnton, L. Turner, S. Rojevsky, and H. C. Berg.  On torque and tumbling in swimming escherichia coli.  _Journal of bacteriology_ , 189(5):1756–1764, 2007.
  * Degrave et al. [2022] J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al.  Magnetic control of tokamak plasmas through deep reinforcement learning.  _Nature_ , 602(7897):414–419, 2022.
  * Dhatt-Gauthier et al. [2023] K. Dhatt-Gauthier, D. Livitz, Y. Wu, and K. J. M. Bishop.  Accelerating the design of self-guided microrobots in time-varying magnetic fields.  _JACS Au_ , 3(3):611–627, 2023.  doi: 10.1021/jacsau.2c00499.  URL https://doi.org/10.1021/jacsau.2c00499.
  * Fawzi et al. [2022] A. Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli.  Discovering faster matrix multiplication algorithms with reinforcement learning.  _Nature_ , 610(7930):47–53, 2022.  doi: 10.1038/s41586-022-05172-4.  URL https://doi.org/10.1038/s41586-022-05172-4.
  * Felfoul et al. [2016] O. Felfoul, M. Mohammadi, S. Taherkhani, D. de Lanauze, Y. Zhong Xu, D. Loghin, S. Essa, S. Jancik, D. Houle, M. Lafleur, L. Gaboury, M. Tabrizian, N. Kaou, M. Atkin, T. Vuong, G. Batist, N. Beauchemin, D. Radzioch, and S. Martel.  Magneto-aerotactic bacteria deliver drug-containing nanoliposomes to tumour hypoxic regions.  _Nature Nanotechnology_ , 11(11):941–947, Nov 2016.  ISSN 1748-3395.  doi: 10.1038/nnano.2016.137.  URL https://doi.org/10.1038/nnano.2016.137.
  * Finkbeiner et al. [2023] J. Finkbeiner, S. Tovey, and C. Holm.  Generating minimal training sets for machine learned potentials, 2023\.
  * Gumbel [1954] E. J. Gumbel.  Statistical theory of extreme values and some practical applications. lectures by emit j. gumbel. national bureau of standards, washington, 1954. 51 pp. diagrams. 40 cents.  _The Aeronautical Journal_ , 58(527):792–793, 1954.  doi: 10.1017/S0368393100099958.
  * Hansen et al. [2004] E. A. Hansen, D. S. Bernstein, and S. Zilberstein.  Dynamic programming for partially observable stochastic games.  In _AAAI_ , volume 4, pages 709–715, 2004.
  * Harris et al. [2020] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant.  Array programming with numpy.  _Nature_ , 585(7825):357–362, Sep 2020.  ISSN 1476-4687.  doi: 10.1038/s41586-020-2649-2.  URL https://doi.org/10.1038/s41586-020-2649-2.
  * Hartl et al. [2021] B. Hartl, M. Hübl, G. Kahl, and A. Zöttl.  Microswimmers learning chemotaxis with genetic algorithms.  _Proceedings of the National Academy of Sciences_ , 118(19):e2019683118, 2021.  doi: 10.1073/pnas.2019683118.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2019683118.
  * Heek et al. [2023] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee.  Flax: A neural network library and ecosystem for JAX, 2023.  URL http://github.com/google/flax.
  * Hosseinidoust et al. [2016] Z. Hosseinidoust, B. Mostaghaci, O. Yasa, B.-W. Park, A. V. Singh, and M. Sitti.  Bioengineered and biohybrid bacteria-based systems for drug delivery.  _Advanced Drug Delivery Reviews_ , 106:27–44, 2016.  ISSN 0169-409X.  doi: https://doi.org/10.1016/j.addr.2016.09.007.  URL https://www.sciencedirect.com/science/article/pii/S0169409X16302629.  Biologically-inspired drug delivery systems.
  * Howse et al. [2007] J. R. Howse, R. A. Jones, A. J. Ryan, T. Gough, R. Vafabakhsh, and R. Golestanian.  Self-motile colloidal particles: from directed propulsion to random walk.  _Physical review letters_ , 99(4):048102, 2007\.
  * Hsu et al. [2016] A. Hsu, A. Wong-Foy, B. McCoy, C. Cowan, J. Marlow, B. Chavez, T. Kobayashi, D. Shockey, and R. Pelrine.  Application of micro-robots for building carbon fiber trusses.  In _2016 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS)_ , pages 1–6, 2016.  doi: 10.1109/MARSS.2016.7561729.
  * Hu and Wellman [2003] J. Hu and M. P. Wellman.  Nash q-learning for general-sum stochastic games.  _Journal of machine learning research_ , 4(Nov):1039–1069, 2003.
  * Ibarz et al. [2021] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine.  How to train your robot with deep reinforcement learning: lessons we have learned.  _The International Journal of Robotics Research_ , 40(4–5):698–721, Jan. 2021.  ISSN 1741-3176.  doi: 10.1177/0278364920987859.  URL http://dx.doi.org/10.1177/0278364920987859.
  * Jakob et al. [2022] W. Jakob, S. Speierer, N. Roussel, and D. Vicini.  DR.JIT.  _ACM Transactions on Graphics_ , 41(4):1–19, jul 2022.  doi: 10.1145/3528223.3530099.  URL https://doi.org/10.1145%2F3528223.3530099.
  * Jarrell et al. [2021] K. F. Jarrell, S.-V. Albers, and J. N. d. S. Machado.  A comprehensive history of motility and archaellation in archaea.  _FEMS microbes_ , 2:xtab002, 2021.
  * Jiang et al. [2010] H.-R. Jiang, N. Yoshinaga, and M. Sano.  Active motion of a janus particle by self-thermophoresis in a defocused laser beam.  _Physical review letters_ , 105(26):268302, 2010\.
  * Khalil et al. [2014] I. S. M. Khalil, V. Magdanz, S. Sanchez, O. G. Schmidt, and S. Misra.  Biocompatible, accurate, and fully autonomous: a sperm-driven micro-bio-robot.  _Journal of Micro-Bio Robotics_ , 9(3):79–86, Aug 2014.  ISSN 2194-6426.  doi: 10.1007/s12213-014-0077-9.  URL https://doi.org/10.1007/s12213-014-0077-9.
  * Kim et al. [2016] H. Kim, J. Ali, U. K. Cheang, J. Jeong, J. S. Kim, and M. J. Kim.  Micro manipulation using magnetic microrobots.  _Journal of Bionic Engineering_ , 13(4):515–524, 2016.  ISSN 1672-6529.  doi: https://doi.org/10.1016/S1672-6529(16)60324-4.  URL https://www.sciencedirect.com/science/article/pii/S1672652916603244.
  * Kiran et al. [2021] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. A. Sallab, S. Yogamani, and P. Pérez.  Deep reinforcement learning for autonomous driving: A survey, 2021.
  * Kriegman et al. [2020] S. Kriegman, D. Blackiston, M. Levin, and J. Bongard.  A scalable pipeline for designing reconfigurable organisms.  _Proceedings of the National Academy of Sciences_ , 117(4):1853–1859, 2020.  doi: 10.1073/pnas.1910837117.  URL https://www.pnas.org/doi/abs/10.1073/pnas.1910837117.
  * Lavergne et al. [2019] F. A. Lavergne, H. Wendehenne, T. Bäuerle, and C. Bechinger.  Group formation and cohesion of active particles with visual perception-dependent motility.  _Science_ , 364(6435):70–74, Apr. 2019.
  * Li et al. [2022] M. Li, A. Pal, A. Aghakhani, A. Pena-Francesch, and M. Sitti.  Soft actuators for real-world applications.  _Nature Reviews Materials_ , 7(3):235–249, Mar 2022.  ISSN 2058-8437.  doi: 10.1038/s41578-021-00389-7.  URL https://doi.org/10.1038/s41578-021-00389-7.
  * Littman [1994] M. L. Littman.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_ , pages 157–163. Elsevier, 1994.
  * Lymburn et al. [2021] T. Lymburn, S. D. Algar, M. Small, and T. Jüngling.  Reservoir computing with swarms.  _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , 31(3):033121, 03 2021.  ISSN 1054-1500.  doi: 10.1063/5.0039745.  URL https://doi.org/10.1063/5.0039745.
  * Macenski et al. [2022] S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall.  Robot operating system 2: Design, architecture, and uses in the wild.  _Science Robotics_ , 7(66):eabm6074, 2022.  doi: 10.1126/scirobotics.abm6074.  URL https://www.science.org/doi/abs/10.1126/scirobotics.abm6074.
  * Mandal et al. [2018] P. Mandal, G. Patil, H. Kakoty, and A. Ghosh.  Magnetic active matter based on helical propulsion.  _Accounts of chemical research_ , 51(11):2689–2698, 2018.
  * Medina-Sánchez et al. [2016] M. Medina-Sánchez, L. Schwarz, A. K. Meyer, F. Hebenstreit, and O. G. Schmidt.  Cellular cargo delivery: Toward assisted fertilization by sperm-carrying micromotors.  _Nano Letters_ , 16(1):555–561, 2016.  doi: 10.1021/acs.nanolett.5b04221.  URL https://doi.org/10.1021/acs.nanolett.5b04221.  PMID: 26699202.
  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.  doi: 10.1038/nature14236.  URL https://doi.org/10.1038/nature14236.
  * Muiños-Landin et al. [2021] S. Muiños-Landin, A. Fischer, V. Holubec, and F. Cichos.  Reinforcement learning with artificial microswimmers.  _Science Robotics_ , 6(52):eabd9285, 2021.  doi: 10.1126/scirobotics.abd9285.  URL https://www.science.org/doi/abs/10.1126/scirobotics.abd9285.
  * Murphy et al. [2008] R. R. Murphy, S. Tadokoro, D. Nardi, A. Jacoff, P. Fiorini, H. Choset, and A. M. Erkmen.  _Search and Rescue Robotics_ , pages 1151–1173.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.  ISBN 978-3-540-30301-5.  doi: 10.1007/978-3-540-30301-5_51.  URL https://doi.org/10.1007/978-3-540-30301-5_51.
  * Nachum et al. [2017] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans.  Bridging the gap between value and policy based reinforcement learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/facf9f743b083008a894eee7baa16469-Paper.pdf.
  * Nelson and Pané [2023] B. J. Nelson and S. Pané.  Delivering drugs with microrobots.  _Science_ , 382(6675):1120–1122, 2023.  doi: 10.1126/science.adh3073.  URL https://www.science.org/doi/abs/10.1126/science.adh3073.
  * Nelson et al. [2010] B. J. Nelson, I. K. Kaliakatsos, and J. J. Abbott.  Microrobots for minimally invasive medicine.  _Annual Review of Biomedical Engineering_ , 12(1):55–85, 2010.  doi: 10.1146/annurev-bioeng-010510-103409.  URL https://doi.org/10.1146/annurev-bioeng-010510-103409.  PMID: 20415589.
  * Nikolaou and Tovey [2021] K. Nikolaou and S. Tovey.  ZnNL, July 2021.  URL https://github.com/zincware/ZnNL.
  * Oliehoek et al. [2016] F. A. Oliehoek, C. Amato, et al.  _A concise introduction to decentralized POMDPs_ , volume 1.  Springer, 2016.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 488–489, 2017.  doi: 10.1109/CVPRW.2017.70.
  * Qin et al. [2023] K. Qin, Z. Zou, L. Zhu, and O. S. Pak.  Reinforcement learning of a multi-link swimmer at low Reynolds numbers.  _Physics of Fluids_ , 35(3), 03 2023.  ISSN 1070-6631.  doi: 10.1063/5.0140662.  URL https://doi.org/10.1063/5.0140662.  032003\.
  * Romanczuk et al. [2012] P. Romanczuk, M. Bär, W. Ebeling, B. Lindner, and L. Schimansky-Geier.  Active brownian particles.  _The European Physical Journal Special Topics_ , 202(1):1–162, Mar 2012.  ISSN 1951-6401.  doi: 10.1140/epjst/e2012-01529-y.  URL https://doi.org/10.1140/epjst/e2012-01529-y.
  * Schmidt et al. [2020] C. K. Schmidt, M. Medina-Sánchez, R. J. Edmondson, and O. G. Schmidt.  Engineering microrobots for targeted cancer therapies from a medical perspective.  _Nature Communications_ , 11(1):5618, Nov 2020\.  ISSN 2041-1723.  doi: 10.1038/s41467-020-19322-7.  URL https://doi.org/10.1038/s41467-020-19322-7.
  * Schulman et al. [2017a] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel.  Trust region policy optimization, 2017a.
  * Schulman et al. [2017b] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms, 2017b.
  * Schulman et al. [2018] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation, 2018.
  * Shen et al. [2023] H. Shen, S. Cai, Z. Wang, Z. Ge, and W. Yang.  Magnetically driven microrobots: Recent progress and future development.  _Materials & Design_, 227:111735, 2023.  ISSN 0264-1275.  doi: https://doi.org/10.1016/j.matdes.2023.111735.  URL https://www.sciencedirect.com/science/article/pii/S0264127523001508.
  * Silflow and Lefebvre [2001] C. D. Silflow and P. A. Lefebvre.  Assembly and motility of eukaryotic cilia and flagella. lessons from chlamydomonas reinhardtii.  _Plant physiology_ , 127(4):1500–1507, 2001.
  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.  doi: 10.1038/nature16961.  URL https://doi.org/10.1038/nature16961.
  * Stefanec et al. [2022] M. Stefanec, D. N. Hofstadler, T. Krajník, A. E. Turgut, H. Alemdar, B. Lennox, E. Şahin, F. Arvin, and T. Schmickl.  A minimally invasive approach towards “ecosystem hacking” with honeybees.  _Frontiers in Robotics and AI_ , 9, 2022.  ISSN 2296-9144.  doi: 10.3389/frobt.2022.791921.  URL https://www.frontiersin.org/articles/10.3389/frobt.2022.791921.
  * Su et al. [2019] H. Su, C.-A. Hurd Price, L. Jing, Q. Tian, J. Liu, and K. Qian.  Janus particles: design, preparation, and biomedical applications.  _Materials Today Bio_ , 4:100033, 2019.  ISSN 2590-0064.  doi: https://doi.org/10.1016/j.mtbio.2019.100033.  URL https://www.sciencedirect.com/science/article/pii/S2590006419300596.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. [1999] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In S. Solla, T. Leen, and K. Müller, editors, _Advances in Neural Information Processing Systems_ , volume 12. MIT Press, 1999.  URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.
  * Tan [1993] M. Tan.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  In _Proceedings of the tenth international conference on machine learning_ , pages 330–337, 1993.
  * Tovey [2021] S. Tovey.  ZnVis: A Visualisation Package for Particle Simulations, 2021.
  * Tovey et al. [2023] S. Tovey, S. Krippendorf, K. Nikolaou, and C. Holm.  Towards a phenomenological understanding of neural networks: data.  _Machine Learning: Science and Technology_ , 4(3):035040, sep 2023.  doi: 10.1088/2632-2153/acf099.  URL https://dx.doi.org/10.1088/2632-2153/acf099.
  * Wadhwa and Berg [2022] N. Wadhwa and H. C. Berg.  Bacterial motility: machinery and mechanisms.  _Nature reviews microbiology_ , 20(3):161–173, 2022.
  * Wang et al. [2019] X. Wang, X.-Z. Chen, C. C. J. Alcântara, S. Sevim, M. Hoop, A. Terzopoulou, C. de Marco, C. Hu, A. J. de Mello, P. Falcaro, S. Furukawa, B. J. Nelson, J. Puigmartí-Luis, and S. Pané.  Mofbots: Metal–organic-framework-based biomedical microrobots.  _Advanced Materials_ , 31(27):1901592, 2019.  doi: https://doi.org/10.1002/adma.201901592.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.201901592.
  * Weik et al. [2019] F. Weik, R. Weeber, K. Szuttor, K. Breitsprecher, J. de Graaf, M. Kuron, J. Landsgesell, H. Menke, D. Sean, and C. Holm.  Espresso 4.0–an extensible software package for simulating soft matter systems.  _The European Physical Journal Special Topics_ , 227:1789–1816, 2019.
  * Zeng et al. [2023] Y. Zeng, Z.-Y. Zhou, E. Rinaldi, C. Gneiting, and F. Nori.  Approximate autonomous quantum error correction with reinforcement learning.  _Phys. Rev. Lett._ , 131:050601, Jul 2023.  doi: 10.1103/PhysRevLett.131.050601.  URL https://link.aps.org/doi/10.1103/PhysRevLett.131.050601.
  * Zhang et al. [2018] K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Başar.  Fully decentralized multi-agent reinforcement learning with networked agents, 2018.
  * Zhang et al. [2021] K. Zhang, Z. Yang, and T. Başar.  Multi-agent reinforcement learning: A selective overview of theories and algorithms, 2021.
  * Zhuang et al. [2015] J. Zhuang, R. Wright Carlsen, and M. Sitti.  ph-taxis of biohybrid microsystems.  _Scientific Reports_ , 5(1):11403, Jun 2015.  ISSN 2045-2322.  doi: 10.1038/srep11403.  URL https://doi.org/10.1038/srep11403.


# Characteristic Cohomology of Mumford-Tate domains

Mohammad Reza Rahmati, Khurram Shabbir
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

In this text we pretend to find generators and relations for the universal
characteristic cohomology of MT-domains of Hodge structures by some structure
theorem via toric stacks. We also compare a uniformization theorem of toric
stacks with the corresponding structure on MT-domains.

###### Key words and phrases:

Deligne-Mumford Stacks, Toric variety, Mumford-Tate domains, Gerbe, Root
Stack, Stacky fan, Cohomology of toric stacks

###### 1991 Mathematics Subject Classification:

14D23

## Introduction

Assume $D$ is a period domain of pure Hodge structures defined by Griffiths,
with the period map $(\Delta^{*})^{n}\to\Gamma\setminus D$, where $\Gamma$ is
the monodromy group. The domain $D$ can be understood as the orbit of a
polarized Hodge structure $(H_{\mathbb{Q}},F^{\bullet},Q)$, under the action
of a reductive group $G$ presering the polarization form $Q$. We treat a
partial compactification of $\Gamma\setminus D$ so that the period map is
extended over $\Delta^{n}$. There are different theories of compactifications
for period domains $D$. Kato and Usui provide the toroidal compactifications
for any period domain $D$, which is not Hermitian symmetric in general and
show these are moduli spaces of log Hodge structures. This partial
compactification is given by using toroidal embedding associated to the cone
generated by the data of the monodromies. The Mumford-Tate (MT)-domains of a
Hodge structure

$\phi:\mathbb{S}(\mathbb{R})\to Gl(V,\mathbb{R})$

is the orbit under the MT-group of the Hodge structure $\phi$. They are
reductive Lie algebraic group defined over $\mathbb{Q}$.

Mumford-Tate domains can be considered as a toric stack, by a definition using
Stacky-fans. The structure is compatible with the Kato-Usui generalized
toroidal compactification as we will see below by the definition. In this way
we may regard some additional data on MT-domains which preserves the
stabilizers of the points under a Lie group action. A toric stack is a simple
generalization of the definition of toric variety, but we will concern the
toric action as a groupoid action. There are several definitions of toric
Stacks in the literature. A toric stack is the stack quotient of a normal
toric variety $X$, by a subgroup $G$ of its Torus $T$, cf. [GS]. The main task
of this text is to match this definition with how to express the definition
with that of period domains of Griffiths, Mumford-Tate domains, and their
boundary points via the Kato-Usui generalized toroidal compactification. As an
application one can consider the universal characteristic cohomology of
Mumford-Tate domains as a type of stack cohomology (orbifold cohomology).

A simple way to define a toric stack is by the definition of a stacky fan
$(\Sigma,\beta,N)$, where $\Sigma$ is a fan, $\beta:\mathbb{Z}^{n}\to N$ a
homomorphism of abelian groups with finite cokernel, and $N$ is a lattice in
$\mathbb{R}^{n}$. Toric stacks are Deligen-Mumford stacks, that is they can be
understood similar to usual manifolds. Such stacks are also smooth as DM
stacks and their boundary points have codimension 1 as a sub-stack. Toric
stacks satisfy a simple uniformization structures as a global quotient via the
structure of their Chow group or equivalently their Picard group of line
bundles. We explain this by root sheaves or root of line bundles associated to
rays in their fan.

In this text we show that MT-domains of Hodge structure are smooth DM-stacks
and in this way satisfy the same uniformization theorem as general smooth DM-
stacks. We propose to apply the structure theorems of toric stacks to Kato-
Usui partial toric compactification of period and Mumford-Tate domains. To
this target, we briefly mention some basic results on toric stacks as smooth
Deligne-Mumford stacks from the references, [FMN] and [J]. Our strategy is to
combine this literature with Hodge theory as much as possible. The Proposition
2.1 and part (4) of Section 3 seem to have new contributions. The basic
material on MT-domains in use are from [TH], [KP], [GGK]. We refer to [WF] for
preliminaries on Toric varieties.

##  1\. Toric Stacks

Toric stacks are generalization of the notion of toric varieties and their
torus actions. The point is to upgrade the action of the torus to a groupoid
action. The action $\rho$ is equivalently thought of a functor $\rho:BG\to
Sets$, from the group $G$ regarded as a one-object groupoid, denoted by $BG$.
This functors sends the single object to the set $X$. The notion of groupoids
and their action can be generalized over arbitrary schemes. Let $S$ be a base
scheme. The action of the groupoid $\mathcal{G}$ on an $S$-stack is a morphism
$\mathcal{G}\times_{S}X\to X$ with two arrows $\eta_{x}:e\times x\Rightarrow
x,\ (g_{1}.g_{2})\times x\Rightarrow g_{1}.(g_{2}.x)$. The example we are
interested in are Deligne-Mumford torus. A Deligne-Mumford torus is a Picard
stack over $Spec(\mathbb{C})$ which is obtained as a quotient $[T_{L}/G]$,
with $\phi:L\to N$ is a morphism of finitely generated abelian groups such
that $\ker(\phi)$ is free and $coker(\phi)=:G$ is finite, and $T_{L}$ is a
torus associated to the lattice $L$. Any Deligne-Mumford (DM) torus is
isomorphic as Picard stack to $T\times BG$, where $T$ is a torus and $G$ is a
finite abelian group.

A smooth toric Deligne-Mumford stack is a smooth separated DM-stack $X$
together with an open immerssion of a Deligne-Mumford torus
$\imath:\mathcal{T}\hookrightarrow X$ with dense image such that the action of
$\mathcal{T}$ on itself extends to an action $\mathcal{T}\times X\to X$. In
this case a morphism is a morphism of stacks which extends a morphism of
Deligne-Mumford tori. A toric orbifold is a toric DM-stack with generically
trivial stabilizer. A toric DM-stack is a toric orbifold iff its DM-torus is
an ordinary torus. The same as way to define toric varieties by combinatorial
data of their fan, a toric stack may also be defined by a stacky fan.

A stacky fan is a pair $(\Sigma,\beta)$, where $\Sigma$ is a fan on a lattice
$L$ and $\beta:L\to N$ is a homomorphism to a lattice $N$, so that
$coker(\beta)$ is finite. A stacky fan gives rise to a toric stack as follows.
Let $X_{\Sigma}$ be the toric variety associated to $\Sigma$. The map
$\beta^{*}:N^{*}\to L^{*}$ induces a homomorphism of Tori, $T_{\beta}:T_{L}\to
T_{N}$, by naturally identifying $\beta$ with the induced map on lattices of
1-parameter subgroups. Since $coker(\beta)$ is finite, $\beta^{*}$ is
injective, so $T_{\beta}$ is surjective. Let $G_{\beta}=\ker T_{\beta}$. Note
that $T_{L}$ is the torus on $X_{\Sigma}$, and $G_{\beta}\subset T_{L}$ is a
subgroup. The action of $G_{\beta}$ on $X_{\Sigma}$ is induced by the
homomorphism $G_{\beta}\to T_{L}$. If $(\Sigma,\beta)$ is a stacky fan, we
define the toric stack $\mathcal{X}_{\Sigma,\beta}$ to be
$[\mathcal{X}/G_{\beta}]$, with the torus $T_{N}=T_{L}/G_{\beta}$. This
construction is also reversable in a trivial way, [GS].

One may expect that toric DM-stack can be represented by a larger $Z$ with
larger group $G$. For instance, the classifying stack $B\mu_{3}=[*/\mu_{3}]$
is a toric DM stack by the stacky fan $(N=\mu_{3},0,0)$. It is isomorphic to
$[\mathbb{C}^{\times}/\mathbb{C}^{\times}]$ where $\mathbb{C}^{\times}$ acts
by $(\lambda,x)\to\lambda^{3}x$. It is simply possible that a toric stack
correspond to different stacky fans. Motivated by the study of gerbes one
introduces the notion of extended stacky fan. An extended stacky fan is a
triple $(N,\Sigma,\beta^{e})$ where $N,\Sigma$ are the same as the stacky fan
$\Sigma$, and $\beta^{e}:\mathbb{Z}^{m}\to N$ is determined by
$b_{1},...,b_{n}$ and additional elements $b_{n+1},...,b_{m}$. Then the toric
DM-stack is defined by $[Z^{e}/G^{e}]$, where
$Z^{e}=Z\times(\mathbb{C}^{\times})^{m-n}$, and $G^{e}$ acts on $Z^{e}$
through the homomorphism $\alpha^{e}:G^{e}\to(\mathbb{C}^{\times})^{m-n}$
determined by the extended stacky fan, [J].

A coherent sheaf on a DM stack $[Z/G]$ is a $G$-equivariant sheaf on $Z$. As
usual a line bundle on $[Z/G]$ is given by a character $\chi$ of $G$. Let $b$
be a positive integer. We denote by $\sqrt[a]{L/X}$ the fiber product

(1) $\begin{CD}\sqrt[a]{L/X}@>{}>{}>B\mathbb{C}^{*}\\\ @V{}V{}V@V{}V{\wedge
b}V\\\ X@>{}>{L}>B\mathbb{C}^{*}\end{CD}$

where $\wedge b$ is the map that sends an invertible sheaf to its $b$-th
power. More explicitly an object of $\sqrt[a_{1}]{L/X}$ over $f:S\to X$ is a
couple $(M,\phi)$ where $M$ is an invertable sheaf on $S$ and $\phi:M^{\otimes
b}\stackrel{{\scriptstyle\cong}}{{\rightarrow}}f^{*}L$ is an isomorphism. Let
$X$ be a smooth algebraic stack. Assume $D=(D_{1},...,D_{n})$ are $n$
effective Cartier divisors and $a=(a_{1},...,a_{n})\in\mathbb{N}_{>0}^{n}$.
Then the $a$-th root of $(X,D)$ is the fiber product

(2) $\begin{CD}\sqrt[a]{D/X}@>{}>{}>[\mathbb{A}^{n}/(\mathbb{C}^{*})^{n}]\\\
@V{}V{}V@V{}V{\wedge a}V\\\
X@>{}>{L}>[\mathbb{A}^{n}/(\mathbb{C}^{*})^{n}]\end{CD}$

$\wedge
a:[\mathbb{A}^{n}/(\mathbb{C}^{*})^{n}]\to[\mathbb{A}^{n}/(\mathbb{C}^{*})^{n}]$
is the stack morphism $x_{i}\to x_{i}^{a_{i}}$ and
$\lambda_{i}\to\lambda_{i}^{a_{i}}$, where $x_{i},\lambda_{i}$ are coordinates
in $\mathbb{A}^{n}$ and $(\mathbb{C}^{*})^{n}$ respectively. The structure
morphism $X\to\bar{X}$ into its coarse moduli space factors canonically via
the toric morphisms $X\to X^{\text{rig}}\to X^{\text{can}}\to\bar{X}$ where
the first map is an abelian gerbe over $X^{\text{rig}}$, and is a fiber
product of roots of toric divisors as in (2), and the last is the minimal
orbifold having $\bar{X}$ as a coarse moduli space. For instance, the stack
$\mathbb{P}(w)\cong[(\mathbb{C}^{n+1}\setminus 0)/\mathbb{C}^{*}]$ is a toric
DM-stack with deligne-Mumford torus
$[\mathbb{C}^{n+1}/\mathbb{C}^{*}]\cong\mathbb{C}^{n}\times B\mu_{d}$. It is
canonical if $gcd(w_{0},...,\hat{w_{i}},...,w_{n})=1$. It is an orbifold if
$gcd(w_{0},...,w_{n})=1$, [FMN].

######  Proposition 1.1.

[FMN] (1) Let $X$ be toric Deligne-Mumford stack with Deligne-Mumford torus
isomorphic to $T\times BG$. Given $G=\prod_{j=1}^{l}\mu_{b_{j}}$. There exists
$L_{j}$ in $Pic(X^{rig})$ such that $X$ is isomorphic as a $G$-gerbe over
$X^{rig}$ to

(3)
$\sqrt[b_{1}]{L_{1}/X^{rig}}\times_{X^{rig}}...\times_{X^{rig}}\sqrt[b_{l}]{L_{l}/X^{rig}}$

where the classes
$(L_{1},...,L_{l})\in\prod_{j=1}^{l}Pic(X^{rig})/b_{j}Pic(X^{rig})$ are
unique. Moreover the reduced closed substack $X\setminus\mathcal{T}$ is a
simple normal crossing divisor.

(2) Let $X$ be a smooth toric Deligne-Mumford stack with DM-torus isomorphic
to $T\times BG$, with $\bar{X}$ as coarse moduli. denote by $n$ the number of
rays in the fan of $\bar{X}$. Then there exists a unique
$(a_{1},...,a_{n})\in\mathbb{N}_{>}^{n}$ such that

$X^{\text{rig}}\cong\sqrt[a_{1}]{D_{1}^{can}/X^{can}}\times_{X^{can}}...\times_{X^{can}}\sqrt[a_{n}]{D_{n}^{can}/X^{can}}$

where $D_{i}$ is the divisor corresponding to the ray $\rho_{i}$.

##  2\. Higher structure on Mumford-Tate domains

We apply the structure theorems of section 1 to the structure on period and
Mumford-Tate domains. To begin with let $V$ be finite dimensional
$\mathbb{Q}$-vector space, and $Q$ a non-degenerate bilinear map $Q:V\otimes
V\to\mathbb{Q}$ which is $(-1)^{n}$-symmetric for some fixed $n$. A Hodge
structure is given by a representation

$\phi:\mathbb{U}(\mathbb{R})\to
Aut(V,Q)_{\mathbb{R}},\qquad\mathbb{U}(\mathbb{R})=\left(\begin{array}[]{cc}a&-b\\\
b&a\end{array}\right),\ a^{2}+b^{2}=1$

It decomposes over $\mathbb{C}$ into eigenspaces $V^{p,q}$ such that
$\phi(t).u=t^{p}\bar{t}^{q}.u$ for $u\in V^{p,q}$, and
$\overline{V^{p,q}}=V^{q,p}$. It is well-known that
$Ad(\phi):\mathbb{U}(\mathbb{R})\to Aut(\mathfrak{g}_{\mathbb{R}},B)$ defines
a weight $0$ Hodge structure, where $B$ is the Killing form. Then
$\mathfrak{g}_{\mathbb{C}}$ has a decomposition

$\mathfrak{g}=\mathfrak{g}_{\phi}^{-}\oplus\overline{\mathfrak{g}_{\phi}^{-}}\oplus\mathfrak{h}_{\phi},\qquad\mathfrak{g}^{-}:=\oplus_{i>0}\mathfrak{g}_{\phi}^{-i,i}$

and
$T_{\phi,\mathbb{R}}D\otimes\mathbb{C}=\mathfrak{g}_{\phi}^{-}\oplus\overline{\mathfrak{g}_{\phi}^{-}},\
T_{\phi}D=\mathfrak{g}_{\phi}^{-}$. The Mumford-Tate group $M_{\Phi}$ of the
Hodge structure $\phi$ is the smallest $\mathbb{Q}$-algebraic subgroup
containing the image of $\phi$, and is a reductive Lie group, that satisfies
the almost product decomposition

(4) $M_{\Phi}=M_{1}\times...\times M_{l}\times T$

The associated Mumford-Tate domains $D_{i}$ (the $M_{i}(\mathbb{R})$-orbit of
$\phi(s_{0})$) have a splitting of the

(5) $D_{M_{\Phi}}=D_{1}\times...\times D_{l}$

Assume $\Phi:S\to\Gamma\setminus D$ is a variation of Hodge structure and let
$T_{1},...,T_{n}$ be generators of the monodromy group $\Gamma$. Then the
partial compactification $\Gamma\setminus D_{\sigma}$ is given by the cone

$\sigma=\displaystyle{\sum_{j=1}^{n}\mathbb{R}_{\geq 0}N_{j},\qquad N_{j}=\log
T_{j}}$

in the lie algebra $\mathfrak{g}:=Lie(G_{\mathbb{C}})\subset Hom(V,V)$. Here a
boundary point is a nilpotent orbit associated to a face $\sigma$. Set $\
\Gamma(\sigma)^{\text{gp}}=\exp(\sigma_{\mathbb{R}})\cap G_{\mathbb{Z}},\ \
\Gamma(\sigma)=\exp(\sigma)\cap G_{\mathbb{Z}}$. The monoid $\Gamma(\sigma)$
defines the toric variety

(6) $D_{\sigma}:=Spec([\mathbb{C}[\Gamma(\sigma)^{\vee}])_{an}\cong
Hom(\Gamma(\sigma)^{\vee},\mathbb{C})$

with the torus

(7) $T_{\sigma}:=Spec(\mathbb{C}[\Gamma(\sigma)^{\vee\text{gp}}])_{an}\cong
Hom(\Gamma(\sigma)^{\vee\text{gp}},\mathbb{G}_{m})\cong\mathbb{G}_{m}\otimes\Gamma(\sigma)^{\text{gp}}$

Let $\Sigma$ be a maximal fan in $\mathfrak{m}:=Lie(M)$. Consider the
injective map $\beta=\emph{e}:\log\gamma\mapsto\gamma$ restricted to
$\Sigma=\mathbb{Z}\langle N_{1},...,N_{r}\rangle$. The map e is an isomorphism
onto its image $\mathfrak{m}_{\mathbb{Z}}$ a maximal lattice in
$\mathfrak{m}$. The triple
$(D_{M},\emph{e}:\Sigma_{\mathbb{Z}}\to\mathfrak{m}_{\mathbb{Z}},\Sigma)$ is a
toric stack. On the lattices e is an isomorphism, mentioning that the generic
stabilizer is trivial.

######  Theorem 2.1.

The triple
$(D_{M},\emph{e}:\Sigma_{\mathbb{Z}}\to\mathfrak{m}_{\mathbb{Z}},\Sigma)$ is a
smooth toric Deligne-Mumford stack. This stacky structure decomposes as

(8) $D_{M_{\Phi}}=D_{1}\times...\times D_{l}$

where

$D\cong\sqrt[b_{1}]{L_{1}/D}\times_{D}...\times_{D}\sqrt[b_{l}]{L_{l}/D}\cong\sqrt[d]{L/Z}$

$D_{i}\cong\sqrt[b_{1i}]{L_{1i}/D_{i}}\times_{D_{i}}...\times_{D_{i}}\sqrt[b_{li}]{L_{ki}/D_{i}}\cong\sqrt[d_{i}]{L_{i}/Z_{i}}$

where $b_{ij}\in\mathbb{N}_{>0}$ and $d=gcd(b_{1},...,b_{l}),\
d_{i}=gcd(b_{1i},...,b_{1k})$. Moreover $D_{M_{\Phi}}$ is rigid.

###### Proof.

The first part is by the definition and the construction above. The
decomposition in (8) follows from proposition 1.1 and the uniqueness of
splitting in (5). It is rigid because the map
$\emph{e}:\Sigma_{\mathbb{Z}}\to\mathfrak{m}_{\mathbb{Z}}$ is an isomorphism
onto its image. ∎

##  3\. Cohomology

(1) Differentiable stacks are stacks over the category of differentiable
manifolds. They are the stacks associated to Lie groupoids. A groupoid
$X_{1}\rightrightarrows X_{0}$ is a Lie groupoid if $X_{0}$ and $X_{1}$ are
differentiable manifolds, structure maps are differentiable, source and target
maps are submersion. There is associated a simplicial nerve $X_{p}$ of
$p$-fold cartesian product of $X_{1}$ over $X_{0}$. Then we get an associated
co-simplicial object $(\Omega^{q}(X^{\bullet},\
\partial=\sum_{i=0}^{p}(-1)^{i}\partial_{i}^{*})$. The cohomology groups
$H^{k}(X,\Omega^{q})$ are called the Cech cohomology groups of the groupoid
$X=[X_{1}\rightrightarrows X_{0}]$. The double complex
$A^{pq}:=\Omega^{q}(X_{p})$ is called the de Rham complex of the stack
$\mathcal{X}$, and its cohomologies are called de Rham cohomologies of
$\mathcal{X}$. The cohomology of the DM-stack $X$ is defined to be
$H_{DR}^{n}(X)=H_{DR}^{n}(X_{1}\rightrightarrows X_{0})$, for any groupoid
atlas $X_{1}\rightrightarrows X_{0}$ of the stack $X$.

If $G$ is a Lie group then $H^{k}(BG,\Omega^{0})$, is the group cohomology of
$G$ calculated with differentiable cochains. When the stack $X$ is a quotient
stack there is a clear explanation of its cohomology as an equivariant
cohomology. There is a well-known generalization of the de Rham complex to the
equivariant case, namely the Cartan complex $\Omega_{G}^{\bullet}(X)$, defined
by

(9)
$\Omega_{G}^{\bullet}(X):=\displaystyle{\bigoplus_{2k+i=n}(S^{k}\mathfrak{g}^{\vee}\otimes\Omega^{i}(X))^{G}}$

where $S^{\bullet}\mathfrak{g}^{\vee}$ is the symmetric algebra on the dual of
the Lie algebra of $G$. The group $G$ acts on $\mathfrak{g}$ by adjoint
representation on $\mathfrak{g}^{\vee}$ and by pull back of differential forms
on $\Omega^{\bullet}(X)$. The Cartan differential is $d_{DR}-\imath$ where
$\imath$ is the tensor induced by the vector bundle homomorphism
$\mathfrak{g}_{X}\to T_{X}$ coming from differentiating the action. If $G$ is
compact, the augmentation is a quasi-isomorphism, i.e

(10) $H_{G}^{i}(X)\stackrel{{\scriptstyle\cong}}{{\longrightarrow}}H^{i}(Tot\
\Omega_{G}^{\bullet}(X_{\bullet}))$

for all $i$ and the groupoid $X_{\bullet}$, [KB].

######  Proposition 3.1.

[KB] If the Lie group $G$ is compact, there is a natural isomorphism

$H_{G}^{i}(X)\stackrel{{\scriptstyle\cong}}{{\longrightarrow}}H_{DR}^{i}(G\times
X\rightrightarrows X)=H_{DR}^{i}([X/G])$

As a corollary $H_{DR}^{*}(BG)=(S^{2*}\mathfrak{g}^{\vee})^{G}$ for a compact
lie group $G$. If $G$ is not compact, then $H_{DR}([X/G])$ is still equal to
equivariant cohomology. This fact holds for equivariant cohomology in general.

(2) Let $X$ be a complex manifold and $W\subset X$ a holomorphic sub-bundle.
Then $I=W^{\perp}\subset T^{*}(X)$ is also holomorphic and let $\bar{I}$ be
its conjugate. Let
$\mathcal{I}^{\bullet\bullet}\subset\mathcal{A}^{\bullet\bullet}$ be the
differential ideal of sections of $I\oplus\bar{I}$ in the algebra of smooth
differential forms on $X$. The characteristic cohomology of $X$ denoted
$H^{*}_{\mathcal{I}}(X)$ is the cohomology of the double complex
$\subset\mathcal{A}^{\bullet\bullet}/\mathcal{I}^{\bullet\bullet}$.

Let $D$ be a period domain for a PHS $(V,Q,\phi)$ of weight $n$ and set
$\Gamma_{\mathbb{Z}}=Aut(V_{\mathbb{Z}},\mathbb{Q})$. In the tangent bundle
$TD$ there is a homogeneous sub-bundle $W$ whose fiber at $\phi$ is
$W_{\phi}=\mathfrak{g}_{\phi}^{-1,1}=\\{\psi\in
T_{\phi}D:\psi(F_{\phi}^{p})\subset F_{\phi}^{p+1}\\}$ defined by, namely
infinitesimal period relations (IPR). There is a natural inclusion
$T_{F^{\bullet}}\check{D}\subset\oplus_{p}Hom(F^{p},V_{\mathbb{C}}/F^{p})$. We
shall assume that, the canonical sub-bundle $W\subset T\check{D}$ given by the
infinitesimal period relation is defined by

$W_{F^{\bullet}}=T_{F^{\bullet}}\check{D}\cap(\oplus_{p}Hom(F^{p},F^{p-1}/F^{p}))$

The bundle $W\to\check{D}$ is acted by $G_{\mathbb{C}}$, and the action of
$G_{\mathbb{R}}$ on $W\to D$ leaves invariant the metric given by Cartan
killing form at each point. With the identification
$T_{\phi}D=\oplus_{i>0}\mathfrak{g}^{-i,i}$ we have
$W_{\phi}=\mathfrak{g}_{\phi}^{-1,1}$. In our case, take $X=D_{M}\subset D$ to
be a Mumford-Tate domain, and $W_{M}\subset TD_{M}$ is the infinitesimal
period relation. Denote by $\Lambda_{M}^{\bullet\bullet}$ the complex of
$G(\mathbb{R})$-invariant forms in
$A^{\bullet\bullet}(D_{M})/\mathcal{I}^{\bullet\bullet}$ with the operator
$\delta:\Lambda_{M}^{\bullet}\to\Lambda^{\bullet+1}$.
$H^{*}(\Lambda_{M}^{\bullet},\delta_{M})$ is called the universal
characteristic cohomology, [GGK].

######  Proposition 3.2.

[GGK] The universal characteristic cohomology i.e the cohomology of
$A^{\bullet\bullet}(D_{M})/\mathcal{I}^{\bullet\bullet},\delta:\Lambda_{M}^{\bullet}\to\Lambda^{\bullet+1}):=H^{*}(\Lambda_{M}^{\bullet},\delta_{M})$
is equal to the equivariant cohomology $H^{*}(I^{\bullet\bullet})$. Moreover,
$H^{2p-1}(\Lambda_{M}^{\bullet},\delta_{M})=0$ and
$H^{2p-1}(\Lambda_{M}^{\bullet},\delta_{M})=(\Lambda^{p,p}){{}^{\mathfrak{m}^{0,0}}}$.

The universal characteristic cohomology is equal to the de Rham cohomology of
the stack $D_{M}$. In case $D_{M}=D$ and $M=G$, the universal characteristic
cohomology is generated by the chern forms of Hodge bundles. According to the
proposition 3.2 and the remark after that, the universal characteristic
cohomology can be understood as the stack cohomology of the Mumford-Tate
domains. One can ask; hat are the conditions that the chern forms of Hodge
bundles on $D_{M}$ generate the characteristic cohomology? In case this holds
what are the relations?.

(3) Keeping the notation as section (1), let $P\to B$ be a principal
$(\mathbb{C}^{\times})^{m}$-bundle. The group $G^{e}$ acts on the fiber
product $P\times_{(\mathbb{C}^{\times})^{m}}Z^{e}$ via the map $\alpha^{e}$,
where $\alpha^{e}$ is given by the stacky fan. Each $\theta\in M=N^{*}$, gives
$\chi_{\theta}:(\mathbb{C}^{\times})^{m}\to(\mathbb{C}^{\times})$ and assume
the line bundle $\psi_{\theta}=P\times_{\chi_{\theta}}\mathbb{C}$ be given by
the ray $\theta$. Then one knows that

(11)
$H_{orb}^{*}([(P\times_{\alpha^{e}}Z^{e})/G^{e}]=\dfrac{H^{*}(B)[N]\otimes\mathbb{Q}[N]^{\Sigma^{e}}}{<\sum_{i}c_{1}(\psi_{\theta})+\theta(b_{i})y^{b_{i}}>_{\theta\in
M}}$

where $H_{orb}^{*}$ is the orbifold Cohomology ring, $H^{*}$ the usual
cohomology ring, $<.>$ means the ideal generated by ., and
$\mathbb{Q}[N]^{\Sigma^{e}}$ is $\oplus_{c\in N}y^{c}$ where $y$ shows formal
variables and the multiplication $y^{c}.y^{c^{\prime}}=y^{c+c^{\prime}}$ when
there is $\sigma\in\Sigma$ such that $\beta^{e}(c)$ and
$\beta^{e}(c^{\prime})\in\sigma$, and $0$ otherwise, [J].

(4) We apply the situation in (3) to the homogeneous manifold $G/V$ of the
principal bundle $V\hookrightarrow G\to G/V=D$ and the homogeneous vector
bundle $\mathcal{E}$ associated to a representation $\rho:V\to Gl(H)$, for $H$
a complex vector space. The formula in (11) translates into

(12)
$H_{orb}^{*}([\mathcal{E}=G\times_{V}H])=\dfrac{H^{*}(D_{M})[\mathfrak{m}]^{\Sigma}}{<\sum_{i}c_{1}(\psi_{\lambda})+\lambda(b_{i})y^{b_{i}}>_{\lambda}}$

The line bundles $\psi_{\lambda}$ are coming from the action of the torus $T$,
i.e the roots of the representation $\rho$. The equation (12) explains that
the relations in characteristic cohomology can be given by polynomial
equations in terms of first chern classes of line bundles associated to the
co-characters of a maximal torus $T$ in $G$. The same method can be applied to
the principal bundle $P\hookrightarrow G_{\mathbb{C}}\to\check{D}$ where $P$
is a parabolic subgroup. The curvature form $\Theta$ of a homogeneous manifold
of this type can always be expressed as

$\Theta=A\wedge^{t}\bar{A}-B\wedge^{t}\bar{B}$

where $A,B$ are matrices of $(1,0)$-forms. Using Cartan structure equation one
obtains the following explicit form of $\Theta$,

$\Theta=\displaystyle{-\sum_{\alpha,\beta\in\Delta_{+}-\Phi}[e_{\alpha},e_{\beta}]_{\mathfrak{v}}\otimes\omega^{\alpha}\wedge\bar{\omega}^{\beta}}$

where $\Delta_{+}$ is a system of positive roots for the root decomposition
$\mathfrak{g}=\mathfrak{h}\oplus\bigoplus_{\alpha\in\Delta}\mathfrak{g}_{\alpha},\
e_{\alpha}\in\mathfrak{h}^{\vee}\text{ the root vectors},\
\omega^{\alpha}=e_{\alpha}^{\vee}$ and $\mathfrak{v}=Lie(V)$, $\Phi$ the
corresponding sub-root system for $V$.

## References

  * [FMN] B. Fantechi, E. Mann, F. Nironi, Smooth toric Deligne-Mumford Stacks, J. reine angew. Math. 648, 2010, 201-244
  * [GGK] M. Green, P. Griffiths, M. Kerr, Mumford-Tate domains, Bollettino dell’ UMI (9) III (2010), 281-307
  * [J] Y. Jiang, The orbifold cohomology ring of simplicial toric stack bundles, Illinois J. of Math. Vol 52, No 2, 2008, 493-514
  * [FMN] B. Fantechi, E. Mann, F. Nironi, Smooth toric Deligne-Mumford Stacks, J. reine angew. Math. 648, 2010, 201-244
  * [GGK] M. Green, P. Griffiths, M. Kerr, Mumford-Tate domains, Bollettino dell’ UMI (9) III (2010), 281-307
  * [GS] A. Geraschenko, M. Satriano, Toric Stacks I, Theory of stacky fans, Trans. Amer. Mat. soc., Vol. 367, No 2, 2014, 1033-1071
  * [KP] M. Kerr, G. Pearlstein, Boundary components of Mumford-Tate domains, arxiv preprint
  * [KB] K. Behrend, Cohomology of Stacks, University of British Columbia, Lectures given at the school and conference on intersection theory and moduli, Sep. 9-27, 2002
  * [TH] T. Hayama, On the boundary of moduli spaces of log Hodge structures, II: Non-trivial torsors, Nagoya Math. J.Volume 213 (2014), 1-20.
  * [WF] W. Fulton, Introduction to toric varieties, princetorn University press, Princetorn, New Jersey 1993