By its turn, the class *MMint* #1 ::GroupRef is the derived correspondent of MOISE+::SubGroup. In the context of *MMint* #1 ::GroupDef the derivation of *MMint* #1 ::GroupRef is specified by the invariant GD3 shown in the bottom right of Figure 8. The invariant establishes that for each instance sg of *MMint* #1 ::GroupDef (referenced by subGroup), there must exist (be created) an instance gr of *MMint* #1 ::GroupRef pointing at sg and having the attributes gr.min = sg.supRoleDef.min and gr.max = sg.supRoleDef.max.

As long as they are more expressive, the classes RoleRef and GroupRef are used in the articulation graph replacing the references roleDef and subGroup present in *MMint* #1 . Therefore the replaced references are marked to be left out during the merge step at the end of the iteration. In addition, the attributes max and min in the class *MMint* #1 ::RoleDef are marked once the same information is now represented as attributes of RoleRef and GroupRef in the articulation graph.

 .

A second peculiarity of MOISE+ concerns the possibility of defining *links* and *compatibilities* between *roles* as part of a *group specification*. In this regard, there are two observation to be made. Firstly, *links* and *compatibilities* are new kinds of *role relation*, not present in *MMint* #1 . In fact, we observe that the *link* and *compatibility* concepts, in essence, differ from the *dependency* and *correspondence* concepts found in AGR. On the one hand, in MOISE+, a *link* enable the *acquaintance*, *communication*, or *authority* between roles; and a *compatibility* indicates that an agent playing a role can also play another role. On the other hand, in AGR, a *dependency* indicates that an agent only can play a role if it previously commits itself to another one; and a *correspondence* means that to play a role automatically implies to play another role.

The second observation is about the nature of the *group definition* concept behind the notions of *links* and *compatibilities*. By default, a *group definition* in MOISE+ does not enable the *compatibility* or any *link* between roles. In other words, if not stated explicitly, an agent playing a role do not have permission to play another role, or even to interact with agents playing another role, neither in the same nor in other groups. If *compatibilities* and *links* are needed, this must be explicitly specified in the *group definition*.

Conversely, in AGR and STEAM a *group definition* does not implies *a priori* any restriction regarding *compatibility* and *link* among the roles. With the exception of explicit *dependencies* relationships and cardinality restrictions, in AGR and STEAM specifications the agents are free to play the roles they want and are not blocked with respect to interacting with any agent playing some other role.

In the articulation presented in Figure 8, these observations are made explicit by means of the derived attributes allowsComm, allowsAcqu and allowsComp in the context *group definitions*. For MOISE+::GroupSpecification, these attributes have the value false. This represents respectively the *communication*, *acquaintance* and *compatibility* restrictions existing in MOISE+. On the other side, for *MMint* #1 ::GroupDef the three attributes assume the value true indicating the absence of the respective restrictions in AGR and STEAM.

Despite their opposite nature, we note that AGR and STEAM *group definitions* can be expressed in MOISE+. To this end, one has to explicitly define *communication*, *acquaintance*, and *compatibilities* relationships between all roles in a *group definition*. However, the converse is not always possible without losing information.

Ending the comparison, in MOISE+ there is a third form of *role relationship*: the *inheritance*. In MOISE+ metamodel this relationship is represented by the reference superRole involving instances of the class MOISE+::Role. In the articulation between MOISE+ and *MMint* #1 , an alternative form of representing inheritance could be as a subclass of RoleRel. As the inheritance relation does not have a direct effect on the behavior of the agents as the other role relations, being only a way of simplifying role definitions in MOISE+, we have opted to preserve the representation of this relation as the reference superRole rather than defining a new subclass for RoleRel.

Finally, concluding the iteration, we merge the metamodels taking into account the identified correspondences. The result is show in Figure 9.

**Figure 9.** Integrated metamodel *MMint* #2 .

#### **6. Organizational Interoperability Approach**

Adopting an organization-centered perspective [4], the engineering of MAS can be described as a process that starts with the creation of an *organizational specification* written in conformance to an *organizational model*. This specification is a prescription of the desired patterns of joint activity that should occur inside the MAS towards some desired purpose. Once the organizational specification is done, it is used as the input to an *organizational infrastructure*. In general, what we mean by organizational infrastructure is some kind of middleware supposed to interpret the specification and reify the *organization* of the MAS outside the agents. In this respect, it should maintain an internal *organizational state* and offer to the agents an interface for accessing and modifying this state. The information maintained in the organizational state contains a list of the members of the organization, what roles they are playing, what groups are active in the organization, among others. Finally, with the organizational infrastructure materializing the desired agent organization, it is time to develop application domain agents (not necessarily by the same designer of the organization specification) that can enter and interact inside it by accessing the available organizational interface.

Regarding *organizational infrastructures* there are several approaches for the engineering of (open) agent organizations [12,13,36,47,51–54]. On the one hand, the availability of a wide range of diverse models and infrastructures has made the development of agent organizations feasible. On the other hand, such a diversity introduced an important new interoperability challenge for agent designers: How to deal with heterogeneous organizational models and infrastructures? Whenever an agent is build to enter some MAS it has to be able to interact with the other participants using a particular agent communication language as well as to understand received messages against a given domain ontology. Besides this, if the MAS was designed as an agent organization, the entering agent has also to be able to access a particular organizational infrastructure and to interpret its underlying organizational model. In this way, the agent design can become tailored to a particular organizational approach.

For instance, suppose that several e-business applications designed as open agent organizations are available on the Internet. In addition, assume that these applications are heterogeneous regarding the organizational technology applied to build them. To put it in more concrete terms, let us suppose two agents organizations: One built upon the S-MOISE+ [53] organizational middleware, based on the MOISE+ model, and the other by using MADKIT [54] platform, based on ARG model. In this setup (and assuming a shared common agent communication language and domain ontology), the agent designers face the following problem: The native S-MOISE+ agents do not interoperate with the MADKIT platform, and vice-versa. Thus, it is not directly possible, for instance, to write an agent code that enter both e-business agent organizations in the search of products and/or services on behalf of its users. Such fact limits the range of applicability of S-MOISE+ and MADKIT agents which, in turn, limits the idea of open MAS.

As mentioned in the Introduction, four basic approaches can be envisioned for this organizational interoperability problem. One of them is to bridge the interface between the external agents and the agent organization by means of model mapping (Adaptation). By using such mappings it is possible to provide adapted copies of the specification and state of a given organizational model/infrastructure "understood" by the external agents.

In what follows we describe MAORI—a *Model-based Architecture for ORganizational Interoperability* [19]. MAORI is an experimental framework for providing organizational interoperability following the line of adaptation. Its main objective is to show how the integration of organizational models presented in this paper can possibly be used in a solution for the problem of organizational interoperability.

#### *6.1. MAORI Overview*

MAORI is structured along three layers, as it may be seen in Figure 10:

#### *6.2. Model Integration Layer*

M2M layer is composed of *metamodels* and *transformations*. For each organizational model *OMi*, there is a corresponding metamodel *MMi*. The metamodel *MMint* is the conceptual integration of all *MMi*, as described in Section 5.

The transformations are functions that implement the morphisms between the integrated metamodel *MMint* and the particular metamodels *MMi* (Section 3.4). There are two types of transformations. One type is **transf**(*f rom* : *MMi*) : *MMint*, which converts from *MMi* to *MMint*. The other type is **transf**(*f rom* : *MMint*) : *MMi*, which converts from *MMint* to *MMi*. In this way, M2M main functionality is to provide transformations that can be combined to translate specifications and states between organizational models/infrastructures.

#### *6.3. Organizational Interoperability Layer*

ORI layer works as an extension of organizational infrastructures. In order to enable heterogeneous agents in the same organization, ORI adds two basic components to the organizational infrastructures: *providers* and *adapters*.

Providers are responsible for exporting the organizational specification/state of agent organizations. In this case, to export means to use **transf**(*f rom* : *MMi*) : *MMint* to convert the specification/state from from a source *MMi* to the integrated metamodel *MMint*. Adapters are responsible for importing the organizational specification/state that was exported by a provider. The import is done by using **transf**(*f rom* : *MMint*) : *MMi*.

Imagine a scenario where an agent functions on a given organizational infrastructure and consider an agent organization implemented on a different organizational infrastructure. If the agent wants to participate in the organization, an adapter has to be instantiated in the organizational infrastructure of the entering agent. Initially, the responsibility of the adapter is to locate the appropriate provider, establish a connection with it, ask for the organizational specification/state and finally translate this specification/state to the target organizational infrastructure of the entering agent. In this way, for each heterogeneous agent organization there will be an organizational provider. Connected to this provider, there will be several organizational adapters, one for each organizational infrastructure in which there could be external heterogeneous agents.

**Figure 10.** MAORI framework (redesigned from [19]).

#### *6.4. MAORI Implementation*

MAORI was implemented in the Java programming language. The metamodels in M2M layer were coded using the Eclipse Modeling Framework (EMF) [55]. Regarding the transformations, there were first prototyped in the Atlas Transformation Language (ATL) [56] and then ported to Java for performance reasons. An implementation was developed as a proof of concept of the ORI layer considering the MADKIT and S-MOISE+ organizational infrastructures.

To evaluate MAORI, some agent organizations were developed. One is the example of a group of agents that wants to write a paper and use for this purpose an explicit organization to help them to collaborate. The organization consists in a group composed of: One agent in the role of *coordinator* (who controls the process and writes the introduction and conclusion of the paper), one to five agents in the role of *collaborator* (who writes the paper sections) and one agent in the role of *librarian* (who compiles the bibliography). Taking this simple example, some experiments were performed. One of them considered an organization composed of five agents—one coordinator (Eric), three collaborators (Greg, Joel and Mark) and one librarian (Carol). Initially the organization was started in the MADKIT platform. In addition, in MADKIT, the agents Eric and Carol were started and, after that, in the S-MOISE+ platform an organizational adapter was started to import the organization state. The three remaining agents (Greg, Joel and Carol) are started in S-MOISE+. They perceived and enter the organization by requesting the role of collaborator. At this point, the interaction begins: The agents in S-MOISE+ are now members of an organization running in MADKIT. More details about MAORI may be found in [19].

#### **7. Related Work**

The proposition of abstract structures, such as patterns, and integrated metamodel to enable interoperability among organization-centered multiagent systems is somehow new. Therefore, related work must be considered in several areas from business to services, including multiagent systems. In the following we contextualize our work within this broad scenario.

In the multiagent systems area, Pechoucek and Marik [57] adopted a model-driven approach to propose a general metamodel for developing multiagent systems. The metamodel is defined as a Platform Independent Model (PIM) considering the MDA abstraction levels definition [18]. In order to identify a unified metamodel to support the development of agent-based systems, they considered seven views: Multiagent view; Agent view; Behavioural view; Organization view; Role view; Interaction view and Environment view. They adopted the top down approach to develop the metamodels that represent each of the views aforementioned after analysing some existing agent-oriented modeling languages, methodologies and programming languages. Moreover, it was conceived to be used independently of the agent-oriented methodology, modeling and programming languages. Nevertheless, their main purpose was to support the development of MAS using a model-driven approach than providing means for interoperability among MAS. Our work presents some similarities with theirs since the proposed integrated metamodel could be used independently of the Organization Model adopted to design and implement an organization-centered MAS. In addition, the abstract structures where defined based on organization-centered multiagent systems dimensions, in a similar approach of theirs when stating their metamodel concerning some views. Nevertheless, although using a model driven approach, we adopted it to define the way integration occurs using the bottom-up approach to define the integrated metamodel, based on existing Agent Organization Models and their underlying metamodels. By doing that we foster interoperability to organization-centered MAS during design time, by providing means of transforming the design of a MAS with an underlying organization model into another one, or during execution time, as presented in Section 6.

Muramatsu and colleagues [58] provided organizational interoperability by using organizational artifacts within the environment where the MAS is situated. They adopted a normative language to describe the organizational structure in artifacts. In this sense, their work is similar to ours while adopting a common language (in our case a common metamodel) to describe several organizational models.

Isern and colleagues [59] classified organizational structures according to organizational paradigms, such as (i) hierarchy, (ii) holarchy, (iii) coalition, (iv) team, (v) congregation, (vi) society, (vii) federation and (viii) market, to support the design of MAS using existing agent-oriented methodologies and organizational models. The main purpose of their work is to provide information for MAS developers that would like to adopt an organization approach to develop MAS and did not know what Organization Model or agent-oriented methodology to choose. Their work is related to ours in the sense they adopted metamodels' characteristics of existing organizational models and existing agent-oriented methodologies to classify such organizational structures as patterns.

Karaenke et al. [60] proposed an inter-organizational interoperability architecture based on multiagent systems, web services and semantic web technologies. In their work, the MAS did not present an underlying organizational model and agents adopt the "head body" paradigm to include web services technologies to provide interoperability among enterprise information systems. Therefore, interoperability is focused on system-to-system communication using web services technologies, which limits the kind of systems that may participate in such communication. Our proposition is

broader in the sense that it provides a solution for interoperability among open organization-centered MAS independently of their underlying organizational Model.

A template description for agent-oriented patterns was given by Oluyomi and colleagues [61]. Based on a classification scheme, they organized agent technology concepts into categories and then identified agent-oriented pattern description templates for each category. Eight agent-oriented pattern templates were described to support the modeling of multiagent systems. Examples of the conformity between the proposed templates and their adoption during the design phase of existing agent-oriented methodologies were provided. Comparing with our proposition, their work adopts a similar approach when considering categories (in our case we adopted dimensions) to guide the patterns definition. In addition, their work is situated in the model level instead of the metamodel level as ours, since their objective is to improve communication among AOSE developers. Nevertheless, the adoption of an integrated metamodel obtained via model driven transformations combined with organizational dimensions give to our patterns high level of formality whenever compared with theirs.

Chella et al. [62] defined agent-oriented patterns to develop multiagent system to support robot programming. The proposed patterns were created based on an existing layered architecture for programming robots [63]. Patterns are described considering three aspects: The problem description; the definition of the solution in terms of MAS models and the description of the solution in terms of implementation. They just define some patterns for an specific domain based on the templates proposed by [61]. The only relation between their work and our is that pattern definition is based on some criteria that can be classified as dimension or category.

Organizational interoperability and integration issues are not new concerns for the administrative practice and research, specially after the wide acceptance and use of Information and Commmunication Technologies in their business models [64]. Several frameworks to provide organization interoperability were defined and even in this domain some dimensions were considered to define such frameworks.

#### **8. Conclusions and Future Work**

The research reported in this work has consisted in the use Model Driven Engineering techniques to address the *organizational interoperability problem*: How can we provide means for a set of agents, immersed in a common environment, to evolve, reason, decide and interact with each other based on organizational concepts, since their organizational models may differ? In order to achieve this goal, we have proposed an abstract and integrated view of the main concepts that have been used to specify agent organizations, based on the analysis of several organizational models present in the literature. In this model, we captured the recurring modeling concepts, that were coherently combined into an abstract conceptual structure. We have then presented an adaptation-based solution for the organizational interoperability problem, when we have defined the mappings between different organizational models, by using this abstract conceptual structure. We have built our abstract conceptual structure based on six organizational models (STEAM, MOISE+, AGR, OPERA, TEAMS, ISLANDER), presented in Section 2. For brevity, in Section 5 we illustrated the application of our integration method using three of these models (MOISE+, AGR, STEAM), and concerning exclusively the structural dimension.

A first extension of this work would be to build an integrated metamodel that could cope with OPERA and TEAMS. Moreover, we could evaluate how the other organizational models mentioned in Section 2.2.7 would affect our abstract conceptual structure. Concerning the MAORI framework, described in Section 6, we have tested its use by interoperating two organizational infrastructures, S-MOISE+ and MADKIT. A second extension of this work would be to test the franework with other organizational infrastructures, like AMELI [13] and ORA4MAS [52]. Finally, we would like to test our model driven approach to solve other MAS interoperability problems, like the ones mentioned in Section 1.

**Author Contributions:** Conceptualization, L.R.C, J.S.S. and O.B.; Formalization, implementation and experiments, L.R.C and A.A.F.B.; Supervision, J.S.S and O.B.; Writing and proofreading the paper, all authors.

**Funding:** "This research was partially funded by FAPEMA grant number 127/04 and CAPES, Brazil, Grant 1511/06-8". Jaime Sichman was partially supported by CNPq and FAPESP, Brazil. Anarosa Alves Franco Brandão was partially supported by grant #010/20620-5 and #014/03297-7, São Paulo Research Foundation (FAPESP), Brazil.

**Conflicts of Interest:** The authors declare no conflicts of interest.

#### **References**

c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **ARPS: A Framework for Development, Simulation, Evaluation, and Deployment of Multi-Agent Systems**

#### **Thiago Coelho Prado and Michael Bauer \***

Department of Computer Science, Western University, London, ON N6A 3K7, Canada<EMAIL_ADDRESS>**\*** Correspondence<EMAIL_ADDRESS>Tel.: +1-519-661-3562

Received: 18 September 2019; Accepted: 18 October 2019; Published: 23 October 2019

**Abstract:** Multi-Agent Systems (MASs) are often used to optimize the use of the resources available in an environment. A flaw during the modelling phase or an unanticipated scenario during their execution, however, can make the agents behave not as planned. As a consequence, the resources can be poorly utilized and operate sub-optimized, but it can also bring the resources into an unexpected state. Such problems can be mitigated if there is a controlled environment to test the agents' behaviour before deployment. To this end, a simulated environment provides not only a way to test the agents' behaviour under different common scenarios but test them as well in adverse and rare state conditions. With this in mind, we have developed ARPS, an open-source framework that can be used to design computational agents, evaluate them in a simulated environment modelled after a real one, and then deploy and manage them seamlessly in the actual environment when the results of their evaluation are satisfactory.

**Keywords:** multi-agent systems; discrete event simulator; interoperability; agent and multiagent applications

#### **1. Introduction**

In environments where resource management is critical, software agents can be employed to optimize those resources. When the task to accomplish this is too complex to be carried out by a single agent, multiple interacting agents can be used to achieve effectiveness. The Multi-Agent System (MAS) approach has been successfully applied in many domains, including: helping controllers in Air Traffic Control for making decisions based on the aircrafts' fuel availability, flight plan, weather, and any other relevant data [1]; optimizing distributed generators in smart grids for energy production, storage, and distribution [2]; or finding better arrangements for the components in manufacturing plants to increase the throughput and optimize material usage [3].

Many frameworks and toolkits enable the implementation of a MAS. They are not designed to provide seamless means of evaluation of the outcome of the agents in the system under certain scenarios before deployment. A flaw during the design and implementation of the agents, or an unpredicted state of the environment where they are acting, can not only interfere with the achievement of their goals, but can also bring the environment into a unexpected situation with unforeseen consequences.

Some work with MASs has made use of simulations to aid deployed agents to update their plans according to the current state of the environment. This means that the simulation is used as a planner tool, meant to help the agent to make a decision under a specified near future scenario rather than using the simulation in the process of designing the agent behaviour before deployment. This strategy has been applied to the field of multiple Unmanned Aerial Vehicles (UAV) [4], where communication is required for coordination and failures related to it can make the entities unreachable. The system simulates possible scenarios where communication is unavailable and an action by the UAV is expected. Another example is the simulation component used to detect conflicts and inconsistencies of resource allocation during the high-level planning in a manufacturing plant [5].

There is no general purpose MAS framework, to the best of our knowledge, that integrates the process of validation of the agents in a simulated environment before their deployment. To address this, we have developed ARPS, an open-source framework available at https://gitlab.com/arps/arps/ under MIT LICENSE [6], to seamlessly design, implement, assess the agents, and deploy the MAS after the results meet established criteria. ARPS stands for some of the core properties of agents: autonomy, reactivity, proactivity, and social ability. We use the management of resources of a data centre to illustrate our approach and how it can be used in other domains.

In Section 2 we cover related work. Following this, Section 3 describes the background and architecture of the framework. The process of the implementation of the MAS to manage an experimental scenario is shown in Section 4. In Section 5 we discuss the findings, limitations, and future directions.

#### **2. Related Work**

There are multiple toolkits, platforms, and frameworks available for creating a MAS. In this section, we will describe a few of them. The works here by no means represent the only alternatives for creating MASs. For other options, refer to surveys on this topic, such as the one presented in [7].

The works reviewed here implement common aspects of a MAS to work in an actual environment, such as communication, interoperability, storage, security, and resource discovery. Therefore, the users can focus on the definition of the agents and their behaviour, how they are organized, and how they interact to solve a problem.

Among the popular MAS frameworks, JADE (also known as Java Agent Development Framework) [8] was created to address the problem of interoperability and provide an environment for the development of agents. It has no domain dependent requirement as seen in the other solutions. According to the authors, such dependencies were obstacles for the adoption of MAS technologies at the time of its conception. JADE simplifies the implementation of multi-agent systems through a middleware compliant with the FIPA (Foundation for Intelligent Physical Agents) specifications [9], a standard proposed for interoperability of agents. The JADE authors argue that is industry-driven and currently the most known FIPA-compliant agent platform in the academic and industrial community.

The A-Globe platform [10] is designed for testing experimental scenarios featuring agents' position that requires a Geographical Information System (GIS, though agents may suffer from communication inaccessibility either because of the spatial distance of the agents or by broken links. Because it is a closed-environment, interoperability is not one of the concerns of this platform. Hence, it is not fully compliant with the FIPA-specifications on inter-platform communication, albeit it provides compliance with the Agent Communication Language (ACL), a structure for composing messages exchanged by agents.

Based on the fulfillment of requirements such as robustness, security, and the ability to ensure that a partial solution can be executed when an optimized one is not found due to constraints, DARPA funded a project called Cougaar [11] (Cognitive Agent Architecture). This agent platform was created to offer specialized support for logistics-related problems. This platform is also not FIPA-compliant. It aims to facilitate the development of agent-based applications that are complex, large scale and distributed.

Other solutions offer more components built on top of the existing agent-based platforms. The Jadex BDI Agent System [12] follows the Belief Desire Intention (BDI) model [13] and facilitates intelligent agent construction over other middleware such as JADE. It has been used to build applications in different domains, such as simulation, scheduling, and mobile computing. The programming model of Jadex allows for designing an application as hierarchical decomposition of components interacting via services and thus helps to make complexity manageable. These components can be used in concurrent and dynamic distributed systems. Another example is JaCaMo [14]. It is an interpreter for an extended version of AgentSpeak, a BDI agent-oriented logic programming language [15]. It is a platform that integrates three projects with different MAS-related paradigm

models for development: Jason [16] for agent development; Moise [17] for agent organization; and CArtAgO [18] for environment-oriented programming.

Lastly, the MadKit [19] offers a modular and scalable multiagent platform. Its central aspect is the ability to organize agents in groups and roles aiming at the development of artificial societies. It is closely related to our approach in the sense that has a simulator component. Nonetheless, the simulation and evaluation are not seamless in the workflow, leaving the responsibility of this integration to the user.

Previous work has often presented MAS tools that provide a base infrastructure to implement agents. These approaches also looked to help address other specific problems, such as interoperability by defining standards, robustness and security. In some cases, the work sought to introduce approaches to enrich the design process, provide clear definition of how agents are organized and what their roles are. However, these solutions lack a component that would allow a developer to assess the behaviour of the agents in a simulated environment before actual deployment. Further, this step should be as seamless as possible, i.e., it should take no or few modifications to alter agents in the simulated environment in order to make them run in the actual environment. This is the gap that ARPS fills.

#### *Agent-Based Modelling (ABM)*

One of the main solutions available to simulate complex systems is known as Agent-based modelling (ABM). It is employed in domains such as social sciences, biology, ecology, engineering, and economics. There are many platforms to implement ABM [20,21]. Among them, we list a few widely adopted examples. The Swarm package [22] provides object-oriented libraries of reusable components for building models and analyzing, displaying, and controlling experiments on those models. NetLogo [23] is a software that provides packages to simulate multi-agent in environments. It has a significant user community, it is highly documented, and they provide many demos. MASON [24] is a discrete event multi-agent simulation toolkit. It was designed to serve as the basis for a wide range of multi-agent simulation tasks ranging from swarm robotics to machine learning to social complexity environments. The Repast platform, initially developed as a tool to be used in social sciences, is a family of agent-based modelling and simulation platforms. Currently, not only does it provide a package to be used in regular environments, such as desktops, and laptops [25], but also has an advanced version to be used in HPC environments [26] to simulate more demanding scenarios.

The ABM and MAS concepts are closely related to each other since both are agent-based. The difference is that the modelling in ABM aims to gain insight about emergent properties in complex adaptive systems while MAS focus on actual agents [27,28].

Our framework aims to provide an environment to combine ABM properties, such as the ability to model agents in a simulated environment and observe their behaviour, with MAS's ability to implement and deploy the agents evaluated during the simulation in the action environment. The integration of both approaches can yield positive results. The framework in [29], features this combination. It enables mobile agents to work simultaneously both in the actual and virtualized agent platform to enable large-scale simulation to observe unknown emergence behaviour. This is accomplished by having the agents in the actual environment collecting data in real-time, and improving the simulation, which in turn can have its outcome used in decision making. In our case, we need a separate simulator component to enable the study of the agents' behaviour without disrupting the environment before their deployment. This then enables a developer to leverage the construction of reliable physical agents by evaluating their interactions and the effects of their actions. Because simulation can involve the compression of time, it is possible to simulate many different scenarios efficiently. Also, during the simulation, it is possible to create unexpected scenarios to see how the agents perform. This can be desirable in areas that already employ ABM for resource management during a disaster, such as [30], that could be extended to have software agents that could direct resources where it is necessary. We are aiming to combine the characteristics of both simulation and development to offer an alternative for creating solutions in resource management using MAS.

Figure 1 illustrates our workflow to achieve this integration. A user/admin can create the environment, its resources, and the agents' models, and define how they are organized. The MAS environment is generated, and the user can simulate a scenario. After analyzing the simulation results, the user can decide to refine the models or apply the agents' models to the actual environment. The output of this environment can also be used as feedback to improve the existing models and create a more reliable simulation.

As we have discussed, none of the previoius MAS frameworks have a simulator component that allows this workflow without having to reimplement the concepts defined during the simulation into the actual environment.

**Figure 1.** Workflow for MAS conception, evaluation, and deployment.

#### **3. Our Approach**

Below we describe the background of the framework, our design choices, and present the architecture.

#### *3.1. Background*

Data centres present complex dynamic environments where many resources are allocated to provide guaranteed, reliable services. These resources are not only related directly to the computer systems, such as processing power, storage capacity, and network, but also to the supporting equipment and environmental control. Data centre administrators face a daunting task in trying to manage them optimally. There are impacts when these resources are misconfigured or overallocated on the total cost of ownership because of excess power consumption or idle resources. Data centre operators can also incur financial penalties due to the broken guarantees related to service delivery.

One proposed way to tackle this problem is the adoption of autonomic computing architectures and strategies. An autonomic approach aims to embody the idea of self-management, which in turn can be realized by decomposing it into other sub-properties, such as self-configuration, self-healing, self-optimization, and self-protection [31]. This separation of concerns can be managed by decentralized autonomous agents that may interact with each other, optimizing local resources to achieve global optimization, as exemplified in [32], where a MAS manages the number of hosts available to process workloads depending on the demand.

One problem faced in the employment of MASs in data centres is the impracticality of having an actual data centre available to evaluate the effectiveness of the policies governing agents due to costs and security concerns. In some cases, data centre simulators, such as that described in [33], have been developed to evaluate the possible impacts of the autonomic agents policies in the data centre before implementing and deploying them in the real environment. Even so, there is a gap, however, in the development between the simulation step and the implementation and deployment of the actual agents. To address this problem, a framework was developed using initially the concepts of the aforementioned simulator with the additional feature of employing the assessed policies during the simulation to drive the software agents that will manage the actual resources in the environment.

Contrary to most of the works related to MAS platforms presented in the previous sections, our approach was first developed to solve a domain-specific problem focused on self-management of resources. During the development, the generic components were extracted to allow more flexibility during the implementation and the test of the proposed solutions. This resulted in the general purpose ARPS framework to enable MAS.

The generalization of the scenarios where our MAS framework can apply, although it is not limited by, is illustrated in Figure 2. As can be seen, there is a complex system with resources (*Rn*), grouped by environments (*Environmentm*), that are affected by external events (*E*). The events occur at a variable or fixed interval in this system. Without any management, these resources can be in a suboptimal state. To overcome this problem, agents (*Ai*), driven by policies, are employed to monitor resources or modify them using available touchpoints. The agents can be reactive, proactive. They can act in isolation, or they can communicate with each other for cooperation, coordination, or negotiation.

**Figure 2.** Scenario for resources being modified by external events while agents manage them.

The framework is being currently developed in Python 3, a high-level multiplatform language that can be deployed in myriad host platforms, including Internet of Things (IoT) devices [34,35]. The users can install the framework from source code or as a Python package, available at https://pypi.org/ project/arps/. This means that, when implementing the agents for a specific domain, the user needs to implement all the components using Python. Albeit there is no single programming language that can be applied in every domain effectively, Python has been suggested as an alternative for a general programming language to be adopted by the scientific community and it has been used by researchers in areas not related to technology or engineering, such as psychology and astronomy [36,37].

#### *3.2. Architecture*

The ARPS framework is composed of four main components: agent manager, agents, discovery service/yellow pages service and Discrete Event Simulator (DES). The architecture and relationship of the first three are illustrated in Figure 3. The agent manager has three main aspects: the management of the availability of resources and policies related to an environment, agent life-cycle, and simulation

when it is running for this purpose. It acts as a container for the agents. This container groups resources logically by some criteria, like resource similarity, or accessibility. Agent managers can be distributed across systems. The agent is an entity driven by policies that will manage one or more resources. It can interact with all other agents available by the discovery service. Since each agent manager can be deployed in distributed manner, the discovery service is a directory where its agents are registered when created and their location is made available to be discovered by other requesters, so the agent from one agent manager can exchange messages with agents from others agent managers. Lastly, the DES is a component available for the evaluation of the agents, and its integration with the others will be explained in the following sections.

**Figure 3.** Architecture of the ARPS framework.

#### 3.2.1. Interoperability

The agent manager, agents, and discovery service implement the RESTful architecture style for interoperability. This is done using HTTP methods, and the payload is in JSON format. This RESTful API employs uniform resource access using the format http://hostname:port/resource/, where the *hostname:port* are the host name and listening port to access the API respectively, and the *resource* is a web resource. The HTTP methods (GET, POST, PUT, and DELETE) are available for the clients. The payload of the POST and PUT methods, as well as the possible HTTP response codes, were omitted for simplification purposes. Their descriptions are available at https://gitlab.com/arps/arps/wikis/ home. Below we present the API to perform the requisitions.

The agent manager API is intended to fulfill requests by the user. The summary in Table 1 shows how the policies that can be used to drive agents' behaviour can be listed, the agents that are currently running in its environment, the touchpoints that agents can monitor/control, and, when there are agents created just for the purpose of monitoring resources, a timestamped log containing the state of the resources can be retrieved.

The life-cycle of the agents is controlled by creation, modification, inspection, and termination methods, as seen in Table 2.

**Table 1.** REST API for environment.

**Table 2.** Agent Manager API for agent life-cycle.

Lastly, the REST API is available only when the agent manager is created to run the simulator is seen in Table 3.

**Table 3.** Agent Manager API in simulator mode.

The agents have their API described in Table 4. It is not only accessed with the intent of message exchange among the agents, but it can also be called by users or other applications.

The API provided by the discovery service, shown in Table 5, is used to register, unregister, and list the running agents in the system.

**Table 5.** Discovery Service.

We are aware of the existence of FIPA-HTTP to make compliant FIPA agents using HTTP [38]. This standard, however, only supports the POST method with the semantics of the message embedded in a multi-part content sent in the body of the message. This defeats the purpose of the semantics presented by the HTTP methods, and the already available HTTP status codes related to each HTTP method. Some studies that support this view of interoperability in MAS, using Resource Oriented Architecture (ROA), can be found in [39–41],

There are many advantages of using RESTful architecture style for interoperability. The system is open, so applications that the system designers did not take into account a priori can be integrated into the existing system since the web resources are accessed uniformly. Secure communication can be implemented over HTTPS. A cache can be used to save bandwidth and to optimize system communication. Visualization through different devices can be easily implemented by third-party entities across different devices. The support of the MAS for the REST API can also be used to extend the API and make it compliant with FIPA-HTTP if required.

#### 3.2.2. Agent Model

The intelligent agent model, as described by Russel & Norvig [42], is used to define the agent. Thus, besides the conventional definition of the agent, as an entity perceiving the environment through sensors and modifying it through actuators, it has also the components to achieve reasoning. To this end, the agent's behaviour is driven by a set of policies that are executed by its control loop. A policy can be activated either by interaction with other agents, or internally by a fixed interval. Each policy has access to the touchpoints available in the environment provided by the agent during its creation. Another characteristic is that policies can be dynamically added or removed.

The policies can be reactive or proactive. The reactive policy follows the Event Condition Action (ECA) rule. This model is used to create simple reflex agents. Thus, the agent monitors the environment, and, when the current state matches a condition, a predefined action is performed. Alternatively, the agents can optimize the environment continuously. In this case, their behaviour can be defined in terms of proactive policies. To this end, the user can extend the interface to implement goal-based or utility-based policies.

Another characteristic is related to how agents are organized. The framework defines that an agent has a unidirectional relationship with other agents. This relationship is also dynamic and can be created or removed during the agent lifetime. Since this model does not impose any form of organization and agents communicate using a peer-to-peer architecture, they can be organized hierarchically, or horizontally. Agents can have a relationship established with any other agent available through the yellow pages service. Currently, there is no support to enforce groups or roles.

#### 3.2.3. Simulation

During the simulation, only a single instance of an agent manager is necessary since it will act as the gateway to all other virtual agent managers that would be available in the system, as seen in Figure 4a. Similarly to the actual environment, the format used by the API is http://hostname: port/agent\_manager/resource/, where *agent*\_*manager* antecedes *resource* to identify the virtual agent manager that will perform the requisition.

In the actual environment, the transport system used to exchange messages between agents is implemented using the HTTP protocol, as seen previously. It relies on the physical network to function. During the simulation, however, this protocol is substituted by a global bus that will serve as a discovery service and communication layer at the same time. The content sent from one agent is put directly into the queue of messages of the receiver agent. The difference between real and simulated communication is seen in Figure 4b. This has the advantage of removing the overhead of communication. Additionally, it gives the freedom to the user to apply models into this layer to improve the reliability of the agents to evaluate them under intermittent communication, increased latency, or corrupted messages.

The resource is another component that has to be overridden by the models that describe how they behave in the real environment. For example, in our case, we can model the behaviour of the computational resources to have a certain load accordingly to the task that arrived in the system. As well, our model supports resources that affect other resources indirectly. Revisiting the previous example, we can create a model that increases the energy consumption based on the load in the system when the computational resources are utilized.

The main component of the DES is the events generator. It supports both deterministic and stochastic models. The former uses a log file containing the events—when the last event is completed the simulation is terminated—while the latter uses a stochastic generator implemented by the user—the termination needs to be invoked explicitly. Each event has two actions. The main action that is executed every step while the event is still unfinished, and a post-action, when the event exits the system. Figure 4c showing the resource being modified by an event during the simulation and in Figure 4d showing the queue of deterministic events illustrates the DES component working along with the agents

**Figure 4.** Agent Manager in Simulator Mode.

When finished, the results of the simulation are made available in the CSV format. We have chosen this format since it is supported by a myriad of statistical tools, such as Pandas (https:// pandas.pydata.org/), R (https://www.r-project.org/), or spreadsheet-like apps such as LibreOffice (https://www.libreoffice.org/). We believe these third-party tools are better equipped to fulfill the needs of the user.

#### **4. Demonstration of the Framework**

In this section, we will illustrate how the ARPS framework can be used to solve the problem of resource management. In the next section, we describe the components needed to be implement to enable the MAS. Following this, we present an example in a specific domain: management of resources in data centres.

#### *4.1. ARPS Framework Usage*

The modelling of the MAS for resource management can be accomplished by following the steps:

The usage of the framework can be summarized as the implementation of some interfaces, and the creation of the configuration files needed by the agent managers and agents.

We have created the abstraction of the resources that provides the touchpoints to be accessed by sensors and actuators, seen in Figure 5.

Starting with the sensors and a minimal set of the configuration files (files that will be described in detail later in this section), it is possible to execute the agents in the actual environment to collect data since a set of monitoring policies related to the implemented resources are made available automatically. Therefore, it is possible to initially create agents with the only the purpose of gathering data periodically. This data provides insight about the actual resources . This will be essential during the modelling of the environment used by the simulator. This model can be later updated when comparing the simulated environment with the actual environment.

**Figure 5.** Resources, Sensors, and Actuators structure diagrams.

Once the resources and sensors are implemented, the next step is the implementation of the actuators. The actuators are used by the policies that drive the behaviour of the agents. These policies can be created by implementing the Policy interface, illustrated by a reflex policy in Figure 6. Since our approach is based on autonomic computing concepts [43], other types of policies, employing utility functions, or goal-based approaches can be implemented. The method *condition(event)* can be implemented to use optimization algorithms, defined by the user, that will compute the ideal parameters and then use the results to modify the resources through their actuators. Goal-based approaches would require the user to model the states and use the method *condition(event)* to search for the best action to perform to achieve a better state. Both policies would be executed as periodic

policies, so the event is just a time event indicating that it is time for the policy to evaluate the current state and executes its actions based on the perceived environment.

**Figure 6.** Policy structure diagram.

Since the policies encapsulate the perception and modification of the resources, and these resources can be modified by external events, the next step is the modelling of the these events. The events arrival in the system can be stochastic or deterministic and it can be created by implementing the interface EventQueueLoader. The events are encapsulated in the SimEvent class, which in turn contains the SimTask that will be executed during the simulation. The SimTask *main* method should provide the behaviour that will modify a resource, while the *pos* method will contain all the finalization process to be executed, like resource release, after the task is finished. The relationship between the DES components is illustrated in Figure 7.

**Figure 7.** Simulator structure diagram.

Given the structure present in Figure 8, the final step before running the MAS in the actual or simulated environment is the creation of the configuration files, in JSON format, containing the implemented files previously described in this section. Both configuration files are similar in their structure. The file *simulation.conf* contains the paths of the files related to the fake resources, and the DES component classes, while the file *real.conf* contains only the actual resources. The remaining classes, like policies, sensors, and actuators, remain the same for both configuration files. Therefore, the transition of the agents from the simulated environment to the real environment can be done seamlessly.

After the instantiation of the agent manager in simulation mode, using the *simulation.conf* configuration file, the REST API presented in the previous section can be used to create the agents, organize them, and run, stop, and collect the result of the simulation.

Based on the result, only the policies classes need to be modified. Then, using the *real.conf* configuration file, the agents can be deployed in the actual environment. According to the behaviour observed in the actual environment, improvements can be made in models used by the simulated environment. Thus, it is only necessary to modify existing fake resources, simulation tasks and policies. Incrementally, new resources, touchpoints, and policies can be added into the MAS to improve the system.

**Figure 8.** The minimal structure required.

#### *4.2. Example: Data Centre Management*

As previously discussed in Section 3, we are researching ways to address the problems in trying to optimize energy efficiency in the data centre while ensuring that other performance metrics are met using MAS. To illustrate this, we provide a minimal example to cover the basic setup of the ARPS framework. The source code is available at https://gitlab.com/arps/arps/tree/main/arps/examples/ computational\_resources\_management.

This example by no means provides an in-depth analysis of resource management in data centres and only serves to illustrate how the MAS can be realized. To keep this example simple, we will manage only two resources from a single host: the CPU and the Energy Monitor. Additional resources, such as temperature sensors of the environment, cooling system, and multiple hosts, can be added iteratively as the need for better understanding the system arises.

The CPU provides means to both read its current utilization or modify its frequency. It is driven by two governors: performance and powersave. This enables dynamic frequency scaling. The maximum frequency is 2.9 GHz in each one of its four cores and when in performance mode, the frequency is adjusted dynamically. Conversely, in powersave mode, the frequency stays closer to the minimum frequency available. The Energy Monitor only provides an interface to read the estimated power consumption in watts using the tool *PowerTOP* [44]. This tool was developed by Intel, and provides estimates on power consumption.

To understand how energy consumption is affected when the CPU has a certain workload, we collected the data using the sensors. As described in the previous section, when a resource is implemented, a monitor policy related to it is made available by the Agent Manager. In this case, it is possible to create agents executing the CPUMonitorPolicy and EnergyMonitorPolicy policies periodically. The collected data were summarized into two charts, where the Y-axis represents the normalized value of the CPU workload and the estimated power. The X-axis is the time in seconds. In Figure 9, the CPU is using a performance governor, while in Figure 10 the CPU is using powersave governor. As expected, there is a correlation between workload and energy consumption in performance mode since the CPU adjusts its frequency during the execution. The same does not apply in the powersave mode, where the energy consumption almost never goes over a certain value.

**Figure 9.** Correlation of energy consumption and CPU workload using performance governor.

**Figure 10.** Correlation of energy consumption and CPU workload using powersave governor.

To reduce energy consumption, for example, we can use reactive policies implementing the method *condition(event)* of the Policy interface to evaluate the workload in the CPU. The action performed by the policy can be the adjustment of the CPU's frequency to a lower level. In this example, we considered two different strategies for saving energy. The strategies involve three policies; these policies are presented in Figure 11. The first adjusts the CPU to powersave mode when the utilization is over 80% while in the second sets the CPU to powersave mode when the utilization is over 50%. The third returns the CPU to performance mode when the utilization is under 50. The first strategy is composed of policies PowersaveDynamicScalingPolicy1 and PerformanceDynamicScalingPolicy; strategy two uses PowersaveDynamicScalingPolicy2.

We have chosen batch jobs to represent the external tasks that modify the environment to evaluate how the agents will manage energy consumption. In this case, we opted to use the deterministic model of the job arrival using an external file of events, where each row represents one event arriving in the system containing the arrival time, duration of the task, and workload. The SimTask *main* method, seen in Figure 12, allocates the CPU resource, and consequently causes additional power consumption proportionately to the workload when the CPU is performance mode and more steady when the CPU is in powersave mode, as mentioned previously. The SimTask *pos* method would release the resource as soon as the task is completed.

```
class PowersaveDynamicScalingPolicy1 (Policy ):
    def condition ( self , event ):
         return s el f . cpu . read () > 80
    def action ( self ):
         self . cpu freq . set ( governor=' powersave ' , max frequency=0.9)
class PowersaveDynamicScalingPolicy2 (Policy ):
    def condition ( self , event ):
         return s el f . cpu . read () > 50
    def action ( self ):
         self . cpu freq . set ( governor=' powersave ' , max frequency=0.9)
class PerformanceDynamicScalingPolicy (Policy ):
    def condition ( self , event ):
         return s el f . cpu . read () <= 50
    def action ():
         self . cpu freq . set ( governor=' per formance ' , max frequency=2.9)
```
**Figure 11.**  Example implementation of a task to update the energy monitor based on CPU workload.

The first analysis suggests that the average energy consumption with the first strategy is 9 Watts while for the second one it is 7.57 Watts. Without taking any other resources correlated to energy consumption and CPU, we can infer that the second one is better. However, other resources or metrics, such as a metric on Quality of Service (QoS), can be affected by the CPU dynamic frequency scaling of strategy two; these could be included in the model for further investigation.

When comparing the policies in the real environment, a similar behaviour is observed. The average energy consumption in the first policy is 13.75 Watts while in the second one is 12.34 Watts. These results come from a very limited set of experiments and comparisons and so no significance can be determined. Again, this was meant as example to illustrate how the ARPS framework can be used.

#### **5. Discussion**

We presented ARPS, a framework to enable evaluation of a MAS before deployment in an actual environment. This framework is the result of a general purpose MAS solution developed to be as flexible as possible to provide management of a data centre. It has proven useful in a specific domain, and we believe that others in the MAS community can benefit from this additional option when developing solutions to their problems. To this end, we illustrated how it could be applied using our specific case.

Due to the decision of using the RESTful architecture style, agents can be deployed manually by a user in an environment that would contain a single agent, without the need of an agent manager. This unintended feature is possible because agents are self-contained and all the operations are made available through the API. The interaction can be done using web clients widely available on the Internet. The only requirement is that the platform can run Python applications. Thus, a user can instantiate an agent to control an autonomous robot developed in a platform such as a RaspberryPI, or it can deploy an agent in devices of an IoT environment.

We have identified other features that could enrich this framework. The inclusion of a Policy Management Tool can ease the process of creating new policies dynamically using a high-level language. Also, new resources and their touchpoints could be made available on the fly or dynamically removing unavailable resources. This could improve the integration of simulation and deployment even more. We plan to add a learning component to make agents more robust to complex adaptive environments. Further, reliability characteristics, such as fault tolerance, and security concerns, are being addressed and will be included in future versions.

Currently, the framework has some limitations related to features that were set aside due to low priorities. The framework does not come with a visualization component. Thus, it is not possible to visualize the running agents both during the simulation or within the real environment. Also, because we were aiming to have actual agents, we did not evaluate the usage of this framework applied to problems entailing only agent-based modelling simulation (ABMS). Since ABMS and MAS concepts overlaps, this MAS framework could be adapted to be used as ABMS tool. We do not know, however, how it performs when millions of agents are deployed in the environment. The framework does include statistical tools to analyze the results of the simulation or to assess the system in the actual environment since we believe that each user has their own needs and preference for tools. As mentioned, all the data gathered via the framework is made available in CSV format and so other applications that can provide more suitable tools can be used to extract insightful information from the data. Lastly, the framework is not FIPA compliant yet.

**Author Contributions:** Conceptualization, T.C.P. and M.B.; Methodology, T.C.P. and M.B.; Software, T.C.P.; Supervision, M.B.; Visualization, T.C.P.; Writing—original draft, T.C.P. and M.B.; Writing—review & editing, T.C.P. and M.B.

**Funding:** This research is partially supported by CNPq, Brazil grant number 249237/2013-0.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Consensus Algorithms Based Multi-Robot Formation Control under Noise and Time Delay Conditions**

#### **Heng Wei 1,\*, Qiang Lv 1, Nanxun Duo 1, GuoSheng Wang <sup>1</sup> and Bing Liang <sup>2</sup>**

Received: 2 January 2019; Accepted: 6 March 2019; Published: 11 March 2019

**Abstract:** In recent years, the formation control of multi-mobile robots has been widely investigated by researchers. With increasing numbers of robots in the formation, distributed formation control has become the development trend of multi-mobile robot formation control, and the consensus problem is the most basic problem in the distributed multi-mobile robot control algorithm. Therefore, it is very important to analyze the consensus of multi-mobile robot systems. There are already mature and sophisticated strategies solving the consensus problem in ideal environments. However, in practical applications, uncertain factors like communication noise, communication delay and measurement errors will still lead to many problems in multi-robot formation control. In this paper, the consensus problem of second-order multi-robot systems with multiple time delays and noises is analyzed. The characteristic equation of the system is transformed into a quadratic polynomial of pure imaginary eigenvalues using the frequency domain analysis method, and then the critical stability state of the maximum time delay under noisy conditions is obtained. When all robot delays are less than the maximum time delay, the system can be stabilized and achieve consensus. Compared with the traditional Lyapunov method, this algorithm has lower conservativeness, and it is easier to extend the results to higher-order multi-robot systems. Finally, the results are verified by numerical simulation using MATLAB/Simulink. At the same time, a multi-mobile robot platform is built, and the proposed algorithm is applied to an actual multi-robot system. The experimental results show that the proposed algorithm is finally able to achieve the consensus of the second-order multi-robot system under delay and noise interference.

**Keywords:** multi-robot; consensus problem; formation control; noise; time delay

#### **1. Introduction**

In recent years, with the continuous development of computer science, complex network theory and control theory, autonomous mobile robots have received more and more attention [1]. Compared to single mobile robots, multi-mobile robot systems have better stability, higher fault tolerance and higher work efficiency. As a result, they have better application prospects and higher research value in the fields of reconnaissance, patrol, rescue and environmental survey. Formation control of multi-mobile robots is the basis of multi-mobile robot systems, and has become a hotspot in the field of robotics [2].

As part of the design process of multi-robot formation control algorithm, many problems need to be considered, including robot model, external environmental interference, sensor measurement noise, algorithm control precision, and the controllability of different formations [3]. The existing formation control algorithms for multi-robots mainly include the leader-follower algorithm [4], the behavior-based algorithm [5], the graph theory-based method [6], the virtual structure method [7], and the artificial potential field method. The leader-follower algorithm has flexible motion strategy and scalability, but the algorithm cannot form stable and reliable feedback between the follower and the leader. Therefore, the control error of the follower will increase with interference from the environment. In particular, when the leader fails, it can cause the entire multi-robot system to crash. The behavior-based algorithm can effectively reduce the complexity of the entire formation control algorithm, but it has higher requirements in terms of sensor sensing ability and communication ability between robots, and cannot accurately quantify the behavior of robots during operation. Thus, it is difficult to guarantee the system's robustness using the behavior-based algorithm. The virtual structure method is convenient for designing the formation behavior of multi-robot systems, while due to the constraints of rigid structures, it lacks flexibility with respect to obstacle avoidance and formation transformation. The artificial potential field algorithm has a simple structure and can effectively avoid collisions and obstacles, but it is susceptible to interference when maintaining the formation, and it is difficult to perform precise formation control. Moreover, the potential energy function needs to be reset if the formation transformation is performed, leading to a lack of flexibility.

In view of the shortcomings of the traditional formation control algorithm, considering the increase in the number of robots in the multi-robot system and the continuous improvement of the data processing capability of a single robot, the distributed multi-robot control algorithm has attracted the attention of researchers. The distributed multi-robot system can make full use of the data processing resources of the robot and share the pressure of the central processing machine, which has great advantages in terms of flexibility and fault tolerance [8,9]. In addition, solving the consensus problem is the core of the distributed multi-robot control algorithm [10]. There are already mature and sophisticated strategies for solving the consensus problem in ideal environments [11,12]. However, in practical applications, uncertain factors like communication noise, communication delay and measurement error will still lead to many problems in multi-robot formation control. Some algorithms have considered some practical problems. Reference [13] studied the conditions of the system reaching consensus under uniform delay, when the communication structures of second-order multi-robot systems were a directed graph with spanning tree or a strongly connected graph, respectively. However, that paper does not consider the noise condition or consensus under different delay conditions. Reference [14] studied the consensus problem of second-order multi-robot systems under noisy conditions. A control protocol based on distributed sampling data was proposed to achieve system consensus, but the delay condition was not taken into account in the algorithm. Reference [15] studied the consensus of second-order multi-robot systems under non-uniform and multi-time delays using the frequency domain analysis method. Compared with the Lyapunov method, it has lower conservativeness, and the results were extended to higher-order multi-robot systems. However, it did not take noise into consideration, which is unavoidable in practical environments. Reference [16] studied the consensus of second-order multi-robot systems under uniform time delay and noise environments, and designed different control protocols for different types of noise, thus achieving the consensus of the system. These algorithms provide some basic solutions to the second-order system consensus problem, but the problems encountered by multi-robots in practical applications are far more varied than these. On the basis of these algorithms, this paper performs a more in-depth analysis, especially considering the consensus of the second-order system in which there are many different time delays and multiplicative noises in the system, laying the foundations for a formation control algorithm for second-order multi-robot systems that can be truly implemented in real robot systems.

In summary, this paper analyzes the consensus problem of second-order multi-robot systems under various delay and noise conditions. The system character equations are transformed into quadratic polynomials of pure imaginary eigenvalues based on frequency domain analysis, and then solved. Finally, its critical steady state is obtained and verified using Matlab numerical simulation. Compared with existing algorithms, this algorithm has lower conservativeness, and it is easier to extend the results to higher-order multi-robot systems. Since the omnidirectional mobile robot is a fully driven robot, and the horizontal and vertical directions can be separately controlled, it can be constructed as two one-dimensional second-order multi-robot systems. Therefore, experiments

were carried out on a multi-omnidirectional mobile robot platform built in the laboratory using the proposed algorithm [17,18], which verifies the effectiveness of the proposed algorithm.

#### **2. Pre-Preparation and Problem Description**

#### *2.1. Graph Theory*

*<sup>G</sup>* <sup>=</sup> {*V*, *<sup>E</sup>*} represents the communication topology between robots, in which each robot represents a node. *V* is a set of nodes. *E* is a set of edges, representing the connection state between robots. The topology map is represented by a Laplacian matrix, which is *<sup>L</sup>* <sup>=</sup> *<sup>D</sup>* <sup>−</sup> *<sup>A</sup>*. *<sup>D</sup>* is the degree matrix, which represents how many nodes are adjacent to each node. *A* = [*aij*] is the adjacent matrix and *<sup>i</sup>*, *<sup>j</sup>* <sup>∈</sup> *<sup>V</sup>*. *Ni* represents all sets of nodes adjacent to the *<sup>i</sup>* node. If node *<sup>j</sup>* is adjacent to node *<sup>i</sup>*, then *aij* <sup>&</sup>gt; 0. If *aij* <sup>=</sup> *aji* for any *<sup>i</sup>*, *<sup>j</sup>* <sup>∈</sup> *<sup>V</sup>*, the graph is an undirected graph; otherwise, it is a directed graph. If there is a directed path on any two nodes in the graph, the directed graph *G* is strongly connected. If there is a directed path to a node in the graph to any other node, then the directed graph *G* contains a spanning tree. If the undirected graph *G* is strongly connected, it is called a connected graph. When the undirected graph *G* is a connected graph, its Laplacian *L* matrix contains a zero root, and the other eigenvalues are positive real numbers. When a directed graph *G* contains a spanning tree, its Laplacian *L* matrix contains a zero root, and the rest eigenvalue's real part are positive.

#### *2.2. Problem Description*

Suppose the system consists of n omnidirectional robots. The dynamic characteristics of the omnidirectional robot in the x direction are:

$$\begin{cases}
\dot{x}\_i(t) = v\_i(t) \\
\dot{v}\_i(t) = u\_i(t)
\end{cases} \tag{1}$$

where *xi*(*t*) is position, *vi*(*t*) is velocity and *ui*(*t*) is input control. If any *i* robot and *j* robot in the multi-robot system satisfy the identities as follows:

$$\lim\_{t \to +\infty} \left[ \mathbf{x}\_i(t) - \mathbf{x}\_j(t) \right] = 0 \tag{2}$$

$$\lim\_{t \to +\infty} [v\_i(t) - v\_j(t)] = 0 \tag{3}$$

then the multi-robot system (1) has achieved consensus under the control protocol *ui*(*t*) Let the state vector of the *i* robot be *δi*(*t*) = [*xi*(*t*), *vi*(*t*)] *<sup>T</sup>*, then the multi-robot system state vector is *S*(*t*) = [*δ*1(*t*), *δ*2(*t*), *δ*3(*t*),..., *δn*(*t*)]. Rewrite system (1) as:

$$
\dot{\mathbf{S}}(t) = \mathbf{Y} \mathbf{S}(t) \tag{4}
$$

where *<sup>Ψ</sup>* <sup>=</sup> *<sup>I</sup>* <sup>⊗</sup> *<sup>A</sup>* <sup>−</sup> *<sup>L</sup>* <sup>⊗</sup> *<sup>B</sup>*, *<sup>A</sup>* <sup>=</sup> 0 1 0 0 , *B* = 0 0 *k*<sup>1</sup> *k*<sup>2</sup> , ⊗ is Kronecker. When ideally without noise and delay, the control protocol designed in [13] is as follows:

$$u\_i(t) = \sum\_{j \in N\_i} a\_{ij} \left\{ k\_1[\mathbf{x}\_i(t) - \mathbf{x}\_j(t)] + k\_2[v\_i(t) - v\_j(t)] \right\} \tag{5}$$

where *aij* > 0 is the topology weight of the communication between robot *i* and robot *j*, *k*<sup>1</sup> is the position scale factor that needs to be designed, *k*<sup>2</sup> is the velocity scale factor that needs to be designed. Lemmas 1 and 2 give the conditions that the coefficient matrix *Ψ* of control protocol (5) must satisfy when the communication topology of system (4) is undirected graph and directed graph, respectively. **Lemma 1.** *When the communication topology of multi-robot system (4) is connected graph, the coefficient matrix Ψ has a double zero root, and the real part of other eigenvalues is negative*.

**Proof.** Let there be an orthogonal matrix *Q*, such that:

$$\mathbf{Q}^T \mathbf{L} \mathbf{Q} = \text{diag}\{0, \lambda\_{2\prime}\lambda\_{3\prime}, \dots, \lambda\_{n}\} \tag{6}$$

where 0, *λ*2, *λ*3, ... , *λ<sup>n</sup>* is the eigenvalue of the Laplacian matrix *L*, and *λ<sup>i</sup>* >0(*i* = 2, 3, ... , n). Formula (7) is obtained from Formula (6):

$$(\mathbf{Q}\otimes\mathbf{I}\_2)^T\mathbf{P}(\mathbf{Q}\otimes\mathbf{I}\_2) = \text{diag}\{\mathbf{A}, \mathbf{A} - \lambda\_2\mathbf{B}, \mathbf{A} - \lambda\_3\mathbf{B}, \dots, \mathbf{A} - \lambda\_\text{ll}\mathbf{B}\}\tag{7}$$

The determinant of Formula (7) is obtained:

$$\left| \text{diag} \left\{ A, A - \lambda\_2 \mathbf{B}, A - \lambda\_3 \mathbf{B}, \dots, A - \lambda\_n \mathbf{B} \right\} \right| = s^2 \prod\_{2}^{n} s^2 + \lambda\_i k\_2 s + \lambda\_i k\_1 = 0 \tag{8}$$

Because there is *s*<sup>2</sup> in Formula (8), there must be a double zero root in the eigenvalue. By solving polynomial equation *s*<sup>2</sup> + *λik*2*s* + *λik*<sup>1</sup> = 0, we can get:

$$s\_1 = \frac{-\lambda\_i k\_2 + \sqrt{(\lambda\_i k\_2)^2 - 4\lambda\_i k\_1}}{2}$$

$$s\_2 = \frac{-\lambda\_i k\_2 - \sqrt{(\lambda\_i k\_2)^2 - 4\lambda\_i k\_1}}{2}$$

Based on this analysis, when (*λik*2) <sup>2</sup> > 4*λik*1, obviously <sup>−</sup>*λik*<sup>2</sup> <sup>±</sup> (*λik*2) <sup>2</sup> <sup>−</sup> <sup>4</sup>*λik*<sup>1</sup> < 0, so the eigenvalues *s*<sup>1</sup> and *s*<sup>2</sup> are negative. When (*λik*2) <sup>2</sup> < 4*λik*1, because *<sup>λ</sup>ik*<sup>2</sup> > 0, so <sup>−</sup>*λik*<sup>2</sup> < 0, the eigenvalues *s*<sup>1</sup> and *s*<sup>2</sup> have negative real parts. Lemma 1 is proved. -

**Lemma 2.** *When the communication topology of multi-robot system (4) is directed graph and contains spanning tree, <sup>k</sup>*<sup>1</sup> <sup>∈</sup> (0, *<sup>k</sup>*0*k*<sup>2</sup> <sup>2</sup>) *the coefficient matrix Ψ has a double zero root, and the real part of other eigenvalues is negative. Where k*<sup>0</sup> = min *λi*=0 *λi*<sup>2</sup> real(*λi*) imag(*λi*) .

**Proof.** The characteristic determinant of system (4) is obtained by Formula (8), assuming that there are polynomial equations:

$$s^2 + s(a+bj) + k(a+bj) = 0\tag{9}$$

where *a* > 0, *k*, *a*, *b* ∈ *R*. Let *s* = *jw*:

$$-w^2 - bw + ka + (aw + kb)j = 0\tag{10}$$

Solving Formula (10), we can get:

$$\begin{cases} -w^2 - bw + ka = 0\\ aw + kb = 0 \end{cases} \tag{11}$$

Thus, solved:

$$\begin{cases} \begin{array}{c} k\_1 = 0\\ k\_2 = \frac{a(a^2 + b^2)}{b^2} \end{array} \tag{12}$$

Document [13] proves that when 0 <sup>&</sup>lt; *<sup>k</sup>* <sup>&</sup>lt; *<sup>a</sup>*(*a*2+*b*2) *<sup>b</sup>*<sup>2</sup> , the roots of Formula (9) are all on the left open half plane. Formula (8) is modified according to Formula (9):

$$s^2\prod\_{2}^n s^2 + \lambda\_i k\_2 s + \lambda\_i \frac{k\_1}{k\_2} k\_2 = 0\tag{13}$$

The analysis shows that when *k*<sup>0</sup> = min *λi*=0 *λi*<sup>2</sup> real(*λi*) imag(*λi*) , *<sup>k</sup>*<sup>1</sup> <sup>∈</sup> (0, *<sup>k</sup>*0*k*<sup>2</sup> <sup>2</sup>), the coefficient matrix *Ψ* has a double zero root, and the real part of other eigenvalues is negative. Lemma 2 is proved. -

#### **3. Consensus Analysis of Multi-Robot with Various Delays and Noise Conditions**

In the previous section, we analyzed the conditions under which second-order systems achieve consensus in an ideal environment. However, in real environments, due to noise interference and communication differences between different robotic hardware, the above control protocols need to be improved. Assuming that there are *D* kinds of different delays in the system, the multi-agent system (4) can be changed to:

$$\dot{\mathbf{S}}(t) = (I \otimes A) \cdot \mathbf{S}(t) - \sum\_{d=1}^{D} \left( L\_d \otimes \mathbf{B} \right) \cdot \mathbf{\tilde{s}}(t) \cdot \mathbf{S}(t - \tau\_d) \tag{14}$$

where *ζ*(*t*) is the communication noise or measurement noise between robots, *τij* is the transmission delay, which represents the time taken by *i* robot to receive and process information transmitted by *j* robot, *L<sup>d</sup>* is the Laplacian matrix corresponding to the sub-topological graph of the robot node when the delay is *<sup>τ</sup>d*, and *<sup>D</sup>* ∑ d=1 *L<sup>d</sup>* = *L*.

**Theorem 1.** *If system (14) is a connected graph, the system can achieve consensus when the system delay τ<sup>d</sup> is less than τ*max *under the action of noise ζ*(*t*)*. Among them:*

$$\begin{cases} \text{ } \tau\_{\text{max}} = \left[ \arctan \left( \frac{k\_2}{k\_1} w\_{\text{max}} \right) \right] / w\_{\text{max}}\\ \text{ } w\_{\text{max}} = \sqrt{\frac{\lambda\_{\text{max}}^2 k\_2^2 \tilde{\varepsilon}^2(t) + \zeta(t) \sqrt{\tilde{\varepsilon}^2(t) \left( \lambda\_{\text{max}}^2 k\_2^2 \right)^2 + 4 \lambda\_{\text{max}}^2 k\_1^2}}{2}} \end{cases} \tag{15}$$

**Proof.** Using the frequency domain analysis method for analysis, the Laplace transform of Equation (14) can be obtained:

$$\mathcal{S}(\mathbf{s}) = (s\mathbf{I}\_{2\mathbb{N}} - (\mathbf{I}\_{\mathbb{N}} \otimes \mathbf{A}) + \sum\_{\mathbf{d}=1}^{D} (\mathbf{L}\_d \otimes \mathbf{B}) \zeta(t) e^{-\tau\_{\mathbf{d}}\mathbf{s}})^{-1} \mathcal{S}(\mathbf{0}) \tag{16}$$

$$\text{Let } \mathbf{G}\_{\mathsf{T}}(s) = \mathsf{s}\mathbf{I}\_{2n} - (\mathsf{I}\_{n} \otimes \mathsf{A}) + \sum\_{\substack{\mathbf{d}=1\\\mathbf{d}\_{1}=1}}^{D} (\mathsf{L}\_{\mathsf{d}} \otimes \mathbf{B})\_{\mathsf{T}}^{\tau}(\mathsf{t}) e^{-\mathsf{T}\_{\mathsf{d}}\mathbf{s}}, \text{ so the eigenvalues of the determinant } |\mathsf{G}\_{\mathsf{T}}(s)| \text{ is}\\
\sum\_{\substack{\mathbf{d}=1\\\mathbf{d}\_{1}=1}}^{D} (\mathsf{L}\_{\mathsf{d}} \otimes \mathbf{B})\_{\mathsf{T}}^{\tau}(\mathsf{t}) e^{-\mathsf{T}\_{\mathsf{d}}\mathbf{s}}.$$

are the eigenvalues of the system. Lemma 1 proves that multi-robot system (4) achieves the conditions of consensus. According to Lemma 1, how can the eigenvalues of system (14) be kept in the negative half-plane under the interference of time delay *τ* and noise *ζ*(*t*) relative to the system (4)? Because the measurement noise and communication noise are uncertainties in real environments, it is impossible to carry out accurate quantitative analysis. Therefore, only when the system delay *τ* increases to a value under the action of noise *ζ*(*t*) does a non-zero eigenvalue of the system appear on the virtual axis for the first time, while the time delay *τ* is the critical value for the system to maintain stability.

Assuming that the eigenvalue of the system is on the imaginary axis, let *s* = *jw* be the eigenvalue; then *α* = *α*<sup>1</sup> ⊗ [1, 0] *<sup>T</sup>* <sup>+</sup> *<sup>α</sup>*<sup>2</sup> <sup>⊗</sup> [0, 1] *<sup>T</sup>* is the eigenvector corresponding to the eigenvalue, and if *α* <sup>=</sup> 1, *<sup>α</sup>*1, *<sup>α</sup>*<sup>2</sup> <sup>∈</sup> <sup>C</sup>*n*, then:

$$\left[jwI\_{2n} - (I\_n \otimes A) + \sum\_{d=1}^{D} (L\_d \otimes \mathcal{B})\zeta(t)e^{-jw\tau\_d}\right]a = 0\tag{17}$$

The imaginary eigenvalues of the system appear in pairs in conjugate form. This paper only analyzes the case where *w* > 0. Formula (17) left multiplied by *α<sup>H</sup>* is:

$$a^H \left[ jw\mathbf{I}\_{2n} - (\mathbf{I}\_n \otimes \mathbf{A}) + \sum\_{\mathbf{d}=1}^D (\mathbf{L}\_d \otimes \mathbf{B}) \zeta(t) e^{-jw\tau\_d} \right] a = 0 \tag{18}$$

Because each line of the left matrix of Formula (17) is zero, so *jwα*<sup>1</sup> = *α*2, and substituting it into Formula (18):

$$\sum\_{d=1}^{D} \beta\_d \zeta(t) e^{-jw\tau\_d} = \frac{w^2}{k\_1 + jwk\_2} \tag{19}$$

where *<sup>β</sup><sup>d</sup>* <sup>=</sup> *<sup>α</sup>H*(*Ld*⊗*I*2)*<sup>α</sup> <sup>α</sup>H<sup>α</sup>* . Replace A with B in Formula (19):

$$F(w) = \sum\_{d=1}^{D} \beta\_d \zeta(t) e^{jw\tau\_d} = \frac{w^2}{k\_1 - jwk\_2} \tag{20}$$

Take module operation on both sides of the upper equal sign:

$$\|F(w)\| = \|\sum\_{d=1}^{D} \beta\_d \zeta(t) e^{-jw\tau\_d}\| < \|\sum\_{d=1}^{D} \beta\_d \zeta(t)\| = \frac{a^H (L \otimes I\_2) a}{a^H a} \zeta(t) \le \lambda\_{\text{max}} \zeta(t) \tag{21}$$

Let *w*max = *λ*2 max*k*<sup>2</sup> <sup>2</sup>*ζ*2(*t*)+*ζ*(*t*) *ζ*2(*t*)(*λ*<sup>2</sup> max*k*<sup>2</sup> 2) 2 +4*λ*<sup>2</sup> max*k*<sup>2</sup> 1 <sup>2</sup> get *w* ≤ *w*max, upper formula establishment. From Formula (20):

$$\theta(w) = \arg\text{z}[F(w)] = \arctan(\frac{k\_2}{k\_1}w) \tag{22}$$

where *<sup>θ</sup>*(*w*) <sup>∈</sup> [0, 2*π*). Let *<sup>τ</sup>*(*w*) = *<sup>θ</sup>*(*w*) *<sup>w</sup>* , *<sup>a</sup>* <sup>=</sup> *<sup>k</sup>*<sup>2</sup> *k*1 , deriving for *τ*(*w*), we can obtain:

$$M\_1(w) = \frac{d\tau(w)}{dw} = \frac{1}{w^2} M\_2(w) = \frac{1}{w^2} \left[ \frac{aw}{a^2 w^2 + 1} - \arctan(aw) \right] \tag{23}$$

Deriving for *M*2(*w*) we can obtain:

$$\frac{dM\_2(w)}{dw} = -\frac{2a^3 w^2}{\left(a^2 w^2 + 1\right)^2} < 0\tag{24}$$

*M*2(*w*) is decreasing, so when *w* > 0, *M*2(*w*) < *M*2(0) = 0, so *M*1(*w*) < 0; that is, *τ*(*w*) is also decreasing. So *τ*(*w*) ≥ *τ*(*w*max) = *τ*max. When *τ<sup>d</sup>* < *τ*max, when *τ<sup>d</sup>* < *τ*max, we can get:

$$\tau(w) = \frac{\theta(w)}{w} = \frac{\arg z \left(\sum\_{d=1}^{D} \beta\_d \zeta(t) e^{jw\tau\_d} \right)}{w} \le \frac{\max[w\tau\_d]}{w} < \frac{w\tau\_{\max}}{w} < \tau\_{\max} \tag{25}$$

That is to say, it contradicts *τ*(*w*) ≥ *τ*(*w*max). Therefore, when *τ<sup>d</sup>* < *τ*max, the eigenvalues of the system can be maintained in the left half plane, and the consensus of system (14) can be achieved. Theorem 1 is proved. -

**Theorem 2.** *If system (14) is a directed graph and there is a spanning tree, the system can achieve consensus when the system delay <sup>τ</sup><sup>d</sup> is smaller than the <sup>τ</sup>*max *under the action of noise <sup>ζ</sup>*(*t*)*, and k*<sup>1</sup> <sup>∈</sup> (0, *<sup>k</sup>*0*k*<sup>2</sup> <sup>2</sup>)*. Among them:*

$$\begin{cases} \mathsf{r}\_{\text{max}} = \min\_{\|\lambda\_i\| \neq 0} \left[ \left[ \arctan \left( \frac{k\_2}{k\_1} w\_i \right) - \arg z(\lambda\_i) \right] / w\_i \right] \\\ w\_i = \sqrt{\frac{\lambda\_i^2 k\_2^2 f^2(t) + \zeta(t) \sqrt{\xi^2(t) \left( \lambda\_i^2 k\_2^2 \right)^2 + 4 \lambda\_i^2 k\_1^2}}} \end{cases} \tag{26}$$

*where* arg*z*(*λi*) <sup>∈</sup> (−*<sup>π</sup>* <sup>2</sup> , *<sup>π</sup>* 2 ).

**Proof.** Lemma 2 proved that, when the communication topology of multi-robot system (4) is directed graph and contains spanning tree, *<sup>k</sup>*<sup>1</sup> <sup>∈</sup> (0, *<sup>k</sup>*0*k*<sup>2</sup> <sup>2</sup>), the coefficient matrix *Ψ* has a double zero root, and the real part of the other eigenvalues is negative. The same analysis is performed using the frequency domain analysis method. Similar to the proof of Theorem 1, only when the system delay *τ* increases to the value under the action of noise *ζ*(*t*) does a non-zero eigenvalue of the system first appear on the imaginary axis, while the delay *τ* is the critical value for the system to maintain stability. Take modulo operation on Formula (20):

$$\|\|F(w)\|\| = \|\frac{w^2}{k\_1 - jwk\_2}\|\tag{27}$$

Let *w* be a function of *F*(*w*); then the above formula can be written as follows:

$$w = \sqrt{\frac{\|F(w)\|^2 k\_2^2 + \sqrt{\left(\|F(w)\|^2 k\_2^2\right)^2 + 4\|F(w)\|^2 k\_1^2}}{2}} \tag{28}$$

Then we can get that *w* is an incremental function about *F*(*w*). From Formula (20):

$$\begin{cases} \operatorname{argz}[F(w)] = \arctan(\frac{k\_2}{k\_1}w) \\\\ \operatorname{argz}[F(w)] \le \operatorname{argz}(\sum\_{d=1}^D \beta\_d) + \max(w\tau\_m) \end{cases} \tag{29}$$

So:

$$
\arctan(\frac{k\_2}{k\_1}w) - \arg z(\sum\_{d=1}^D \beta\_d) \le \max(wr\_{\mathfrak{m}}) \tag{30}
$$

Because *<sup>β</sup><sup>d</sup>* <sup>=</sup> *<sup>α</sup>H*(*Ld*⊗*I*2)*<sup>α</sup> <sup>α</sup>H<sup>α</sup>* , so *<sup>D</sup>* ∑ d=1 *β<sup>d</sup>* = *λi*,where *λ<sup>i</sup>* is the non-zero eigenvalue of Laplace matrix *L*. So *F*(*w*)≤*ζ*(*t*)*λi*. Because *w* is an incremental function about *F*(*w*), so:

$$w(\|F(w)\|) \le w(\|\zeta(t)\lambda\_i\|) = w\_i = \sqrt{\frac{\lambda\_i^2 k\_2^2 \zeta^2(t) + \zeta(t)\sqrt{\zeta^2(t)(\lambda\_i^2 k\_2^2)^2 + 4\lambda\_i^2 k\_1^2}}{2}} \tag{31}$$

When *τ<sup>d</sup>* < *τ*max, we can get:

$$\begin{split} \max(w\tau\_d) < w\_l \tau\_{\max} &= \min\left[ \left[ \arctan(\frac{k\_2}{k\_1} w\_l) - \arg z (\sum\_{d=1}^D \beta\_d) \right] / w\_l \right] w\_l \\ &\le \arctan(\frac{k\_2}{k\_1} w) - \arg z (\sum\_{d=1}^D \beta\_d) \end{split} \tag{32}$$

We can find that this contradicts Formula (30), so when *τ<sup>d</sup>* < *τ*max, the eigenvalue of the system can be maintained in the left half plane, and the consensus of system (14) can be achieved. Theorem 2 is proved. -

#### **4. Simulation Verification**

In this section, two sets of Matlab/Simulink numerical simulation experiments are carried out to verify the consensus of the system described in Theorems 1 and 2 when the communication topology is undirected graph and directed graph under the conditions of noise and various delays.

**Experiment 1.** *Let system (14) consist of four robots whose communication topology is shown in Figure 1*.

**Figure 1.** Experiment 1 system communication topology.

As can be seen from Figure 1, the time delay between robots 1 and 2 is *τ*1, between robot 2 and robot 3 it is *τ*2, between robot 3 and robot 4 it is *τ*1, between robot 4 and robot 1 it is *τ*2. If the adjacent communication weight *aij* is 1, then the Laplace matrix *L* is:

$$L = \begin{pmatrix} 2 & -1 & -1 \\ -1 & 2 & -1 \\ & -1 & 2 & -1 \\ -1 & & -1 & 2 \end{pmatrix} \tag{33}$$

We can get *λ*max = 4. Assume that the communication noise or measurement noise is white noise with a maximum amplitude of two. According to Theorem 1, *τ*max = 0.226.

In the first group of Experiment 1, set *τ*<sup>1</sup> = 0.21, *τ*<sup>2</sup> = 0.22, and the initial posture is assumed to be (1,0), (2,0), (3,0), (4,0). The simulation results are shown in Figure 2.

To verify Theorem 1 and compare with the first group of experiments, in the second group of experiments, set *τ*<sup>1</sup> = 0.23, *τ*<sup>2</sup> = 0.24 under the same conditions. The simulation results are shown in Figure 3.

According to Experiment 1, system (14) satisfying lemma 1 can achieve consensus when all *τ<sup>d</sup>* are less than *τ*max, and the system will diverge when all *τ<sup>d</sup>* are greater than *τ*max, which cannot achieve consensus; thus Theorem 1 is verified.

**Figure 2.** Experiment 1 Group 1 simulation results; *x* is position, *v* is velocity. (**a**) Trajectory of *x* changing with time; (**b**) Trajectory of *v* changing with time.

**Figure 3.** *Cont.*

**Figure 3.** Experiment 1 Group 2 simulation results; *x* is position, *v* is velocity. (**a**) Trajectory of *x* changing with time; (**b**) Trajectory of *v* changing with time.

**Experiment 2.** *Let system (14) consist of four robots whose communication topology is shown in Figure 4.*

**Figure 4.** Experiment 2 system communication topology.

The time delay from Robot 1 to Robot 2 is *τ*1, from Robot 2 to Robot 3 it is *τ*2, from Robot 3 to Robot 4 it is *τ*3, from Robot 4 to Robot 1 it is *τ*4. If the adjacent communication weight *aij* is 1, then the Laplace matrix *L* is:

$$L = \begin{pmatrix} 2 & -1 \\ -1 & 1 & -1 \\ & -1 & 2 \\ -1 & & -1 & 1 \end{pmatrix} \tag{34}$$

Then *k*<sup>0</sup> = min *λi*=0 *λi*<sup>2</sup> real(*λi*) imag(*λi*) = <sup>5</sup>×<sup>2</sup> <sup>1</sup> <sup>=</sup> 10, so *<sup>k</sup>*<sup>1</sup> <sup>∈</sup> (0, 10*k*<sup>2</sup> <sup>2</sup>). Assume that the communication noise or measurement noise is white noise with a maximum amplitude of two. Set *k*<sup>1</sup> = 1, *k*<sup>2</sup> = 1, according to Theorem 2, *τ*max = 0.137.

In the first group of experiment 2, set *τ*<sup>1</sup> = 0.13, *τ*<sup>2</sup> = 0.12, *τ*<sup>3</sup> = 0.11, *τ*<sup>4</sup> = 0.1, and the initial posture is assumed to be (1,0), (2,0), (3,0), (4,0). The simulation results are shown in Figure 5.

To verify Theorem 2 and compare with the first group of experiments, in the second group of experiments, set *τ*<sup>1</sup> = 0.14, *τ*<sup>2</sup> = 0.141, *τ*<sup>3</sup> = 0.142, *τ*<sup>4</sup> = 0.143 under the same conditions. The simulation results are shown in Figure 6.

**Figure 5.** Experiment 2 Group 1 simulation results; *x* is position, *v* is velocity. (**a**) Trajectory of *x* changing with time; (**b**) Trajectory of *v* changing with time.

**Figure 6.** *Cont.*

**Figure 6.** Experiment 2 Group 2 simulation results;*x* is position, *v* is velocity. (**a**) Trajectory of *x* changing with time; (**b**) Trajectory of *v* changing with time.

According to experiment 2, system (14) satisfying Lemma 2 can achieve consensus when all *τ<sup>d</sup>* are less than *τ*max, and the system will diverge when all *τ<sup>d</sup>* are greater than *τ*max, which cannot achieve consensus; thus Theorem 2 is verified.

#### **5. Physical Experiment Verification**

To verify the proposed formation control algorithm, we did the experiment based on a pre-constructed multi-mobile robot research platform built by our laboratory, which was constructed with a self-designed three-wheeled omnidirectional robot carrying an UWB (Ultra-Wide Band) ranging module. The system is shown in Figure 7, the omnidirectional robot is shown in Figure 8, and the performance parameters are shown in Table 1. Because the consensus of the second-order system is analyzed in the theoretical analysis part, the speed and position are consistent, and while the omnidirectional robot is a fully driven robot, the horizontal and vertical directions can be controlled separately. Because the velocity control in a given direction is a second-order system, therefore, multi-omni-directional mobile robots can be decomposed into two one-dimensional second-order multi-robot systems. Therefore, omni-directional robots are used to verify the proposed algorithm. In the experiment, it is possible to determine whether the algorithm is valid based on whether the speed and the position of the robot after final stabilization are consistent. In the data acquisition part, the external positioning data of the robot are collected by the UWB positioning system built by myself, and the speed of the robot itself is collected by the encoder on the wheel of the robot and transmitted to the central processing computer via Wi-Fi for processing. The ranging error between the UWB ranging modules used in the experiment is 7 cm. Experiments were carried out in an indoor environment with length × width of 4 m × 5 m in order to verify the effectiveness of the proposed algorithm.

**Figure 7.** Multi-robot research platform.

**Figure 8.** Omnidirectional robots. (**a**) Top view; (**b**) Side view.

It should be pointed out that when the proposed algorithm is applied to a practical multi-robot system, it is necessary to first determine the maximum communication delay between robots and the maximum amplitude of the noise environment, and then design *k*<sup>1</sup> and *k*<sup>2</sup> based on this. At the same time, it should be noted that this experiment mainly focuses on verifying whether the system can achieve consensus under the control law. The collision avoidance behavior of the multi-robot system is not the emphasis in this research. Therefore, the collision avoidance algorithm program is written in the bottom control program of the robot in this experiment. When the robot is about to collide, the formation algorithm program will be interrupted, and the collision avoidance behavior will be executed. When a safe distance between the robots has been reached, the formation algorithm program will continue to be executed [17,18].

The communication topology used in the experiment is shown in Figure 4. The adjacent communication weight *aij* is 1, so *<sup>k</sup>*<sup>1</sup> <sup>∈</sup> (0, 10*k*<sup>2</sup> <sup>2</sup>). The central processor logs on each robot remotely through SSH, and obtains the communication delay between two robots whose communication weight A is not zero by PING command. The time delay between robots in the actual communication environment is time-varying, so take its maximum delay. We get *τa*<sup>1</sup> = 0.56*s*, *τa*<sup>2</sup> = 0.043*s*, *τa*<sup>3</sup> = 0.047*s*, *τa*<sup>4</sup> = 0.061*s*. The time taken for each robot to receive data and process them is *τb*<sup>1</sup> = 0.021*s*, *τb*<sup>2</sup> = 0.02*s*, *τb*<sup>3</sup> = 0.021*s*, *τb*<sup>4</sup> = 0.021*s*. Therefore, the time delay between robots is *τ*<sup>1</sup> = 0.077*s*,

*τ*<sup>2</sup> = 0.063*s*, *τ*<sup>3</sup> = 0.068*s*, *τ*<sup>4</sup> = 0.082*s*. Because this experiment is being performed in a laboratory environment, it is assumed that the communication noise is white noise with a maximum amplitude of 2. According to Formula (26) and the moving speed of omnidirectional robot, set *k*<sup>1</sup> = 1, *k*<sup>2</sup> = 1.4. Four omnidirectional robots were placed in arbitrary positions, (0.83,2.20,0), (0.35,1.74,0), (0.67,0.88,0), (0.56,0.48,0), respectively. In the practicality experiment, the robot cannot converge to one point, so Formula (2) is changed to:

$$\lim\_{t \to +\infty} \left[ \mathbf{x}\_i(t) - \mathbf{x}\_j(t) - F\_{\mathcal{P}} \right] = 0 \tag{35}$$

where *Fp* is the formation parameters and *p* = 1, 2, ... , *n*. Since the system only installs a UWB ranging sensor to provide positioning for the robot, the robot will perform pose determination before the experiment starts, and the robot's body coordinate system will be consistent with the global coordinate system. The experimental results are shown in Figure 9. The experimental video address can be found in reference [19].

**Figure 9.** Experiment process screenshots. (**a**) T = 0 s; (**b**) T = 15 s; (**c**) T = 25 s; (**d**) T = 30 s.

The experimental data collected by the UWB positioning system are shown in Figure 10.

The experiments show that the multi-robot system can eventually achieve consensus and form a formation in a variety of time-delay and noise environments, which verifies the effectiveness of the proposed algorithm.

**Figure 10.** Experiment 1 data of UWB positioning system.

#### **6. Conclusions**

Aiming at the consensus of multi-mobile robots under uncertain conditions such as communication delay, communication noise and measurement noise, we used the frequency domain analysis method, transformed the characteristic equation into the quadratic polynomial of the pure imaginary eigenvalue, and then obtained the conditions for achieving consensus under various time delay and noise conditions for a second-order multi-robot system. That is, when the time delays of all robots are less than the maximum time delays, the system can achieve consensus. In this paper, based on two aspects of system communication topology—directed graph and undirected graph—the results were verified by numerical simulation using MATLAB/Simulink, verifying the correctness of the theoretical derivation of the proposed algorithm. Finally, a multi-robot research platform was built, and formation control experiments were carried out in a real laboratory environment. The experimental results showed that the proposed algorithm could effectively make the second-order multi-mobile robot systems consistent. This paper only analyzes the consensus problem of second-order systems, while most existing multi-mobile robot systems are higher-order systems. Therefore, the consensus analysis of higher-order systems under noise and time delay conditions will be the focus of our next research.

**Author Contributions:** H.W. wrote this article and accomplished the algorithm analysis; Q.L. was the principle researcher on this project and provided many valuable suggestions; N.D. performed the simulation; G.W. established the physical experimental system; and B.L. performed the experiment and analyzed the experimental data.

**Funding:** This research received no external funding.

**Acknowledgments:** The work in this paper is supported by the National Natural Science Foundation of China (Grant No. 61663014).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Project Report* **A Multi-Agent Based Intelligent Training System for Unmanned Surface Vehicles**

#### **Wei Han 1,2,\*, Bing Zhang 2, Qianyi Wang 2, Jun Luo 1, Weizhi Ran <sup>3</sup> and Yang Xu <sup>3</sup>**

Received: 10 February 2019; Accepted: 6 March 2019; Published: 15 March 2019

**Abstract:** The modeling and design of multi-agent systems is imperative for applications in the evolving intelligence of unmanned systems. In this paper, we propose a multi-agent system design that is used to build a system for training a team of unmanned surface vehicles (USVs) where no historical data concerning the behavior is available. In this approach, agents are built as the physical controller of each USV and their cooperative decisions used for the USVs' group coordination. To make our multi-agent system intelligently coordinate USVs, we built a multi-agent-based learning system. First, an agent-based data collection platform is deployed to gather competition data from agents' observation for on-line learning tasks. Second, we design a genetic-based fuzzy rule training algorithm that is capable of optimizing agents' coordination decisions in an accumulated manner. The simulation results of this study demonstrate that our proposed training approach is feasible and able to converge to a stable action selection policy towards efficient multi-USVs' cooperative decision making.

**Keywords:** unmanned surface vehicles; multi-agent system; training system; genetic-based fuzzy rule learning; intelligent autonomous control; modeling and simulation

#### **1. Introduction**

The modeling and design of multi-agent systems for applications in the evolving intelligence of unmanned systems is interesting and promising [1–4], especially in situations where traditional methods can be costly, dangerous, or even impossible to realize. Several applications can be found in a very broad range of domains such as energy [5–7], security [8], robotics [9], and resource management [10]. A multi-agent system consists of autonomous agents that interact in an environment to achieve specific goals [11]. An autonomous agent, in this sense, is able to perceive its environment and perform actions using actuators.

Over the years, there have been several studies that have proposed principles for designing multi-agent systems, as well as approaches to coordinate the individual behavior of agents [12–16]. In most multi-agent application domains, a priori specification of the optimal agents' behavior is difficult due to the complexity and/or dynamics of the environment [11]. In such environments, it is natural for agents to adapt or learn optimal actions that maximize performance on-line. However, one of the key challenges is the need to build a simulation platform that can be used for fast training so as to gather enough training data to promote the intelligent development process.

In the context of multi-surface vehicles' modeling and design [17–19], an unmanned surface vehicles (USVs) system is one of the development trends of modern weapon equipment [20]. The unmanned surface vehicle system has become an important means of information confrontation, precision strikes, and special combat tasks in future war [21]. Over the years, unmanned surface vehicles have long been applied to varying kinds of military applications in the real world. Some of the application domains include anti-mine warfare, submarine warfare, reconnaissance, and surveillance [22–24]. Due to the open and complex dynamic combat environment, the USV combat system must develop in the direction of autonomy and synergy, and the future surface unmanned combat should have a strong autonomous capacity, be able to be controlled autonomously, and complete complex and diverse combat tasks independently or collaboratively in a complex uncertain environment [25]. However, human knowledge is difficult to apply directly to the coordination of USVs.

We propose a multi-agent-based intelligent training system for unmanned surface vehicles (USVs). We focus on the problem of training agents of a multi-agent-based unmanned surface vehicles system where no historic data concerning the agent behavior is available. Using the team learning framework, we provide approaches for learning the rule base for multi-agent systems, designing the learning environment for simulating cooperative and competitive agents' behavior, and gathering competition data from agents' observation for off-line learning tasks. To this end, this paper first presents a decentralized coordination platform and simulator design based on a multi-agent architecture. Secondly, a USV-based agent model is presented. The paper also proposes a data collection platform for agent learning and USV training. Lastly, a fast learning and training algorithm design based on the agents' rule base is presented.

To evaluate our approach, we used an island-conquering scenario where two teams of unmanned surface vehicles compete to conquer islands in an environment. We model this case study as a partially-observable stochastic game where one team has to learn the behavior that maximizes their returns against a human-controlled team. Our empirical evaluations show that our approach to learning the knowledge base of a multi-USV system, when applied to the trained team, was able to find a knowledge base (KB) to achieve better performance.

We first present the problem this study seeks to address in Section 2. Section 3 discusses the multi-surface vehicle training system design. Finally, in Section 4, we discuss our experiments and analysis of the simulation results followed by the conclusion of our study in Section 5.

#### **2. Multi-Agent-Based USVs' Training Problem**

We first discuss the multi-agent-based intelligent training system learning problem of this study.

In general, a stochastic game is an extension of the Markov decision process (MDP) to the multi-agent context. In such a game, agents may have conflicting goals, and their joint actions determine rewards and transition between states. Stochastic games usually assume agents as having a complete view of their states, whereas the partial observation case is discussed under the more general partially-observable stochastic games (POSGs) domain. Our study is performed under the POSG case.

**Theorem 1.** *A POSG is a tuple* < *X*, *U*1...*Un*,*O*1...*On*, *f* , *R*1...*Rn* > *where:*

In this study, we consider a team of multiple unmanned surface vehicles that must be trained to perform a given mission *<sup>m</sup>* in a mission space D ⊂ <sup>R</sup>*<sup>n</sup>* consisting of opposite forces. Let *<sup>H</sup>*(*m*) be the objective function dependent on the mission. Given that a team *T* consists of *N* USVs, the dynamics of the *i* th USV is given by:

$$\begin{aligned} \dot{x}\_i &= f\_i(x\_i(t), u\_i(t)) \\ \dot{x} &= 1, 2, 3..., N \end{aligned} \tag{1}$$

where *xi*(*t*) <sup>∈</sup> <sup>R</sup>*<sup>n</sup>* and *ui*(*t*) <sup>∈</sup> <sup>R</sup>*<sup>m</sup>* are the state and control inputs (available actions) of the USV, respectively, at time *t*. Depending on the USV model and mission type, a set of constraints (such as obstacle avoidance and the physical limitations of the USV) on the state *xi*(*t*) of a USV are imposed. The control inputs of a USV are bounded according to the limits of the USV's actuators |*ui*(*t*)| < *umax*.

The reward of an USV *i* after taking action *ui*,*<sup>t</sup>* ∈ *Ui* in stage *t* is denoted as *ri*,*t*+1. The individual policies *hi* : *X* × *Ui* → [0, 1] of the USVs in a team form the joint policy **h**. In the case of team learning, the joint policy is defined by the single learner. Since the reward of each USV in a team depends on the joint action, the respective USV rewards depend on the joint policy:

$$R\_i^{\mathbf{h}}(\mathbf{x}) = \mathbb{E}\left\{\sum\_{t=0}^{T} \gamma^t r\_{i,t+1} | \mathbf{x}\_0 = \mathbf{x}, \mathbf{h} \right\} \tag{2}$$

where *γ* is the discounting factor and *T* is the length of the horizon. The training objective for the intelligent training system is to find a policy that maximizes Equation (2) such that the trained team of USVs can outperform its rival USVs, while adapting to the changing dynamics of it's operating environment.

#### **3. Multi-Agent-Based Training System Design**

In this section, we discuss the multi-agent-based training system design of our study, as well as the ideas involved in training the agents. To start with, the section first presents the architecture of the multi-agent-based intelligent training system. Next, we present the USV agent model for cooperative training and competition. Finally, the section presents the agent data collection platform for gathering competition for the agent information sharing and learning algorithm.

#### *3.1. System Architecture*

The architecture of the system used in this study is presented in Figure 1. We decouple the agents' control from the main environment in order to allow different implementations of controllers. In this case, we consider the multi-agent simulation platform as a server, and any attached controller becomes a client. Each USV agent selects its action based on the observed information from the environment. Based on the data visualization model of the USV, the data collection platform forms and mounts the corresponding resource tree structure of the USV. Historical data are provided to the upper machine learning algorithm by the data collection platform after data fusion. The obtained algorithm controller is then loaded into the USV.

The multi-agent cooperative reasoner component serves as the policy used to select an action for a controlled agent using the agent's local observation. In this study, we model the reasoner as an FISdecomposed into a genetic fuzzy trees (GFT). Thus, the reasoner maintains a KB, which is used for the mapping fuzzy values of the observation of an agent to an action for execution in the POSG.

The learner component of the intelligent control model facilitates KB learning, tuning, or optimizing. It seeks to find the policy that exhibits the designed behavior of the controlled team as specified by the reward function. Thus, gradient-based approaches such as neural networks (NN) [26], as well as non-gradient-based approaches can be employed for this purpose. As already indicated in our usage of GFT for the reasoner component, we use the GA in this study to learn the fuzzy rules

and tune the MFsof a GFT. Each candidate KB is applied to the control problem for the whole of a simulation episode (we use round and episode interchangeably) as found in the Pittsburgh approach. Hence, the rewards that are received each time step are aggregated and assigned as the fitness value of the candidate KB used at the end of the round/episode.

**Figure 1.** The architecture of the multi-agent based intelligent training system.

#### *3.2. Agent Design*

The USV agent model that is used by the intelligent training system for cooperative training and competition is equipped with a radar and weapon system. The radar of a USV can detect a hostile threat within it's detection range. In this work, the weapon system of a USV consists of a calibrated gun with limited forward turning angle and firing range. A USV can turn in both directions (left or right) and with limited speed and turning radius *r*. In this study, only surge, sway, and yaw are used to describe the USV's movement at sea, as shown Figure 2). Therefore, the kinematic relationship between the USV position in the global inertial frame XYZ and the boat body-fixed frame xyz can be defined as:

$$\begin{aligned} \chi &= \iota \cos(\psi) - \upsilon \sin(\psi) \\ \chi &= \upsilon \sin(\psi) + \upsilon \cos(\psi) \\ \psi &= \lambda \end{aligned} \tag{3}$$

The location and orientation of USVs in the coordinates of the environment are represented by (*x*, *y*) and *ψ* in the Earth-fixed reference frame, while *u*, *v*, and *λ* represent the velocity of surge, sway, and yaw in the body-fixed reference frame, respectively.

**Figure 2.** Schematic diagram of the USV model and planar motion where *XEOEYE* is the Earth-fixed reference frame and *XBOBYB* denotes the body-fixed reference frame.

#### *3.3. Data Collection Model*

Agents' observation can persist in the course of training to enable off-line operations. Data collected during training can be useful for agents' cooperative decision making. Furthermore, the gathering of training data means that different agents' learning algorithm can be employed both on-line and off-line for agents' performance improvement. Moreover, the data can be organized to enable communication between agents. The gathered domain data (in resource forms) can be used for purposes such as improving agents' performance using state-of-the-art deep reinforcement learning (DRL) algorithms. Hence, the data collector component is not only necessary, but also can extend the scope of learning algorithms that can be employed for intelligent training.

The data collection model is designed using a training and control platform consisting of loosely-coupled components and a common resource model. One such paradigm for implementing modular services is the Future Airborne Capability Environment's (FACE (http://www.opengroup. org/face)) common operating environment approach. In Figure 3, we show the components of FACE with regards to our study. Their descriptions, in the context of our study, are as follows:

**Figure 3.** The architecture of the agent data collection platform based on Future Airborne Capability Environment (FACE) components.

#### *3.4. Multi-Agent-Based USV Training Algorithm Design*

The intelligent control model, as depicted in Figure 4, is responsible for performing learning and reasoning. It controls a team of agents in the simulation environment by sending agent actions to the environment every time step. Thus, at time step *t*, it sends the joint actions *ut* to the environment for execution where the action for agent *i*, *i* ≤ *n*, is *ui*,*<sup>t</sup>* ∈ *ut*. The intelligent control model consists of a learner and a reasoner that employs fuzzy logic and genetic algorithms (GA) known as genetic fuzzy systems (GFS) [27]. In this technique, GA is used to learn and tune the rule base and membership function of an FIS, respectively. To do this, an initial population of solutions, or strings, is created to encode the rule base and membership functions.

Classical approaches such as Michigan [28], Pittsburgh [29], and iterative rule learning [30] are mostly used to derive GFS fuzzy rules. We give an instance to illustrate how the rule base is encoded in this study using the Pittsburgh approach. Suppose *X*<sup>1</sup> and *X*<sup>2</sup> are input variables each with linguistic terms {*term*1, *term*2} and output variable *Y* with terms {*a*1, *a*2} with an arbitrary rule base of an FIS as follows:

Assume we assign codes zero and one to *a*<sup>1</sup> and *a*<sup>2</sup> respectively. The chromosome 0110 is obtained. This approach can represent an if-then rule in a single digit. This implies that each chromosome encodes possible outputs of a set of rules.

Similarly to tuning membership functions, each digit in the string corresponds to some endpoint of a membership function. GA can then be used to tune the parameters as part of the evolution process; such as found in [31], where initial parameters of a triangular MF T (*α*, *β*, *γ*) are tuned using:

$$
\mu\_{i+1} \leftarrow (\mu\_i + \delta\_i) - \eta\_i \tag{4}
$$

$$
\beta\_{i+1} \leftarrow (\beta\_i + \delta\_i) \tag{5}
$$

$$
\gamma\_{i+1} \leftarrow (\gamma\_i + \delta\_i) - \eta\_i \tag{6}
$$

where *δ* and *η* are the tuning coefficients, whereas *α*, *β* and *γ* parameterize T. *δ* shifts the MF to the right or left, whereas *η* shrinks or expands the MF. Therefore, every MF has two tuning parameters that can be optimized using GA.

**Figure 4.** The architecture of the multi-agent-based intelligent training system. KB, knowledge base. GFT, genetic fuzzy trees.

Although a GFS can be used to find KBs that optimize the fitness function, it is inefficient to use a single GFS for a complex control problem. In such a case, there is an increase in GA search space complexity, the tendency to have redundant rules, and KB size. Therefore, using genetic fuzzy trees (GFT) in [2] helps to mitigate this problem. A GFT is, essentially, an ensemble of GFSs arranged in a tree structure according to a logical and conditional sequence of execution. A GFT enables the decomposition of a complex fuzzy control system (FCS) or GFS into logical sections with each node in the tree focusing on an aspect of the control problem. Each GFS in a GFT defines its own KB. Hence, the KB of a GFT consisting of *<sup>m</sup>* <sup>∈</sup> <sup>R</sup> GFSs *<sup>G</sup>* <sup>=</sup> {*gfs*1, *gfs*2, ..., *gfsm*} may have the genetic structure (represented as a concatenated string) *gfsrb* <sup>1</sup> *gfsrb* <sup>2</sup> ...*gfsrb <sup>n</sup> gfsm f* <sup>1</sup> *gfsm f* <sup>2</sup> ...*gfsm f <sup>n</sup>* where *gfsrb <sup>i</sup>* and *gfsm f <sup>i</sup>* , *i* ≤ *m*, denote the RB and MF string/genome, respectively, of a given GFS. This structure can be seen in Figure 5.

**Figure 5.** The knowledge base genetic structure of a genetic fuzzy tree. Each chromosome or member of the GA population takes this form.

Algorithm 1 shows how beginning with an initial population, the KB of a GFT can be learned/optimized using the GA process. In Line 2, the initial population of the GFT KB is generated as the current population. This can be based on a predefined KB set or randomly generated. In Lines 3–4, the current population is passed on to the the FIS to be used for the control task. After all these KBs have taken turns performing the control task and have been evaluated, they are then subjected to the GA operators (Lines 6–17). This operation continues till the termination condition is reached. The resulting candidate KBs are returned as the best performing or most suitable of the KBs.

**Algorithm 1** Procedure: GFS procedure.

**Input:** GA hyperparameters **Output:** Best set of GFT KBs 1: Initialize *t* = 0 2: Generate initial population as P<sup>0</sup> 3: Set current population C*<sup>t</sup>* := P<sup>0</sup> 4: Run the fuzzy control system using C*<sup>t</sup>* 5: Evaluate the members of C*<sup>t</sup>* 6: **while** *Not termination condition* **do** 7: <sup>P</sup>*<sup>t</sup>* :<sup>=</sup> *selection*(C*t*) 8: <sup>O</sup>*<sup>t</sup>* :<sup>=</sup> *crossover*(P*t*) 9: C*temp* := *mutate*(O*t*) 10: **if** elitism **then** 11: C*temp* := *applyElitism*(C*temp*) //keeps a percentage of previous chromosomes 12: **end if** 13: *t* := *t* + 1 14: C*<sup>t</sup>* := C*temp* 15: Run the fuzzy control system using C*<sup>t</sup>* 16: Evaluate the members of C*<sup>t</sup>* 17: **end while** 18: Return best-performing candidate solutions

Non-stationarity is an inherent problem in the multiple learning agents' environment. In this section, we present an approach for incorporating agent-induced non-stationarity awareness into a GFT based on the framework of [32].

Since the root of the GFT has the strongest impact on the learning process or action selection, we consider it to be the main component for addressing this problem. Unlike the other GFSs in the GFT, which can be modeled to address specific aspects of a control problem (domain dependent), we consider the root GFS to be a special case.

We regard the sub-trees that are formed under the root GFS as the elements of the BR function co-domain. Thus, a BR function, in the GFT case, selects a sub-policy to be used for selecting an action for the agent. The influence function is also represented as an FIS (or GFS in our case, since GA is applied to learn the KB), which takes PGFsas input variables and a BF as output the variable. This is motivated by the idea that an influence function maps beliefs to possible best responses and can be designed to use deductive reasoning. These are properties that an FIS exhibits due to fuzzy logic. Thus, the root of the GFT takes incomplete/partial observations and produces its belief of another agent's policy. This connotes that the terms of the output variables then serve as possible opponent strategies as perceived by the reasoning agent.

In the case of the PGFs that serve as input variables to the GFT root FIS, the history of observations concerning the opponent or another agent is maintained and used by each PGF variable to produce an input value. This PGF input value can then be fuzzified and used in the inference process. The problem that arises is then how to design a PGF to capture the adaptation dynamics of another agent. In the work of [33], an agent *i* estimated the model of agent *j* as:

$$\mathcal{O}\_j^i(\boldsymbol{u}\_j) = \frac{\mathbf{C}\_j^i(\boldsymbol{u}\_j)}{\sum\_{\boldsymbol{a} \in \mathcal{U}\_j} \mathbf{C}\_j^i(\boldsymbol{u}\_j)} \tag{7}$$

where *C<sup>i</sup> j* (*uj*) is the frequency that agent *i* observed agent *j* taking action *uj*. Therefore, given the history of observations, Equation (7) can be extended to include the observation of agent *i* as:

$$\mathcal{O}\_{\dot{\jmath}}^{i}(o\_{i\prime}, u\_{\dot{\jmath}}) = \frac{\mathbb{C}\_{\dot{\jmath}}^{i}(o\_{i\prime}, u\_{\dot{\jmath}})}{\sum\_{a \in \mathcal{U}\_{\dot{\jmath}}} \mathbb{C}\_{\dot{\jmath}}^{i}(o\_{i\prime}, u\_{\dot{\jmath}})} \tag{8}$$

where *C<sup>i</sup> j* (*oi*, *uj*) is the frequency that agent *i* observed agent *j* taking action *uj* when its local observation was *oi*. *σ*ˆ*<sup>i</sup> j* (*oi*, *uj*) becomes the input value of agent *i*'s PGF for monitoring a specific action *uj* of agent *j*. With the input variables (PGFs) and output variable (BF) determined for the root FIS, GA can be used to find the best response for agent-induced non-stationarity and thereby stabilize learning in the POSG.

#### **4. Multi-Agent-Based Simulator Design**

The simulator used for the experiment was developed using the Java programming language. The fuzzy control library [34] was extended for the design and encoding of fuzzy rules for learning purposes as implemented by the intelligent control model of our architecture. In this section, we present the experiment conducted for this study and follow it up with the analysis of the simulation results.

#### *4.1. Environment Setup*

To evaluate the effectiveness of the proposed learning system, a scenario of multiple boats competing for conquering more islands while engaging in combat was adopted. In this scenario, the environment set in a maritime setting consisted of *N* islands and two teams of unmanned surface vehicles (boats) referred to as blue and red forces. The boats were equipped with radar and guns for detecting and shooting enemies, respectively. The guns were set to have a fixed left-to-right traversal angle and shooting range, as shown in Figure 6. Since both teams had conflicting goals, a team achieved its goal by contending with opponent boats. Each team had information of the location and number of islands and their states, whether conquered or unconquered by the team. An island is said to be conquered by a team if a member of the team moves to the coordinate of the island and stays there for that time step or no opponent boats move to that particular island conquered by the team. If two opponent boats occupy an island at the same time, the island is not awarded to any team for the elapsed time steps. For our experiment, we implemented two controllers for both teams. The controller for the blue force was made to use fixed rules provided by humans, whereas the red force controller, which we sought to train, had to learn the best rules that maximized their performance. At the beginning of each time step, the simulation environment received a batch of commands from the (ally and enemy) controllers of both teams and updated the environment.

**Figure 6.** A snapshot of the island-conquering scenario. In the middle are the 5 islands to be conquered by both teams. The circles around the boats are their detection ranges, and the forward looking arc is the the firing regions of the boats.

The possible actions of each agent are also explained below:

#### *4.2. Competition Objective*

The goal of each team was to conquer all islands and destroy opponents, while staying alive. The performance of each team was evaluated after each time step using the function:

$$R\_t^f = A\_t^f \times p + I\_t^f - D\_t^{f'} \times p \tag{9}$$

where *A<sup>f</sup> <sup>t</sup>* is the destruction suffered by the opponent team *f* as a result of attacks from team *f* in time step *t*, *I f <sup>t</sup>* the conquered island points received by team *<sup>f</sup>* , *<sup>D</sup><sup>f</sup> <sup>t</sup>* the damaged caused by the opponent *f* to the team being assessed *f* , and *p* the points awarded for boat attacks/destruction. The winner of an encounter is decided after the end of the episode. The team with the highest score is declared a winner of that encounter.

#### *4.3. Data Collector for Training and Learning*

As mentioned earlier, gathering of training data is useful in a number ways. The cooperation of agents can be enhanced when agents share their observation. Furthermore, data gathered during training can be used later by other learning algorithms such as deep learning methods for analysis and performance improvement. The resource model shown in Figure 7 is used for on-line data gathering during training. The data that are sent to a controller as feedback take this form. The main components are as follows:

**Figure 7.** The simulation feedback resource structure of the island-conquering case study.

#### *4.4. Training and Learning Algorithm*

In order to provide a realistic virtual environment and opponent for the red force to train against, the blue force control algorithm uses human-defined rules. Table 1 shows an example encoding rules used for the blue force task selection during simulation. The GFT of the red force control algorithm is illustrated in Figure 8. The details of each of the GFSs of the GFT can also be seen in Table 2.

**Table 1.** Example encoded task selection rules used by the enemy team.

**Figure 8.** The genetic fuzzy tree structure used for training the ally team. The rectangles are FISs, whiles the circles represent predefined actions in the simulation.

**Table 2.** Details of the GFSsthat constitute the GFT used for training the ally team.

The Assignment GFS was designed to consider opponent-induced non-stationarity with the following PGFs:

The linguistic terms of each PGF are low, moderate, and high. Each PGF first computes the action probabilities for each detected agent. A softmax is then computed over the reported action probabilities, and the maximum is selected as the PGF input value. The belief function output variable selects the perceived opponent strategy in each IF-THEN rule. The other input variables are: