# Ontology-Based Feedback to Improve Runtime Control for Multi-Agent
Manufacturing Systems

Jonghan Lim1, Leander Pfeiffer2, Felix Ocker2, Birgit Vogel-Heuser2, and Ilya
Kovalenko1 1Jonghan Lim and Ilya Kovalenko are with Department of Industrial
and Manufacturing and Department of Mechanical Engineering, Pennsylvania State
University, State College, USA (e-mail: {jxl567; iqk5135}@psu.edu). 2Leander
Pfeiffer, Felix Ocker and Birgit Vogel-Heuser are with Institute of Automation
and Information Systems, Technical University of Munich, Munich, Germany
(e-mail: {leander.pfeiffer; felix.ocker; vogel-heuser}@tum.de).

###### Abstract

Improving the overall equipment effectiveness (OEE) of machines on the shop
floor is crucial to ensure the productivity and efficiency of manufacturing
systems. To achieve the goal of increased OEE, there is a need to develop
flexible runtime control strategies for the system. Decentralized strategies,
such as multi-agent systems, have proven effective in improving system
flexibility. However, runtime multi-agent control of complex manufacturing
systems can be challenging as the agents require extensive communication and
computational efforts to coordinate agent activities. One way to improve
communication speed and cooperation capabilities between system agents is by
providing a common language between these agents to represent knowledge about
system behavior. The integration of ontology into multi-agent systems in
manufacturing provides agents with the capability to continuously update and
refine their knowledge in a global context. This paper contributes to the
design of an ontology for multi-agent systems in manufacturing, introducing an
extendable knowledge base and a methodology for continuously updating the
production data by agents during runtime. To demonstrate the effectiveness of
the proposed framework, a case study is conducted in a simulated environment,
which shows improvements in OEE during runtime.

##  I Introduction

Manufacturing systems have become increasingly complex in recent years due to
various factors, including globalization, the spread of new technology, the
rising demand for customized products, and the growing concern about
sustainability. As a result, manufacturers are facing the challenge of
managing the performance of their manufacturing systems to ensure that they
operate efficiently and effectively, which can be achieved by improving
Overall Equipment Effectiveness (OEE) [1]. A key approach to improve OEE is to
enable runtime control, managing the performance of the manufacturing system
in real-time, including continuous monitoring and decision making [2].

Decentralized methods allow for greater flexibility and adaptability in
complex manufacturing environments and quickly respond to changes on the plant
floor [3]. One such decentralized strategy is the use of Multi Agent System
(MAS) to control various types of complex systems. MAS has been applied to
various aspects of manufacturing systems, such as product design, production
planning, and control [4]. These systems perform well in dynamic environments
with increased uncertainty and complexity.

Despite the advantages of improved flexibility and adaptability, coordinating
the activities of agents requires extensive communication and computational
efforts during runtime with a shared understanding of the environment. To
overcome these challenges, Semantic Web Technologies (SWT) have been proposed
to support more efficient communication and knowledge sharing [5]. Ontologies,
enabled by SWT, play a crucial role in facilitating communication and
interoperability in MASs, particularly in the manufacturing domain [6]. SWT
can provide improved communication and cooperation among agents by providing a
common language and structure.

While online communication load and computational effort in MAS have been
reduced by using SWT [5], there are still challenges when ensuring effective
runtime control of the MAS in manufacturing. One challenge lies in the ability
of the MAS to adapt to changes in the production system and its environment,
capturing all the necessary data for the agents to accurately measure the OEE
of each machine. (C1). Another challenge involves the need for technical
expertise for both customers and engineers to interact with agents in the MAS,
as it requires a deep understanding of the system’s architecture and
communication protocols (C2). There is also a challenge of continuously
updating agents relevant to previous and current states of the
resources/parts/processes in the system (C3).

The main contribution of this work is the design of the ontology, i.e.
Knowledge Base (KB), for MAS in manufacturing to enable agents to access and
update the KB during runtime and provide continuous decision support based on
a global context. The contributions of this paper over the previous work
include: (1) an extendable KB that effectively accumulates the MAS’s state and
history of production to accurately measure the OEE, (2) an ontology-based
interface, enabling customers and engineers to facilitate communication with
the MAS during runtime, and (3) the implementation of a mechanism where agents
within the manufacturing system continuously update the KB with their current
state and the latest history of their actions.

The rest of the manuscript is organized as follows. Section II provides
background regarding ontologies and MASs in manufacturing. Section III
describes an ontology-based multi-agent framework to improve runtime control
of manufacturing systems. Section IV explains the implementation of the
proposed framework Section V presents a case study to demonstrate the proposed
framework. Section VI summarizes the paper and discusses future work.

##  II Background

Ontologies and multi-agent systems have been implemented in manufacturing
systems to enable more flexible, adaptable, and efficient production systems.

###  II-A Ontologies in Manufacturing

An ontology is understood as an “explicit specification of a
conceptualization” [7]. As such, ontologies can be used to represent knowledge
about production systems. The ontology consists of the terminological
component (TBox), the schema, and the assertional component (ABox), which
includes the instance level. Ontologies can be utilized to efficiently locate,
categorize, represent, and reuse knowledge that already exists inside various
information resources [8].

MASON (MAnufacturing’s Semantics ONtology) is one of the first examples of
ontology in the manufacturing domain [9] to formalize and share data using the
Web Ontology Language (OWL). Ontologies have also been applied in specific
applications within the manufacturing field. For example, Dinar et al. [10]
proposed an ontology for Design for Additive Manufacturing (DFAM) to model 3D
printed components. In the steel manufacturing industry, Common Reference
Ontology for Steelmaking (CROS) was proposed as a solution to the problem of
semantic interoperability [11]. One study presented an OWL-based manufacturing
resource capability ontology (MaRCO) [12], which aims to describe the
capabilities of manufacturing resources in a common formal resource model. The
project AutomationML ontology (AMLO) interlinks and integrates heterogeneous
data in industrial systems design by providing a semantic representation of
the AutomationML standard [13]. While these studies have developed ontologies
that provide information about the combined capabilities of resources and
integrate heterogeneous data, they do not focus on applications during
runtime.

###  II-B Multi-Agent Systems in Manufacturing

A variety of multi-agent architectures have been developed to achieve system-
level control of manufacturing systems [14, 15, 16]. These multi-agent
architectures use several software agents to make high-level decisions for
different manufacturing system components [3, 17]. The high-level decisions
made by the agents influence the overall performance of the manufacturing
system [18]. Therefore, the design of these software agents is crucial in
understanding and enhancing the performance of the manufacturing system.

Recently proposed MAS architectures for manufacturing systems contain
instances of product agents (PAs) and resource agents (RAs) [3, 19, 20]. A PA
is an agent which makes decisions for a specific component in the production
system. An RA is a high-level controller for a resource (e.g., robot, machine)
on the shop floor. An example proposed in [3] presents a MAS architecture
revolving around PAs that have the ability to understand their surroundings,
plan, and activities, and request necessary actions from the RAs. Similarly,
Bi et al. [20] developed a model-based RA architecture that incorporates risk
assessment and improves throughput in dynamic manufacturing environments.
While these architectures provide data structures and requirements for
coordinating the behavior of agents, their aim is not on utilizing the
historical knowledge for improving the runtime control of the manufacturing
system.

###  II-C Ontologies and Multi-Agent Systems in Manufacturing

The introduction of ontologies in MASs allows agents to make decisions based
on their shared understanding of the domain. Formalizing knowledge is key to
enabling agents to dynamically adapt to system changes [15]. Thus, it has been
suggested that the use of SWTs would improve the MAS of manufacturing systems
[6].

ADAptive holonic COntrol aRchitecture (ADACOR) [21] is one of the early
efforts for using an ontology and MAS in manufacturing systems. This project
aims to develop an ontology for distributed manufacturing systems that include
components and procedures to assist in scheduling and monitoring. Vrba et al.
[22] suggested using a central ontology in the form of an Resource Description
Framework (RDF) database. A production plan agent provides each PA with a
possible production plan and then single agents schedule a plan by
collaborating with available resources. An ontology framework for automatic
initialization of multi-agent production systems using SWT was proposed in
prior work [5]. The ontology is effectively queried to produce an automaton
for initializing the individual agents. As a result, the online communication
load and the computational effort are reduced for PAs and RAs. However, in the
framework proposed runtime information from the agents is not fed back to the
ontology, which would allow for adapting and evolving over time as the
environment changes.

In summary, several architectures have been proposed for using SWT and MAS for
controlling manufacturing systems. However, none of these works aim to utilize
an ontology for MAS in manufacturing to improve runtime control by feeding
information back into the ontology. To the best of our knowledge, none of the
existing works addresses the combination of utilizing an extendable KBs to
capture comprehensive historical information (C1), providing an interface for
customers and engineers for agent communication during runtime via the KB
(C2), and updating the MAS during runtime (C3) to provide decision support for
the agents in a global context.

##  III Framework for Runtime Control of Agents

In this section, we introduce an ontology-enhanced multi-agent framework for
manufacturing that contains an extendable KB, an interface for engineers and
customers, and runtime control through communication and coordination between
agents and the ontology.

###  III-A Design of the Framework

Figure 1: Graphical Abstract of the Framework

The novel architecture presented in Figure 1 provides a general overview of an
ontology-based multi-agent framework. An ontology is designed based on the
architecture of a MAS that includes both PAs and RAs using the Belief-Desire-
Intention (BDI) paradigm. This ontology stores the knowledge about the
production process, the resources, and the interactions. Section IV-B offers a
detailed discussion on how the knowledge is stored within the ontology.
Engineers input the initial knowledge of a manufacturing system in the
spreadsheets and then export it into the ontology using the Ontology Builder
during the “offline” phase, depicted in Figure 1. Using the constructed
ontology, agents in MAS communicate with the ontology through the Application
Programming Interface (API) during runtime, i.e. the “online” phase. The
agents are indicated as PAs and RAs in the box of MAS in Figure 1. Both
customers and engineers also have the ability to retrieve and update the KB
using the API as shown in Figure 1. Some of the essential functions provided
by the API are introduced in Section III-C.

The BDI architecture includes beliefs, desires, and intentions, where beliefs
refer to the manufacturing environment, desires define goals, and intentions
are plans derived from beliefs to fulfill desires [3]. Therefore, an ontology
based on the BDI architecture can provide a structured and organized way to
represent the knowledge required for both PAs and RAs to operate effectively
in the MAS.

The framework introduces three essential capabilities, offering main benefits
over existing approaches. First, the framework utilizes an ontology as a
centralized KB to accumulate the production history of the MAS. This knowledge
provides agents with holistic environment models, offering a comprehensive
understanding of the system’s current state and history (C1). The agents are
able to make their respective decentralized decisions in a global context. The
framework also allows engineers and customers to easily communicate with the
agents in the MAS through the use of ontology (C2). They can retrieve and
update the manufacturing environment without needing knowledge of the system
architecture, thus providing flexibility and scalability. Additionally, the
framework provides capabilities for communication and coordination between
agents and the ontology during the runtime of the manufacturing process (C3).
The agents can directly query the ontology for adjustments in response to
global changes in resources, product features, or other factors.

###  III-B The Basic Ontology

Figure 2: Overview of the Ontology

The core concepts in the manufacturing system that are defined in the ontology
include these classes: feature, process, resource, and specification. This is
based on the Product Process Resource (PPR) model, in which resources execute
processes on products to realize features [23].

The specification class refers to a detailed description of the desired
product, including the specific features, requirements, and the expected
deadline, as specified by the customer. The process class contains information
on the production processes. These include both the processPlans, which is a
set of operations and procedures that transform inputs into finished products
including the use of resources, as well as processExecutions, which represent
real-world execution of the respective process plans. The resource class
encompasses physical assets or capabilities used in the production process
such as robots and buffers. The feature refers to the various characteristics
of a product that are defined in order and used to describe the desired
product. The initial TBox of the ontology that includes this information is
shown in Figure 2.

Figure 3: Relationship among Products, Processes, Features, and Resources

To ensure that the KB accurately represents not only the static MAS, but also
the dynamically changing environment, further classes, and properties have to
be defined. To align with the BDI architecture and capture the PA objectives,
we propose an objective function that considers a set of use-case specific
metrics which is shown in Figure 3. This is represented in the ontology by the
PA specific class objectiveFunction. The link between the objective function
and the PA is created by the hasObjectiveFunction property. The
objectiveFunction then has several coefficients linked to it by the
hasCoefficient property. The value of the coefficient is given by the data
property hasValue and the type of metric by the coefficientFor datatype
property. Their respective performance also needs to be evaluated to allow for
an evaluation of different process plans. This is done by introducing a
performance class and the hasPerformance property. The hasPerformance property
has two subclasses, realPerformance and expectedPerformance to further enable
transparency when it comes to historic decision-making. The last metric for
each product is the deadline. By utilizing the previously mentioned classes,
we can accurately represent both the beliefs (via environment models and
expected performances) and desires (via objective function and deadline) of
the BDI architecture.

###  III-C Modeling the System during Runtime

Figure 4: Example of Runtime Information Ontology

To support runtime control of MAS utilizing the historical data, runtime
information is added to the KB. We developed a concept called
processExecution, establishing its relationship with product, resource, and
performance, to provide agents information during runtime. A processExecution
represents a physical event, in which a processPlan gets executed.

An example of runtime data is shown in Figure 4. A PA adds each process
execution to the ontology and displays its intention in a globally accessible
format. To connect each process execution to different parts of the base
ontology, a new set of object properties is introduced. The
hasProcessExecution property associates a product with execution, while the
runsOnResource property indicates which resource is utilized. The
runsProcessPlan property links the process plan to the execution.

The hasStatus property captures the different stages of the execution by
assigning a literal value. Initially, the PA proposes the execution in the
ontology and the status changes to planned. When the execution starts, the
status changes to running and then to successful if completed successfully. If
unexpected events occur and the execution does not finish as planned, the
status changes to errored with a hasErrorMessage property. Other important
datatype properties for the execution include plannedStartTime and
plannedEndTime, as well as realStartTime and realEndTime. The actual
performance of the execution may differ from the expected performance of the
process, which is captured by the realPerformance property. By utilizing this
model, we can incorporate past, present, and future events into an ontology,
which further contributes to the belief component of the BDI architecture. As
a result, these events form the basis of intentions for the various agents
within the MAS. The process execution model facilitates the integration and
retrieval of a wide range of knowledge and capabilities during runtime. The
following are some of the essential interactions implemented:

  * •

addPlannedExecutionData: This query adds data on the product that will be
processed, planned start and end times, and the resources that will be
utilized.

  * •

updateExecutionData: This query updates execution data such as status, real
start times, and end times.

  * •

getProductStatus: This query is used to retrieve product-related information
such as the status, duration, and deadline.

  * •

getResourceHistory: This query allows retrieval of the historical data of the
resources utilized in the manufacturing processes.

  * •

changeResourcePerformance: This query allows modification of the performance
such as energy costs, emissions, and quality.

By utilizing queries, various interactions can be implemented to retrieve and
modify information such as resource history and performance. The agents can
use this information to improve runtime control and OEE of machines in the
system.

##  IV Implementation

###  IV-A Dynamically Building the Knowledge Base

The approach for dynamically building the KB is based on prior work [5]. The
initial ontology is created using spreadsheets to reduce dependence on
proprietary tools. The spreadsheets are converted into CSV files using a
macro. Finally, the transformation of these files into a formalized ontology
using OWL is implemented in Python using the package Owlready2 [24]. By
leveraging this package, the ontology creation process was made more
accessible.

###  IV-B Agent-Ontology Communication during Runtime

Figure 5: Communication between Agents and Ontology

We developed an API to facilitate communication between the MAS and the
ontology. This API allows interacting with the ontology, querying and updating
the ontology using a HyperText Transfer Protocol (HTTP) requests. The PAs and
RAs from the MAS request specific actions through this API, which then
utilizes a REpresentational State Transfer (REST)-interface to pass the
corresponding query to the ontology. When a PA or RA from the MAS retrieve or
update information from an ontology, the agents send a request to the REST
API. The API then converts the request to SPARQL Protocol And RDF Query
Language (SPARQL) queries, allowing agents to retrieve or manipulate data in
the ontology. We utilize the Stardog [25] knowledge graph platform to run the
SPARQL queries outlined in III-C. For instance, if an agent requests
information about a product’s status, the API will execute the appropriate
SPARQL query. The response is provided in JavaScript Object Notation (JSON) to
an agent. The format is optimized for the agents, meaning that it is
structured in a way that makes it easy for the agents to use the data.
Moreover, customers and engineers can also interact with the ontology to
retrieve or update information during runtime. Note that in the context of
this framework, there is no communication delay between the agents and
ontology. A graphical representation of this communication between agents and
ontology is provided in Figure 5.

###  IV-C Querying the Knowledge Base

Figure 6: Querying the Knowledge Base

The ontology was designed to store information about processes, resources,
relationships, and performance data. To register runtime information to an
ontology based on this design, we utilize SPARQL to interact with an ontology
as described in Section IV-B. The overall process of querying some of the
knowledge and capabilities for a PA and an RA is shown in Figure 6. The solid
line represents the request flow from the agents to the ontology, while the
dotted line represents the response flow from the ontology back to the agents.
An INSERT operation allows new data to be added, thus extending the KB. For
example, to register a new planned process execution, the data includes the
properties of the execution such as its ID, part name, and planned start time
and end time. For UPDATE operations, INSERT and DELETE operations can be
combined. When a process in the MAS is executed, we update the start time and
status of a process. The SELECT operation can retrieve information during
runtime. For example, an RA can use this operation to retrieve information
about the resource’s past activities. The SPARQL query illustrated in
LABEL:lst:sparql retrieves information associated with successful process
executions run on a particular resource. The query selects distinct values for
the execution ID, emissions, costs, quality, and start and end time. The
question mark (“?”) is used as a prefix to represent variables that will be
assigned values during the execution of the query. It accomplishes this by
searching for a resource identified by the <resource> URI and filtering for
executions with a status of "successful". The query then looks for performance
data including emissions, costs, and quality. This information can be used by
an RA to make decisions about allocating resources to improve the
manufacturing process.

Listing 1: SPARQL Query for retrieving a resource’s history.

⬇

1SELECT DISTINCT ?execution ?emissions ?costs

2 ?quality ?startTime ?endTime

3 WHERE

4{

5 ?execution :runsOnResource <resource>;

6 :hasStatus "successful";

7 :realPerformance [

8 :emissions ?emissions;

9 :costs ?costs;

10 :quality ?quality

11 ];

12 :realStartTime ?startTime;

13 :realEndTime ?endTime.

14} ORDER BY ?startTime

##  V Case Study

In this case study, we examine the proposed framework and demonstrate how the
MAS utilizes the ontology to improve runtime control.

###  V-A Case Study Setup and Workflow

Figure 7: Layout of a Small Manufacturing System

To evaluate the feasibility, a case study is set up in a small manufacturing
system environment, which is composed of four identical machines (M1, M2, M3,
and M4), two robots (R1 and R2), and three buffers (B1, B2, and B3). The
system has 6 RAs representing R1, R2, M1, M2, M3, and M4. R1 transfers the
part from B1 to B2. R2 transfers the part from B2 to a machine. When a
manufacturing process P1 is completed, a part is transferred from a machine to
B3 to exit the system. The performance of the machines is shown in Table I.

The objective of PA and RA is to enhance the runtime control of MAS in
manufacturing by utilizing the knowledge in the ontology. Specifically, the
goal of PAs is to fulfill the product requirements based on the
objectiveFunction, which is described in Section III-B. The goal of RAs is to
increase their OEE while limiting energy utilization. In this example, the RAs
try to increase their OEE by ensuring that each resource has an uptime over
50% while limiting the total energy cost to 450 kWh. Each machine’s
performance is adjusted by controlling the duration and energy cost by the
RAs. In this case study, the assumption is made that 5% of energy cost from
the previous cost results in a one-minute decrease in duration. The layout of
a small manufacturing system is depicted in Figure 7.

A new manufacturing process is initiated once a new part enters the system
beginning at B1. As a part enters the system, product data are initialized in
the ontology as shown in Figure 3. This data also contains information such as
planned start and end time, provided in Figure 4. When R1 RA queries from the
ontology that a process P1 is planned, R1 moves the part from B1 to B2. Once
the part has been moved to B2, the PA updates the data for P1 by specifying
the machine to be run on. It should be noted that the ontology only provides
each PA with possible plans, and does not compute optimal plans. Therefore,
the PA identifies the machine to be used by evaluating the objectiveFunction,
as depicted in Figure 3. Then, the PA selects a machine that allows parts to
exit the system as fast as possible, taking into account the time required for
previously running parts inside the machines. When R2 RA detects from the
ontology that a part is planned to be processed on a machine such as M2, R2
moves the part to M2. After the part is processed, the PA updates the ontology
by setting the hasStatus to “successful” and realEndTime to the current time
using the UPDATE Query shown in Figure 6. Finally, R2 picks up the part from
M2 to B3 to exit the system.

TABLE I: Peformance of the Machines Machines | M1 | M2 | M3 | M4
---|---|---|---|---
Duration (minutes) | 20 | 18 | 15 | 17
Energy Costs (kWh) | 100 | 110 | 120 | 115

###  V-B Case Study Scenario

In the case study, we created a scenario of how RAs use the ontology to adjust
the performance levels of each machine. The RAs of each machine queries the
ontology to obtain historical data about the status, energy costs, quality,
and process time of each machine. Specifically, the RAs query the ontology
using the SPARQL query shown in LABEL:lst:sparql to retrieve the performance
of each completed process for their respective machines. The query provides
RAs with the uptime and real performance over the expected performance data of
each machine.

Suppose that after an initial set of operations, the uptime of M1 is below the
threshold of 50% (e.g. M1: 48%, M2: 60% M3: 80% M4: 70%) while the performance
efficiency was the best at M1 (e.g. M1: 95%, M2: 93% M3: 80% M4: 85%). To
improve the uptime of M1, The M1 RA increases the energy cost and reduces the
duration, while the M3 RA decreases the energy cost and increases the
duration. For example, the RA of M1 reduces the duration of M1 from 20 to 18
minutes and increases the energy cost by 10.25% (from 100 to 110.25) while M3
increases the duration from 15 to 16 minutes and decreases the energy by 5%
(from 120 to 114) using the API function called changeResourcePerformance.
With these adjustments, the uptime of M1 not only exceeds 50%, leading to
improvements in OEE but also ensures that the products are processed in the
most high-performing machine available, thereby maximizing productivity.

In this case study, we showcase how a small manufacturing system exhibits
nominal behavior by utilizing ontology for monitoring the system’s status in a
global context. The agents involved in the system continuously monitor the
ontology during runtime to make informed decisions, leading to improved
runtime control of the manufacturing process.

##  VI Summary and Outlook

In this paper, we presented an ontology-based approach for implementing MAS in
manufacturing to enable agents and users to access and update the KB during
runtime. The framework proposed for implementing an ontology for MAS in
manufacturing has several advantages over existing approaches. First, the
extendable KB accumulates the system’s state and history of production data,
enabling agents to make informed decisions based on this data to measure the
OEE (C1). Second, providing an API enables customers and engineers to
communicate with the MAS during runtime without requiring detailed technical
knowledge of the system (C2). Lastly, the agents are continuously updating the
KB with their current state and their latest actions, which lead to
improvements of OEE (C3).

Future research will include evaluating this ontology-based framework by
incorporating additional criteria that focus on improving system efficiency
and effectiveness to further validate its benefits over existing approaches.
Also, we plan to explore the application of this ontology-based framework to a
real-world testbed, aiming to achieve improved OEE through ontology
utilization during runtime with MAS. This testbed will enable us to prove the
effectiveness of the framework in a practical manufacturing environment.

## References

  * [1] S. Nakajima, “Introduction to TPM: Total Productive Maintenance.” _Productivity Press_ , p. 129, 1988.
  * [2] R. Iannone and M. E. Nenni, “Managing OEE to Optimize Factory Performance,” _Operations Management_ , pp. 31–50, 2013.
  * [3] I. Kovalenko, D. Tilbury, and K. Barton, “The Model-based Product Agent: A Control Oriented Architecture for Intelligent Products in Multi-agent Manufacturing Systems,” _Control Engineering Practice_ , vol. 86, pp. 105–117, 2019.
  * [4] J.-H. Lee and C.-O. Kim, “Multi-agent Systems Applications in Manufacturing Systems and Supply Chain Management: A Review Paper,” _International Journal of Production Research_ , vol. 46, no. 1, pp. 233–265, 2008.
  * [5] F. Ocker, I. Kovalenko, K. Barton, D. Tilbury, and B. Vogel-Heuser, “A Framework for Automatic Initialization of Multi-Agent Production Systems Using Semantic Web Technologies,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 4330–4337, 2019.
  * [6] M. Obitko and V. Marik, “Ontologies for Multi-agent Systems in Manufacturing Domain,” _13th International Workshop on Database and Expert Systems Applications, IEEE Computer Society_ , pp. 597–602, 2002.
  * [7] T. R. Gruber, “A Translation Approach to Portable Ontology Specifications,” _Knowledge acquisition_ , vol. 5, no. 2, pp. 199–220, 1993.
  * [8] M. Hadzic, E. Chang, P. Wongthongtham, and T. Dillon, _Ontology-based multi-agent systems_. Springer, 2009.
  * [9] S. Lemaignan, A. Siadat, J.-Y. Dantan, and A. Semenenko, “MASON: A Proposal for an Ontology of Manufacturing Domain,” _IEEE Workshop on Distributed Intelligent Systems: Collective Intelligence and Its Applications_ , pp. 195–200, 2006.
  * [10] M. Dinar and D. W. Rosen, “A Design for Additive Manufacturing Ontology,” _Journal of Computing and Information Science in Engineering_ , vol. 17, no. 2, 2017.
  * [11] Q. Cao, S. Beden, and A. Beckmann, “A Core Reference Ontology for Steelmaking Process Knowledge Modelling and Information Management,” _Computers in Industry_ , vol. 135, p. 103574, 2022.
  * [12] E. Järvenpää, N. Siltala, O. Hylli, and M. Lanz, “The Development of an Ontology for Describing the Capabilities of Manufacturing Resources,” _Journal of Intelligent Manufacturing_ , vol. 30, no. 2, pp. 959–978, 2019.
  * [13] O. Kovalenko, I. Grangel-González, M. Sabou, A. Lüder, S. Biffl, S. Auer, and M.-E. Vidal, “Automationml Ontology: Modeling Cyber-physical Systems for Industry 4.0,” _IOS Press Journal_ , vol. 1, 2018.
  * [14] P. Leitão, “Agent-based Distributed Manufacturing Control: A State-of-the-art Survey,” _Engineering Applications of Artificial Intelligence_ , vol. 22, no. 7, pp. 979–991, 2009.
  * [15] P. Leitao, V. Marik, and P. Vrba, “Past, Present, and Future of Industrial Agent Applications,” _IEEE Transactions on Industrial Informatics_ , vol. 9, no. 4, pp. 2360–2372, 2013.
  * [16] H. J. Yoon and W. Shen, “A Multiagent-Based Decision-Making System for Semiconductor Wafer Fabrication With Hard Temporal Constraints,” _IEEE Transactions on Semiconductor Manufacturing_ , vol. 21, no. 1, pp. 83–91, 2008.
  * [17] M. Bi, I. Kovalenko, D. M. Tilbury, and K. Barton, “Dynamic Resource Allocation Using Multi-Agent Control for Manufacturing Systems,” _IFAC-PapersOnLine_ , vol. 54, no. 20, pp. 488–494, 2021.
  * [18] J. Barbosa, P. Leitão, E. Adam, and D. Trentesaux, “Dynamic Self-organization in Holonic Multi-agent Manufacturing Systems: The ADACOR Evolution,” _Computers in Industry_ , vol. 66, pp. 99–111, 2015.
  * [19] A. M. Farid and L. Ribeiro, “An Axiomatic Design of a Multiagent Reconfigurable Mechatronic System Architecture,” _IEEE Transactions on Industrial Informatics_ , vol. 11, no. 5, pp. 1142–1155, 2015.
  * [20] M. Bi, I. Kovalenko, D. M. Tilbury, and K. Barton, “Dynamic Distributed Decision-making for Resilient Resource Reallocation in Disrupted Manufacturing Systems,” _International Journal of Production Research_ , pp. 1–21, 2023.
  * [21] S. Borgo and P. Leitão, “The Role of Foundational Ontologies in Manufacturing Domain Applications,” _OTM Confederated International Conferences, Springer_ , pp. 670–688, 2004.
  * [22] P. Vrba, M. Radakovič, M. Obitko, and V. Mařík, “Semantic Technologies: Latest Advances in Agent-based Manufacturing Control Systems,” _International Journal of Production Research_ , vol. 49, no. 5, pp. 1483–1496, 2011.
  * [23] J. Nielsen, “Information Modeling of Manufacturing Processes: Information Requirements for Process Planning in a Concurrent Engineering Environment,” Ph.D. dissertation, KTH, 2003.
  * [24] J.-B. Lamy, “Owlready: Ontology-oriented Programming in Python with Automatic Classification and High Level Constructs for Biomedical Ontologies,” _Artificial intelligence in medicine_ , vol. 80, pp. 11–28, 2017.
  * [25] Stardog Union, “Stardog: The Enterprise Knowledge Graph Platform.” [Online]. Available: https://www.stardog.com/


# LiCS: Navigation using Learned-imitation on Cluttered Space

Joshua Julian Damanik, Jae-Won Jung, Chala Adane Deresa, and Han-Lim Choi J.
J. Damanik, J.-W. Jung, C. A. Deresa, and H.-L. Choi were with the Department
of Aerospace Engineering, Korea Advanced Institute of Science and Technology,
Daejeon, Republic of Korea. E-mail: (joshuad, jjwon13, czchal99,
<EMAIL_ADDRESS>
###### Abstract

In this letter, we propose a robust and fast navigation system in a narrow
indoor environment for UGV (Unmanned Ground Vehicle) using 2D LiDAR and
odometry. We used behavior cloning with Transformer neural network to learn
the optimization-based baseline algorithm. We inject Gaussian noise during
expert demonstration to increase the robustness of learned policy. We evaluate
the performance of LiCS using both simulation and hardware experiments. It
outperforms all other baselines in terms of navigation performance and can
maintain its robust performance even on highly cluttered environments. During
the hardware experiments, LiCS can maintain safe navigation at maximum speed
of $1.5\ m/s$.

###### Index Terms:

imitation-learning, indoor navigation, cluttered environment, UGV.

##  I Introduction

Navigation within cluttered indoor environments poses a substantial challenge
for Unmanned Ground Vehicles (UGVs). Ensuring robust and rapid navigation in
such cluttered spaces is vital for applications that range from warehouse
automation to search and rescue missions. Traditional navigation systems often
encounter difficulties in these environments due to tight spaces and numerous
obstacles [1, 2].

The recent availability of benchmarking datasets [3, 4] for navigation in
cluttered environments has facilitated significant advancements in learning-
based navigation systems, particularly through the use of reinforcement
learning (RL) [5] and imitation learning (IL) [6]. RL shows promise but can
result in unexpected behaviors and requires extensive reward function
engineering for effective training [7].

Imitation learning (IL), in contrast, aims to replicate the behavior of an
expert, be it a human or optimal control algorithm. This complex task can be
simplified into a supervised learning model known as Behavior Cloning (BC).
However, BC faces limitations due to its assumption that data in the training
dataset are sampled independently of the environment [8]. In practice, actions
taken during training influence future states, leading to compounded errors in
the learned policy [9].

To mitigate these challenges, techniques like SMILe [9] and DAgger [10]
combine BC with sequential online learning to maintain policy performance.
Although effective, these methods require ongoing expert interaction during
training, which can be resource-intensive. Offline BC, alternatively,
emphasizes careful planning of the demonstrations to ensure comprehensive
exploration coverage. A key strategy involves introducing controlled noise to
the input controls, which has been shown to enhance the robustness and
generalizability of the policies [11, 12, 13].

(a) Without noise

(b) With noise

Figure 1: Demonstrations with exploration noise allow policy to learn how to
effectively act on various states

In this letter, we introduce Learned-imitation on Cluttered Space (LiCS), an
end-to-end local navigation method to tackle the challenge of navigation in
cluttered environments. Our approach utilizes a Transformer-based neural
network for imitation learning [14]. During demonstrations, we inject Gaussian
noise $\mathcal{N}(0,\sigma^{2})$ to ensure the expert demonstrations cover a
broad range of states and induce knowledge of the policy to recover from
imminent collision (Fig. 1). This method enables the system to learn and
replicate an optimization-based baseline algorithm, adapting it to robustly
and rapidly handle cluttered environments. Our proposed method demonstrates a
robust capability to navigate a UGV equipped with LiDAR at speeds up to
$1.5m/s$ through narrow passages, validated in both simulation and real-world
experiments.

##  II Proposed System

Figure 2: Training pipeline diagram of the Learned-imitation on Cluttered
Space model.

The proposed system is a learning-based local planner for a differential robot
equipped with LiDAR and odometry sensors. It consists of two main components:
a Transformer-based neural network trained through imitation learning and a
safety check layer using geometric calculations to ensure the safety of the
input control. For a complete system stack, a global planner (e.g., Djikstra
or A*) is required. Optionally, SLAM can be employed for complex navigation
environments and tasks, such as exploration.

###  II-A Transformer-based neural network

The network used for the imitation learning is depicted in the Fig. 2.
Employing both transformer encoder and decoder, the network processes raw
LiDAR sensor data and normalized local goal $\hat{P}$, derived from a global
path obtained from the global planner, to provide outputs of linear and
angular velocity ($v$ and $\omega$) for the robot. The LiDAR scan data is a
vector with length of $H$ data, and a global path is a list of Cartesian
points $P^{\prime}=[(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{k},y_{k})]$ from
the current robot position towards the global goal point.

To derive the local goal $P_{g}$, points in the global path are transformed
from the origin $O^{\prime}(0,0)$ to the robot frame
($P_{i}^{\prime}\rightarrow P_{i}$), selecting the closest point on the global
path that has distance $||P_{i}||\geq L,\ \forall i\in 1,2,\ldots,k$, where
$L$ is the lookup distance of the robot. The normalized local goal is
calculated by dividing the transformed local goal point with its magnitude.

$\displaystyle P_{g}$
$\displaystyle=\operatorname{arg\,min}_{P_{i}:\|P_{i}\|\geq L}\|P_{i}\|$ (1)
$\displaystyle\hat{P_{g}}$ $\displaystyle=\frac{P_{g}}{||P_{g}||}$ (2)

The transformer encoder’s architecture is derived from the ViT (Vision
Transformer) [15]. As illustrated in Fig. 2, the encoder inputs raw LiDAR scan
data, which consists of distance measurements obtained sequentially increasing
laser angles. The input $X\in\mathbb{R}^{H\times 1}$ is first reshaped into a
2D matrix of patches $X_{p}\in\mathbb{R}^{N\times D}$, where $N$ is numbers of
patches and $D=H/N$ is the length of the patches. Following reshaping, the
input passes through a LiDAR embedding layer consisting of a linear network.
After embedding, trainable positional embeddings are added. Unlike the
standard ViT where class token row is added during the embedding process, we
omit this step as our focus is encoding the observation data for navigation,
not classification. The remainder of encoder structure mirrors that of the
standard transformer [14].

The transformer decoder processes the normalized local goal as input. During
embedding, the the input is passed through a fully connected layer to match
the dimension of encoder’s output. In contrast to the standard transformer,
position encoding and masked multi-head attention layers are omitted due to
the non-sequential nature of the input. However, the encoder-decoder attention
layers are retained, allowing the network to learn the relationship between
the LiDAR scan and the normalized local goal, which guides the generation of
robot commands through a linear transformation.

To facilitate the deployment on embedded systems, we have minimized the number
of layers in both the encoder and decoder to three, ensuring the model remains
lightweight and operational without requiring GPU (Graphics Processing Unit).

###  II-B Behavior cloning

Given a dataset $\mathbb{D}$ consisting of sets of state-action pairs
$\\{(s,a^{*})\\}$ from simulation using an expert controller, our goal is to
train a policy $\pi_{\theta}(s)$ with parameters $\theta$ that imitates the
expert policy $\pi^{*}$. Behavior cloning (BC) reduces the imitation learning
into a supervised learning task aimed at minimizing the following objective:

$\arg\min_{\theta}\mathop{\mathbb{E}_{(s,a^{*})\sim\mathbb{D}}}[l(a^{*},\pi_{\theta}(s))]$
(3)

In our proposed system, we employ an MSE loss function to calculate the
discrepancy between the expert action and the learned policy
$l(a^{*},\pi_{\theta}(s))$. The network, as shown in Fig. 2, is trained using
simulation of differential robot model. We deploy the model of cluttered
environment for robot navigation as proposed in [3]. During training,
exploration noise modeled as Gaussian function with standard deviation
$\sigma$ is added to the output velocity action, modifying the input action in
the simulation to include noise rather than the optimal one.

$\displaystyle v$ $\displaystyle=v^{*}+\mathcal{N}(0,\sigma^{2})$ (4)
$\displaystyle\omega$ $\displaystyle=\omega^{*}+\mathcal{N}(0,\sigma^{2})$ (5)

Using the LiDAR sensor data $X$ and normalized local goal $\hat{P_{g}}$ as
input for the neural network model, and the optimal output velocity action
($v^{*},\omega^{*}$) provided by the baseline algorithm as the target value,
we trained the proposed network in a supervised manner using MSE loss of the
predicted and optimal velocity action (Eq. 3).

###  II-C Safety check layer

The output of the neural network is often unpredictable, especially in
unexplored domains, potentially leading to unsafe behavior and collisions with
obstacles. To mitigate this risk, we introduce a safety check layer that takes
inputs from observation data (LiDAR or costmap generated by the global
planner) and the neural network’s output velocity action. The computation is
performed in the sensor domain, enabling fast and efficient calculation and
allowing real-time supervision of the model output before passing input
control into the motor controller.

Movement safety is predicted using geometric calculations [16]. We model the
robot into a polygon centered at the middle point along the axle of the robot.
While the shape can be arbitrary, $h$ and $l$ represents maximum distances
between two points along the $y-$ and $x$-axis, respectively. For the sake of
simplicity, we assume the robot’s shape is a rectangle with dimensions $l$ and
$h$.

(a) Linear

(b) Radial

Figure 3: ROI illustration for safety check layer during linear and radial
movement.

####  II-C1 Linear motion with constant velocity ($|v|>0,\omega=0$)

During linear motion, the robot moves forward or backward along the $x$-axis
relative to its frame of reference, at a constant velocity ($|v|>0,\omega=0$).
The safety check for linear motion involves determining whether any obstacles
lie within a predefined ROI (Region of interest) directly ahead of the robot.
This ROI is defined as a rectangular area extending from the front or back of
the robot up to a certain distance (maximum observable distance) and covering
the width of the robot.

The robot’s movement is defined as ”unsafe” if there exists a point with
coordinate $(x,y)$ that lies withing the ROI, satisfying the following
conditions:

$\begin{cases}xv>0\\\ |y|\leq h/2\end{cases}$ (6)

####  II-C2 Radial motion with constant velocity ($|v|>0,|\omega|>0$)

During radial motion, the robot turns at a constant angular velocity
($\omega$), causing it to move along a circular arc with a turning radius $R$.
The safety check for radial motion involves determining whether any obstacles
lie within a predefined ROI along this arc. The turning radius $R$ is
calculated as the ratio of the linear velocity ($v$) to the angular velocity
($\omega$).

Given the arbitrary width of the robot, let $R_{o}$ be the outer turning
radius and $R_{i}$ be the inner turning radius. The ROI is calculated by
generating two polygons of the robot at initial and final positions (after
$\Delta t$) and connecting the most outer ($r=R_{o}$) and inner ($r=R_{i}$)
points of the polygons with arcs. The robot’s movement is defined as ”unsafe”
if there exists a point that lies in the ROI. For rectangle robot, we define
the outer and inner radii as follows:

$\displaystyle R_{o}$ $\displaystyle=\sqrt{(R+h/2)^{2}+(l/2)^{2}}$ (7)
$\displaystyle R_{i}$ $\displaystyle=R-h/2$ (8)

##  III Related works

Optimization-based methods have long been foundational in robotics for
autonomous navigation, utilizing established algorithms to ensure reliable
performance. The Dynamic Window Approach (DWA) [1] emphasizes collision
avoidance by dynamically calculating optimal velocity within feasible velocity
space, thus reducing computational overhead for quicker responses. Similarly,
Elastic Bands (EBand) [2] generate paths through a series of connected points,
adjusting these paths via simulated elastic forces to avoid obstacles and
minimize travel distance. Another notable optimization approach is the Free-
Space Motion Tube (FSMT) technique [16], which defines a robot’s maneuvers
through adaptive curvature-based motion, allowing efficient navigation in
cluttered environments.

On the other hand, learning-based methods leverage advanced machine learning
techniques to enhance navigational capabilities. The End-to-End (E2E)
algorithm [5] uses Twin Delayed Deep Deterministic Policy Gradient (TD3) to
learn navigation policies directly from raw sensor inputs, allowing robots to
adapt to diverse environments without predefined rules. Learning from
Hallucination (LfH) [17] simulates highly constrained obstacle configurations
during training by defining a hallucination function of obstacle
configuration. Then the Learning from Learned Hallucination (LfLH) [18] used
self-supervised learning to generate motion plan based on the LfH and Inventec
[19] extends the LfLH by incorporating a finite state machine to manage
recovery behaviors and introduces safety measures to ensure robust navigation
in constrained spaces. A hybrid approach is seen in Adaptive Planner Parameter
Learning from Reinforcement (APPLR) [20], which combines traditional planning
with reinforcement learning. APPLR dynamically adjusts planner parameters at
each time step, merging the strengths of classical motion planning with
adaptive learning to handle various navigation scenarios effectively.

These diverse approaches highlight the evolution of autonomous navigation
techniques, from traditional algorithms ensuring computational efficiency to
advanced learning-based methods offering adaptability and robustness, along
with hybrid systems that integrate the best of both worlds.

##  IV Experiment Result

This section describes the implementation and evaluation of the proposed
system, both in the simulation and through hardware experiments, using a
differential drive UGV model with four wheels and equipped with a 2D LiDAR
sensor.

###  IV-A Simulation result

The simulation was performed in Gazebo Classic, utilizing worlds provided by a
benchmark dataset [3]. The proposed method generated 300 random worlds with
different navigation difficulty metrics derived based on distance to closest
obstacle, visibility, dispersion, and turtuosity of every points along the
traversable path.

A total 66 test worlds, referred as benchmark worlds, were selected to
evaluate our algorithm against the baseline methods. The proposed LiCS
algorithm was trained on the remaining environments using the proposed
training pipeline. The code for APPLR, EBand, E2E, LfH, and Fast-DWA was
sourced from the BARN Challenge public repository
111https://github.com/Daffan/the-barn-challenge, while the code for FSMT and
Inventec algorithms was taken from the authors’ public repositories
222https://github.com/romulortr/barn-kul-fm 333https://github.com/inventec-ai-
center/inventec-team-barn-challenge-2023.

Each algorithm were subjected to three trials in the benchmark environments.
Metrics recorded include average score, success rate, and average traversal
time $T$. The score metric, adapted from [4], incorporated traversal time as
follows:

$Score=\mathbbm{1}_{succ}\frac{T^{*}}{\text{clip}(T,2T^{*},8T^{*})}$ (9)

The optimal traversal time $T^{*}$ is calculated from the shortest path length
$L^{*}$ of each world, as provided by the dataset, divided by the maximum
velocity of UGV ($2\ m/s$). Lower traversal times indicate more efficient and
agile navigation. The results of these simulations are summarized in Table I.

TABLE I: Performance on benchmark worlds. Bold and square brackets indicate the best and second best performers Algorithm | Average score | Success (%) | Average Time (s)
---|---|---|---
Fast-DWA [1] | 0.2306 | 0.8636 | 25.7320
EBand [2] | 0.3256 | 0.9495 | 16.7633
APPLR [20] | 0.3357 | 0.8889 | 16.8034
E2E [5] | 0.3712 | 0.7424 | 5.1905
LfH [18] | 0.4239 | 0.9141 | 11.1017
Inventec [19] | 0.4242 | [0.9899] | 13.8181
FSMT [16] | 0.4654 | 1.0000 | 12.2152
LiCS444without safety check and dynamic inflation | 0.4455 | 0.9091 | 8.5064
LiCS555without safety check | [0.4874] | 0.9747 | [7.8487]
LiCS | 0.4894 | 0.9798 | 8.0182

Our proposed algorithm, LiCS, demonstrated the highest average score,
showcasing robust performance across metrics. LiCS achieved a success rate of
97.98%. Although E2E exhibited the shortest average traversal time of 5.19
seconds, it suffered from a lower success rate in comparison. LiCS provided a
balanced approach with an average time of 8.02 seconds, effectively combining
speed and safety. To provide a clearer visualization, we plotted the average
traversal times (over three trials) for each algorithm and the average score
by grouping test worlds into 6 bins in Fig. 4

To evaluate the performance limits of these algorithms, we selected 16 of most
challenging worlds from the BARN dataset, where most algorithms had previously
scored the lowest. This selection aimed to rigorously test the robustness and
adaptability of each algorithm under difficult navigation scenarios. Each
algorithm was tested over three iterations in these hard worlds. The results
of this evaluation are summarized in Table II.

TABLE II: Performance on 16 most challenging worlds Algorithm | Average score | Success (%) | Average Time (s)
---|---|---|---
Fast-DWA [1] | 0.1010 | 0.5210 | 36.2100
APPLR [20] | 0.1151 | 0.4792 | 31.65
E2E [5] | 0.1250 | 0.25 | 5.6100
LfH [17] | 0.2142 | 0.5210 | 14.5500
EBand [2] | 0.2194 | 0.6875 | 19.54
Inventec [19] | 0.2932 | 0.9375 | 21.5000
FSMT [16] | 0.3831 | 0.8750 | 14.0878
LiCS666without safety check and dynamic inflation | 0.3167 | 0.6458 | 8.9144
LiCS777without safety check | 0.4583 | [0.9167] | 8.4039
LiCS | [0.4162] | 0.8438 | [7.8345]

In these challenging environments, LiCS consistently outperformed the other
algorithms, achieving the highest and second highest average scores and still
maintaining high success rate. Although E2E had the shortest average traversal
time of 5.61 seconds, LiCS demonstrated a balanced performance with an average
time of 7.83 seconds.

Through the process of comparing the performance of LiCS with other
algorithms, it becomes evident that our proposed algorithm strikes a balance
between high success rates, low collision rates, and reasonable completion
times. Unlike some algorithms that excel in one aspect but lag in others
(e.g., E2E’s low collision rate but poor success rate in hard worlds), LiCS
maintains a strong overall performance, making it suitable for a wide range of
real-world applications.

(a)

(b)

Figure 4: (a) Average scores of algorithms for each benchmark worlds group.
(b) Average traversal times across benchmark worlds.

###  IV-B Hardware Experiment Result

The robot used for hardware experiment is identical to the one used in
simulation experiment, hence allowing the performance comparison between the
simulation and real implementation (sim-to-real). The algorithm was deployed
in Jackal robot equipped with Hokuyo UST-20LX LiDAR and Intel i3 CPU
controller, operating without any GPU support.

We conducted tests on three tracks constructed from card boards, each offering
different navigation difficulties. The first track was the easiest and the
third track was the most difficult. Minimal fine-tuning was applied to the
algorithm post-simulation. The identical neural model trained in the
simulation was used for the hardware experiments. However the LiDAR sensor
differed in resolution between the simulation ($720\times 1$) and real
hardware setups ($1081\times 1$). To overcome this discrepancy, we scale down
the hardware sensor data size by linearly sampling of the real sensor data to
match the required model input size. Additional adjustments were made to the
velocity settings and the inflation radius of the global cost map to optimize
the performance.

(a) Track I

(b) Track II

Figure 5: Robots navigating the first and second tracks, delineated by cardboard boundaries. The red lines indicate the robot’s future trajectory, and the black arrows show the direction of movement. TABLE III: Hardware Experiment Results Trial | Track 1 (s) | Track 2 (s) | Track 3 (s)
---|---|---|---
1 | 32 | 37 | X
2 | 31 | 37 | X
3 | 32 | 40 | X
4 | 27 | 29 | X
5 | 30 | 32 | X
Average | 30.4 | 35 | X

The results of the hardware experiments are presented in Table III. The table
lists traversal times for each trial, with ”X” indicating a failure to
complete the track caused by collision. Our algorithm successfully navigated
track 1 and 2 but failed on track 3. The difficulty increased progressively
across the tracks. Particularly, track 3 featured narrow path requiring sharp
turns, which proved too challenging. During the experiments, several
parameters were adjusted, including maximum linear velocity, angular velocity,
and inflation radius. For the first three trials on tracks 1 and 2, the
maximum linear and angular velocities were set to 1.0 m/s and 1.0 rad/s,
respectively. In subsequent trials, the maximum linear velocity was increased
to 1.5 m/s, but the optimal performance was achieved at 1.3 m/s. At higher
velocities, the robot moved too aggressively, necessitating frequent
corrective maneuvers, which ultimately increased traversal times. On track 3,
the robot frequently collided with obstacles or became stuck at tight corners
due to the demanding navigation requirements.

##  V Conclusion

In this study, we introduced the Learned-imitation on Cluttered Space (LiCS)
algorithm, a novel imitation learning-based approach for navigating Unmanned
Ground Vehicles (UGVs) through complex, cluttered indoor spaces. This approach
utilizes a Transformer-based neural network and combining behavior cloning
with robust safety checks, LiCS was designed to optimize navigation by
learning from expert demonstrations while adapting to dynamic and
unpredictable conditions. It was trained under noisy conditions to generalize
across various scenarios. The safety layer integrated into LiCS effectively
mitigated potential hazards, preventing collisions and ensuring stable
operation under diverse conditions.

The simulation result demonstrated that LiCS provides a significant
improvement over baseline methods. It achieved the lowest average traversal
time with high success rate, especially in challenging environments
characterized by tight spaces. The hardware experiments further validated the
simulation results, with LiCS performing reliably on real robots. Although it
encountered difficulties in the most challenging track, which highlighted
potential limitations in real-world sensor discrepancies and dynamic
responses, the overall success in simpler tracks confirmed its practical
utility and effectiveness.

This study’s findings suggest that the LiCS algorithm represents a promising
advancement in the field of autonomous navigation for UGVs, particularly in
scenarios where traditional methods struggle.

## Acknowledgment

This work was supported by Unmanned Vehicles Core Technology Research and
Development Program through the National Research Foundation of Korea (NRF),
Unmanned Vehicle Advanced Research Center (UVARC) funded by the Ministry of
Science and ICT (MSIT), the Republic of Korea (#2020M3C1C1A0108237512).

## References

  * [1] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,” _IEEE Robotics & Automation Magazine_, vol. 4, no. 1, pp. 23–33, 1997.
  * [2] S. Quinlan and O. Khatib, “Elastic bands: Connecting path planning and control,” in _[1993] Proceedings IEEE International Conference on Robotics and Automation_. IEEE, 1993, pp. 802–807.
  * [3] D. Perille, A. Truong, X. Xiao, and P. Stone, “Benchmarking Metric Ground Navigation,” Nov. 2020, arXiv:2008.13315 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2008.13315
  * [4] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martin-Martin, and S. Savarese, “Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 713–720, Apr. 2020. [Online]. Available: https://ieeexplore.ieee.org/document/8954627/
  * [5] Z. Xu, B. Liu, X. Xiao, A. Nair, and P. Stone, “Benchmarking reinforcement learning techniques for autonomous navigation,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 9224–9230.
  * [6] J. Zhang and K. Cho, “Query-Efficient Imitation Learning for End-to-End Autonomous Driving,” May 2016, arXiv:1605.06450 [cs]. [Online]. Available: http://arxiv.org/abs/1605.06450
  * [7] Y. Li, “Deep Reinforcement Learning: An Overview,” Nov. 2018, arXiv:1701.07274 [cs]. [Online]. Available: http://arxiv.org/abs/1701.07274
  * [8] A. Beygelzimer, V. Dani, T. Hayes, J. Langford, and B. Zadrozny, “Error limiting reductions between classification tasks,” in _Proceedings of the 22nd international conference on Machine learning - ICML ’05_. Bonn, Germany: ACM Press, 2005, pp. 49–56. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1102351.1102358
  * [9] S. Ross and J. A. Bagnell, “Efﬁcient Reductions for Imitation Learning,” 2010.
  * [10] S. Ross, G. J. Gordon, and J. A. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” Mar. 2011, arXiv:1011.0686 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1011.0686
  * [11] M. Green and J. B. Moore, “Persistence of excitation in linear systems,” _Systems & Control Letters_, vol. 7, no. 5, pp. 351–360, Sep. 1986. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/0167691186900526
  * [12] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg, “DART: Noise Injection for Robust Imitation Learning,” 2017.
  * [13] L. Ke, J. Wang, T. Bhattacharjee, B. Boots, and S. Srinivasa, “Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. Xi’an, China: IEEE, May 2021, pp. 6185–6191. [Online]. Available: https://ieeexplore.ieee.org/document/9561662/
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” Jun. 2021, arXiv:2010.11929 [cs]. [Online]. Available: http://arxiv.org/abs/2010.11929
  * [16] R. T. Rodrigues, N. Tsiogkas, N. Huebel, and H. Bruyninckx, “Clutter-resilient autonomous mobile robot navigation with computationally efficient free-space features,” in _The International Symposium of Robotics Research_. Springer, 2022, pp. 522–537.
  * [17] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Toward agile maneuvers in highly constrained spaces: Learning from hallucination,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1503–1510, 2021.
  * [18] Z. Wang, X. Xiao, A. J. Nettekoven, K. Umasankar, A. Singh, S. Bommakanti, U. Topcu, and P. Stone, “From agile ground to aerial navigation: Learning from learned hallucination,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 148–153.
  * [19] H. Mandala and G. Christmann, “The barn challenge 2023–autonomous navigation in highly constrained spaces–inventec team,” _arXiv preprint arXiv:2307.14580_ , 2023.
  * [20] Z. Xu, G. Dhamankar, A. Nair, X. Xiao, G. Warnell, B. Liu, Z. Wang, and P. Stone, “Applr: Adaptive planner parameter learning from reinforcement,” in _2021 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2021, pp. 6086–6092.

|  Joshua Julian Damanik received the B.S. degree in engineering physics from
Institut Teknologi Bandung, Indonesia, in 2018, and the M.S. degree in
aerospace engineering from the Korea Advanced Institute of Science and
Technology (KAIST), Daejeon, South Korea, in 2021, where he is currently
pursuing the Ph.D. degree in aerospace engineering KAIST. His current research
interests include robotics learning and control, and data mining.
---|---
|  Jae-Won Jung received the B.S. degree in mechanical engineering from
SungKyunKwan University, Suwon, South Korea in 2018. He is currently pursuing
the M.S. degree in aerospace engineering from the Korea Advanced Institute of
Science and Technology (KAIST), Daejeon, South Korea. His current research
interests include robotics control, and Machine Learning.
---|---
|  Chala Adane Deresa received the B.S. degree in aerospace engineering from
the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South
Korea, in 2024, where he is currently pursuing the M.S. degree in aerospace
engineering. His current research interests include robotics estimation and
control, and spacecraft autonomy.
---|---
|  Han-Lim Choi (Senior Member, IEEE) received the B.S. and M.S. degrees in
aerospace engineering from the Korea Advanced Institute of Science and
Technology (KAIST), Daejeon, South Korea, in 2000 and 2002, respectively, and
the Ph.D. degree in aeronautics and astronautics from the Massachusetts
Institute of Technology (MIT), Cambridge, MA, USA, in 2009. Then, he studied
at MIT as a Postdoctoral Associate until he joined KAIST, in 2010. He is
currently a Professor of aerospace engineering at KAIST. His current research
interests include estimation and control for sensor networks and decision
making for multi-agent systems. He was a recipient of the Automatic
Applications Prize, in 2011 (together with Dr. Jonathan P. How).
---|---